Some Articles off of the Web






Friday, 17 April 1998

Computer digests essay exam, spits out 'consistent' grade

The time-honored college essay - where students are supposed to exhibit concise writing and persuasive arguments - could soon be read and graded by the artificial intelligence of a new computer program.

Some regard it as a savior for overworked teachers, and others see it as reassurance for students suspicious that their poor grades come from bored, ignorant or vengeful teachers.

Critics say it misses the point - that essays teach communication between people.

``I think it's a terrible idea. Education is not about just spewing back information but assimilating it in language,'' said Mary Burgan, executive director of the American Association of University Professors.

The Intelligent Essay Assessor software, in development for 10 years, is not yet ready for widespread use in the classroom. Its creators have applied for a patent and want feedback from educators.

``The program has perfect consistency in grading, an attribute that human graders almost never have,'' said Darrell Laham, a doctoral student at the University of Colorado who helped with the development. ``The system does not get bored, rushed, sleepy, impatient or forgetful.''

Thomas Landauer, a university psychology professor who worked on the software, presented it yesterday at a meeting of the American Educational Research Association in San Diego. Peter Foltz, an assistant psychology professor at New Mexico State University, also helped.

This program does more than just count words or analyze mechanics and grammar, like earlier essay scorers. Landauer said it mathematically determines how well students understand the material they are taught, determining mathematically what words should appear in an essay.

© COPYRIGHT 1998 Associated Press

Back to Top

Inside Tech 04/15/98- Updated 10:16 PM ET

Software makes the grade on essays

by Mary Beth Marklein

A computer may not be able to write the Great American Novel, but new software can evaluate a student essay about as well as a teacher can,researchers say.

The Intelligent Essay Assessor software, being presented Thursday at a meeting of education researchers in San Diego, mimics the way a teacher grades essays. The software focuses on content rather than writing ability, but ''it turns out the two are highly correlated. Students who know content write well,'' says University of Colorado, Boulder, psychology professor Thomas Landauer, who has been working on the project for 10 years.

The idea is that ''a good essay is an essay that is similar to other essays that have been good,'' says researcher Peter Foltz, New Mexico State University, Las Cruces.

Here's how it works: The teacher feeds the computer material from textbooks and other sources so that it can ''learn'' the topic and relationships between words. Then the teacher submits a model essay written by an expert or a sampling of essays that have been graded by a human. That provides the software with a basis for comparison.

It has been tested on about 2,000 students, including sixth-graders, undergraduates and first-year medical students, and often produces the same grades that humans give, Foltz and Landauer say.

Researchers say the program examines about a dozen predictors of essay quality.

Essays that don't conform -- they could be extremely poor or remarkably good -- are flagged so humans can decide the grade. For instance, an essay that lists words relevant to the topic but doesn't use complete sentences would be flagged. So would an essay found to have sentence patterns too similar to those in the textbook. The software also provides feedback on what the essay left out, a feature that could help students writing term papers.

''We haven't dared'' use the software for creative writing, Landauer is quick to note. But he hopes the software will prompt teachers to use more essay tests, which he says ''encourage a better kind of learning than . . . a multiple-choice test.'' And because the software's methodology is consistent and efficient, he says it could lighten the load for educators, especially those with 300 to 400 students a semester.

But biology professor James Perley, president of the American Association of University Professors, says he would rather have smaller classes. And if the software were overused, he says, ''we (could) produce a whole population of people who can type things into computers but have no interaction with humans who can help them understand the importance of what they wrote.''

© COPYRIGHT 1998 USA TODAY, a division of Gannett Co.Inc.

Back to Top


New software grades essays

By Dave Curtin
Denver Post Higher Education Writers

April 16 - College professors, bleary-eyed and heavy-lidded from plowing through an endless sea of term papers, may be saved.

University of Colorado psychology professor Thomas Landauer and CU doctoral student Darrell Laham have developed new computer software that grades essay tests with artificial intelligence.

Landauer, Laham and fellow researcher Peter Foltz of New Mexico State University will present the Intelligent Essay Assessor today to the American Educational Research Association in San Diego.

The revolutionary software eliminates grueling late-night grading sessions and their potentially unpredictable results.

"The software does not get bored, rushed, sleepy, impatient or forgetful,'' Laham said.

Educators have historically lauded essay exams because they provide a better assessment of students' knowledge than other types of tests. But they've fallen out of favor in some academic corners because they are time-consuming and difficult to grade fairly and accurately for large classes, Landauer said. The software solves that problem.

"From sixth-graders to first-year medical students, we get consistently good results,'' said Landauer, who has worked on the technology for 10 years.

"Essay tests encourage students to study in a different way so they learn a subject in a way they can express it and talk about it,'' Landauer said. "If you don't test them in that way, they tend not to acquire that skill.

"But essay tests have become less and less practical because of the growth in very large classes and higher student-tofaculty ratios. They're hard to grade, they're open to bias and they don't get used as much.''

The gigabyte grader is ready for use but isn't commercially available yet. The researchers have applied for a patent.

Here's how it works: The computer first "learns'' the material when an electronic textbook on disk is fed into its memory. It takes the text and determines mathematically what words should appear in the essay. But the software allows the student to use different words that mean the same thing.

Next it's given a sample of human-graded essays as benchmarks. Students type their essays into a computer and receive the same grade as the human-graded essays to which they are most closely matched.

"It tries to mimic the function of the human brain,'' Laham says.

In tests of the software, grades varied no more than when two instructors graded the same essay.

"It agrees with a human instructor as well as two instructors agree with one another,'' Landauer said. "We've found that with well over 1,000 essays.''

In one test, both the software and faculty members graded essays from 500 psychology students at CU-Boulder.

"The correlation between the two scores was very high - it was the same correlation as if two humans were reading them,'' said Landauer.

The system requires about 20 times the memory of an ordinary personal computer.

The software doesn't correct grammar and spelling, which existing programs already do. But it can detect originality and even alert professors to possible plagiarism.

"If it finds an essay that is unusual, unlike any other it's seen, it passes it on to a human to look at,'' Landauer said. "Then we can see if we have an undiscovered genius or someone talking gibberish.''

New Mexico State's Foltz said the software offers immediate feedback and suggestions on how to improve the essays.

But skeptics say that a teacher who doesn't grade papers doesn't connect with students and that students want to know that a human will read their paper.

Margaret Whitt, director of first-year English at the University of Denver, likened the essay grader to a computer diagnosing a patient.

"It takes away a teacher's wonderful opportunity to find out firsthand how much progress a student has made and where they still need help,'' Whitt said. "The thought of taking a student paper and reducing it to technology - I don't even want to hear this. This takes away the individual delight and joy of seeing where a student's mind is on a particular day.''

As an example, Whitt said she had a student who wrote, "I once had a boyfriend that was just like Hamlet. So when I discovered Hamlet in this play it made me so mad I wanted to reach into the text and slap his face.'' Said Whitt, "That says something about that student's personality and life and where she is in beginning to see what Shakespeare means to her. Knowing that about her will make me a better teacher.''

© COPYRIGHT 1998 Denver Post

Back to Top