Essay-Grading Software Can Read Teachers' Minds

Comment | Print | RSS

• Login to Rate

So would teachers welcome the introduction of software that could grade students' test-prep essays? Would they reject that software, claiming that while computers might be fine at grading multiple-choice tests, they can't grade something as complex as a writing sample? And how would they respond if they were told that while a computer can't necessarily grade an essay better than a trained educator can, a computer can do a darned good job of reading a trained educator's mind?

I confess that I cannot read a teacher's mind, so I'm very curious to know how K through 12 teachers greet the news that data scientists have recently created automated essay-grading algorithms that deliver scores that are shockingly similar to those issued by human educators. A three-man team of data analytics junkies was just awarded $60,000 for creating just such an algorithm as part of a competition sponsored by the William and Flora Hewlett Foundation and hosted by Kaggle (the site that "makes data science a sport"). According to the press release announcing the winners:

Participants competed in the Automated Student Assessment Prize (ASAP) to develop software that could score students’ essays used in state standardized tests that had already been individually graded by educators. The winning team came closest to replicating how the tests were graded by the trained experts.

So the idea wasn't really to create an algorithm that can assess what score an essay should get, but to correctly guess what score an essay would get.

I can see how this could be a great thing as far as standardized testing goes. Typically each writing sample is graded by two teachers separately. If one teacher co-graded with the software, instead of a second teacher, then it would allow the school district to either employ half the number of teachers to grade the writing samples or give each teacher fewer essays to grade, so that they could spend more time on each one. And that would be a good thing.

Barbara Chow, education program director at the Hewlett Foundation, told the New York Times that teachers who grade essays on standardized tests spend an average of three minutes reading each essay. (If one of my terrifying high school English teachers had told my honors English class this, I think we all would have killed ourselves right then and there.)

Ultimately, the Hewlett Foundation's hope is that automated grading software will enable educators to include more writing components into standardized tests:

This is important because standardized testing has had a significant impact on classroom practice. The goal is for students to acquire critical thinking and communication skills that writing requires -- all without the burden of added time and cost to the system.

This innovative software also has great potential in classroom use. Its purpose is not to replace teachers, but to give them tools to be able to assign more writing in the classrooms. In today’s writing classes, students only write an average of three essays a semester. With up to forty students in each class, essays take too long to grade. This technology will allow teachers to assign more writing and give quicker feedback.

I can get behind the idea of coupling one human with one algorithm to grade essays for standardized tests. However, the idea that this software could be used in a class taught by only one teacher concerns me -- because it sounds like the only way this would save teachers' time is if the software did all the grading. And to me, this is a bad idea. After all, the written word is meant to be read by people, not by software. Despite the success of this software, I still don't think that it could it do a better job than my 12th grade English teacher at determining how well I supported my assertions about James Joyce's symbolic use of color in Portrait of the Artist as a Young Man.

Tell us, education CIOs. How would the educators in your school respond to increased use of automated student assessment technology in the classroom? Would it improve or injure education? Would it give teachers the opportunity to spend less time preparing their students to pass standardized tests and more time teaching them other things?