In spite of assurances, the testing company hired for testing Oklahoma elementary students is using highly suspect methods.

Currently, writing tests still must be hand graded by real people sitting in a room somewhere. There is little known about who they are and how well they have been trained, especially since book companies have jumped into the testing business only in the past several years.

After having seen first-hand just how much effort it takes to produce a consistent grading process on a massive numbers scale, I cannot believe that these test scores are valid.

My Experience as a Grading Hired Gun

I have four years of personal experience at grading essays for a large service, the Educational Testing Service, which has scored the SAT and the now large array of Advanced Placement course exams each year. Unlike the service that is grading the elementary essays, they put a remarkable amount of effort into grading essays.

Let’s, look at just how much effort and expense goes into a good, credible essay-grading process:

For the grading of essays for AP exams, thousands of teachers fly in ahead of time and engage in rigorous training for several days on the specific questions that they will be scoring.

Readers are organized into a table of 10 readers. Those readers are made up of half high school AP teachers who have graded their students’ essays all year, the other half are college profs who teach the college course that the high school AP course mirrors.

The ETS has a very good quality control system in place that they have perfected over the decades, using table leaders who back-grade some of their readers’ scoring and huge servers to check for grading consistency of those essays at the end of each day.

Actual grading trends of each reader, each table of readers, and each table leader, is checked against the larger numbers of the reading of that same exam question. It is impressive how they can detect when a reader, table, or table leader are drifting in their scores

But, that is the elite process of essay-grading that is not matched by any other company. The money made from fees that students and school districts pay for that testing is lucrative business; but, cannot be done on the cheap. If their reputation ever starts to suffer, the money will go away like a puff of smoke.

Even with all that impressive effort, it is still a human effort that has its quirks. No college slices a test score so thinly that the student is judged as pass/fail.

Which leads us back to Oklahoma and the complete mess of the 5th and 8th grade writing tests. There is not credibility in the process of testing Oklahoma’s 5th and 8th grade writing.

Undeserved Bad Reputation of Oklahoma Elementary Kids and Schools

Because of my experiences at seeing just how delicate the essay test grading process really is, I have not been able to believe that an old text book hustler company like CTB/McGraw-Hill could develop a testing process with integrity over just a few years like they have.

Earlier this weekend, two education blogs, Oklahoma Education Truths, and Rob Miller’s A View From the Edge raised serious questions about the test results from the standpoint of statistics and reviews of the scored exams.

For some impressive specific details about why the tests are suspect, see yesterdays post in Miller’s blog showing how the tests are going wrong.

Today The Oklahoman and The Tulsa World ran a story about how more and more elementary and middle school teachers, principals, and their superintendents are raising serious questions about the quality controls of CTB/McGraw-Hill who have the contract with the Oklahoma State Department of Education to grade writing tests of our 5th and 8th graders.

A host of schools across the state say their fifth- and eighth-grade writing test scores are deeply flawed, but state education officials are standing by the scores issued by controversial vendor CTB/McGraw-Hill.

After discovering abnormally high rates of students receiving the same scores, school officials all over the state are questioning whether the company’s readers properly scored the tests. They also question widespread reductions in scores for “plagiarism” for students they say simply followed instructions to cite directly from reading passages.

Local school officials raised concerns with the vendor and the Oklahoma Education Department in a meeting May 28 but were shocked by the responses they received.

“We have been told all we can do is request a re-score, but if the testing vendor decides not to change the score, we have to pay $125,” said Rick Cobb, assistant superintendent for curriculum and instruction at Moore Public Schools.

“At $125 a pop, 80 tests alone would be $10,000. No district in Oklahoma has $10,000 to throw around right now, but we want the kids to get the scores they deserve.”

Clearly there are problems, yet the SDE doubles down on the process, not because it holds water, but because among Republican political consultants who are advising Supt. Janet Barresi, doubling down works when Republicans are wrong.

It’s time for a change.