It does not necessarily lead to an arms race. Once the students learn to write to the gradind software, the schools can claim the students are getting better, and point to an 'objective, unbiased' measuring stick. Everyone wins except for society, i.e. sll of us.
I'm assuming there will be competition among the companies who develop such software where, periodically, schools will re-evaluate their provider and select the best one. Company A will demonstrate their competitive advantage by showing how Company B's software incorrectly grades as "Excellent" an essay that looks like English but reads as gibberish.
Company B will correct their software, but meanwhile Company C has come along and introduced a sophisticated analyzer which is able to grade the presence and quality of the logical argument being made in the essay. Company C will demonstrate how Company A and B's software grades as "Excellent" an essay which is correct English and isn't gibberish but makes no logical sense.
After an indeterminate number of iterations with the software getting better and better at finding issues, the only way to game it will be to write a great essay.
You are forgetting the negative effect of the ridiculous ranking systems that determine teachers performance evaluations. Why would a school select software which will make thier grades go down? If your grades go down it decimates the schools ranking because these are based on year on year improvements. You would select the software which makes it easiest to teach to the test so the kids 'do well'
