what a week

I now know what a full week as an ETS TOEFL iBT Writing test rater feels like. The job has its fascinating side, but there's no getting around the fact that an observer would see little more than a big, fat Kevin hunched over his tiny netbook, staring intently at the screen, mumbling to himself, and clicking a mouse every two or three minutes.

Any sort of standardized language test is going to involve what I might call "the corralling of subjectivity" in an attempt to make the rating standards appear objective. To work at ETS (and this was also true when I used to work at Smoo) means to think as ETS does. Weirdly enough, this Borg-like mentality seems to produce results, because as the week has worn on, I've felt more and more in tune with my supervisors, despite the fact that I was working with a different supervisor-- called a "scoring leader"-- every day.

ETS is understandably touchy about company secrets, but it's no secret to note that the two TOEFL iBT writing sections-- respectively called Integrated and Independent-- are scored on a scale of 0 to 5. Except perhaps for 0 and 1, each number represents a range of results, and rating well means getting in tune with ETS's rhythm and mindset. It took me a couple days to figure out how to score the Integrated Writing section; now, I score it quite well, and can plow through batches of student work at a pace of about one essay every two minutes. I'm still slow with Independent Writing, but that's because I haven't rated much of it.

The job has its disadvantages. It's a bit unnerving, for example, that your scoring leader can see, down to the second, exactly what your pace is, which is one reason why you can't afford to slack off. Irregularities in your rhythm get noticed, as one scoring leader told me. It's also unnerving when you receive a call from the scoring leader to discuss an "adjacent" (varying by 1 point) or "discrepant" (varying by >1 point) score. In many cases, the work I do overlaps with the work someone else is doing; as a result, student essays usually receive at least two scores-- possibly even a third if arbitration is required. Ideally, the scores from both raters should be the same. The whole operation is part of a massive bureaucracy once you factor in the raters for other tests: GRE, SAT, and so on.

Still, there are perks. You get to crawl inside the minds of the students taking the TOEFL, for instance, and you learn very quickly who's who: Asians routinely make certain kinds of mistakes; francophones and germanophones make others. You also learn that people on different continents learn different TOEFL strategies: certain stock phrases are easy to associate with certain regions of the world. You also begin to notice that people with higher ratings tend to be from Western Europe, where they share cultural assumptions with Americans and have an easier time with English than, say, East Asians do. (Imagine the reverse situation: being an American taking an essay test in Japanese! Not easy.)

Besides the perks, I've also been happy, for the most part, with my scoring leaders. Only one struck me as a bit of a ditz, not to mention a bit snippy on the phone, but even she was helpful when she had to be. That's important, because it makes the job less of a drag. I also like that my day begins without having to call anyone right away: we're all supposed to "calibrate" first before we phone our respective scoring leaders. That can take close to an hour, which gives an introvert like me time to get into the day's rhythm before I make that call.

As ETS employees, we're referred to in the way you'd refer to machines: our work day begins with "calibration" (essentially, it's like taking a short version of the certification exam all over again, every day); once we've successfully calibrated ourselves, we move on to "production," i.e., the actual scoring of essays. If you fail calibration, then, depending on what type of essays you're scoring you can re-calibrate either once or twice more. If you fail... you're done for the day, and you're paid only for the hours you've logged in calibration.

Today, my final day this week, I successfully calibrated in the morning, and worked until we ran out of essays in the mid-afternoon. We then had to switch over to rating a different type of essay, which meant-- you guessed it-- calibrating first. As it turned out, I failed the calibration twice, but that was fine: it was 5PM, so my day was done. I got my full eight hours. My scoring leader was sympathetic: we'd been rating one type of essay for most of the day, meaning that it was hard to readjust our brains to scoring a different type of essay. I was told that I wasn't the only one under my supervisor's watch to fail calibration.

So the week ended on a weird note.

It's a strange job, all in all. Because you have to work at a fast, steady pace, you can't afford to lose focus. A wandering mind is an unproductive mind. At the same time, despite the job's mental intensity, an intensity driven by your membership in a hierarchical hive mind, it's the most sedentary work you could wish for-- just eight hours of sitting while your buttocks expand like glaciers getting ready to calve. I seem to have taken fairly well to the work, but I also know that, in the long run, it'll drive me crazy if I do it for more than a year. Here's hoping I won't need to do this job for even that long.


_