Thursday, March 3, 2011

The promise and fear of an assembly line for knowledge work

Last week, together with Amanda Michel from ProPublica, we were presenting at the CAR 2011 conference (CAR stands for Computer-Assisted Reporting), on how to best use Mechanical Turk for a variety of tasks pertaining to data-driven journalism.

We discussed issues of quality assurance, how TurkIt-like workflow-based tasks can generate nice outcomes, and briefly touched upon the CrowdForge work from Niki Kittur and the team at CMU, showing that crowdsourcing can potentially generate intellectual outcomes comparable to those of trained humans.

The discussion after the session was a mix of excitement and fear. We have observed in the past how "assembly line" work for industrial production lead to massive productivity improvements and was the basis for much of the progress in the 19th and 20th century. But that was for mechanical work. Yes, it replaced centuries old crafts of the blacksmiths, carpenters, potters, but that was just part of progress.

What happens if we see now the assembly line extended into tasks that were traditionally considered creative and intellectual in nature? What would be the effect of an assembly line for knowledge work?

A few months back, I quoted Marx and Engels who, back in 1848, wrote in their Communist manifesto:
the work of the proletarians has lost all individual character, and, consequently, all charm for the workman. ... [The workman] becomes an appendage of the machine, and it is only the most simple, most monotonous, and most easily acquired knack, that is required of him
(Btw, TIME magazine liked that connection enough to put it into their own article about Mechanical Turk.)

But how likely it is to see this style of work to be extended further in the intellectual field? Are these Mechanical Turk experiments something generalizable, or just cute proof-of-concept experiments?

I was reminded of this question today, when I realized that many intellectual tasks are already commoditized:

The article "Inside the multimillion-dollar essay-scoring business: Behind the scenes of standardized testing" gives a dreadful view of now essays are being scored for the standardized tests.

Based on the description of the article, the (human-based) scoring process "goes too fast; relies on cheap, inexperienced labor; and does not accurately assess student learning." Needless to say, the workers were not exactly enthusiastic about their work. Match that with the computer-assisted scoring of essays, and you have an MTurk-like environment for much more intellectually-demanding tasks...

After reading this essay-scoring mill story, I started feeling a little bit uneasy. The MTurk-style work seems too far away to be in my future, so the discussion is always, ahem, academic. But the essay scoring brought the concept a little bit too close for comfort.