Saturday, May 28, 2011

Crowdsourcing Education

This is a guest blog post by Lindsey Wright.

Crowdsourcing Education

Technology and the popularity of online classes are radically transforming the educational process on both national and international scales. Society sees an increased failure of traditional schools in meeting goals to fully educate student populations to produce future generations of innovative, intellectual community leaders and workers. It is time to look at online learning and other alternative choices in education as possible solutions to complex obstacles and challenges of traditional learning routes.

Home Schooling Expands

As elementary and secondary schools continue to lag behind academically, home schooling movements continue to expand. The National Household Education Survey Program found a 36 percent increase in home-schooled populations for reasons other than health from 2003 to 2007 and a 74 percent increase from 1999 to 2007. Parents cite a lack of moral education and safety fears as primary reasons to abandon the traditional school model. However, if parents enrolled their children in an online school these problems could be eliminated. For instance, overworked, tired parents would not have the additional responsibility of planning lessons and students would be held accountable for their learning. Students would also have fewer distractions to concentrate on their studies, thus eliminating discipline problems.

Who's Watching the Kids?

Additionally, in 2002, only 7 percent of U.S. families survived on one spouse’s income, according to the Population Reference Bureau. As such, parents are spending a more substantial amount of time at work, leaving little time for parental school involvement. Overwhelmed teachers routinely contact parents primarily for discipline problems and not for building academic progress. As the primary years lay the foundation for future academic success, the educational and business community must look for new ways to transform education methods to ensure students will have the keys to higher learning opportunities and a lifetime of success. With the wealth of information found on the Internet, online classes could utilize educational Web sites and software that engage students in ways beyond the traditional classroom.

High School Dropouts Increasing

According to Valiance for Excellent Education , 1.2 million students annually fail to graduate from school. Yet due to the rapid growth of knowledge, businesses and industry are demanding innovative workers equipped with advanced skills in communication, mathematics and technology. Students who fail to gain these skills are less likely to earn decent salaries, forcing the government to spend more on poverty fighting programs such as Medicaid. Online classes would aid in dropout prevention by providing students with individualized instruction, allowing them to focus on gaining the practical, innovative working skills they need.

Teacher Attainment Issues

A study by the National Education Association found that 50 percent of new teachers quit teaching within a five year period, due to low salaries, heavy pressures of administrators and poor working conditions. Though teaching can be a fulfilling career, new teachers often lack the support of veteran co-workers and supervisors, leaving the new teacher lost in a maze of bureaucracy and hopelessness. This task can become overwhelming for teachers lacking support from parents, administration and the community at large. By streamlining the curriculum through online classes, teachers would be freed to devote more time to the art of teaching and have more time to spend learning from peers and gaining the support they need, eliminating the decline in teacher attrition rates.

The New Economy

The global economy is radically different from what it was even fifty years ago, when America was a leading global power of industry and commerce. Today, international consumer and industry markets are the norm, as seen with technological leaders of Google and Microsoft, whose headquarters are not located in the United States, but in Ireland. American industries, forced by heavy government regulations, are relocating facilities to countries with friendlier business climates, such as India. A young person without a solid education will have severe difficulties in this new highly competitive job market. However the use of Web chats, videos and other learning projects will transition students into this global economy, and provide engaging activities to keep students interested. For example, instead of learning foreign languages through the traditional methods of worksheets and repetitive instruction, students could learn through interactive distance learning, eventually conversing with foreign students.

Popularity of Online classes

Colleges are already embracing online classes, with the use of popular programs such as Blackboard. Flexibility, availability of classes anywhere in the country and streamlined instruction are benefits of this new technology. Instead of being lazily spoon-fed direction instruction by teachers and faculty members, students can take control of their education through individualized instruction with unlimited access to the Internet from the convenience of their own home. This also allows students to save time money, and gas while pursuing a bachelor’s or master’s degree.

Online Classes - The Solution?

The future of education depends on the radical transformation of attitudes and traditional methods associated with it. These are if not changed, it is likely that the education in the U.S. will be doomed to miserable failure. With online and distance learning taking a strong hold in the education community, expansion of these services could decrease dropout rates, eliminate disciplinary problems and reduce new teacher attrition and instability. Likewise, schools would no longer have funding problems, as much could be saved by the elimination of transportation and utility costs. Parents would also be more aware of their child's academic progress and would have the added peace of mind of knowing that their child was in a safe environment. School uniforms and other required classroom supplies would also be eliminated from the family budget.

In order for the United States to again become recognized as a global leader, we must look to online classes and distance learning as viable options to solving the issues that plague our education system. The traditional learning model is no longer functional in achieving educational goals for this country. Technology is making this revolution possible, if we believe in its potential. We cannot continue to use traditions and methods of the past if we are to educate the students of tomorrow.

Friday, May 13, 2011

Pay Enough or Don't Pay at All

No good deed goes unpunished

A while back, we have been working with Dahn Tamir on identifying spam tasks and requesters on the Mechanical Turk platform. Dahn took the lead and build a task on MTurk in which Turkers could see the (other) newly posted tasks on MTurk, and flag the obvious spam ones. Since this was not a task from which he could benefit, he asked workers to rate as many tasks as possible without submitting the task as "completed", to keep the costs down. Workers were happy to collaborate, and effectively work for free, in order to clear the market. We were collecting data nicely.

And then, I received some minimal funding for the project ($1,000 to be exact). At that point, I thought that it would be a nice gesture to actually start paying the workers. So, we created a new task, we calibrated for time to pay around 7 dollars an hour, and we posted the task. We were expecting workers to be happy. They were doing the work for free before; now they would not only help clean the market, but they would also get paid for this!

The result? A few positive messages with a thank-you note. But also a big backlash: "You, fat cat academic, with all the grants, you want us to work for peanuts?". "Hey, big prof, would you like to be paid minimum wage for your work?". "Yeah, we should be the slaves doing all the grunge work for your research, so that you can get the fame."

I was shocked. What happened? I tried to remind the workers that they were doing the same task for free before, but it did not really make a difference. Actually, it made things worse.

Market norms vs. social norms

Then, I remembered. Dan Ariely, in this book "Predictably Irrational" has warned about this. There are the social norms and the market norms. When no money is involved, the exchanges operate using social norms. Once you put a price on a task, it becomes part of a market norm. It can be measured and compared.

When the workers were not getting paid, they were working towards a noble goal: Clean the market from the spammers. By putting a price on the task of classifying spam tasks, we essentially told the workers how much we value their work: minimum wage. Instead of offering their priceless help, they were being valued as unskilled workers, like every other worker in the market. Money and altruism do not mix.

Somebody must have studied that before

Needless to say, examining the influence of money on performance and motivation is not a new topic. A wonderful paper that deals is the "Pay Enough or Don't Pay at All" by Gneezy and Rustichini, published back in 2000, titled (625 citations so far, according to Google Scholar). Instead of trying to describe the paper myself, I will just list here the succinct abstract:

Economists usually assume that monetary incentives improve performance, and psychologists claim that the opposite may happen. We present and discuss a set of experiments designed to test these contrasting claims. We found that the effect of monetary compensation on performance was not monotonic. In the treatments in which money was offered, a larger amount yielded a higher performance. However, offering money did not always produce an improvement: subjects who were offered monetary incentives performed more poorly than those who were offered no compensation. Several possible interpretations of the results are discussed.

I would encourage anyone to read the paper, as it contains extensive discussion of various models and explanations. I will definitely do no justice if I claim that I covered fully the content of the paper here. However, I would like to highlight some parts below.

Gneezy and Rustichini extended research in psychology from the 1970's, which examined the difference between intrinsic and extrinsic motivation.

Psychologists study behavior modification through conditioning (in the case of the behaviorist school) or learning (for the cognitive school). We do not. To illustrate the difference, we may consider the classic experiment reported in Deci [1971]. He had college students play with a puzzle in three successive sessions. In the first session participants were left to play freely. In the second session subjects in one group received payment if they solved the puzzle, while the control group did not. In a third session the subjects were again left to play freely. The amount of time spent on free activity in the first and third session was taken as a measure of intrinsic motivation. Deci found that in the third session the experimental group spent less time than the control group playing with the puzzle, and he concluded that the reward offered had decreased the intrinsic motivation of subjects in the first group over the three sessions.

That was a result from research in the 1970's. Gneezy and Rustichini wanted to also examine the effect of money in non-sequential environments. So, they conducted the following experiments:

Effect of additional payment on a paid task

At the beginning of the experiment, each student was promised a fixed payment of NIS 60 for participation. (NIS = New Israeli Shekel, at the time of the experiment, 3.5 NIS = $1.) They were then told that the experiment would take 45 minutes, and they would be asked to answer a quiz consisting of 50 problems taken out of a psychometric test used to scan applicants to the university. [...] In the four different treatments subjects were promised different additional payments for each correct answer.
  • In the first group no mention was made of any additional payment.
  • In the second group subjects were promised an additional payment of 0.1 NIS per question answered correctly.
  • In the third group subjects were promised an additional payment of 1 NIS per question answered correctly and
  • In the fourth group subjects were promised an additional payment of 3 NIS per question answered correctly
[...] The average number of correct answers was:
  • 28.4 in the first group (no additional payment)
  • 23.1 in the second group (additional 0.1 NIS per correct answer).
  • 34.7 in the third group, (additional 1 NIS per correct answer). 
  • 34.1 in the fourth group, (additional 3 NIS per correct answer). 
In other words, performance-based payment improved performance. But offering just a small additional financial incentive, actually decreased performance compared to the case of providing no financial incentives.

Effect of payment on unpaid tasks

We had 180 high-school students around the age of 16 participating with three treatment levels [collecting money for a charitable purpose]....
  • In the first treatment, the students were told about the importance of collecting money for the society, that the results of the collection would be published, so that the amount collected by each pair would become public knowledge. 
  • In the second treatment, after the same speech, each pair was promised 1 percent of the amount that the two of them collected. 
  • In the third treatment, each pair was promised 10 percent of the amount they collected.
In the second and third treatments it was made clear that the payment was made from funds additional to the donation, provided by the researchers. The average amount collected was:
  • 238.67 for groups in the first treatment (with no payment).
  • 153.67 in the second group (1 percent of the collected amount).
  • 219.33 in the third group (10 percent of the collected amount).
In this case, where there was no initial payment and the task had an altruistic purpose, providing financial incentives actually reduced performance.

Additional literature

There is significant literature for anyone interested (thanks Sonny!). A few pointers to start:


Essentially, Gneezy and Rustichini found that:
  • Paying more indeed increases performance, compared to paying less. 
  • However, paying nothing may actually be better than paying!
Section IV of the paper has a very nice discussion on how to interpret and model the process. Here are a few explanations in increasing power of explaining the observed phenomenon:
  • Paying something removes the intrinsic motivation for a task, and replaces it with the external motivation for money.
  • Incomplete contract: the piece-wise or performance-based payment changes the original meaning of the contract, which implied that high-performance is part of the task.
  • Paying small amounts compared to the originally implied value of the task devalues the task (e.g., take back a glass bottle to help recycling vs. for getting 5 cents back)

Relevance to crowdsourcing 

I found the results pretty interesting, with significant implications for micro-crowdsourcing. While volunteers may be great for various tasks (e.g., in citizen science applications, such as the Galaxy Zoo), migrating such applications to a paid crowdsourcing application may have a significant downside. Paying small rewards to workers will be counterproductive. The work of volunteers is, indeed, priceless.

Furthermore, with the low level of payments on Mechanical Turk, we are stuck at the worst possible status. We pay, and we do not pay enough. But how can we pay more, when every attempt to increase the price to reasonable levels is followed by attempts of scammers to game the system and get paid for doing nothing?

Tuesday, May 10, 2011

Crowdsourcing: Lessons from Henry Ford

Last month, during the WWW2011 conference, I participated in the panel "The computer is the new sewing machine: benefits and perils of crowdsourcing"

Below you can find my slides:

The main points that I wanted to make:
  • It is common to consider crowdsourcing as the "assembly line for knowledge work" and think of the workers as simple cogs in a big machine. It is almost a knee-jerk reaction to think negatively about the concept. However, it was the proper use of the assembly line (together with the proper automation) by Henry Ford that led to the first significant improvement in the level of living for the masses
  • Crowdsourcing suffers a lot due to significant worker turnover: Everyone who experimented with large tasks on MTurk knows that the participation distribution is very skewed. A few workers contribute the majority of the work, while a large number of workers contribute only minimally. Dealing with these hit-and-run workers is a pain, as we cannot apply any statistically meaningful mechanism for quality control.
  • We ignore the fact that workers give back what they are given. Pay peanuts, get monkeys. Pay well, and get good workers. Needless to say, reputation and other quality signaling mechanisms are of fundamental importance for this task.
  • Keeping the same workers around can give significant improvements in quality. Today on MTurk we have a tremendous turnover of workers, wasting significant effort and efficiencies. Whomever builds a strong base of a few good workers can pay the workers much better and, at the same time, generate a better product for lower cost than relying on an army of inexperienced, noisy workers.
Yes, at the end, crowdourcing is not about the crowd. It is about the individuals in the crowd. And we can now search for these valuable individuals very effectively. Crowdsourcing is crowdsearching.