tag:blogger.com,1999:blog-71185634030274676312024-03-19T00:58:18.609-04:00A Computer Scientist in a Business SchoolRandom thoughts of a computer scientist who is working behind the enemy lines; and lately turned into a double agent.Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comBlogger247125tag:blogger.com,1999:blog-7118563403027467631.post-50773400305452753502024-01-18T15:21:00.002-05:002024-01-22T18:22:01.763-05:00The PiP-AUC score for research productivity: A somewhat new metric for paper citations and number of papers<p><span style="color: #374151; font-family: inherit; white-space-collapse: preserve;">Many years back, we </span><a href="https://www.behind-the-enemy-lines.com/2018/11/distribution-of-paper-citations-over.html" style="font-family: inherit; white-space-collapse: preserve;" target="_blank">conducted some analysis</a><span style="color: #374151; font-family: inherit; white-space-collapse: preserve;"> on how the number of citations for a paper evolves over time. We noticed that while the raw number of citations tends to be a bit difficult to estimate, if we calculate the percentile of citations for each paper, based on the year of publication, we get a number that </span><a href="https://arxiv.org/abs/2103.16025" style="font-family: inherit; white-space-collapse: preserve;" target="_blank">stabilizes very quickly</a><span style="color: #374151; font-family: inherit; white-space-collapse: preserve;">, even within 3 years of publication. That means we can estimate the future potential of a paper rather quickly by checking how it is doing against other papers of the same age. The percentile score of a paper is a very reliable indicator of its future.</span></p><p><span style="font-family: inherit;"><span style="color: #374151; white-space-collapse: preserve;">To make it easy for everyone to check the percentile scores of their papers, we created a small app at </span></span></p><p style="text-align: center;"><span style="font-family: inherit;"><span style="color: #374151; font-size: large; white-space-collapse: preserve;"><b><a href="https://scholar.ipeirotis.org/">https://scholar.ipeirotis.org/</a> </b></span></span></p><p><span style="font-family: inherit;"><span style="color: #374151; white-space-collapse: preserve;">that allows anyone to search for a Google Scholar profile and then calculate the percentile scores of each paper. We then take all the papers for an author, calculate their percentile scores, and sort them in descending order based on their scores. This generates a plot like this, with the paper percentile on the y-axis and the paper rank on the x-axis.</span></span></p><p></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiy2BlC9BfVlKRVI5ogoSPRnSuomBI5XMMWnSPRhAwgqde4DAdJXLTW6pOYkQyz8P_gIElgm92eSGi-NVp2FFuvliE-x8sFlltP9s50ypa-SoqpnmFD7LA9q1GMRDJfdWIquI1XbERlGTbIn7oyBpKWMTak3JDQrd1IxKtoC98kvN5iKdT87yLtiXyog0s" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit;"><img alt="" data-original-height="808" data-original-width="804" height="240" src="https://blogger.googleusercontent.com/img/a/AVvXsEiy2BlC9BfVlKRVI5ogoSPRnSuomBI5XMMWnSPRhAwgqde4DAdJXLTW6pOYkQyz8P_gIElgm92eSGi-NVp2FFuvliE-x8sFlltP9s50ypa-SoqpnmFD7LA9q1GMRDJfdWIquI1XbERlGTbIn7oyBpKWMTak3JDQrd1IxKtoC98kvN5iKdT87yLtiXyog0s" width="239" /></span></a></div><span style="font-family: inherit;"><br /><br /></span></div><p></p><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(69,89,164,.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; margin: 1.25em 0px; white-space-collapse: preserve;"><span style="font-family: inherit;">Then, an obvious next question came up: How can we also normalize the x-axis, which shows the number of papers?</span></p><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(69,89,164,.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; margin: 1.25em 0px; white-space-collapse: preserve;"><span style="font-family: inherit;">Older scholars have more years to publish, giving them more chances to write high-percentile papers. To control for that, we also calculated the percentiles for the papers published, by using a dataset of around 15,000 faculty members at top US universities. The plot below shows how the percentiles for the number of publications evolve over time.</span></p><p></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiyuT3QRzyWSzBaG34vWVB6YXSw4AGBo1EHUDuBWrMDck3dVnsZB_FO0ZpJxWedvtWxt6eoRPtT0DLR80b3HpdIaFR4qL9LmCDC8Jwk4V1hK4SaotG9TMDL382H0J2zZHMSzdwweun26arZAJ-WoKLudzDovwF_2_UdUbvjF7ysGJHgVHWtYegiL0dWa1o" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit;"><img alt="" data-original-height="908" data-original-width="1156" height="240" src="https://blogger.googleusercontent.com/img/a/AVvXsEiyuT3QRzyWSzBaG34vWVB6YXSw4AGBo1EHUDuBWrMDck3dVnsZB_FO0ZpJxWedvtWxt6eoRPtT0DLR80b3HpdIaFR4qL9LmCDC8Jwk4V1hK4SaotG9TMDL382H0J2zZHMSzdwweun26arZAJ-WoKLudzDovwF_2_UdUbvjF7ysGJHgVHWtYegiL0dWa1o" width="306" /></span></a></div><span style="font-family: inherit;"><br /><br /></span></div><span style="font-family: inherit;"><span style="color: #374151; white-space-collapse: preserve;">Now, we can use the percentile scores for the number of papers published to normalize the x-axis as well. Instead of showing the raw number of papers on the x-axis, we normalize paper productivity against the percentile benchmark shown above. The result is a graph like this for the superstar Jure Leskovec</span><br /></span><div><span style="font-family: inherit;"><br /></span></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjcO7NuKjr8wT1WRMFD2k-BQX2DDALQVkxpncuRoA_TSLqs0z76oGr2kH-rb2CpiPKYiXsemBR-K7vjsn8XxXPhf2M_2zJ4N4MofbuKv9DHxEZTBrwzZh4ElEPL-I-6b5b9Geku_lxMrH3khCG9c30VCcF6a54Y4FuBi7EuL-be9oCTecA6v5po-xsV1Ww" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit;"><img alt="" data-original-height="812" data-original-width="804" height="240" src="https://blogger.googleusercontent.com/img/a/AVvXsEjcO7NuKjr8wT1WRMFD2k-BQX2DDALQVkxpncuRoA_TSLqs0z76oGr2kH-rb2CpiPKYiXsemBR-K7vjsn8XxXPhf2M_2zJ4N4MofbuKv9DHxEZTBrwzZh4ElEPL-I-6b5b9Geku_lxMrH3khCG9c30VCcF6a54Y4FuBi7EuL-be9oCTecA6v5po-xsV1Ww" width="238" /></span></a></div><span style="font-family: inherit;"><br /><span style="color: #374151; white-space-collapse: preserve;">and a less impressive one for yours truly:</span></span></div><div><div class="separator" style="clear: both; text-align: center;"><span style="font-family: inherit;"><br /></span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjg_3k00YZu7wIZQOSinqJD4WlQv4YaNODVde-9vdLgcIivSyPQoVfDOImkEOwbIrCe3tw07yNz_WlkqXODQQmyTdi0ebXkv0PR7e-ssQePSLl59QhJFj64HQXGiBg9HfsjsqXwzd9mtLN4nKo-TYkpUZt2bytIDndiR0b8vllej-BPRd2YOx10Mqir8Lg" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit;"><img alt="" data-original-height="812" data-original-width="812" height="240" src="https://blogger.googleusercontent.com/img/a/AVvXsEjg_3k00YZu7wIZQOSinqJD4WlQv4YaNODVde-9vdLgcIivSyPQoVfDOImkEOwbIrCe3tw07yNz_WlkqXODQQmyTdi0ebXkv0PR7e-ssQePSLl59QhJFj64HQXGiBg9HfsjsqXwzd9mtLN4nKo-TYkpUZt2bytIDndiR0b8vllej-BPRd2YOx10Mqir8Lg" width="240" /></span></a></div><div><span style="font-family: inherit;"><br /></span></div><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(69,89,164,.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; margin: 1.25em 0px; white-space-collapse: preserve;"><span style="font-family: inherit;">Now, with a graph like this, with the x and y axes being normalized between 0 and 1, we have a nice new score that we have given the thoroughly boring name "Percentile in Percentile Area Under the Curve" score, or PiP-AUC for short. It is a score that ranges between 0 and 1, and you can play with different names to see their scores.</span></p><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(69,89,164,.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; margin: 1.25em 0px; white-space-collapse: preserve;"><span style="font-family: inherit;"><strike>At some point, we may also calculate the percentile scores of the PiP scores, but we will do that in the future. :-) </strike></span> UPDATE: If you are also curious about the percentiles for the PiP-AUC scores, here is the distribution:</p><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(69,89,164,.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; margin: 1.25em 0px; white-space-collapse: preserve;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj6VCaQZJxRrmmoxGt8Z3zPeITrQmul73Lgbqb_PmGfuErYW8R51sXp7iQGGx_IhTbQUaquucuW65-mSpo90jHKuw3gNwCZf1JfT-kk78KFHmQjx2XOKLMDqEVreG8ycONkuntiLJac2z37er6isqx9MsfHSE92x9rLtauRZY8h60R9VpRm4mPT0Co9kn0" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1306" data-original-width="1308" height="240" src="https://blogger.googleusercontent.com/img/a/AVvXsEj6VCaQZJxRrmmoxGt8Z3zPeITrQmul73Lgbqb_PmGfuErYW8R51sXp7iQGGx_IhTbQUaquucuW65-mSpo90jHKuw3gNwCZf1JfT-kk78KFHmQjx2XOKLMDqEVreG8ycONkuntiLJac2z37er6isqx9MsfHSE92x9rLtauRZY8h60R9VpRm4mPT0Co9kn0" width="240" /></a></div><br />The x-axis shows the PiP-AUC score, and the y-axis shows the corresponding percentile. So, if you have a PiP-AUC score of 0.6, you are in the top 25% (i.e., 75% percentile) for that metric. With a score of 0.8, you are in the top 10% (i.e., 90% percentile), etc.<p></p><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(69,89,164,.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; margin: 1.25em 0px; white-space-collapse: preserve;">In general, the tool is helpful when trying to understand the impact of newer work published in the last few years. Especially for people with many highly cited but old papers, the percentile scores are very helpful for quickly finding the newer gems. I also like the PiP-AUC scores and plots, as they offer a good balance of overall productivity and impact. Admittedly, it is a strict score, so it is not especially bragging-worthy most of the time :-)</p><p style="--tw-border-spacing-x: 0; --tw-border-spacing-y: 0; --tw-ring-color: rgba(69,89,164,.5); --tw-ring-offset-color: #fff; --tw-ring-offset-shadow: 0 0 transparent; --tw-ring-offset-width: 0px; --tw-ring-shadow: 0 0 transparent; --tw-rotate: 0; --tw-scale-x: 1; --tw-scale-y: 1; --tw-scroll-snap-strictness: proximity; --tw-shadow-colored: 0 0 transparent; --tw-shadow: 0 0 transparent; --tw-skew-x: 0; --tw-skew-y: 0; --tw-translate-x: 0; --tw-translate-y: 0; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; color: #374151; margin: 1.25em 0px; white-space-collapse: preserve;"><i>(With thanks to Sen Tian and Jack Rao for their work.)</i></p></div>Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-44713660486445202662022-10-18T22:53:00.002-04:002022-10-18T22:53:10.523-04:00 Tell these fucking colonels to get this fucking economist out of jail.<p style="text-align: justify;"><span style="background-color: white; color: #050505; font-size: 15px; white-space: pre-wrap;"><span style="font-family: Roboto;">Today is October 18th. It is 41 years since Greece voted for Andreas Papandreou with a 48% vote percentage to be elected as prime minister, fundamentally changing the course of history for Greece. Positively or negatively, this is still debated, but the change was real.</span></span></p><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;">On October 6th, Roy Radner passed away at the age of 95. He was a faculty member at our department and a famous microeconomist with a highly distinguished career. Many others have written about him and his accomplishments as an economist and academic, so I will not try to do the same. </span></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;">But Roy also played an important role in making that election in 1981 possible. Why? Let me tell you his story.</span></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; margin: 0.5em 0px 0px; overflow-wrap: break-word;"><div dir="auto"><div style="text-align: justify;"><span style="color: #050505; font-size: 15px; white-space: pre-wrap;"><span style="font-family: Roboto;"><br /></span></span></div><span style="color: #050505; font-size: 15px; white-space: pre-wrap;"><span style="font-family: Roboto;"><a name='more'></a></span></span></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;"><br /></span></div><div dir="auto"><div style="text-align: justify;"><span style="font-family: Roboto;">When I joined Stern in 2004, Roy Radner came to my office, telling me (lovingly) that he dislikes data mining, but I should not take that personally.</span></div><span class="x3nfvp2 x1j61x8r x1fcty0u xdj266r xhhsvwb xat24cr xgzva0m xxymvpz xlup9mm x1kky2od" style="display: inline-flex; height: 16px; margin: 0px 1px; vertical-align: middle; width: 16px;"><span style="font-family: Roboto;"><img alt="🙂" height="16" referrerpolicy="origin-when-cross-origin" src="https://static.xx.fbcdn.net/images/emoji.php/v9/ta5/1.5/16/1f642.png" style="border: 0px; text-align: justify;" width="16" /></span></span><div style="text-align: justify;"><span style="font-family: Roboto;"><br /></span></div><div style="text-align: justify;"><span style="font-family: Roboto;">He also wanted to connect with me, so he shared a story with me. So, he started talking:</span></div></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;">Roy:</span></div><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;"><br /></span></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">"I had a friend from Greece. But he died a few years back. "</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">[...]</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">"He hired me for my first job at Berkeley. A great economist and a great department chair. Strong Trotskyist. Back in the day, especially at Berkeley, economists were not afraid to declare their political views."</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">[...]</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">"I visited him in Greece, coming from Italy by ferry and then driving a long way down the Western part of Greece. He had a nice Polish mother and an American wife. He also had a young son; I loved playing with him."</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">[...]</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">“At some point, he left Berkeley and returned to Greece to start a new economics research center after the prime minister invited him.”</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">[...]</span></i></div><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;">(NB: At this point, I understand he is talking about Andreas Papandreou, and I am starstruck listening to all the first-hand stories about him.)</span></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">[...]</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">“Well, when the dictatorship came, they arrested him. And there were rumors that the colonels may execute him.”</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;"><br /></span></i></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">“But he was a famous economist, very well-respected. The idea that a fellow academic may be executed, because of his beliefs, in a Western, allied country was unbelievable.”</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;"><br /></span></i></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">“So, the unthinkable happened. For the first time in history, 250 economists agreed on something. We wrote a letter demanding that the dictators release Andreas Papandreou immediately.”</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;"><br /></span></i></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">“We wrote a letter to the US President, Lyndon Johnson, asking him to intervene and get Andreas Papandreou out of jail."</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;"><br /></span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">"As part of a committee, Kenneth Galbraith, Kenneth Arrow, and I go to the White House to deliver the message. Johnson agrees to see us for five minutes.”</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;"><br /></span></i></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">“Panos, you may not be familiar with US Presidents, but Johnson was a rough Texan. He was not known for being gentle and polite, and his language was not exactly… presidential.”</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;"><br /></span></i></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;">“So, after we talked to Johnson, he rolled his eyes, he picked up the phone, and said:”</span></i></div><div dir="auto" style="text-align: justify;"><i><span style="font-family: Roboto;"><br /></span></i></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><i><b><span style="font-family: Roboto;">“Tell these fucking colonels to get this fucking economist out of jail.”</span></b></i></div><div dir="auto" style="text-align: justify;"><i><b><span style="font-family: Roboto;"><br /></span></b></i></div><div dir="auto" style="text-align: justify;"><i><b><span style="font-family: Roboto;">(and the rest is history)</span></b></i></div><div dir="auto" style="text-align: justify;"><i><b><span style="font-family: Roboto;"><br /></span></b></i></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;">This is how Roy has changed the history of Greece. </span></div><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;"><br /></span></div><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;">By getting Johnson to tell the fucking colonels to get that fucking economist out of jail. So that the fucking economist can then be a three-times prime minister of Greece and one of the most consequential prime ministers of the modern Greek republic.</span></div><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;"><br /></span></div></div><div class="x11i5rnm xat24cr x1mh8g0r x1vvkbs xtlvy1s x126k92a" style="background-color: white; color: #050505; font-size: 15px; margin: 0.5em 0px 0px; overflow-wrap: break-word; white-space: pre-wrap;"><!--more--><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;"><br /></span></div><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;">NYTimes: <a href="https://timesmachine.nytimes.com/timesmachine/1967/05/08/83601370.html?pageNumber=1&fbclid=IwAR30IsWXH8qaWatSYCN4JoGuq5dBcZc41aQj_Sn9_W8KAb3cRXJvAezcCG4" target="_blank">Johnson to Appeal to Save Jailed Son of Papandreou</a></span></div><div dir="auto" style="text-align: justify;"><span style="font-family: Roboto;">NYTimes: <a href="https://timesmachine.nytimes.com/timesmachine/1967/05/03/107188981.html?pageNumber=43&fbclid=IwAR0SzDUnGfo0-RCa2MILF6u7S7msr9SsQCRBubcqk9OQ-ZsPfJEbqSO8qu4" target="_blank">Letters to the Editor of The Times - Andreas Papandreou</a></span></div><div dir="auto" style="font-family: inherit; text-align: justify;"><br /></div></div>Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-68183479059560070462021-11-23T15:22:00.000-05:002021-11-23T15:22:09.547-05:00"Geographic Footprint of an Agent" or one of my favorite data science interview questions<p> Last week we wrote in the Compass blog how we <a href="https://medium.com/compass-true-north/estimating-the-geographic-area-of-agent-18bfd45657c6" target="_blank">estimate the geographic footprint of an agent</a>.</p><p>At the very core, the technique is simple: Use the addresses of the houses that an agent has bought or sold in the past; get their longitude and latitude; and then apply a 2-dimensional kernel density estimation to find what are the areas where the agent is likely to be active. Doing the kernel density estimation is easy; the fundamentals of our approach are material that you can find in tutorials for applying a KDE. There are two interesting twists that make the approach more interesting:</p><p></p><ol style="text-align: left;"><li>How can we standardize the "geographic footprint" <b>score</b> to be interpretable? The density scores that come back from a kernel density application are very hard to interpret. Ideally, we want a score from 0 to 1, with 0 being "completely outside of the area of activity" and 1 being "as important as it gets". We show how to use a percentile transformation of the likelihood values to create a score that is normalized, interpretable, and very well calibrated.</li><li>What are the metrics for evaluating such a technique? We show how we can use the concept of "recall-efficiency" curves to provide a common way to evaluate the models.</li></ol><p></p><p><a href="https://medium.com/compass-true-north/estimating-the-geographic-area-of-agent-18bfd45657c6" target="_blank">You can read more in the blog post. </a></p><span><a name='more'></a></span><p>Despite its simplicity, this topic ended up being an amazing interview question. I think it has a great question for separating candidates that have a deeper knowledge of data science from those that have only a superficial understanding.</p><div>The typical question during the interview is:</div><p></p><blockquote>"You have a property where the owner wants to sell the house. How can you determine if the property is <i>geographically </i>relevant to a particular agent? For each agent, we know all their past transactions, and we can assume that future behavior is captured well by their past behavior. For all their past transactions, you can get the address, zip code, longitude, and latitude. <i>Do not try, for now, to figure out if the agent is the <b>best </b>one among many candidate agents; that is a harder problem. </i>Just figure out if the agent is active in the location where the property lies."</blockquote><p>I am very surprised by how many people will start on the wrong foot because they will try to actively shoehorn the problem into a binary classification problem: Indicative answers:</p><p></p><ul style="text-align: left;"><li>"I will find all the properties that the agent has <b>not </b>transacted in the past, and treat them as negatives".</li><li>"I will keep showing properties to the agent and when they say no, I will mark them as negatives"</li><li>"I will add features like bathrooms, bedrooms, price, and then predict if it is relevant or not"</li></ul><div>In my experience, once people start like that, they rarely get to a solution that works. The key problem is that they have no <i><b>easy </b></i>way of evaluating the model they propose. </div><div><br /></div><div>So, a typical follow-up question is:</div><div><blockquote>How would you evaluate your approach?</blockquote></div><div>This now gets interesting. Common superficial answers are:</div><div><ul style="text-align: left;"><li>I would use precision and recall, or AUC.</li><li>I would measure how often agents accept my recommendations</li></ul></div><div>If they recommend precision and recall, I ask how they are going to measure recall (relatively easy, hopefully, the answer is a temporal training/test split), and precision. Precision gets tricky, as there is no "base rate" and there are no clear "negatives".</div><div><br /></div><div>If someone recommends measuring the agent reactions, I acknowledge that this is correct, in principle, but ask them to propose an offline evaluation, so that we can test the approach before releasing it to the agents. </div><div><br /></div><div>This is typically the point where the better candidates, even they veered off track will start to move towards a better solution (not necessarily KDE, it can also be clustering, convex hulls, histograms on zipcodes as vector representations, etc), while the one-trick ponies will reveal their lack of substance.</div><div><br /></div><div>Kind of sorry that I have to retire this question from my question bank. Still impressed that such a simple question had such a strong discriminatory power.</div><div><br /></div><div><br /></div><p></p><p></p>Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-73388689879495034142019-11-18T16:27:00.004-05:002019-11-18T17:08:13.821-05:00Mechanical Turk, 97 cents per hour, and common reporting biases<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The New York Times has an article about Mechanical Turk in today's print edition: "<a href="https://www.nytimes.com/interactive/2019/11/15/nyregion/amazon-mechanical-turk.html">I Found Work on an Amazon Website. I Made 97 Cents an Hour</a>". (You will find a couple of quotes from yours truly).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The content of the article follows the current zeitgeist: Tech companies exploiting gig workers. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
While it is hard to argue that there are tasks on MTurk that are really bad, I think that the article paints an unfairly gloomy picture of the overall platform.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Here are a few of the issues:</div>
<div>
<ul>
<li style="text-align: justify;"><b>Availability and survivorship bias. </b>While the paper does describe accurately the cesspool of low-paying tasks that are available on Mechanical Turk, it fails to convey the fact that these tasks are available on the platform <i><b>because nobody wants to work on them</b></i>. The tasks that are easily available for everyone are the ones for which nobody competes to grab: low-paying, badly designed tasks.</li>
<li style="text-align: justify;"><b>The activity levels of workers follow a power-law</b>. We have plenty of evidence that a significant part of the work on MTurk is done by a small minority of workers. While it is hard to have a truly accurate measurement of what percent of the workers do what percent of the tasks, <a href="https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture)">the 1% rule</a> is a good approximation. For example, in my demographic surveys, where I explicitly limit the participation to only once per month, 50% of the responses come from 5% of the participants. Expect the bias to be much stronger in other, more desirable tasks. Such a heavily biased propensity to participate introduces strong sampling problems when trying to find the right set of workers to interview.</li>
<li style="text-align: justify;"><b>Doing piecemeal work while untrained results in low pay</b>. This is a pet peeve of mine, for all the articles of the type "I tried working on MTurk / driving Uber / delivering packages / etc / and I got a lousy pay". Well, if you work piecemeal on <b><i>any </i></b>task, the tasks will take a very long time initially, and the hourly wage will suck. This will hold for Turking, coding, lawyering, or anything else. If someone decides to become a freelance journalist, the first few articles will result in abysmally bad hourly wages as well; expert freelance writers often charge 10x the rates that beginner freelancer writers charge, if not more. I am 100% confident that the same applies to MTurk workers as well: <b><i>Experienced workers make 10x what beginners make.</i></b></li>
</ul>
</div>
<div style="text-align: justify;">
Having said that, I do agree that Amazon could prohibit tasks that are obviously paying very little (as a rule of thumb, it is impossible to get paid more than minimum wage when the HIT is paying less than 5c/task). But I also think that regular workers are smart enough to know that and avoid such tasks. </div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-4604544329103673672018-11-16T13:32:00.000-05:002018-11-16T13:32:28.511-05:00Distribution of paper citations over timeA few weeks ago we had a discussion about citations, and how we can compare the citation impact of papers that were published in different years. Obviously, older papers have an advantage as they have more time to accumulate citations.<br />
<br />
To compare papers, just for fun, we ended up opening the profile page of each paper in Google Scholar, and we analyzed the paper citations years by year to find the "winner." (They were both great papers, by great authors, fyi. It was more of a "Lebron vs. Jordan" discussion, as opposed to anything serious.)<br />
<br />
This process got me curious though. Can we tell how a paper is doing at any given point in time? How can we compare a 2-year-old article, published in 2016, with 100 citations against a 10-year-old document, published in 2008, with 500 citations?<br />
<br />
To settle the question, we started with the profiles of faculty members in the top-10 US universities and downloaded about 1.5M publications, across all fields, and their citation histories over time.<br />
<br />
We then analyzed the citation histories of these publications, and, for each year, we ranked the papers based on the number of citations received over time. Finally, we computed the citation numbers corresponding to different percentiles of performance.<br />
<br />
<b>Cumulative percentiles</b><br />
<b><br /></b>
The plot below shows the number of citations that a paper needs to have at different stages to be placed in a given percentile.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieNZ9OnfTGH6iayyv92b7V3B0yfSBP323qB-_gA1vtRRzvydwahPCGs9YqxLE4TbO5XaX-4qChkuOtXa7mjVwXulLrMaTi_qsIuI4IvHoY5wAnN_4WQ1xqPfJRm_g6N1Dxt70poaYJmN8/s1600/cummulative-citations.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="800" data-original-width="1600" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieNZ9OnfTGH6iayyv92b7V3B0yfSBP323qB-_gA1vtRRzvydwahPCGs9YqxLE4TbO5XaX-4qChkuOtXa7mjVwXulLrMaTi_qsIuI4IvHoY5wAnN_4WQ1xqPfJRm_g6N1Dxt70poaYJmN8/s640/cummulative-citations.png" width="640" /></a></div>
<br />
A few data points, focusing on certain age milestones: 5-years after publication, 10-years after publication, and lifetime.<br />
<br />
<ul>
<li><b>50% line:</b> The performance of a "median" paper<b>.</b> The median paper gets around 20 citations 5 years after publication, 50 citations within 10 years, and around 100 citations in its lifetime. <b><i>Milestone scores: 20,50,90</i></b></li>
<li><b>75% line:</b> These papers perform "better," citation-wise than 75% of the remaining papers with the same age. Such papers get around 50 citations within 5 years, 100 citations within 10 years of publication, and around 200 citations in their lifetime. <b><i>Milestone scores: 50,100,200</i></b></li>
<li><b>90% line:</b> These papers perform better than 90% of the papers in their cohort. Around 90 citations within 5 years, 200 citations within 10 years, and 500 citations in their lifetime. <b><i>Milestones scores: 90,200,500</i></b></li>
</ul>
<div>
<b><i><br /></i></b></div>
<br />
<b>Yearly percentiles and peak years</b><br />
<b><br /></b>
We also wanted to check at which point papers reach their peak, and start collecting fewer citations. The plot below shows the percentiles based on the yearly numbers of accumulated citations. The vast majority of papers tend to reach their peak 5-10 years after publication; the number of yearly citations starts declining after 5-10 years.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtt_fDOV9b7Vn-fN6FUHwYGc3HxLu47YFu_qrzS6FDq0NQn5VfijcIZbinB9uoy6IgXYkaM8a4oMcuWNvtUEiBIQSjSTd6kwW4iRmxTlPR8O0BvM2nUInSPZcEEHUDtofXCAifOEZIBBo/s1600/yearly-citations.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="800" data-original-width="1600" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtt_fDOV9b7Vn-fN6FUHwYGc3HxLu47YFu_qrzS6FDq0NQn5VfijcIZbinB9uoy6IgXYkaM8a4oMcuWNvtUEiBIQSjSTd6kwW4iRmxTlPR8O0BvM2nUInSPZcEEHUDtofXCAifOEZIBBo/s640/yearly-citations.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Below is the plot of the peak year for a paper based on the paper percentile:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilL3XfeLMMSsmDek5ZEChyphenhyphenOq93hWv1aFXP2hf5T8eruq72IuzVoZ9EucMWDM84wOCqbD9-8N9-nLFP-4MaEhQLlH_Be14HmMp1ElTgEcf293vcVWCUJnuMss85JjAaMBe3PtgsxH630pA/s1600/best-year.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="338" data-original-width="631" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilL3XfeLMMSsmDek5ZEChyphenhyphenOq93hWv1aFXP2hf5T8eruq72IuzVoZ9EucMWDM84wOCqbD9-8N9-nLFP-4MaEhQLlH_Be14HmMp1ElTgEcf293vcVWCUJnuMss85JjAaMBe3PtgsxH630pA/s400/best-year.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
There is an interesting effect around the 97.5% percentile: After that level, it seems that a 'rich-gets-richer' effect kicks in, and we effectively do not observe a peak year. The number of <b><i>citations per year</i></b> keeps increasing. You could call these papers the "classics".<br />
<br />
What does it take to be a "classic"? 200 citations at 5 years or 500 citations at 10 years.Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-86396724444437973912018-01-29T09:53:00.001-05:002018-01-31T09:07:57.320-05:00How many Mechanical Turk workers are there?<i>TL;DR: There are about 100K-200K unique workers on Amazon. </i><i>On average, there are 2K-5K workers active on Amazon at any given time, which is equivalent to having 10K-25K full-time employees. On average, 50% of the worker population changes within 12-18 months. </i><i>Workers exhibit widely different patterns of activity, with most workers being active only occasionally, and few workers being very active. Combining our results with the results from <a href="https://arxiv.org/abs/1712.05796">Hara et al</a>, we see that MTurk has a yearly transaction volume of a few hundreds of millions of dollars.</i><br />
<i><br /></i>
<i>For more details read below, or take a look <a href="http://www.ipeirotis.com/?publication=demographics-and-dynamics-of-mechanical-turk-workers">at our WSDM 2018 paper</a>.</i><br />
<br />
<div style="text-align: center;">
--</div>
<br />
<b>Question</b><br />
<br />
A topic that frequently comes up when discussing Mechanical Turk is "how many workers are there on the platform"?<br />
<br />
In general, this is a question that is very easy for Amazon to answer, but much harder for outsiders. Amazon claims that there are 500,000 workers on the platform. How can we check the validity of this statement?<br />
<br />
<div style="text-align: center;">
--</div>
<br />
<b>Basic capture-recapture model</b><br />
<br />
A common technique for this problem is the <b>capture-recapture</b> technique, that is widely used in the field of ecology, to measure the population of a species.<br />
<br />
The simplest possible technique is the following:<br />
<ul>
<li><b>Capture/marking </b>phase: Capture $n_1$ animals, mark them, and release them back. </li>
<li><b>Recapture </b>phase: A few days later, capture $n_2$ animals. Assuming there are $N$ animals overall, $n_1/N$ of them are marked. So, for each of the $n_2$ captured animals, the probability that the animal is marked is $n_1/N$ (from the capture/marking phase).</li>
<li><b>Calculation</b>: On expectation, we expect to see $n_2 \cdot \frac{n_1}{N}$ marked animals in the recapture phase. (Notice that we do not know $N$.) So, if we actually see $m$ marked animals during the recapture phase, we set $m = n_2 \cdot \frac{n_1}{N}$ and we get the estimate that:<br /><br /><div style="text-align: center;">
$N = \frac{n_1 \cdot n_2}{m}$.</div>
</li>
</ul>
<div>
In our setting we adapted the same idea, where "capture" and "recapture" correspond to participating in a demographics survey. In other words, we "capture/mark" MTurk users that complete the survey in one day. Then, in another day, we also "recapture" by surveying more workers and we see how many workers overlap in the two surveys.<br />
<br />
<div style="text-align: center;">
--</div>
<b><br /></b>
<b>First (naive) attempt</b><br />
<br />
We decided to apply this technique to estimate the size of the Mechanical Turk population. We considered as "capture" period the set of surveys running over a period of 30 days. Then we considered as "recapture" period, the surveys that we ran on another 30-day period. The plot below shows the results.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0m23rFIOGqrEIgS6hdLVNX1U3UwZMHqFENXLcOFwdc5uaGepy8VJdFEvKThI2GmvJjVUe-z6F0yaJeDUpAxRT-oJdI7nWuNZsEljvwDdaR8ZnCNvZJuuATZeb9AfPZt1DtN9E_Og-YQc/s1600/population+estimates.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1017" data-original-width="1580" height="410" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0m23rFIOGqrEIgS6hdLVNX1U3UwZMHqFENXLcOFwdc5uaGepy8VJdFEvKThI2GmvJjVUe-z6F0yaJeDUpAxRT-oJdI7nWuNZsEljvwDdaR8ZnCNvZJuuATZeb9AfPZt1DtN9E_Og-YQc/s640/population+estimates.PNG" width="640" /></a></div>
<br /></div>
<div>
The x-axis shows the beginning of the recapture period, and the y-axis the estimate of the number of workers. The color of each dot corresponds to the difference in time between the capture-recapture periods: black is a short time, and red is a long time.<br />
<br />
If we focus on the black-color dots (~60 days between the surveys), we get a (naive) estimate of around 10K-15K workers. <i>(Warning: this is incorrect.)</i><br />
<br />
While we could stop here, we see some results that are not consistent with our model. Remember, that color encodes time between samples: black is for short time (~2 months) between samples, red is for long time (~2yrs) between samples. Notice that, as the time between the two periods increases, the estimates are becoming higher, and we get the "rainbow cake" effect in the plot. For example, for July 2017, our estimate is 12K workers if we compare with a capture from May 2017, but the estimate goes up to 45K workers if we compare with a sample from May 2015. Our model, though, says that the time between captures should <b><i>not </i></b>affect the population estimates. This indicates that there is something wrong with the model.<br />
<br />
<div style="text-align: center;">
--</div>
<br />
<b>Assumptions of basic model</b><br />
<br /></div>
<div>
The basic capture-recapture estimation described above relies on a couple of assumptions. Both of these assumptions are violated when applying this technique to an online environment.</div>
<div>
<ul>
<li><b>Assumption of no arrivals / departures ("closed population")</b>: The vanilla capture-recapture scheme assumes that there are no arrivals or departures of workers between the capture and recapture phase.</li>
<li><b>Assumption of no selection bias ("equal catchability")</b>: The vanilla capture-recapture scheme assumes that every worker in the population is equally likely to be captured.</li>
</ul>
<div>
In ecology, the issue of closed population has been examined under many different settings (birth-death of animals, immigration, spatial patterns of movement, etc.) and there are many research papers on the topic. Catchability, by comparison, has received <i>comparatively </i>less attention. This is reasonable, as in ecology the assumption of closed population is problematic in many settings. By comparison, assuming that the probability of capturing an animal is uniform among similar animals is reasonable. Typically the focus is on segmenting the animals into groups (e.g., nesting females vs hunting males) and assign different catchability heterogeneity to groups (but not to individuals). </div>
<div>
<br /></div>
<div>
In online settings though, the assumption of equal catchability is more problematic. First we have the <b>activity bias</b>: Workers exhibit very different levels of activity: A worker who works every day is much more likely to see and complete a task, compared to someone who works once a month. Similarly, we have a <b>selection bias</b>: Some workers may like to complete surveys, while others may avoid such tasks.<br />
<br />
So, to improve our estimates, we need to use models that alleviate these assumptions.</div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
--</div>
</div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: left;">
<b>Endowing workers with survival probabilities </b></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
We can extend the model, allowing each worker to have a certain survival probability, to allow workers to "disappear" from the platform. If we see the plot above, we can see that the population estimate increases as the time between two samples increases. This hints that workers leave the platform, and the intersection of capture-recapture becomes smaller over time. </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
If we account for that, we can get an estimate that the "half-life" of a Mechanical Turk worker is between 12-18 months. <b>In other words, approximately 50% of the Mechanical Turk population changes every 12-18 months. </b></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: center;">
--</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<b>Endowing workers with propensity to participate</b></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
We can also extend the model by associating a certain <i>propensity </i>for each worker. The propensity is the probability that a worker is active and willing to participate in a task, at any given time.<br />
<br />
In our work, we assumed that the underlying "propensity to participate" follows a Beta distribution across the worker population, and the parameters of the Beta distribution are unknown. When we assume that follow a Beta distribution, then the probability that a worker participates in the survey k times, follows a <a href="https://en.wikipedia.org/wiki/Beta-binomial_distribution">Beta Binomial distribution</a>. Since we know how many workers participated k times in our surveys, it is then easy to estimate the underlying parameters of the Beta distribution.<br />
<br />
Notice that we had to depart from the simple "two occasion" model above, and instead use multiple capturing periods over time. Intuitively, workers that have high propensity to participate will appear many times in our results, while inactive workers will appear only a few times.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
By doing this analysis, we can observe that (as expected) the distribution of activity is highly skewed: A few workers are very active in the platform, while others are largely inactive. A nice property of the Beta distribution is its flexibility: Its shape can be pretty much anything: uniform, Gaussian-like, bimodal, heavy-tailed... you name it. </div>
<div style="text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO6aQ7E-A8quBKIufXusElm8gEfEfNS7NdZsD16BkRPLxe3BICrIpTvybtkUUA30_MzU9TmD9nVRVTshMU2RLCJ0URFRm-OIQuitZ378_svgVlhlgDQFVjOTgwzddJ-F2GbaT_8MsbP8w/s1600/beta-histogram.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="521" data-original-width="978" height="340" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO6aQ7E-A8quBKIufXusElm8gEfEfNS7NdZsD16BkRPLxe3BICrIpTvybtkUUA30_MzU9TmD9nVRVTshMU2RLCJ0URFRm-OIQuitZ378_svgVlhlgDQFVjOTgwzddJ-F2GbaT_8MsbP8w/s640/beta-histogram.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: center;">
<br />
<div style="text-align: left;">
<br />
In our analysis, we estimated that the propensity distribution follows a Beta(0.3,20) distribution. We plot above the "inverse CDF" of the distribution (Inverse CDF: "what percentage of the workers have propensity higher than x").<br />
<br />
As you can see, the propensity follows a familiar (and expected) pattern. Only 0.1% of the workers have propensity higher than 0.2, and only 10% have propensity higher than 0.05.<br />
<br />
Intuitively, a propensity of 0.2 means that the worker is active and willing to participate 20% of their time (this is roughly equivalent to full-time level of activity; full-timer employees work around 2000 hrs per year, out of 24*365 available hours in a year). A propensity of 0.05 means that the worker is active and available approximately 24 hr * 0.05 ~ 1 hour per day.<br />
<br /></div>
--<br />
<br /></div>
<div style="text-align: left;">
<b>How big is the platform?</b></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
So, how many workers are there? Under such highly skewed distributions, giving an exact number for the number of workers is rather futile. The best that you can do is give a ballpark estimate, and hope to be roughly correct on the order of magnitude. What our estimates are showing is that <b>there are round 180K distinct workers in the MTurk platform</b>. This is good news for anyone who is trying to reach a large number of distinct workers through the platform. </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Our analysis also allows us to estimate how many workers are active and willing to participate in our task at any given time. For that, we estimate that around <b>2K to 5K workers are available, at any given time</b>. If we want to convert this number to full-time employee equivalence, this is equivalent to 10K-25K full-time workers.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The latter part also allows us to give some low and high estimates on the transaction volume of MTurk. </div>
<div style="text-align: left;">
<ul>
<li><b>Lower bound</b>: Assuming 2K workers active at any given time, this is 2000*24*365=17,520,000 work hours in a year. If we assume that <a href="https://arxiv.org/abs/1712.05796">the median wage is \$2/hr</a>, this is roughly <b>\$35M/yr </b>transaction volume on Amazon Mechanical Turk (with Amazon netting ~\$7M in fees).</li>
<li><b>Upper bound</b>: Assuming 5K workers active at any given time, this is 5000*24*365=43,800,000 work hours in a year. If we assume average wage of \$12/hr, this is around <b>\$525M/yr</b> transaction volume (with Amazon netting ~$100M in fees).</li>
</ul>
<div>
I understand that a range of \$35M to \$500M may not be very helpful, but these are very rough estimates. If someone wanted my own educated guess, I would put it somewhere in the middle of the two, i.e., transaction volume of a few hundreds of millions of dollars.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-84068740347514533472017-01-17T11:42:00.000-05:002017-01-17T16:08:41.997-05:00Why was my Amazon Mechanical Turk registration denied?(<a href="https://www.quora.com/Why-was-my-Amazon-Mechanical-Turk-registration-denied/answer/Panos-Ipeirotis?srid=wO">This is my answer to a question posted on Quora</a>)<br />
<br />
Mechanical Turk is a platform for work. Workers get paid, which makes now Amazon a payment processor. Payment processors are moving money on behalf of other people, and therefore are under heavy scrutiny from the US government for issues related to money laundering (AML), counter-terrorism, tax compliance, etc.<br />
<br />
One of the key things that is required from financial institutions is to have a <a href="https://en.wikipedia.org/wiki/Customer_Identification_Program">“Customer Identification Program” (CIP),</a> also known as <a href="https://en.wikipedia.org/wiki/Know_your_customer">“Know Your Customer” (KYC) </a>process. The CIP/KYC is a set of procedures that the financial institution needs to follow to establish that they know the true identity of a customer. The processes that each financial institutions follows vary, and the exact processes are rarely available to the public, as they are considered security measures. Furthermore, the practices are regularly monitored by regulators (OCC, Fed, FinCEN, etc) and change over time to follow best practices.<br />
<br />
In your particular case, the most likely reason is that Amazon was not able to verify your identity.<br />
<br />
If you are in the US, Amazon most probably can get your SSN and other personal details and verify whether you are a real person. However, even if you live in the US, if you have no credit history, no bank accounts, and so on, the verification will come back with low confidence. Following standard risk management processes, Amazon could plausibly reject such applications, as part of their CIP processes: it is better to have a false negative (rejecting a normal account) than having a false positive (e.g., accepting an account that will be involved in money laundering or tax-evasion schemes).<br />
<br />
For other countries, the ability of Amazon, to follow CIP/KYC processes that conform to the US regulations, varies. I assume, for example, that the cooperation of US with UK or Australian authorities is much smoother compared to, say, Chinese authorities. So, if you live outside the US, the probability of having your account approved depends on how robust is the ability of Amazon to verify individual identities in your country.<br />
<br />
Given that Amazon gets paid by requesters, I assume their focus is to establish CIP processes first in regions where potential requesters reside, which is not always the place where workers reside. This also means that you are more likely to be approved if you first register as a requester (assuming this is an option for you), and then try to create the worker account.<br />
<br />Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-92130853776941783612016-03-13T14:10:00.003-04:002016-08-02T11:39:37.585-04:00AlphaGo, Beat the Machine, and the Unknown UnknownsIn Game 4, of the 5-game series between AlphaGo and Lee Sedol, the human Go champion, Lee Sedol <a href="http://www.nytimes.com/aponline/2016/03/13/world/asia/ap-as-skorea-game-human-vs-computer.html">managed to get his first win</a>. According to the NY Times article:<br />
<br />
<blockquote class="tr_bq">
Lee had said earlier in the series, which began last week, that he was unable to beat AlphaGo because he could not find any weaknesses in the software's strategy. But after Sunday's match, the 33 year old South Korean Go grandmaster, who has won 18 international championships, said he found two weaknesses in the artificial intelligence program. <b>Lee said that when he made an unexpected move, AlphaGo responded with a move as if the program had a bug, indicating that the machine lacked the ability to deal with surprises.</b></blockquote>
<br />
<hr width="50%" />
<br />
This part reminded me of one of my favorite papers: <b><a href="http://www.ipeirotis.com/?publication=beat-the-machine-challenging-humans-to-find-a-predictive-models-unknown-unknowns">Beat the Machine: Challenging Humans to Find a Predictive Model’s “Unknown Unknowns”</a>. </b><br />
<br />
In the paper, we tried to use humans to "beat the machine" and identify vulnerabilities in a machine learning system. The key idea was to reward humans whenever they identify cases where the machine fails, while also being confident that it provides the correct answer. In other words, we encouraged humans to find "unexpected" errors, not just cases where naturally the machine was going to be uncertain.<br />
<br />
<hr width="50%" />
<br />
As an example case, consider a system that detects adult content on the web. Our baseline machine learning system had an accuracy of ~99%. Then, we asked Mechanical Turk workers to do the following task: Find web pages with adult content that the machine learning system classifies as non-adult with high confidence. The humans had no information about the system, and the only thing they can do was to submit a URL and get back an answer.<br />
<br />
The reward structure was the following: Humans get \$1 for each URL that the machine misses, otherwise they get \$0.001. In other words, we provided a strong incentive to find problematic cases.<br />
<br />
After some probing, humans were quick to uncover underlying vulnerabilities: For example, adult pages in Japanese, Arabic, etc., were classified by our system as non-adult, despite their obvious adult content. Similarly for other categories, such as hate speech, violence, etc. <b>Humans were quickly able to "beat the machine" and identify the "unknown unknowns".</b><br />
<br />
<hr width="50%" />
<br />
Simply told, humans were able to figure out what are the likely cases that the system may have missed during training. At the end of the day, the training data is provided by humans, and no system has access to all possible training data. We operate in an "open world" while training data implicitly assume a "closed world".<br />
<br />
As we see from the AlphaGo example, since most machine learning systems rely on existence of training data (or some immediate feedback for their actions), machines may get into problems when they have to face examples that are unlike any examples they have processed their training data.<br />
<br />
We designed our Beat The Machine system to encourage humans to discover such vulnerabilities early.<br />
<br />
In a sense, our BTM system is s like hiring hackers to break into your network, to identify security vulnerabilities before they become a real problem. The BTM system applies this principle for machine learning systems, encouraging a period of intense probing for vulnerabilities, before deploying the system in practice.<br />
<br />
Well, perhaps Google hired Lee Sedol with the same idea: Get the human to identify cases where the machine will fail, and reward the human for doing so. Only in that case, AlphaGo managed to eat its cake (figure out a vulnerability) and have it too (beat Lee Sedol, and not pay the \$1M prize) :-)Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-19565406162079001762016-02-29T10:33:00.001-05:002016-03-01T01:21:25.250-05:00A Cohort Analysis of Mechanical Turk Requesters<div style="text-align: justify;">
In my last post, I examined the number of "active requesters" on Mechanical Turk, and concluded that there is a significant decline in the numbers over the last year. The definition of "active requester" was: "<i>A requester is active at time X if he has a HIT running at time X</i>". A potential issue with this definition is that an improvement in the speed of HIT completion (e.g., due to increased labor supply) could drive down that number.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
For this reason, I decided to perform a proper <a href="https://en.wikipedia.org/wiki/Cohort_analysis">cohort analysis</a> for the requesters on Mechanical Turk. In the cohort analysis that follows, we will examine how many requesters that have first appeared in the platform on a given month (say September 2015), are still posting tasks in the subsequent months.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Here is the resulting "layer cake plot" that indicates that happens in each cohort. Each of the layers corresponds to requesters that were first seen on a given month. (<a href="https://gist.github.com/ipeirotis/ce26e0e76a5192f89c2e">code</a>, <a href="https://gist.github.com/ipeirotis/6af638e971537b0f9524">data</a>) <span style="font-size: x-small;">(<a href="http://tomblomfield.com/post/81105143223/customer-churn-can-kill-your-startup">Read this post, if want a little bit more background on how the plot should "look like".</a>)</span></div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1Sca6QiHNYwXgPbRSR_QD-9-XhW8yjQgcRPwf_vEfiU8wniLNyFemUJOMcT-m6ooVwC4rSQOmhmc7uiuYQv1jaOpRnZwF53NF1tF5LSajDChvqLp4coiK1HRMmsvlxP3RhU3BkyAvfiA/s1600/mturk-cohort-analysis.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1Sca6QiHNYwXgPbRSR_QD-9-XhW8yjQgcRPwf_vEfiU8wniLNyFemUJOMcT-m6ooVwC4rSQOmhmc7uiuYQv1jaOpRnZwF53NF1tF5LSajDChvqLp4coiK1HRMmsvlxP3RhU3BkyAvfiA/s640/mturk-cohort-analysis.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
For example, the bottom layer corresponds to all the requesters that were first seen on May 2014 (the first month that the new version of MTurk Tracker started collecting data). We can see that we had ~2700 "new" requesters on that month. (The May-2014 cohort obviously contains all prior cohorts in our dataset, as we do not know when these requesters really started posting.) Out of these requesters, approximately 1700 also posted a task on June 2014 or later, approximately 1000 of these have posted a task on March 2015 or later, and approximately 500 have posted a task on February 2016.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The layer on top (slightly darker blue) illustrates the evolution of the June 2014 cohort. By stacking them on top of each other, we can see the composition of the requesters that have been active in every single month.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As the plot makes obvious, until March 2015, the acquisition of new requesters every month was compensating for the requesters that were lost from the prior cohorts. However, starting March 2015, we start seeing a decline in the overall numbers, as the total decline in requesters from prior cohorts dominates the acquisition of new requesters. So, the cohort analysis supports the conclusions of the prior post, as the trends and conclusions are very similar (always good to have a few robustness checks).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Of course, a more comprehensive cohort analysis would also analyze the revenue generated by each cohort, and not just the number of active users. That requires a little bit more digging in the data, but I will do that in a subsequent post.</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-39877745925145725372016-02-26T23:31:00.001-05:002016-02-28T14:07:42.222-05:00The Decline of Amazon Mechanical Turk<div style="text-align: justify;">
It seems that after years of neglect, Mechanical Turk starts losing its appeal. In our latest measurement, we see Mechanical Turk losing 50% of its requesters in a YoY measurement.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
A few days ago, Kristy Milland (aka SpamGirl) asked me if there is a way to see the active requesters on Mechanical Turk over time. I did not have this dashboard on Mechanical Turk tracker, but it was an important metric, so I decided to add it in the MTurk Tracker website.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So, now MTurk Tracker has a tab called "<a href="http://www.mturk-tracker.com/#/activerequesters">Active Requesters</a>" which shows how many requesters are "active" on Mechanical Turk at any given time. The definition of "Active at time X" means "had a task that was running on MTurk before time X and after time X".</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Here is the chart for the active requesters between Jan 1, 2015 and February 28, 2016: </div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkU9t7DoEl3oJrwQkmuiFvIVxj3W-76NGpJyCxvWHLVLDWAWgejNy6XImMtF4GnLJCEg96gxk5iiM3AwuGZ9s2fTQrfvRNPD_xkYAHMRmRWDkKoWWb3Gf_ZiQ_lQkhwN5ZwWB228TzgMI/s1600/mturk-active-requesters.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="310" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkU9t7DoEl3oJrwQkmuiFvIVxj3W-76NGpJyCxvWHLVLDWAWgejNy6XImMtF4GnLJCEg96gxk5iiM3AwuGZ9s2fTQrfvRNPD_xkYAHMRmRWDkKoWWb3Gf_ZiQ_lQkhwN5ZwWB228TzgMI/s640/mturk-active-requesters.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: justify;">
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As you can see, starting March 2016 (that is before the announcement of price increases), we see a decline in the number of active requesters. Interestingly, when the fee increases are announced, we see a small "valley" around the period of fee increases. The numbers remain stable until November, but after that we see a steady decline.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Overall, we observe a YoY decline of almost 50% in terms of active requesters.</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
What is driving the decline? Hard to tell. Perhaps requesters abandon crowdsourcing in favor of more automated solutions, such as deep learning. Perhaps requesters with long running jobs build their own workforce (eg using UpWork). Perhaps they use alternative platforms, such as Crowdflower. Or perhaps my own metric is flawed, and I need to revise it.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But, unless we have a bug in the code, the future does not seem promising for Mechanical Turk. And this is a shame.</div>
<div style="text-align: justify;">
<br /></div>
<div>
<div style="text-align: justify;">
<br /></div>
</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-25246272100640135572015-06-10T12:38:00.001-04:002015-06-10T12:48:16.386-04:00An API for MTurk DemographicsA few months back, I launched <a href="http://demographics.mturk-tracker.com/">demographics.mturk-tracker.com</a>, a tool that runs continuously surveys of the Mechanical Turk worker population and displays live statistics about gender, age, income, country of origin, etc.<br />
<br />
Of course, there are many other reports and analyses that can be presented using the data. In order to make easier for other people to use and analyze the data, we now offer a simple API for retrieving the raw survey data.<br />
<br />
Here is a quick example: We first call the API and get back the raw responses:<br />
<span style="background-color: white; color: navy; font-family: monospace; font-size: 14px; line-height: 1.21429em; text-align: right;"><br />
</span> <span style="background-color: white; color: navy; font-family: monospace; font-size: 14px; line-height: 1.21429em; text-align: right;">In [1]:</span><br />
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; background-color: white; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px;">
<div class="input_area" style="background: rgb(247, 247, 247); border-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython2" style="background: transparent; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">requests</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">json</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">pprint</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">pandas</span> <span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">as</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">pd</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">from</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">datetime</span> <span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="n" style="box-sizing: border-box;">datetime</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">time</span>
<span class="c" style="box-sizing: border-box; color: #408080; font-style: italic;"># The API call that returns the last 10K survey responses</span>
<span class="n" style="box-sizing: border-box;">url</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="s" style="box-sizing: border-box; color: #ba2121;">"https://mturk-surveys.appspot.com/"</span> <span class="o" style="box-sizing: border-box; color: #666666;">+</span> \
<span class="s" style="box-sizing: border-box; color: #ba2121;">"_ah/api/survey/v1/survey/demographics/answers?limit=10000"</span>
<span class="n" style="box-sizing: border-box;">resp</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">requests</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">get</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">url</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">json</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">json</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">loads</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">resp</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">text</span><span class="p" style="box-sizing: border-box;">)</span></pre>
</div>
</div>
</div>
<br />
<br />
Then we need to reformat the returned JSON object and transform the responses into a flat table<br />
<span style="background-color: white; color: navy; font-family: monospace; font-size: 14px; line-height: 1.21429em; text-align: right;"><br />
</span> <span style="background-color: white; color: navy; font-family: monospace; font-size: 14px; line-height: 1.21429em; text-align: right;">In [2]:</span><br />
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; background-color: white; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px;">
<div class="input_area" style="background: rgb(247, 247, 247); border-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython2" style="background: transparent; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span class="c" style="box-sizing: border-box; color: #408080; font-style: italic;"># This function takes as input the response for a single survey, and transforms it into a flat dictionary</span>
<span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">def</span> <span class="nf" style="box-sizing: border-box; color: blue;">flatten</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">):</span>
<span class="n" style="box-sizing: border-box;">fmt</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="s" style="box-sizing: border-box; color: #ba2121;">"%Y-%m-</span><span class="si" style="box-sizing: border-box; color: #bb6688; font-weight: bold;">%d</span><span class="s" style="box-sizing: border-box; color: #ba2121;">T%H:%M:%S.</span><span class="si" style="box-sizing: border-box; color: #bb6688; font-weight: bold;">%f</span><span class="s" style="box-sizing: border-box; color: #ba2121;">Z"</span>
<span class="n" style="box-sizing: border-box;">hit_answer_date</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">datetime</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">strptime</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"date"</span><span class="p" style="box-sizing: border-box;">],</span> <span class="n" style="box-sizing: border-box;">fmt</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">hit_creation_str</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">item</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">get</span><span class="p" style="box-sizing: border-box;">(</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"hitCreationDate"</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">if</span> <span class="n" style="box-sizing: border-box;">hit_creation_str</span> <span class="ow" style="box-sizing: border-box; color: #aa22ff; font-weight: bold;">is</span> <span class="bp" style="box-sizing: border-box; color: green;">None</span><span class="p" style="box-sizing: border-box;">:</span>
<span class="n" style="box-sizing: border-box;">hit_creation_date</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="bp" style="box-sizing: border-box; color: green;">None</span>
<span class="n" style="box-sizing: border-box;">diff</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="bp" style="box-sizing: border-box; color: green;">None</span>
<span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">else</span><span class="p" style="box-sizing: border-box;">:</span>
<span class="n" style="box-sizing: border-box;">hit_creation_date</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">datetime</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">strptime</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">hit_creation_str</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">fmt</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="c" style="box-sizing: border-box; color: #408080; font-style: italic;"># convert to unix timestamp</span>
<span class="n" style="box-sizing: border-box;">hit_date_ts</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">time</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">mktime</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">hit_creation_date</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">timetuple</span><span class="p" style="box-sizing: border-box;">())</span>
<span class="n" style="box-sizing: border-box;">answer_date_ts</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">time</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">mktime</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">hit_answer_date</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">timetuple</span><span class="p" style="box-sizing: border-box;">())</span>
<span class="n" style="box-sizing: border-box;">diff</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="nb" style="box-sizing: border-box; color: green;">int</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">answer_date_ts</span><span class="o" style="box-sizing: border-box; color: #666666;">-</span><span class="n" style="box-sizing: border-box;">hit_date_ts</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">result</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="p" style="box-sizing: border-box;">{</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"worker_id"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"workerId"</span><span class="p" style="box-sizing: border-box;">]),</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"gender"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"answers"</span><span class="p" style="box-sizing: border-box;">][</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"gender"</span><span class="p" style="box-sizing: border-box;">]),</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"household_income"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"answers"</span><span class="p" style="box-sizing: border-box;">][</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"householdIncome"</span><span class="p" style="box-sizing: border-box;">]),</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"household_size"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"answers"</span><span class="p" style="box-sizing: border-box;">][</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"householdSize"</span><span class="p" style="box-sizing: border-box;">]),</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"marital_status"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"answers"</span><span class="p" style="box-sizing: border-box;">][</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"maritalStatus"</span><span class="p" style="box-sizing: border-box;">]),</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"year_of_birth"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="nb" style="box-sizing: border-box; color: green;">int</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"answers"</span><span class="p" style="box-sizing: border-box;">][</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"yearOfBirth"</span><span class="p" style="box-sizing: border-box;">]),</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"location_city"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">get</span><span class="p" style="box-sizing: border-box;">(</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"locationCity"</span><span class="p" style="box-sizing: border-box;">)),</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"location_region"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">get</span><span class="p" style="box-sizing: border-box;">(</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"locationRegion"</span><span class="p" style="box-sizing: border-box;">)),</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"location_country"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"locationCountry"</span><span class="p" style="box-sizing: border-box;">]),</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"hit_answered_date"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="n" style="box-sizing: border-box;">hit_answer_date</span><span class="p" style="box-sizing: border-box;">,</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"hit_creation_date"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="n" style="box-sizing: border-box;">hit_creation_date</span><span class="p" style="box-sizing: border-box;">,</span>
<span class="s" style="box-sizing: border-box; color: #ba2121;">"post_to_completion_secs"</span><span class="p" style="box-sizing: border-box;">:</span> <span class="n" style="box-sizing: border-box;">diff</span>
<span class="p" style="box-sizing: border-box;">}</span>
<span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">return</span> <span class="n" style="box-sizing: border-box;">result</span></pre>
</div>
</div>
</div>
<div class="prompt input_prompt" style="background-color: white; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: navy; font-family: monospace; font-size: 14px; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
<br /></div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; background-color: white; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px;">
<div class="input_area" style="background: rgb(247, 247, 247); border-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython2" style="background: transparent; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span class="c" style="box-sizing: border-box; color: #408080; font-style: italic;"># We now transform our API answer into a flat table (Pandas dataframe)</span>
<span class="n" style="box-sizing: border-box;">responses</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">flatten</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">item</span><span class="p" style="box-sizing: border-box;">)</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">for</span> <span class="n" style="box-sizing: border-box;">item</span> <span class="ow" style="box-sizing: border-box; color: #aa22ff; font-weight: bold;">in</span> <span class="n" style="box-sizing: border-box;">json</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"items"</span><span class="p" style="box-sizing: border-box;">]]</span>
<span class="n" style="box-sizing: border-box;">df</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">pd</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">DataFrame</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">responses</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">df</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"gender"</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="n" style="box-sizing: border-box;">df</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"gender"</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">astype</span><span class="p" style="box-sizing: border-box;">(</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"category"</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">df</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"household_income"</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="n" style="box-sizing: border-box;">df</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"household_income"</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">astype</span><span class="p" style="box-sizing: border-box;">(</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"category"</span><span class="p" style="box-sizing: border-box;">)</span></pre>
</div>
</div>
</div>
<br />
<br />
We can then save the data to a vanilla CSV file, and see how the raw data looks like:<br />
<span style="background-color: white; color: navy; font-family: monospace; font-size: 14px; line-height: 1.21429em; text-align: right;"><br />
</span> <span style="background-color: white; color: navy; font-family: monospace; font-size: 14px; line-height: 1.21429em; text-align: right;">In [3]:</span><br />
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-color: white; box-sizing: border-box; display: flex; flex-direction: row; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; page-break-inside: avoid;">
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%;">
<div class="input_area" style="background: rgb(247, 247, 247); border-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython2" style="background: transparent; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span class="c" style="box-sizing: border-box; color: #408080; font-style: italic;"># Let's save the file as a CSV</span>
<span class="n" style="box-sizing: border-box;">df</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">to_csv</span><span class="p" style="box-sizing: border-box;">(</span><span class="s" style="box-sizing: border-box; color: #ba2121;">"data/mturk_surveys.csv"</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="o" style="box-sizing: border-box; color: #666666;">!</span>head -5 data/mturk_surveys.csv
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-color: white; box-sizing: border-box; display: flex; flex-direction: column; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; position: relative; z-index: 1;">
<div class="output" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column;">
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: row; padding: 0px; page-break-inside: avoid;">
<div class="output_subarea output_stream output_stdout output_text" style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; line-height: 1.21429em; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<pre style="background-color: transparent; border-radius: 0px; border: 0px; box-sizing: border-box; font-size: inherit; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"></pre>
<pre style="background-color: transparent; border-radius: 0px; border: 0px; box-sizing: border-box; font-size: inherit; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;">,gender,hit_answered_date,hit_creation_date,household_income,household_size,location_city,location_country,location_region,marital_status,post_to_completion_secs,worker_id,year_of_birth
0,male,2015-06-10 15:57:23.072000,2015-06-10 15:50:23,"$25,000-$39,999",5+,kochi,IN,kl,single,420.0,4ce5dfeb7ab9edb7f3b95b630e2ad0de,1992
1,male,2015-06-10 15:57:01.022000,2015-06-10 15:35:22,"Less than $10,000",4,?,IN,?,single,1299.0,cd6ce60cff5e120f3c006504bbf2eb86,1987
2,male,2015-06-10 15:21:53.070000,2015-06-10 15:20:08,"$60,000-$74,999",2,?,US,?,married,105.0,73980a1be9fca00947c59b93557651c8,1971
3,female,2015-06-10 15:16:50.111000,2015-06-10 14:50:06,"Less than $10,000",2,jacksonville,US,fl,married,1604.0,a4cdbe00c93728aefea6cdfb53b8c489,1992</pre>
</div>
</div>
</div>
</div>
<br />
Or we can take a peek at the top countries:<br />
<span style="background-color: white; color: navy; font-family: monospace; font-size: 14px; line-height: 1.21429em; text-align: right;"><br />
</span> <span style="background-color: white; color: navy; font-family: monospace; font-size: 14px; line-height: 1.21429em; text-align: right;">In [4]:</span><br />
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-color: white; box-sizing: border-box; display: flex; flex-direction: row; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; page-break-inside: avoid;">
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%;">
<div class="input_area" style="background: rgb(247, 247, 247); border-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython2" style="background: transparent; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span class="c" style="box-sizing: border-box; color: #408080; font-style: italic;"># Let's see the top countries</span>
<span class="n" style="box-sizing: border-box;">country</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">df</span><span class="p" style="box-sizing: border-box;">[</span><span class="s" style="box-sizing: border-box; color: #ba2121;">'location_country'</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">value_counts</span><span class="p" style="box-sizing: border-box;">()</span>
<span class="n" style="box-sizing: border-box;">country</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">head</span><span class="p" style="box-sizing: border-box;">(</span><span class="mi" style="box-sizing: border-box; color: #666666;">20</span><span class="p" style="box-sizing: border-box;">)</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-color: white; box-sizing: border-box; display: flex; flex-direction: column; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; position: relative; z-index: 1;">
<div class="output" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column;">
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: row; padding: 0px; page-break-inside: avoid;">
<div class="prompt output_prompt" style="box-sizing: border-box; color: darkred; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
Out[4]:</div>
<div class="output_text output_subarea output_execute_result" style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; line-height: 1.21429em; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<pre style="background-color: transparent; border-radius: 0px; border: 0px; box-sizing: border-box; font-size: inherit; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;">US 5748
IN 1281
CA 30
PH 22
GB 16
ZZ 15
DE 14
AE 11
BR 10
RO 10
TH 7
AU 7
PE 7
MK 7
FR 6
IT 6
NZ 6
SG 6
RS 5
PK 5
dtype: int64</pre>
</div>
</div>
</div>
</div>
<br />
I hope that the examples are sufficient to get people started using the API, and I am looking forward to see what analyses people will perform.Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-27900498027710375232015-06-08T20:07:00.001-04:002015-06-08T20:07:54.969-04:00Postdoc Position for Quality Control in Crowdsourcing<div style="text-align: justify;">
The Center for Data Science at NYU invites applications for a post-doctoral fellowship in statistical methodology relating to evaluating rater quality for a new research program in the application of crowdsourcing ratings of human speech production.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Duties and Responsibilities</b>: This is a two-year postdoctoral position in the affiliated with the NYU Center for Data Science. The successful candidate will join a dynamic group of researchers in several NYU Centers including PRIISM, MAGNET, the Stern School of Business, the NYU Medical School and the Department of Communicative Sciences and Disorders. We are seeking highly motivated individuals to develop and test novel statistical and computational methods for evaluating rater quality in crowdsourced tasks. Responsibilities will include development, testing and implementation of statistical algorithms, as well as preparation of manuscripts for academic publication. Advanced knowledge of R is preferred. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Position Qualifications</b>: Candidates will ideally have a doctoral degree in Statistics, Biostatistics, Data Science, Computer Science, or a related field, as well as genuine interests and experiences in interdisciplinary research that integrates study of human speech, citizen science games and computational statistics. Candidates will ideally have expertise in the following areas: Bayesian statistics, numerical methods and techniques, psychometrics and/or knowledge of programming languages. Outstanding computing and communication skills are required.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Please send CV, letter of intent, and three reference letters to Daphna Harel (daphna dot harel at nyu dot edu) by July 31, 2015.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The position is for 2 years (subject to good research progress). The successful candidate will be based at the NYU Center for Data Science, under the primary supervision of NYU faculty members Panos Ipeirotis and Daphna Harel, and will closely work with a multidisciplinary team including NYU faculty members Tara McAllister Byun, R. Luke DuBois, and Mario Svirsky. The position will preferably start by September 2015 (start date negotiable).</div>
<div style="text-align: justify;">
<br /></div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-71409478749934055312015-05-29T10:20:00.001-04:002015-06-02T22:31:40.795-04:00The World Bank Report on Online Labor<div style="text-align: justify;">
I am often asked about statistics and data about the global population of "crowdsourcing" workers, going beyond Mechanical Turk. I am happy to say that from now on I will be able to point everyone to <a href="https://www.dropbox.com/s/97css2nuiihbtx5/Global%20OO%20Study_WB%20Rpt%20Final2.pdf?dl=0">a study from The World Bank</a>, which I was fortunate to participate. The reports examines the global landscape of online labor, identifying the opportunities, and providing statistics about the global landscape.<br />
<br />
The study will be officially released on Wednesday June 3rd, and for those of you willing to attend the launch event through Webex, here is the information:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
---</div>
<div style="text-align: justify;">
<b>When</b>: </div>
<div style="text-align: justify;">
Wednesday, June 3, 2015, 9:00AM - 11:30AM EDT</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Where</b>:<br />
<a href="https://worldbankgroup.webex.com/worldbankgroup/j.php?MTID=m4ac0151d13cd83426d5b5b3614c16f99">Webex URL</a></div>
<div style="text-align: justify;">
Meeting number: 730 125 194</div>
<div style="text-align: justify;">
Meeting password: online1</div>
<div style="text-align: justify;">
Audio connection: 1-650-479-3207 Call-in toll number (US/Canada)</div>
<div style="text-align: justify;">
Access code: 730 125 194</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Title</b>:<br />
The New Online Outsourcing Approach for Jobs, Youth and Women's Empowerment and Services Exports</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Abstract</b>: </div>
<div style="text-align: justify;">
This event will discuss the new online outsourcing (OO) phenomena in the world today, its implications for developing countries, and how your clients can leverage it as an innovative approach for jobs, youth employment and women's empowerment.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
OO refers to the contracting of third-party workers and providers (often overseas) to supply services or perform tasks via Internet-based marketplaces or platforms. Also known as paid crowdsourcing, online work, microwork and other names - these technology-mediated channels allow clients to outsource their paid work to a large, distributed, global labor pool of remote workers, to enable performance, coordination, quality control, delivery, and payment of such services online.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The global OO marketplace today includes numerous emerging and growing platforms; such as Upwork (formerly Elance-oDesk), Crowdflower, CloudFactory, Amazon Mechanical Turk, etc. There are also wide variety of services that can be performed online - such as data entry, digitization, graphics rendering and design, programming and apps development, accounting and legal services, etc. Workers in developing countries can have access and perform jobs from all over the world - as long as they have computer and Internet access. In addition to jobs and income - OO offers workers flexible time and working environment, develop skills for professional, and drive positive social change for youth and women.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The event will share with participants the OO study that covers comprehensively the definition and segments, trends and market size, economic and non-financial impact on workers, and the implications and policy recommendations. In addition the event will show how u can apply the online toolkit to assess the readiness of your client countries for OO.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The World Bank's ICT Unit is excited to share this new global study and toolkit, which was developed in partnership with the Rockefeller Foundation and Dalberg Global Development Advisors.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Who:</b></div>
<div style="text-align: justify;">
<ul>
<li>Chair: Mavis Ampah, Lead ICT Policy Specialist and Practice Lead on Jobs, GTIDR </li>
<li>Siou Chew Kuek, Senior ICT Specialist and TTL, GTIDR </li>
<li>Cecilia Paradi-Guilford, ICT Innovation Specialist and Co-TTL, GTIDR </li>
<li>Saori Imaizumi, ICT Innovation and Education Consultant, GTIDR </li>
</ul>
</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-91119415767640234662015-04-06T15:17:00.001-04:002015-11-04T14:28:05.590-05:00Demographics of Mechanical Turk: Now Live! (April 2015 edition)One of the most common question that I receive is whether I have new data about the demographics of Mechanical Turk workers. The latest data that I had collected were back in 2010, and it was not clear how things have changed since then. The key problem was not that I could not run additional surveys; that would have been trivial. However, the results of the surveys were always changing over time: the aggregate data varied too much across surveys, so I refrained from publishing data that seemed to be unreliable.<br />
<div>
<div>
<br /></div>
<div>
So, I thought of how I tackle two problems at once:</div>
</div>
<div>
<ul>
<li>Make it easy for people to see current data about the demographics of Mechanical Turk workers</li>
<li>Make it easy to understand the inherent variability of the collected data, and potentially understand the source of the variability</li>
</ul>
<div>
For that reason, we built a new site:</div>
<div>
<br /></div>
<div style="text-align: center;">
<a href="http://demographics.mturk-tracker.com/">http://demographics.mturk-tracker.com/</a></div>
</div>
<div style="text-align: center;">
(please also check <a href="http://www.behind-the-enemy-lines.com/2015/06/an-api-for-mturk-demographics.html">the API</a>)</div>
<div>
<br /></div>
<div style="text-align: left;">
The site displays live data about the demographics of the workers, based on a small 5-question survey that users are asked to answer (paying 5 cents for each). To be able to capture the time variability, we post one survey every 15 mins, allowing us to observe changes in the answers over time. We also restrict each worker to be able to answer the survey only once per month.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
A few key results:<br />
<br />
<b><a href="http://demographics.mturk-tracker.com/#/countries" target="_blank">Country</a></b></div>
<div style="text-align: left;">
<br />
Overall, we see that approximately 80% of the Mechanical Turk workers are from the US and 20% are from India.</div>
<div style="text-align: left;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjm-KVHx5m-hphdg3cZ2IufZy0VBNwGtsW0XuYpF-ab3FuJzS1Fx4zAf2dbkkhPgv6V13QHE1Slf49NmX7bqiSp7cafM15jyjGp7M90WDSqHJWXP7cwiKjxXebV0rXTv5P4o3IV5eQA9Ys/s1600/countries.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="242" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjm-KVHx5m-hphdg3cZ2IufZy0VBNwGtsW0XuYpF-ab3FuJzS1Fx4zAf2dbkkhPgv6V13QHE1Slf49NmX7bqiSp7cafM15jyjGp7M90WDSqHJWXP7cwiKjxXebV0rXTv5P4o3IV5eQA9Ys/s1600/countries.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
However, this mix is not stable during the day. Around 8-10am UTC (ie 3am NYC time, 1.30pm India time), there is much higher number of workers from India (~50%), which then goes down to 5% at 8-10pm UTC.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6SNL-AiJTG4Inos8tScmLloSeMNA07GtBl7febIMvmHwQ45jsefFAPOFiEIg80hb4gu4k5i7fw6xdTZyy4yCIrcUjYLscnnfP1yo47l0iaccdriR0dUVSfDqJmIUfZpyjZFdY7lUgg80/s1600/countries-hourly.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6SNL-AiJTG4Inos8tScmLloSeMNA07GtBl7febIMvmHwQ45jsefFAPOFiEIg80hb4gu4k5i7fw6xdTZyy4yCIrcUjYLscnnfP1yo47l0iaccdriR0dUVSfDqJmIUfZpyjZFdY7lUgg80/s1600/countries-hourly.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b><a href="http://demographics.mturk-tracker.com/#/gender" target="_blank">Gender</a></b></div>
<br />
The gender participation seems to be balanced, with roughly 50% males and 50%. The charts that examine variability based on hour of day and day of the week do not show any change in this pattern.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZjjJzqY04fv9lEdZqY5e9gLCmMSwV8-BEIVYYOV6Lfj6LAmGbPZGntp82-DPw2ltCpACML7fdXHB5yWv4hyOUwfZlHd3r7YQUILKOBBMa17dlHyzYTjnwtCTq1QDLQgQwXQNIcIViCsM/s1600/gender.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZjjJzqY04fv9lEdZqY5e9gLCmMSwV8-BEIVYYOV6Lfj6LAmGbPZGntp82-DPw2ltCpACML7fdXHB5yWv4hyOUwfZlHd3r7YQUILKOBBMa17dlHyzYTjnwtCTq1QDLQgQwXQNIcIViCsM/s1600/gender.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b><a href="http://demographics.mturk-tracker.com/#/yearOfBirth" target="_blank">Year of birth</a></b></div>
<div class="separator" style="clear: both; text-align: left;">
<b><br />
</b></div>
<div class="separator" style="clear: both; text-align: left;">
Roughly 50% of the workers are born in the 1980's and are around 30 yrs old. Approximately 20% of the workers are born in the 1990's, and another 20% are born in the 1970's. </div>
<div class="separator" style="clear: both; text-align: left;">
<b><br />
</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYNfX5lWbXTRpgW61J1L1e35w_FAyOLVW2pFLWUcQRbeiYe34d5WEwT6W0iBGTfV-tvqhuyxej3pgBgwTrDPOfeGVpx5Dx0riX5U0292c-_FefMBpZzHCkzLQrYJQgUjijXTAywWiXpXQ/s1600/year-of-birth.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="248" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYNfX5lWbXTRpgW61J1L1e35w_FAyOLVW2pFLWUcQRbeiYe34d5WEwT6W0iBGTfV-tvqhuyxej3pgBgwTrDPOfeGVpx5Dx0riX5U0292c-_FefMBpZzHCkzLQrYJQgUjijXTAywWiXpXQ/s1600/year-of-birth.PNG" width="640" /></a></div>
<b><a href="http://demographics.mturk-tracker.com/#/maritalStatus" target="_blank">Marital Status</a></b><br />
<br />
Approximately 40% of the workers are single, 40% are married, and 10% are cohabitating.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtM59YVnUSyJjxZMi1vGlW8LO3Dke4Ln-Vo_V6PXWadUj5nExSwZAqmx5fM_OAL10qCfdlG1m-cxENj84rSfF1jqLwgSI8VhdwoSka12LbWQmaolDWNQS4jruGyHZayJROUId_264AXl8/s1600/marital-status.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="232" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtM59YVnUSyJjxZMi1vGlW8LO3Dke4Ln-Vo_V6PXWadUj5nExSwZAqmx5fM_OAL10qCfdlG1m-cxENj84rSfF1jqLwgSI8VhdwoSka12LbWQmaolDWNQS4jruGyHZayJROUId_264AXl8/s1600/marital-status.PNG" width="640" /></a></div>
<br />
<b><a href="http://demographics.mturk-tracker.com/#/householdSize" target="_blank">Household Size</a></b><br />
<br />
Approximately 15% live alone. Then 25% have a household size of two and 25% have a household size of three. Around 25% live in a household of four, and around 10% have five or more members in their household.<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjT3j4p3qvgh9ZaTGvTqe9_AA2jSypay2CA9HH9qk6txtzWFEa4-nZty1hqMG8SIIm8l_aN2X8_UAduHbc_muKE3kCxvKKT_mmekd0-oGHbUdSdfafOBZhIXMK5MPhr69C_e-jZBVgNCIg/s1600/hh-size.PNG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="262" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjT3j4p3qvgh9ZaTGvTqe9_AA2jSypay2CA9HH9qk6txtzWFEa4-nZty1hqMG8SIIm8l_aN2X8_UAduHbc_muKE3kCxvKKT_mmekd0-oGHbUdSdfafOBZhIXMK5MPhr69C_e-jZBVgNCIg/s1600/hh-size.PNG" width="640" /></a><br />
<b><a href="http://demographics.mturk-tracker.com/#/householdIncome" target="_blank">Income level</a></b><br />
<br />
The median household income is around \$50K per year for US Turkers, which is on par with the median US household income. Indian workers have considerably lower household income, with most of them being around \$10K/yr.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiu5TbOGGPTFXgrMuu-HWvvKms-_D5UNIU_1qecmt7lYBvfG4GkysR5RYoDB5rvSuZJKO5ZQGUVbM4yHYbPgTLezy9lVqRGfmpTkTSwyu_agstXdu6HVKLiph-VfFxGv9za2tZoaoJnbgw/s1600/hh-income.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="226" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiu5TbOGGPTFXgrMuu-HWvvKms-_D5UNIU_1qecmt7lYBvfG4GkysR5RYoDB5rvSuZJKO5ZQGUVbM4yHYbPgTLezy9lVqRGfmpTkTSwyu_agstXdu6HVKLiph-VfFxGv9za2tZoaoJnbgw/s1600/hh-income.PNG" width="640" /></a></div>
<br />
<br />
<b>Next steps</b><br />
<b><br />
</b> In our next steps, we plan on making the (anonymized) survey responses available through an API, and potentially add a few more graphs of interest. If you have any idea or suggestion, please send it my way.<br />
<br /></div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-45024943310557363512014-06-09T16:07:00.000-04:002014-06-09T16:21:33.254-04:00My Peer Grading Scheme<div style="text-align: justify;">
One of the components that I use in my class is student presentations. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
While I like having students present, I had always a hard time grading the presentations. Plus, many students seemed to target the presentation to me, trying to sound too technical and advanced, leaving the audience in the class bored and uninterested.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
For that reason, I adopted a peer-grading scheme. Students have to present to the class, and get rated by the class, and not me. (Although, I still reserve a small degree of editorial judgement for assigning the grades.) Here is how my scheme works, after a few years of experience.</div>
<ol>
<li style="text-align: justify;"><b>Rating scale:</b> Students assign a grade from 0 to 10 to the presentations.</li>
<li style="text-align: justify;"><b>No self-grading:</b> Students do not grade their own presentations. (<i>Early on, there were students that were assigning 10 to themselves, and lower grade to everyone else. Now they can still grade themselves if they want but the grade is ignored</i>.)</li>
<li style="text-align: justify;"><b>Normalization:</b> All assigned grades are normalized, to have a zero mean and one standard deviation. (<i>This normalization was introduced to fight the problem where a student would try to game the system by assigning low grades to everyone else, hoping to lower the average rating of all other students</i>.)</li>
<li style="text-align: justify;"><b>Grade assignment:</b> The presentation grade is the average of the assigned normalized scores. Formally, each student $s_i$ assigns to presentation $t$ a grade $z(s,t)$. The overall grade of the presentation is the mean value $E[z(*,t)]$ of the $z(s_i,t)$ grades.</li>
<li style="text-align: justify;"><b>Ensuring careful grading by asking students to estimate class rating:</b> One problem with the peer grading scheme was that many students did not take it seriously enough, and assigned random grades (typically, the same grade to everyone). To avoid indifferent grading, I decided to give credit (~10%) based on the correlation of the assigned grades $z(s,t)$ against the mean value $E[z(*,t)]$ (across all presentations $t$). This ensured that students will at least try to figure out what other students will assign to the presentation, and will not assign random grades.</li>
<li style="text-align: justify;"><b>Separate assigned and estimated grades:</b> The problem with introducing the requirement to agree with the class was that some students believed to be better assessors than the rest of the class. So, they felt that their own grade was the correct one, and did not like losing credit for assigning their own "true" grade. To address that issue, I now ask students to assign two grades: their own grade $z_p(s,t)$, and an estimate of the class grade $z_c(s,t)$. The personal grade $z_p$ is used to compute $E(z(*,t)]$ in Step 4, and I use the $z_c$ to compute the correlation in Step 5. </li>
<li style="text-align: justify;"><b>Examine self-grading:</b> Given that the class-estimate grades are not directly used to grade a presentation, students are also asked to provide an estimate of their own grade as part of Step 6. Effectively, students are encouraged to estimate properly their own grade.</li>
</ol>
<div style="text-align: justify;">
The only thing that I have not tried to far is to modify Step 4 to take into consideration the different correlations from Step 5, effectively weighting each student's grades based on their correlation with the rest of the class. However, most students tend to exhibit the same, moderate agreement with the class (typical correlation values are in the 0.4-0.6 range, after rating 15-20 presentations), so in practice I do not expect to see a difference.<br />
<br />
Overall, I am pretty happy with the scheme. Students indeed try to impress the class (and not me), and many presentations are interesting, interactive, and engaging. The grades are also very consistent with the overall feeling that I get for each presentation, so I did not have to practice my "editorial oversight" and adjust the grade very often (only in a couple of cases, where the students ran into technical problems during the presentation). I would be really interested to try this scheme in one of the big MOOC classes that use peer grading, and see if it can instill the same sense of responsibility in peer grading. </div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-57913936752130169002014-04-01T15:17:00.000-04:002014-04-28T21:56:39.234-04:00Online Markets: Selling products vs. selling time<div style="text-align: justify;">
We had an interesting discussion a few days back about online job markets, and why they are not a <b><i>huge </i></b>success so far, when other, comparatively less important products are getting huge valuation and visibility. For example, oDesk <a href="https://www.odesk.com/info/riseofonlinework/">reached a total transaction volume of a billion dollars</a>, for the 10 years of their existence, and roughly 5% to 10% of the volume becomes revenue for the company. Other labor marketplaces have typically even smaller number of report.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
While nobody can ignore a billion dollar of transaction volume, I am puzzled why this number has not skyrocketed. It is very clear that the market serves a purpose: work is a trillion dollar industry. Allowing people to work online allows for better and more efficient access to human capital, alleviates need for immigration, and improves the lives of people involved. It is a no-brainer.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Why does it take so long for online work to takeoff? What is missing?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<div style="text-align: center;">
***</div>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I was puzzled by these questions for long. <a href="http://www.behind-the-enemy-lines.com/2013/07/online-labor-markets-why-they-cant.html">I postulated that there are obstacles that prevent <b><i>employers </i></b>from hiring online</a>, but recently I got some hints that there are obstacles from the worker side as well. I talked with some friends of mine back in Greece, who are making a very comfortable living working through the platform. I asked how they like making US salaries while living in Greece, and their answer was surprising. <b>They did not see online work as a long term solution, but rather as a temporary gig.</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
When I asked why, they both indicated the same problem: There is no room in such markets for career evolution. You end up selling your time, and time is not something that scales. It is very hard to grow your business when you are always a freelancer, without the ability to hire new people, delegate tasks, and build a business. Compare now online work with a market like Amazon and eBay. Both Amazon and eBay allow sellers to effectively build businesses. Currently, online job markets allow workers to just sell their time.</div>
<div>
<div style="text-align: justify;">
<br /></div>
</div>
<div>
<div style="text-align: justify;">
<b>When sellers have a capped growth, the market faces headwinds of growth as it tries to reach maturity.</b></div>
</div>
<div style="text-align: center;">
<div style="text-align: center;">
***</div>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
On a general note, this gives birth to a general hypothesis on what can make a marketplace (hugely) successful: <b>The market should allow sellers to grow, without an obvious ceiling. Otherwise, the best sellers are unlikely to be attracted to participate in the platform, due to the lack of upside.</b></div>
<br />
<div style="text-align: justify;">
Take some marketplace companies and interpret them through this framework:</div>
<ul>
<li style="text-align: justify;"><b>Google Helpouts: </b>Same restrictions on seller growth as all other job marketplaces.</li>
<li style="text-align: justify;"><b>Uber</b>: Obviously, currently the sellers have a cap on growth, which is limited by their time. However, Uber allows the enrollment of limo/taxi agencies, which potentially grow indefinitely.</li>
<li style="text-align: justify;"><b>AirBnB:</b> No obvious seller cap for someone who wants to enter the hospitality business.</li>
<li style="text-align: justify;"><b>TaskRabbit:</b> Very obvious growth cap for the individual sellers of services.</li>
<li style="text-align: justify;"><b>OpenTable</b>: No obvious limit of growth for participating restaurants.</li>
<li style="text-align: justify;"><b>eBay/Amazon</b>: No obvious limit of growth for sellers that sell products online</li>
<li style="text-align: justify;"><b>Etsy</b>: This is an interesting case. On the surface, the company looks like eBay/Amazon. However, the etsy guidelines dictate that "<i><a href="http://www.etsy.com/help/article/483">Everything on Etsy must be Handmade, Vintage, or a Craft Supply</a>.</i>" Unfortunately, this places restrictions on seller growth as it implicitly limits sellers to be (very) small businesses. My bet is that either Etsy will revise this policy down the road, once more and more sellers start hitting their growth ceiling.</li>
</ul>
<div>
<div style="text-align: justify;">
How accurate is the hypothesis? Time will tell...</div>
</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-81764629114212592072014-01-22T06:00:00.000-05:002014-01-22T15:11:56.999-05:00Future of Education: Fighting Obesity or Fighting Hunger?<div style="text-align: justify;">
I have been following with interest the discussion about the future of education.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<div style="text-align: center;">
***</div>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Some people criticize existing educational institutions, indicating that they offer little in terms of real training, and that real learning occurs outside the classroom, by actually doing. "Nobody learns how to build a system in a computer science class." "Nobody learns how to build a company in an entrepreneurship program."</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Others are lamenting that by shifting to training-oriented schemes, we are losing the ability to offer deeper education, on topics that are not marketable. Who is going to study poetry if it has no return on investment? Who is going to teach literature if there is no demand for it?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
These two criticisms seem to be pushing in two different directions.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<div style="text-align: center;">
***</div>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
In reality, we need to address two different needs:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
One need is to really try and democratize education, trying to take the content of the top courses and make it accessible and available to everyone. People that want to learn machine learning, can now take courses from top professors, instead of having to read a book. People can now advance their careers easily, without having to enroll to expensive degree programs.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The other need is to preserve the breadth of education, shielding it from market forces. This need wants to preserve the structure where students during their education get exposed to diverse fields, no matter if there is a market and demand for these fields.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<div style="text-align: center;">
***</div>
</div>
<div style="text-align: center;">
<div style="text-align: justify;">
<br /></div>
</div>
<div style="text-align: justify;">
This tension reminded me about the discussion about genetically modified foods.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Mass production of food pretty much solved the problem of world hunger. A few decades ago, there was a real problem with world hunger. Famine was a real problem in many areas of the world, due to the inability to produce enough food to feed the growing population: floods, droughts, diseases were disrupting production, resulting in shortages. Today, the advances in agriculture allow the abundant production of grains and food: wheat and rice varieties are now robust, resistant to diseases, adaptable to many different climates, and allow us to feed the world.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The advances that solved the problem of world hunger, ended up creating other problems. Processed carbohydrates and causing obesity, diabetes, gout, and many other "luxury" diseases in the developed world. The poor in the developed world are not dying because they are hungry. They are dying by starving themselves from essential ingredients in their diet.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
***</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The parallels are striking. The MOOCs, Khan Academies, and Code Academies of the world are the genetically modified foods for those living in the "third world of education". These courses may not be the most nutritious, and they may not provide all the "nutrition" for their education. However, the choice for many of these people in the "third world of education" is not <i>Stanford vs. a Coursera MOOC</i>. It is <i><b>nothing </b>vs. a Coursera MOOC</i>. Given the choice, take the MOOC at any time.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Those that live in the "developed world of education" can be pickier. They may have access to the genetically modified MOOCs, but if they can afford it, the organic, artisanal, locally sourced education can be potentially better than the mass produced MOOC. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
***</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Horses for <i>courses </i>(pun intended).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-45932186964010055512014-01-20T19:57:00.000-05:002014-02-03T10:53:46.123-05:00Crowdsourcing research: What is really new?<div style="text-align: justify;">
A common question that comes up when discussing research in crowdsourcing, is how it compares with similar efforts in other fields. Having discussed these a few times, I thought it would be good to collect all these in a single place.</div>
<ul>
<li style="text-align: justify;"><b>Ensemble learning</b>: In machine learning, you can generate a large number of "weak classifiers" and then build a stronger classifier on top. In crowdsourcing, you can treat each human as a weak classifier and then learn on top. What is the difference? In crowdsourcing, each judgement has a cost. With ensembles, you can trivially easy create 100 weak classifiers, classify each object, and then learn on top. In crowdsourcing, you have a cost for every classification decision. Furthermore, you cannot force every person to participate, and often you have a heavy-tailed participation: A few humans participate a lot, but from many of them we get only a few judgments.</li>
<li style="text-align: justify;"><b>Quality assurance in manufacturing</b>: When factories create batches of products, they also have a sampling process where they examine the quality of the manufactured products. For example, a factory creates light bulbs, and wants 99% of them to be operating. The typical process involves setting aside a sample for testing and testing if they meet the quality requirement. In crowdsourcing, this would be equivalent to verifying, with gold testing or with post-verification, the quality of each worker. Two key differences: The heavy-tailed participation of workers means that gold-testing each person is not always efficient, as you may end up testing a user a lot, and the the user may leave. Furthermore, it is often the case that a sub-par worker can still generate somewhat useful information, while for tangible products, the product is either acceptable or not.</li>
<li style="text-align: justify;"><b>Active learning:</b> Active learning assumes that humans can provide input to a machine learning model (e.g., disambiguate an ambiguous example) and the answers are assumed to be perfect. In crowdsourcing this is not the case, and we need to explicitly take the noise into account.</li>
<li style="text-align: justify;"><b>Test theory and Item Response Theory:</b> Test theory focuses on how to infer the skill of a person through a set of questions. For example, to create a SAT or GRE test, we need to have a mix of questions of different difficulties, and we need to whether these questions really separate the persons that have different abilities. Item Response Theory studies exactly these questions, and based on the answers that users give to the tests, IRT calculates various metrics for the questions, such as the probability that a user of a given ability will answer correctly the question, the average difficulty of a question, etc. Two things make IRT unapplicable directly to a crowdsourcing setting: First, IRT assumes that we know the correct answer to each question; second, IRT often requires 100-200 answers to provide robust estimates of the model parameters, a cost that is typically too high for many crowdsourcing applications (except perhaps the citizen science and other volunteer based projects).</li>
<li style="text-align: justify;"><b>Theory of distributed systems: </b>This <a href="http://www.amazon.com/Introduction-Distributed-Algorithms-Gerard-Tel/dp/0521794838" target="_blank">part </a>of <a href="http://groups.csail.mit.edu/tds/distalgs.html" target="_blank">CS theory</a> is actually much closer to many crowdsourcing problems than many people realize, especially the work on asynchronous distributed systems, which attempts to solve many coordination problems that appear in crowdsourcing (e.g. agree on an answer). The work on analysis of byzantine systems, which explicitly acknowledges the existence of malicious agents, provides significant theoretical foundations for defending systems against spam attacks, etc. One thing that I am not aware of, is the explicit dealing of noisy agents (as opposed to malicious ones), and I am not aware of any study of incentives within that context that will affect the way that people answer to a given question.</li>
<li style="text-align: justify;"><b>Database systems and User-defined-functions (UDFs)</b>: In databases, a query optimizer tries to identify the best way to execute a given query, trying to return the correct results as fast as possible. An interesting part of database research that is applicable to crowdsourcing is the inclusion of user-defined-functions in the optimization process. A User-Defined-Function is typically a slow, manually-coded function that the query optimizer tries to invoke as little as possible. The ideas from UDFs are typically applicable when trying to optimize in a human-in-the-loop-as-UDF approach, with the following caveats: (a) UDFs were considered to be return perfect information, and (b) the UDFs were assumed to have a deterministic or a stochastic but normally distributed execution time. The existence of noisy results and the fact that execution times with humans can be often long-tailed make the immediate applicability of UDF research in optimizing crowdsourcing operations rather challenging. However, it is worth reading the related chapters about UDF optimization in the database textbooks.</li>
<li style="text-align: justify;"><b>(Update) Information Theory and Error Correcting Codes: </b>We can model the workers are noisy channels, that get as input the true signal and return back a noisy representation. The idea of using advanced error correcting codes to improve crowdsourcing is rather underexplored, imho. Instead we rely too much on redundancy-based solutions, although pure redundancy has been theoretically proven to be a suboptimal technique for error correction. (<a href="http://www.behind-the-enemy-lines.com/2013/07/majority-voting-and-information-theory.html">See an earlier, related blog post.</a>) Here are a couple of potential challenges: (a) The errors of the humans are very rarely independent of the "message" and (b) It is not clear if we can get humans to compute properly functions that are commonly required for the implementation of error correcting codes. See a related e</li>
<li style="text-align: justify;"><b>(Update) Information Retrieval and Interannotator Agreement: </b>In information retrieval, it is very common to examine the agreement of the annotators when labeling the same set of items. My own experience with reading the literature, and the related metrics is that they implicitly assume that all workers have the same level of noise, an assumption that is often violated in crowdsourcing.</li>
</ul>
<div>
<div style="text-align: justify;">
Any other fields and what other caveats that should be included in the list?</div>
</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-27523476172103968712013-10-17T17:48:00.000-04:002013-10-17T17:58:36.285-04:00Badges and the Lake Wobegon effect<br />
<div style="text-align: justify;">
For those not familiar with the term, the <a href="http://en.wikipedia.org/wiki/Lake_Wobegon#The_Lake_Wobegon_effect">Lake Wobegon effect</a> is the case when all or nearly all of a group claim to be above average, and comes from the finctional town where "all the women are strong, all the men are good looking, and all the children are above average."</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Interestingly enough, as Wikipedia states, this effect of the majority of the group thinking that they are performing above-average "<i>has been observed among drivers, CEOs, hedge fund managers, presidents, coaches, radio show hosts, late night comedians, stock market analysts, college students, parents, and state education officials, among others</i>."</div>
<br />
<div style="text-align: justify;">
So, a natural question was whether this effect also appears in an online labor setting. We took some data from an online certification company, similar to <a href="http://smarterer.com/">Smarterer</a>, where people take tests to show how well they know a particular skill (e.g., Excel, Audio Editing, etc.) The tests are not pass/fail but more like a GRE/SAT score: there is no "passing" score, only a percentile indicator that shows what percentage of other participants have a lower score. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Interestingly enough, we noticed a Lake Wobegon effect there as well: Most of the workers that displayed the badge of achievement, have scores above average, giving yet another point for the Lake Wobegon effect.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Of course, this does not mean that all users that took the test performed above average. Test takers have the choice to make their final score public to the world, or keep it private. Given that the user's profile is also used in a site where employers look for potential hires, there is some form of strategic choice in whether the test score is visible or not. Having a low score is often worse than having no score at all.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So, we wanted to see what scores make users comfortable with their performance, and incentivizes them to display their badge of achievement. <a href="http://people.stern.nyu.edu/mk3539/">Marios</a> analyzed the data, and compared the distribution of scores for workers that decided to keep their score private, compared to the workers that made their performance public. Here is the outcome:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWUHYSvFZKPeqY4xkGhFTTNpPzXsaw1QLObw7OBrOWRXF7sGyWrz9XrNX7YPL3ILtb7IB7UfwmGyF2TJU511q1ES69sEPy70SJMsX9I1PW6nBqs38kGgRT-dyX7kjsqEx9HtxeG8zbpDc/s1600/public-private-test-result-ExpertRating.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="488" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWUHYSvFZKPeqY4xkGhFTTNpPzXsaw1QLObw7OBrOWRXF7sGyWrz9XrNX7YPL3ILtb7IB7UfwmGyF2TJU511q1ES69sEPy70SJMsX9I1PW6nBqs38kGgRT-dyX7kjsqEx9HtxeG8zbpDc/s640/public-private-test-result-ExpertRating.PNG" width="600" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
It becomes clear that scores below 50% are not posted often, while scores that exceed 60% have significantly higher odds of being posted online for the world to see. This becomes more clear if we take the log-odds of a worker deciding to make the score public, given the achieved percentile:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiukmhqTUPK-F-acg0hpz5vCOip6KASAtFSLO98j2IVsL66v4CsolsVxIRgEQXUqY9oHBBawly_dOK44zS0cBNgmLvBBfsfB6FFk09p5K_W-xOCyeJPt3heYxg1wuhbnZD2ajPGMaLZo1o/s1600/public-private-test-result-ExpertRating-logodds.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="450" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiukmhqTUPK-F-acg0hpz5vCOip6KASAtFSLO98j2IVsL66v4CsolsVxIRgEQXUqY9oHBBawly_dOK44zS0cBNgmLvBBfsfB6FFk09p5K_W-xOCyeJPt3heYxg1wuhbnZD2ajPGMaLZo1o/s640/public-private-test-result-ExpertRating-logodds.PNG" width="600" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So, in the world of online labor if you ever hire someone who chose to display a certification, you know that there are good chances that you picked a worker that is better than average, at least in the test. (We have some other results on the predictive power of tests in terms of work performance, but this is a topic that cannot fit into the margins of this blog post :-)</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Needless to say, this effect illustrates a direction that will take crowdsourcing, and labor markets in general, out of the race-to-the-bottom, market-for-lemons-style, pricing, where only price can separate the various workers. As education history serves in an offline setting as signaling for the potential quality of the employee, we are going to see more and more globally recognized certifications replacing educational history for many online workers.</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-22689705786155842632013-09-11T08:31:00.000-04:002013-09-11T08:31:39.743-04:00CrowdScale workshop at HCOMP 2013<div>
A public service announcement, to advertise CrowdScale (<a href="http://www.crowdscale.org/">http://www.crowdscale.org/</a>) a cool workshop at <a href="http://www.humancomputation.com/2013/">HCOMP 2013</a> that focuses on challenges that people face when applying crowdsourcing at scale.</div>
<div>
<br /></div>
<div>
A couple of interesting twists from the classic workshop recipe; </div>
<div>
<ul>
<li>The workshop invites submission of short (2-page) position papers <b>which identify and motivate key problems</b> or potential approaches for crowdsourcing at scale, <b>even if there aren’t satisfactory solutions proposed.</b> (Deadline: October 4)</li>
<li>Second, there is a <a href="http://www.crowdscale.org/shared-task">shared task challenge</a>, which also carries a cool $1500 reward for the winner.</li>
</ul>
</div>
<div>
The CfP follows:</div>
<div>
<br /></div>
<div>
<blockquote>
<i>Crowdsourcing at a large scale raises a variety of open challenges:</i><ul>
<li><i>How do we programmatically measure, incentivize and improve the quality of work across thousands of workers answering millions of questions daily? </i></li>
</ul>
<ul>
<li><i>As the volume, diversity and complexity of crowdsourcing tasks increase, how do we scale the hiring, training and evaluation of workers? </i></li>
</ul>
<ul>
<li><i>How do we design effective elastic marketplaces for more skilled work? </i></li>
</ul>
<ul>
<li><i>How do we adapt models for long-term, sustained contributions rather than ephemeral participation of workers?We believe tackling <a href="http://www.crowdscale.org/position-paper-ideas">such problems</a> will be key to taking crowdsourcing to the next level – from its uptake by early adopters today, to its future as how the world’s work gets done. </i></li>
</ul>
<i>To advance the research and practice in crowdsourcing at scale, our workshop <a href="http://www.crowdscale.org/cfp">invites position papers</a> tackling such issues of scale. In addition, we are organizing a <a href="http://www.crowdscale.org/shared-task">shared task challenge</a> regarding how to best aggregate crowd labels on large crowdsourcing datasets released by Google and CrowdFlower.<br />Twitter: <a href="https://twitter.com/search?q=%23crowdscale">#crowdscale</a> • <a href="https://twitter.com/CrowdAtScale">@CrowdAtScale</a><br />Organizers</i><ul>
<li><i><a href="http://www.linkedin.com/in/tatianajosephy">Tatiana Josephy</a> (<a href="https://twitter.com/tatianajosephy">@tatianajosephy</a>), CrowdFlower </i></li>
</ul>
<ul>
<li><i><a href="https://www.ischool.utexas.edu/~ml/">Matthew Lease</a> (<a href="https://twitter.com/mattlease">@mattlease</a>), University of Texas at Austin </i></li>
</ul>
<ul>
<li><i><a href="http://research.google.com/pubs/PraveenParitosh.html">Praveen Paritosh</a> (<a href="https://twitter.com/heuristicity">@heuristicity</a>), Google </i></li>
</ul>
</blockquote>
</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-67947261696359144612013-07-31T09:30:00.000-04:002013-07-31T09:30:03.419-04:00Online labor markets: Why they can't scale and the crowdsourcing solution.<div style="text-align: justify;">
<a href="http://www.behind-the-enemy-lines.com/2012/07/the-disintermediation-of-firm.html" target="_blank">I am a big proponent of outsourcing work using online labor markets</a>. Over the last decade I outsourced hundreds of projects, ranging from simple data entry to big, complex software products. I learned to create project specs, learned how to manage contractors, and learned how to keep projects moving forward. In general, I consider myself competent in managing distributed teams and projects.<br />
<br />
I also met and talked with many people that share my passion for this style of work. We discuss strategies for hiring, for managing the short- and long-term projects, for pricing, for handling legal risks, and other topics of interest. After many such discussions, I reached a striking conclusion: <b>Everyone has a completely different style of managing this process.</b><br />
<br />
<b>This plurality of "best practices" is a bad thing.</b> Having too many best practices means that there are no best practices. The lack of consensus makes it impossible to effectively teach a newcomer of how to handle the process. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>The problem with manual hiring in online labor markets</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
People that want to use contractors for their projects face the following problems:</div>
<ul>
<li style="text-align: justify;"><b>Few people know what they want:</b> Just for fun, go and check random projects on oDesk, eLance, and Freelancer. An endless list of poorly described projects, requests for "clone of Facebook" for $500, and a lot of related crap. It is not a surprise that many of these projects remain open for ever.</li>
<li style="text-align: justify;"><b>Few people know how to hire:</b> Ask any startup CEO how easy is to hire an employee. It is a pain. The art and craft of inferring the match of an individual to a given task is a very hard problem. Few people know how to do it right. Even within Google and Microsoft, with their legendary interviewing processes, interviewing is seen by many as a hard, time-consuming, and unrewarding experience.</li>
<li style="text-align: justify;"><b>Few people know how to manage a project: </b>Even fewer people know how to manage a project. The harrowing fact is that most people believe that they can. Most people hire someone, hoping that the employee will be in their head, will understand what these vague specifications mean, will know everything that is not documented in a project, and will be able to do a great job. Very few people realize that outsourcing a project means that you will need to spend significant amount of time <i>managing </i>the project.</li>
</ul>
<div style="text-align: justify;">
The result of the combination of these factors? <b>Online labor does not scale through manual hiring. </b>(Of course, this is not unique to online outsourcing. Offline hiring has the same problem.) There are simply not enough qualified employers that can hire effectively, who will be able to create demand for jobs for the online labor markets to continue to grow.<br />
<br />
<b>Online hiring vs online shopping</b><br />
<br />
The counter-argument is that labor was always like that. Since the market for labor operates "manually," the transition to electronic hiring will allow for growth. In the same way people were initially afraid of shopping online, they started buying things online, they are going to switch to hiring online.<br />
<br />
I do not buy this argument. When people buy an item online, they buy a standardized product. They are not ordering a bespoke item, which is created according to the customer specifications. Customization is typically limited and allowed on a specific set of dimensions. You can customize your Mac to have a better processor, more memory, and a larger hard disk. But you cannot order a laptop with a 19 in screen, and cannot ask for 96 Gb of memory. <br />
<br />
But in online markets this is what happens. The random customer comes and asks for a web application ("just the functionality of the X website"), and wants this app to be built for $500. It is the same as if someone goes to a computer store and asks for a laptop with a 19 inch screen, with 128Gb of memory, and 10Tb disk. And, since <a href="http://www.amazon.com/Upgrade-OptiPlex-Desktop-DDR2-400-PC2-3200/dp/B002OFNN6C" target="_blank">1Gb of memory costs 7 dollars</a>, it is reasonable to just pay $1000 for 128Gb, right?</div>
<div style="text-align: justify;">
<br />
<b>Lessons from online shopping</b><br />
<br /></div>
<div style="text-align: justify;">
Based on the experience for the transition of shopping from offline to online, let's see how online labor can move forward. </div>
<ol>
<li style="text-align: justify;"><b>Standardize and productize</b>: Currently, in online markets, most people ask for a specific set of tasks. Content generation, website authoring, transcriptions, translations, etc. Many of these can be "productized" and be offered as standardized packages, perhaps with a few pre-set customizations available. (Instead of "select the hard disk size, you have a "select blog post length".) This vertical-oriented strategy is followed by many crowdsourcing companies and offers to the client a clean separation from the process of hiring and managing a task. This vertical strategy works well to create small offerings but it is not clear if there is sufficient demand within each vertical to fuel the growth expected for a startup. This is a topic for a new blog post.</li>
<li style="text-align: justify;"><b>Productize the project creation/management: </b>When a standardized offering is not sufficient, the client is directed into hiring a product manager that will spec the requirements, examine if there is sufficient supply of skills in the market, hire individual contractors, manage the overall process, etc. This is similar to renovating a house. The delivered product is often completely customized, but the client does not seek to hire separately electricians, carpenters, painter, etc. Instead, the owner hires a "general contractor" who creates the master plan for the renovation, procures the materials, hires subcontractors, etc. While it eases some of the problems, this is a process suitable only for reasonably big project.</li>
<li style="text-align: justify;"><b>Become a staffing agency</b>: A problem with all existing marketplaces is that they are not acting as employers, but only as matching agents. Few, if any, marketplaces are guaranteeing quality. Every transaction is a transaction between "consenting adults." Unfortunately, very few potential employers understand that, and hire with the implicit assumption that the marketplace is placing a guarantee on the quality of the contractors. So, if the contractor ends up being unqualified for the task, there is very little recourse. By guaranteeing quality, the employer (who is the one spending the money) gets some minimum level of guarantee about the deliverable. Unfortunately, providing such quality guarantees is easier said than done.</li>
<li style="text-align: justify;"><b>Let contractors build offerings</b>: By observing the emergence of marketplaces like Etsy, you can see that people are becoming more comfortable with ordering semi-bespoke, handcrafted items online, for which they have little information. A potential route is to allow the contractors in online markets to build such "labor products" and price them themselves, in the same way that Etsy sellers are putting up their handcrafted stuff online.</li>
</ol>
<div style="text-align: justify;">
All these approaches are fine, and I expect most current marketplaces to adopt one or more of these strategies over time. However, all of them rely on the same assumption: That hiring, as shopping, will be a human activity. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
What happens, though, if we stop assuming that hiring is a human-mediated effort?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Crowdsourcing practices to the rescue</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I will not pretend that the current state of the crowdsourcing industry offers concrete solutions to the problems listed above. But today's efforts in crowdsourcing move us towards an <b>algorithmically-mediated work environment.</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Of course, like all automatic solutions, the initial environment is much worse than "traditional" approaches.</b> We see that in all the growing pains of Mechanical Turk. It is often easier to just hire a couple of trusted virtual assistants from oDesk to do the job, instead of trying to implement the full solution stack to get things done properly on MTurk.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
However, the initial learning curve starts paying off later. Production environments that rely on a "crowd" <b>need</b> to automate as much as possible the hiring and management of workers. This automation makes the tasks much more scalable than traditional hiring and project management. High-startup costs, then lower marginal costs of adding workers to a process.<br />
<br />
This leads to easier scalability. Of course, the moment the benefits of easier scalability start becoming obvious, it will be too late for players that rely on manual hiring to catch up. It is one of the reasons that I believe that Mechanical Turk has the potential to be the major labor platform, even if this seems a laughable proposition at this point. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I will make a prediction: Crowdsourcing is currently at the forefront of defining the methods and practices in the workplace for the next few decades. Assembly lines and integration of machines in the work environment led to the mass production revolution of the 20th century. The current crowdsourcing practices will define how the majority of people are going to work on knowledge tasks in the future. A computer process will monitor and manage the working process, and hiring manually will be soon a thing of the past, for many "basic" knowledge tasks.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Some will find this prospect frightening. I do not find it any more frightening than having traffic lights regulate traffic in intersections, or having the auto-pilot taking care of my flight.</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-68448262410035138672013-07-28T19:43:00.000-04:002013-07-29T22:04:34.764-04:00Crowdsourcing and information theory: The disconnect<div style="text-align: justify;">
In crowdsourcing, redundancy is a common approach to ensure quality. One of the questions that arises in this setting is the question of equivalence. Let's assume that a worker has a known probability $q$ of giving a correct answer, when presented with a choice of $n$ possible answers. If I want to simulate one high-quality worker workers of quality $q$, how many workers of quality $q' < q$ do we need?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Information Theory and the Noisy Channel Theorem</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Information theory, and the noisy channel theorem, can give an answer to the problem: Treat each worker as a noisy channel, and measure the "capacity" of each user. Then, the sum of the capacities of the different workers should give us the equivalent capacity of a high-quality worker.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
We have that the capacity $C(q,n)$ of a worker with quality $q$, who returns the correct answer with probability $q$, when presented with $n$ choices, is:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
$C(q,n) = H(\frac{1}{n}, n) - H(q, n)$</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
where $H(q,n) = -q \cdot \log(q) - (1-q) \cdot \log(\frac{1-q}{n-1}) $ is the entropy (aka uncertainty) of the worker. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Examples</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The value $H(\frac{1}{n}, n) = \log(n)$ is the initial entropy that we have for the question, when no answers are given. Intuitively, when $n=2$, the initial uncertainty is equal to $\log(2)=1$ bit, since we need one bit to describe the correct answer out of the 2 available. When $n=4$, the uncertainty is equal to $\log(4)=2$ bits, as we need 2 bits to describe the correct answer out of the 4 available.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Therefore, a perfect worker, with quality $q=1$ will have $H(1,n)=0$ entropy, and therefore the capacity of a perfect worker is $\log(n)$.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Can Two Imperfect Workers Simulate a Perfect One?</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Now, here comes the puzzle. Assume that we have $n=4$, and the workers have to choose among 4 possible answers. We also have also two workers with $q=0.85$, that select with 85% probability the correct answer out of 4 available. These workers have each capacity equal to $C(0.85, 4) = 1.15$ bits. At the same time, we have one perfect worker with $q=1$. This worker has a capacity of $C(1,4)=2$ bits. So, in principle, the two noisy workers are sufficient to simulate a perfect worker (and would leave a remaining 0.3 bits to use :-)</div>
<div style="text-align: justify;">
<br />
<b>What am I missing?</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
My problem is that I do not get how to reach this theoretical limit. I cannot figure out how to use these two workers with $q=0.85$, in order to reconstruct the correct answer. Asking two workers to work in parallel will not cut it (still possible for both workers to agree and be incorrect). Sequential processing (get first a worker to select two out of the four answers, then the second one pick the correct out of the two) seems more powerful, but again I do not understand how to operationalize this.<br />
<br />
According to information theory, these two $q=0.85$ workers are equivalent, on average, with one perfect $q=1.0$ worker. (Actually, they seem to carry more information). And even if we avoid perfection, and we set target quality at $q=0.99$, $C(0.99,4)=1.9$. I still cannot see how I can combine two workers with 85% accuracy to simulate a 99% accurate worker.</div>
<div style="text-align: justify;">
<br />
<ul>
<li><b><i>Update 1</i></b> (thanks to Michael Nielsen): Information theory operates over a large amount of transmitted information, so posing the question as "answering a single question" makes it sound more impossible than it should.<br /><br />We need 2 bits of information to transfer the answer for a multiple choice question with n=4 choices. Say that we have a total of N such questions. So, we need 2N bits to transfer perfectly ann the answers. If we have perfect workers, with $q=1$, we have that $C(1,4)=2$, and we need 2N bits / 2 bits/answer = N answers, from these workers.<br /><br />Now, let's say that we have workers with $q'=0.85$. In that case $C(0.85, 4) = 1.15$ bits per answer. Therefore, we need 2N bits / 1.15 bits/answer = 1.74N answers from these 85% accurate workers in order to perfectly reconstruct the answers for these N questions.<br /><br />So, if we get from these 85% workers a total of 100 answers (each one 85% correct), we should be able to reconstruct the 100% correct answer for ~57 (=100/1.74) questions.<br /><br />Of course we should be intelligent of what exactly to ask and get these 100 answers.</li>
</ul>
<br />
I see in Wikipedia, in the article about the <a href="http://en.wikipedia.org/wiki/Noisy-channel_coding_theorem" target="_blank">noisy channel theorem</a>, that "<i>Simple schemes such as 'send the message 3 times and use a best 2 out of 3 voting scheme if the copies differ' are inefficient error-correction methods</i>" and that "<i>Advanced techniques such as <a href="http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_code">Reed–Solomon codes</a> and, more recently, <a href="http://en.wikipedia.org/wiki/Turbo_code">turbo codes</a> come much closer to reaching the theoretical Shannon limit</i>". Unfortunately, my familiarity with such coding schemes is minimal (i.e., I have no clue), so I cannot understand their applicability in a crowdsourcing setting.<br />
<br />
<div>
So, here is my question: What coding schemes should we use in crowdsourcing in order to get closer to the theoretical limits given by Shannon? Or what is the fundamental thing that I miss? Because I do feel that I am missing something...</div>
<div>
<br /></div>
<div>
Any help would be appreciated.</div>
</div>
<div style="text-align: justify;">
<br />
<ul>
<li><b><i>Update 2</i></b> (thanks to the comments by stucchio and syrnik): Information theory predicts that we can <b><i>always</i></b> recover the <b><i>perfect</i></b> answers from noisy workers, given sufficient worker capacity. For anyone that has worked in crowdsourcing, this sounds very strange, and seems practically infeasible. The problem does not seem to be in the assumptions of the analysis; instead it seems to rely on the feasibility of <b><i>implementing a proper encoding scheme on top of human computation</i></b>.<br /><br />The key concept in information theory is the coding scheme that is used to encode the information, to make the transmission of information robust to errors. Information theory does not say <b><i>how</i></b> we can recover this perfect information using a noisy channel. Over time, researchers came up with appropriate encoding schemes that approach very closely the theoretical maximum (see above, Reed-Solomon codes, turbo codes, etc). However, it is not clear whether these schemes are translatable into a human computation setting.<br /><br />Consider this gross simplification (which, I think, is good enough to illustrate the concept): In telecommunications, we put a "checksum" together with each message, to capture cases of incorrect information transmission. When the message gets transmitted erroneously, the checksum does not match the message content. This may be the result of corruption in the message content, or the result of corruption in the checksum (or both). In such cases, we re-transmit the message. Based on the noise characteristics of the channel, we can decide how long the message should be, how long the checksum should be, etc., to achieve maximum communication efficiency.<br /><br />For example, consider using a parity bit, the simplest possible checksum computation. We count the number of 1 bits in the message: if the number of 1's is odd, we set the parity bit to 1, if the number of 1's is even, we set the parity bit to 0. The extra parity bit increases the size of the message but can be used to detect errors when the message gets transmitted over a noisy channel, and reduce the error rate. By increasing the number of parity bits we can reduce the error rate to arbitrarily low levels.<br /><br />In a human computation setting, computing such a checksum is highly non-trivial. Since we do not really know the original message, we cannot compute at the source an error-free checksum. We can of course try to create "meta"-questions that will try to compute the "checksum" or even try to modify all the original questions to have an error-correcting component in them. <br /><br />See now the key difference: In information theory, we have computed <b><i>error-free</i></b> the message to be transmitted with built-in error-correction. Consider now the same implementation in a human computation setting: We ask the user to inspect the previous k questions, and report some statistics about the previously submitted answers. The user now operates on the <b><i>noisy </i></b>message (i.e., the given, noisy answers), therefore even the error-free computation of the checksum is going to be noisy, defeating the purpose of an error-correcting code.<br /><br />Alternatively, we can try to take the original questions, and try to ask them in a way that enforces some error-correcting capability. However, it is not clear that these "meta-questions" are going to have the same level of complexity for the humans, even if in the information-theoretic sense, they carry the same amount of information.<br /><br />It seems that in a human computation setting we have <b><i>noisy computation</i></b>, as opposed to <i><b>noisy transmission. </b></i>Since the computation is noisy, there is a good chance that the computation these "checksums" is going to be correlated with the original errors. Therefore, it is not clear whether we can actually implement the proper encoding schemes on top of human computers, to achieve the theoretical maximums predicted by information theory.<br /><br />Or, at least, this seems like a very interesting, and challenging research problem.</li>
</ul>
</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-41312484663290842392013-06-28T20:34:00.003-04:002013-06-28T20:34:59.822-04:00Facebook implements brand safety, doing it "manually" (crowdsourcing?)<div style="text-align: justify;">
I just read that Facebook <a href="http://newsroom.fb.com/News/658/A-New-Review-Policy-For-Pages-and-Groups" target="_blank">started thinking about</a> <a href="http://integralads.com/our-solutions/brand-safety" target="_blank">brand safety</a>, and will restrict ads from appearing next to content that may be controversial (e.g., adult-oriented content).</div>
<div>
<div style="text-align: justify;">
<br /></div>
</div>
<div>
<div style="text-align: justify;">
I was rather surprised to find out that Facebook has <b><i>not </i></b>been doing that already. It is know that Facebook <a href="http://bits.blogs.nytimes.com/2011/12/05/spot-porn-on-facebook-for-a-quarter-cent-an-image/" target="_blank">has been using crowdsourcing</a> to detect content that violates the terms of service. So, I assumed that the categorization of the content as brand-inappropriate was also part of that process. Apparently not.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Given the similarities of the two tasks (the difference between no-ads-for-brand-safety and violating-terms-of-service is often just part of intensity of the offense), I assume that Facebook is also going to adopt a crowdsourcing-style solution (perhaps with a private crowd), and then they will build a machine learning algorithm on top using the crowd judgements. At least the wording "<i>In order to be thorough, this review process will be manual at first, but in the coming weeks we will build a more scalable, automated way</i>" in the announcement seems to imply that.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Or perhaps, to blow my own horn, Facebook should just use <a href="http://integralads.com/" target="_blank">Integral Ad Science</a>, (aka AdSafe Media). At AdSafe, we built a solution for exactly this problem back in 2009, employing a combination of crowdsourcing and machine learning to detect brand-inappropriate content. We did not go just for porn, but also for other categories, such as alcohol use, offensive language, hate speech, etc. In fact, most of my work in crowdsourcing was inspired, one way or another, through the problems faced when trying to deploy a crowdsourcing solution at scale. Also, except for the academic research, my work with Integral also led to one of the best blog posts that I have written, "<a href="http://www.behind-the-enemy-lines.com/2011/03/uncovering-advertising-fraud-scheme.html" target="_blank">Uncovering an advertising fraud scheme (or, the Internet is for Porn)</a>".</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Perhaps, the next step is to demonstrate how to use <a href="http://project-troia.com/" target="_blank">Project Troia</a>, together with a good machine learning toolkit in order to deploy quickly a system for detecting brand inappropriate content. Maybe Facebook could use that ;-)</div>
</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-17572513870154349862013-06-28T16:57:00.003-04:002013-06-28T17:06:36.641-04:00Mechanical Turk account verification: Why Amazon disables so many accounts<div style="text-align: justify;">
Over the last year, Amazon embarked into a big effort: All holders of an Amazon Payments account (which includes all the Mechanical Turk worker) had to verify their accounts, by providing their social security number, address, full legal name, etc. Users that did not provide this information found their accounts disabled, and unable to perform any financial transaction.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
This led to big changes in the market, as many international workers realized that Amazon could not verify their identity (even if they provided the correct information), and they found themselves locked out of Mechanical Turk.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So, why would Amazon start doing that?</div>
<br />
<ul>
<li><div style="text-align: justify;">
<a href="http://turkrequesters.blogspot.com/2013/01/the-reasons-why-amazon-mechanical-turk.html" target="_blank"><b>Low quality of international workers</b></a>. While there are certainly many high-quality workers outside the US, there is a certain segment of workers that join the market with the sole purpose of getting something for nothing. Especially after Indian workers became eligible to receive cash compensation (instead of just gift cards available to other non-US workers), the number of spam attacks from India went up significantly. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So, <a href="http://www.behind-the-enemy-lines.com/2012/10/why-odesk-has-no-spammers.html" target="_blank">identity verification can help in that front</a>. It is well-known that it is difficult to have a good reputation scheme that allows for cheap generation of identities. When identities are easy to create, every time someone commits a bad action and gets caught, the account gets closed and a new account is created, ready to commit the same bad actions again. This hurts significantly new workers, that are defacto treated as potential spammers, discouraging them to join the market. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I have long criticized the fact that Amazon allowed for easy generation of ids. Even though it seemed that Amazon required SSN numbers, and other information to create an account, this was an effectively optional step. In fact, it was possible to use <a href="http://www.fakenamegenerator.com/" target="_blank">Fake Name Generator</a>, and create plenty of <a href="http://www.behind-the-enemy-lines.com/2012/01/identify-verification-and-how-to-bypass.html" target="_blank">seemingly authentic "US based" accounts</a>, using simply <a href="http://ssa-custhelp.ssa.gov/app/answers/detail/a_id/149/~/social-security's-death-master-file" target="_blank">SSN numbers of dead people</a>. This meant that many fake accounts existed, many of them being "US based" that then used Amazon Payments to forward their earnings to the true puppetmaster holder.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
</li>
<li><div style="text-align: justify;">
<b>Labor law</b>. Even though many (small) requesters are unaware of the fact, when you post jobs on Mechanical Turk, you directly engage into hiring contractors to do some work for you. Many people believe that you are paying Amazon, who then pays the workers but in reality Amazon acts simply as a payment processor. Amazon does not act as an employer; the requester acts as an employer. As discussed in the past, this forces many requesters to <a href="http://www.behind-the-enemy-lines.com/2009/07/is-amazon-mechanical-turk-black-market.html" target="_blank">unknowingly participate in a black market</a>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The moment requesters realize that they are actually employing all these contractors is when some workers end up receiving more than $600 in payments from the requester over the fiscal year. At that point, due to IRS regulations, the requester needs to send a 1099-MISC form to the MTurk worker. Amazon then provides the full information (SSN, address, etc) of the workers to the requester. <b>So Amazon would like to have the correct information, to avoid forcing the requesters to send 1099 forms to fake addresses, with fake names and SSNs.</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I should clarify here that the $600 limit is the point where the employer is <b>forced </b>to send a 1099-MISC form. In principle, a requester may want to send 1099-MISC forms to <b>all </b>workers, and Amazon may want to provide this information on demand. (I doubt that this can be the reason, though).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Finally, there was a new regulation from IRS last year: IRS introduced the concept of a <a href="http://www.irs.gov/uac/Form-1099-K,-Merchant-Card-and-Third-Party-Network-Payments" target="_blank">1099-K form</a>. Since Amazon acts as a defacto payment processor (and not as an employer), Amazon should also report the amount of payments sent to each worker. So, even if no worker have met the $600 limit from a single requester, <b>if the overall payments for a single worker was high enough (specifically <a href="http://www.irs.gov/pub/irs-utl/irdm_section_6050w_faqs_7_23_11.pdf" target="_blank">$20,000/yr or more, and more than 200 requesters</a>) then again Amazon needs to report this information and include valid worker information there.</b></div>
<div style="text-align: justify;">
<br /></div>
</li>
<li style="text-align: justify;"><b>Money laundering</b>: Since Mechanical Turk started becoming a marketplace with significant volume, this may have raised some flags in all the places that monitor financial transactions for money laundering. All US companies need to comply with the infamous <a href="http://en.wikipedia.org/wiki/USA_PATRIOT_Act">US Patriot Act</a>, and for Mechanical Turk the <a href="http://en.wikipedia.org/wiki/USA_PATRIOT_Act,_Title_III">provisions about money laundering and financing of terrorist activities</a> may have been a reason for cleaning up the marketplace from fake worker identities. The basic idea, known as the "<a href="http://en.wikipedia.org/wiki/Know_your_customer">Know Your Customer (KYC)</a>" doctrine, is that Amazon should know from whom they get money and to whom they send the money. Since <a href="http://www.behind-the-enemy-lines.com/2010/02/why-mechanical-turk-allows-only-us.html" target="_blank">Amazon accepts payments from US requesters only</a>, they know where the money come from. Now, with cleaning up the marketplace from fake identities and verifying the existing ones, they also know where the money flows to, so they seem to be more in compliance with the money laundering laws.<br /> </li>
</ul>
<div style="text-align: justify;">
Overall, there are many reasons for Amazon to check and clean up the market from fake accounts and prevent any anonymous activity. For me, this is a good step, despite all the problems that it may generate for workers that have problems proving their identity. Even in India, <a href="http://www.stern.nyu.edu/experience-stern/faculty-research/india-unique-id" target="_blank">the new UID system</a> will eventually allow the legitimate Indian workers to prove their identity without problems.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
One concerns that someone expressed to me was that this direction was <a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2228728" target="_blank">removing the ability of workers to be truly anonymous</a>. I am not exactly sure how this can be a concern, given that it is well established that in the workplace (electronic or not) there is very limited right to privacy. Knowing the true identity of your workers (contractors or employees) is a pretty fundamental right of the employer, and I doubt that the expectation that a worker remains anonymous can be a "<a href="http://www.nolo.com/legal-encyclopedia/workplace-privacy" target="_blank">reasonable expectation of privacy</a>". The only case that I see this happening is if Amazon switches from being a payment processor to being an employer of all the Mechanical Turk workers, but I doubt this will happen anytime soon. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
At the end of the day, markets do not mix well with true and complete anonymity.</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.comtag:blogger.com,1999:blog-7118563403027467631.post-53096329445060350432013-06-18T16:31:00.003-04:002013-06-18T16:31:46.793-04:00Project Troia: Quality Assurance in CrowdsourcingOne of the key problems in crowdsourcing is the issue of quality control. Over the last few years, a large number of methods have been proposed for estimating the quality of workers and the quality of the generated data. A few years back, we have released the <a href="https://github.com/ipeirotis/Get-Another-Label" target="_blank">Get Another Label</a> toolkit, which allowed people to run their data through a command-line interface, and get back estimates of the worker quality, estimates of how well the data have been labeled, and identify the data points that have high uncertainty and therefore may require additional attention.<div>
<br /></div>
<div>
The next step for the Get Another Label was to get it ready to work in more practical settings. The GAL toolkit, assumed that we have all the labels assigned by the workers, we process them, and get the results. In reality, though, most tasks run in an incremental mode. The task is running over time, new data arrive, new workers arrive, and the "load-analyze-output" process was not a good fit. We wanted to have something that gives back estimates of worker quality on the fly, and again on-the-fly identifies the data points that need most attention.</div>
<div>
<br /></div>
<div>
Towards this goal, over the last few months we have been porting the GAL code into a web service, called <a href="http://project-troia.com/" target="_blank">Project Troia</a>. You can load the data <b><i>as the crowdsourced project runs</i></b> and get back the results immediately. This allows for very fast estimation of worker quality, and also allows the quick identification of data points that either meet the target quality, or require additional labeling effort.</div>
<ul>
<li>Supports labeling with any number of discrete categories, not just binary.</li>
<li>Supports labeling with continuous variables.</li>
<li>Allows the specification of arbitrary misclassification costs (e.g., "marking spam as legitimate has cost 1, marking legitimate content as spam has cost 5").</li>
<li>Allows for seamless mixing of gold labels and redundant labels for quality control.</li>
<li>Estimates the quality of the workers that participate in the task and returns the estimates on-the-fly.</li>
<li>Estimates the quality of the data that are returned back by the algorithm and returns the estimate of labeling accuracy on-the-fly.</li>
<li>Estimates a quality-sensitive payment for every worker, based on the quality of the work done so far.</li>
</ul>
<div>
If you are interested in the description of the methods implemented in the toolkit, please take a look at the paper "<a href="http://archive.nyu.edu/handle/2451/31825" target="_blank">Quality-based Pricing for Crowdsourced Workers</a>". Our experiments indicate that when labeling allocation happens following the suggestions of Project Troia, we achieve the target data quality with almost optimal budget, and workers are fairly compensated for their effort. (For details, see the paper :-)</div>
<div>
<br /></div>
<div>
Special thanks to Tagasauris, oDesk, and Google for providing support for developing the software. Needless to say, <a href="http://project-troia.com/documentation/3_API.html" target="_blank">the API is free to use</a>, and the <a href="https://github.com/ipeirotis/Troia-Server" target="_blank">source code is available on Github</a>. We hope that you will find it useful.</div>
<div>
</div>
Panos Ipeirotishttp://www.blogger.com/profile/15283752183704062501noreply@blogger.com