Monday, November 30, 2009

Anchoring and Mechanical Turk

Over the last few days, I have been reading the book Predictably Irrational by Dan Ariely, which (predictably) describes many biases that we exhibit when making decisions. These biases are not just effects of random chance but are rather expected and predictable. Such biases and the "irrationality" of human agents is one of the focuses of behavioral economics; these biases have been also extensively studied in the field of cognitive psychology, which examines the ways that human agents process information.

One of the classic biases is the bias of "anchoring". Dan Ariely in his book shows how he got students to bid higher or lower for a particular bottle of wine: He asked students to write down the last digit of their social security number before placing the bid. As the anchoring theory postulated, students that wrote down a lower number, ended up bidding lower than students with a higher last digit in their SSN.

Why? Definitely not because the last digit revealed anything about their character. It was because the students got "anchored" to the value of the last digit they wrote down. I am certain that the experiment could be repeated by using the middle two digits as anchor, and the results would be similar.

Interestingly enough, at the same time that I was reading the book, I got contacted by Gabriele Paolacci, a PhD student in Italy. In his blog, Experimental Turk, Gabriele has been replicating some of these "classic" cognitive psychology experiments that illustrate these biases. As you might have guessed already, Gabriele has been using Mechanical Turk for these experiments. Gabriele tested the theory of anchoring using Amazon Mechanical Turk, replicating a study from a classic paper. In his own words:

We submitted the “african countries problem” from Tversky and Kahneman (1974) to 152 workers (61.2% women, mean age = 35.4). Participants were paid $0.05 for a HIT that comprised other unrelated brief tasks. Approximately half of the participants was asked the following question:

  • Do you think there are more or less than 65 African countries in the United Nations?
The other half was asked the following question:
  • Do you think there are more or less than 12 African countries in the United Nations?
Both groups were then asked to estimate the number of African countries in the United Nations.

As expected, participants exposed to the large anchor (65) provided higher estimates than participants exposed to the small anchor (12), F(1,150) = 55.99, p<.001. Therefore, we were able to replicate a classic anchoring effect - our participants’ judgments are biased toward the implicitly suggested reference points. It should be noted that means in our data (42.6 and 18.5 respectively) are very similar to those recently published by Stanovich and West (2008; 42.6 and 14.9 respectively).


Stanovich, K. E., West. R. F. (2008). On the relative independence of thinking biases and cognitive ability. Journal of Personality and Social Psychology, 94, 672-695.

Tversky, A., Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124-1131.

Gabriele has more experiments posted in his blog, and I am looking forward for more experiments.

So, here is a question: Definitely we should take similar biases into consideration when collecting data from humans, and when conducting user studies. In a more general setting, can we use such biases more productively, in order to get users to complete tasks that are useful?

Ignore.. (Test)

Just a set of links to the homepages of my students, to be picked up by search engines...

Wednesday, November 18, 2009

Using the NYC Data Mine for an Intro Database Assignment

On October 6th, I was attending the New York Tech Meetup, and there I learned about the NYC Data Mine repository, which contains "many sets of public data produced by City agencies [...] available in a variety of machine-readable formats".

I went over the data sets available there and indeed the data sets were big, comprehensive, and (mostly) well-structured. So, I decided to use these data sets for the introductory database assignment for my "Information Technology in Business and Society" class. It is a core, required class at Stern and the students are mainly non-majors. Still, I wanted to see what they will do with the data.

So, I created an assignment, asking them to get two or more data sets, import them in a database and run some basic join queries to connect the data sets. Then, they had to bring the data into Excel and perform some PivotChart-based analysis. I left the topic intentionally open, just to see what type of questions they will ask.

Here are the results, together with my one-sentence summary of the analysis/results.
Given that this was the first time that I was giving this assignment, and that this was the first time that students were learning about databases, I was pretty happy with the results. Most of them understood well the datasets and wrote meaningful queries against the data.

However, I would like to encourage the analysis of a more diverse set of data: Students seemed particularly attracted to the graffiti dataset and (expectedly) most used the data set with the socio-economic numbers of each borough.

The rather disappointing fact was that many teams took the "easy way out" and joined data based on the borough (Manhattan, Queens, Brooklyn, Bronx, Staten Island), while it would have been much more interesting to see joins based on zip codes, community boards, districts etc. I guess this becomes a requirement for next year.

Finally, I should encourage people to work with really big datasets (e.g., property valuation statistics), instead of the relatively small ones. But perhaps this is something reserved for the data mining class...

Friday, November 6, 2009

Utility of Money and the St. Petersburg Paradox

Consider the following game:

We will flip a fair coin, until a tail appears for the first time.
  • If the tail appears in the first throw, you win $2^1=2$ dollars.
  • If the tail appears in the second throw, you win $2^2=4$ dollars. 
  • If the tail appears in the third throw, you win $2^3=8$ dollars.
  • ...
  • If the tail appears in the $n$-throw, you win $2^n$ dollars.
What is the amount of money that someone should risk to enter this game? (This question works best when given to a person that claims to never play a lottery, roulette, or any gambling game, because the expected return is lower than the bet.)

Computing the expected return of this game, we have:

$E=\frac{1}{2}\cdot 2+\frac{1}{4}\cdot 4 + \frac{1}{8}\cdot 8 + \cdots = 1+1+1+ \cdots =\infty$

In other words, the expected utility is infinity, and a rational player should be willing to gamble an arbitrarily large amount of money to enter this game.

Can you find anyone willing to bet \$1,000 to play this game? Or \$10,000? Or even \$100? Yes, I did not think so.

This paradox is called the St. Petersburg Paradox, posed in 1713 by Nicholas Bernoulli and solved in 1738 by Daniel Bernoulli. Since then, a number of potential explanations appeared.

The most common approach is to use expected utility theory. In this case, we introduce a utility function $U(x)$, which describes the "satisfaction" that someone would get by having $x$ amount of money.

Utility of Money: The basic idea is that people do not bet based on the absolute amounts of the return but rather based on the utility of the award. The value of an additional \$100 when I have \$100 in the bank is much higher compared to the case when I have \$1,000,000 in the bank. This means that the "utility of money" function is a concave function of the available funds.

Just for demonstration, below you can see such a concave utility-of-money function that we have computed as part of a research project:

This concavity also partially explains the "risk aversion" that most people have: they prefer certainty over uncertainty. This means that they will reject even a reasonable bet with positive expected return. Why? Notice that the utility gained by winning is smaller than the decrease in utility that results from losing the bet. The higher the concavity, the higher the risk aversion.

If you want to read more about utility of money and its applications to portfolio management, insurance, and analysis of other cases, take a look at this book chapter.

So, next time that someone claims never to engage in any bet with a negative expected return, give the setting of the Bernoulli paradox with the positive expected return and observe the reactions...