Tuesday, November 23, 2021

"Geographic Footprint of an Agent" or one of my favorite data science interview questions

 Last week we wrote in the Compass blog how we estimate the geographic footprint of an agent.

At the very core, the technique is simple: Use the addresses of the houses that an agent has bought or sold in the past; get their longitude and latitude; and then apply a 2-dimensional kernel density estimation to find what are the areas where the agent is likely to be active. Doing the kernel density estimation is easy; the fundamentals of our approach are material that you can find in tutorials for applying a KDE. There are two interesting twists that make the approach more interesting:

  1. How can we standardize the "geographic footprint" score to be interpretable? The density scores that come back from a kernel density application are very hard to interpret. Ideally, we want a score from 0 to 1, with 0 being "completely outside of the area of activity" and 1 being "as important as it gets". We show how to use a percentile transformation of the likelihood values to create a score that is normalized, interpretable, and very well calibrated.
  2. What are the metrics for evaluating such a technique? We show how we can use the concept of "recall-efficiency" curves to provide a common way to evaluate the models.

You can read more in the blog post. 

Despite its simplicity, this topic ended up being an amazing interview question. I think it has a great question for separating candidates that have a deeper knowledge of data science from those that have only a superficial understanding.

The typical question during the interview is:

"You have a property where the owner wants to sell the house. How can you determine if the property is geographically relevant to a particular agent? For each  agent, we know all their past transactions, and we can assume that future behavior is captured well by their past behavior. For all their past transactions, you can get the address, zip code, longitude, and latitude. Do not try, for now, to figure out if the agent is the best one among many candidate agents; that is a harder problem. Just figure out if the agent is active in the location where the property lies."

I am very surprised by how many people will start on the wrong foot because they will try to actively shoehorn the problem into a binary classification problem: Indicative answers:

  • "I will find all the properties that the agent has not transacted in the past, and treat them as negatives".
  • "I will keep showing properties to the agent and when they say no, I will mark them as negatives"
  • "I will add features like bathrooms, bedrooms, price, and then predict if it is relevant or not"
In my experience, once people start like that, they rarely get to a solution that works. The key problem is that they have no easy way of evaluating the model they propose. 

So, a typical follow-up question is:
How would you evaluate your approach?
This now gets interesting. Common superficial answers are:
  • I would use precision and recall, or AUC.
  • I would measure how often agents accept my recommendations
If they recommend precision and recall, I ask how they are going to measure recall (relatively easy, hopefully, the answer is a temporal training/test split), and precision. Precision gets tricky, as there is no "base rate" and there are no clear "negatives".

If someone recommends measuring the agent reactions, I acknowledge that this is correct, in principle, but ask them to propose an offline evaluation, so that we can test the approach before releasing it to the agents. 

This is typically the point where the better candidates, even they veered off track will start to move towards a better solution (not necessarily KDE, it can also be clustering, convex hulls, histograms on zipcodes as vector representations, etc), while the one-trick ponies will reveal their lack of substance.

Kind of sorry that I have to retire this question from my question bank. Still impressed that such a simple question had such a strong discriminatory power.