During my 15 years at Amazon, I interviewed hundreds of candidates. One shift I had to make was abandoning interview questions that were hypothetical, case or modeling questions, or in any way not behavioral. There are two problems with hypotheticals:
1) They tell you how a candidate will think, but not how they will behave, manage, and lead.
2) They introduce bias because the interviewer tends to have either a single or a small range of “correct” answers in mind.
Here are two of my favorite hypothetical or modeling questions that I asked during my first few years at Amazon, along with their benefits and drawbacks.
1. Imagine that you are the Product Manager of a new idea for a subscription service called Amazon Prime. The way it will work is, for $49 a month, customers get free two-day shipping on every order. Prior to launch, the CFO asks you to prepare a financial model to measure the expected impact of Prime. Describe the model you would build, including the metrics and your assumptions.
2. If you were the CEO of Amazon, tell me about the most important changes or improvements you would make to grow revenue and profitability.
The point of the first question was to determine the analytical capabilities of the candidate. I was looking to see how facile they were with constructing a model designed to measure the impact of new product, feature or promotion.
One of the keys was for them to identify specific metrics that would change, such as average order size, frequency of shopping visits, number or breadth of product categories shopped, etc. I found it useful to see candidates talk about the importance of various metrics for this analysis.
But, one of the problems with this question was that I knew the “right” answer, which was to compare the average (or by segment for extra credit) customer frequency of purchase and the corresponding total gross profit dollars before Prime vs after Prime. The handful of candidates able to zero in on this would get my highest marks. I gave people credit if they were close and their framework was logical and detail-oriented, but there was no way I would hire someone who flubbed it.
The second question differed in that I was assessing a candidate’s ability to identify the right problems and opportunities. It was also a test to see how much they had studied Amazon in advance of the interview.
In retrospect, this question was pretty useless. Almost everyone answered with “…I would improve personalization” and/or “…I would reduce the clutter on the home page.”
If everyone gives nearly the same (bad) answer, then the question is a waste of time. And just like the first question, I was biased in my assessment of the “correct” answer based on my insider knowledge.
Behavioral questions, on the other hand, reduce bias and give the candidate the ability to describe their past work where THEY possessed the requisite insider information to make good decisions.
(cont. in comments)
