How Many Porta Potties Should You Check? The Secretary Problem and Hiring

In his words

"It's much more expensive to hire the wrong person than to spend time interviewing more people."

"Early in the process, it's likely there's somebody better you haven't seen yet. That makes it more comfortable to say no."

"It's not a replacement for your intuition."

The interview

How did you first get into the secretary problem?

I first saw it in a Numberphile video on YouTube. They talked about it in terms of being at a concert or something, and you need to use a porta potty, and you want to check if it's clean. How many of these porta potties would you check before you committed to using one? So it was an interesting math problem: how many do I need to investigate before I start choosing?

That's where I was first introduced to the idea. But it came up when I started to interview candidates. We were trying to budget: how many candidates do we want to interview, and how much time do we want to spend? How many resources can we allocate to the hiring process? And based on that, when do we want to make a decision about who to hire? That's where I started to dig into this a little more, because it had real-world applications.

What were you hiring for?

My role at the company is automation specialist. My company makes AI metal detectors, and my job is to make sure the software that runs on these scanners is working. I write a lot of automated tests, and I evaluate the detection algorithms.

The role we were hiring for was a manual test role, where we'd also want that person to be able to work in the automated test environment. It was a senior SQA engineer role. His job was going to be more to work directly on the scanner, manually, but be able to make some automated contribution.

Did you actually use the porta potty approach on that hire?

I wasn't responsible for the hiring, but I try to make ideas clear for the team. When we were discussing "do we want to hire this person or not," I drew on that knowledge to say, "Okay, well, it depends." How many people do we potentially want to see?

I talked about the most effective strategy. If we have the capacity to interview 10 people, then we'd want to investigate three or four of them, and the next best person after that, we would hire. So I was talking about it in terms of hire or no hire, and we figured out it made sense to pass on this person. Not because we thought they were a bad fit, but because we wanted to explore more options. There might be somebody else out there who is even more qualified.

How did the team react when you said to pass on someone who wasn't a bad fit?

I think people are often inclined to say yes early when hiring, because it's so expensive. It's tough to hire people. It's a lot of work, and it's usually not budgeted into your time, so you're doing it on top of everything else you have to do. So there's an inclination to say yes too early or too quickly when we're not sure about the fit.

But the truth is, it's very expensive to hire somebody who isn't a good fit, because then you need to spend all sorts of time working with them, and firing people is very expensive. It's much more expensive to hire the wrong person than to spend time interviewing more people. When you put it that way, people are more receptive.

People aren't necessarily super stoked when I say, "I think we need to pass." But I try to frame it as: the next person might be a better fit. And we look forward to some of the other resumes we've seen: "I really want to try this guy, he might be a good fit." Framing it in a more positive way can be helpful.

What happened with the candidate you passed on?

I didn't really keep in touch with them, so I don't know where he ended up. But the candidate we said yes to ended up being a very good fit. He's still with the team, and things are going quite well there.

Where in the order did he come?

Funny enough, if the algorithm is "examine the first 37%," that would have been three or four people, because maybe we had capacity for 10, let's say. And he would have been the fourth or fifth. So he actually was the best candidate so far after the cutoff, which is interesting. So there's a pretty high probability that he would have been the best overall. And it turns out he's a pretty good fit.

You also built a visualization of this. What were you trying to show?

Statistics often aren't very intuitive. So the intention was to model and simulate what would happen, to demonstrate with a visualization that the algorithm actually works, statistically.

I built a little JavaScript app that randomly simulates it. It's an abstract idea. It doesn't have to be hiring; it could be anything. How many evaluations do you do? It randomly generates a set of numbers, and you're trying to find the best fit, say the best number. How many do you see, and what's the probability that you get the best one?

Did anything surprise you when you ran it over and over?

Not really, just because I'd been exposed to the idea for a while, and that Numberphile video really laid it out well. I do think the statistics are a little unintuitive. But once you get past that and you can see it happen live... You know those boards with a bunch of little beads in them [Galton boards], where you can watch the normal distribution form in real time? Those are cool. I think this is similar. You learn something in school, and it's an interesting idea, but actually seeing it modeled out is pretty interesting.

Where is the math nonsense for real hiring?

I'd say it's not a replacement for your intuition. If somebody comes along and you're really sure about them, I don't think you need to throw them out of the running. So I wouldn't say you should be beholden to the process. But it's helpful to keep in mind, when you're deciding hire or no hire, that early in the process, it's likely there's somebody better you haven't seen yet. That makes it more comfortable to say no.

Has there been a time your gut said yes but the math said wait?

Did I trust my gut? Definitely. I don't use mathematical models much in my life. This is a special instance that I thought was intellectually interesting. I rely a lot on my intuition. I can't think of a particular time where the math said yes but my gut said no, because I tend to rely a lot more on my gut.