What Freakonomics Taught Me About Asking Better Questions

There are several Freakonomics books sitting on my shelf today.

The first one entered my life much earlier, when a classmate recommended Freakonomics to me in business school. I enjoyed the stories and the unconventional way Steven Levitt and Stephen Dubner approached seemingly ordinary questions.

But what stayed with me wasn't any particular answer in the book.

It was the habit of questioning the answer that seemed obvious.

I began giving copies to members of my team at MHP. Years later, I did the same at Katz Hillel Day School and Jewish Leadership Academy, where we read and discussed the book as a group.

In 2010, I attended the FMI/GMA Supply Chain Conference in San Antonio, where Steven Levitt was the keynote speaker. When I met him, I told him that I had a recurring problem: I kept giving away my copies of Freakonomics.

He inscribed my newly released copy of SuperFreakonomics:

"To Joe, Keep giving this book away!"

I did.

Eventually, because my first two books in the series were signed, I became a little more careful. If I lent one out, I traded the borrower a new copy to get mine back.

But what I've really tried to give away over the years isn't the book.

It's a way of thinking.

Don't Stop at the First Explanation

Organizations are very good at developing explanations.

Enrollment declined because of demographics.

Expenses increased because prices went up.

Employees are unhappy because they want higher salaries.

A process failed because people don't like change.

Customers are complaining because their expectations are unreasonable.

Maybe.

One of the most useful habits I've developed over my career is resisting the temptation to turn a plausible explanation into a conclusion too quickly.

Sometimes the data supports the conventional explanation. Sometimes it doesn't.

And sometimes asking one more question uncovers something you weren't looking for at all.

There was a point in time, while reviewing expenses at a school, I noticed that our water bills were rising dramatically.

Why?

I wasn't satisfied with the answers we were getting, so I kept asking. Eventually, I asked that we physically search the campus for leaks.

We found them.

One was a broken pipe outside, sending water unnoticed into the grass along the edge of the property.

We also discovered a defunct water line that should have been capped, along with an account that should have been discontinued.

But the investigation uncovered something far more consequential.

In one of the buildings, a design or construction defect had allowed moisture to accumulate between the walls behind built-in shelving. Hidden from view, the woodwork had become covered in mold, and the problem was continuing to spread.

A rising number on a utility report had led us into the walls of a building.

And here's what I think is important: I hadn't correctly guessed what we would find.

The water bill may not even have been entirely explained by what we found.

The lesson wasn't that I knew the answer.

It was that we didn't stop looking simply because we had answers that sounded reasonable.

Think About What Happens After You Make the Rule

Our Freakonomics discussions with my teams weren't limited to analyzing data. We spent a lot of time talking about incentives and unintended consequences.

Organizations frequently create policies with the best of intentions.

We want to be flexible. We want to treat people well. We want to create a positive culture. So we design a benefit or policy around how we expect reasonable people to use it.

Then someone finds the outer boundary.

Someone else stretches it a little further.

Eventually, leadership gets frustrated. And too often the response is to replace a generous policy with a restrictive one that punishes everyone, including the overwhelming majority of people who never abused it.

A better question can be asked at the beginning:

If we create this policy, how might people actually respond to it?

That's not about assuming the worst in people.

It's about recognizing that good intentions don't eliminate incentives or unintended consequences. Good policy design requires thinking not only about what we're trying to accomplish, but also about how the system might actually be used.

The goal isn't to write rules around every possible bad actor.

It's to anticipate enough of the unintended consequences that a few exceptions don't eventually force you to design the entire organization around them.

Stop Trying the Same Door

There was another lesson I repeatedly emphasized with my teams.

When an approach isn't working, stop assuming that trying it harder will make it work.

We all have methods that are comfortable. They've worked before. They're familiar. They're easy to execute. Perhaps they're even considered best practice.

So when we hit a wall, our instinct is often to try the same thing again.

Another email.

Another meeting.

Another version of the same process.

Another analysis using the same assumptions.

Sometimes persistence is exactly what's required.

Other times, we're simply pushing harder on a door that isn't going to open.

I've encouraged teams throughout my career to become comfortable asking:

What haven't we tried?

Change the approach. Challenge the assumption. Look at different data. Talk to someone who hasn't been part of the conversation. Turn the problem around. Consider the option that initially seems a little out of the box.

Something that worked somewhere else isn't automatically right here.

Something that worked here five years ago isn't automatically right today.

And something being the easiest thing to try doesn't make it the thing most likely to work.

Data Doesn't Eliminate Judgment

I've spent much of my career in finance and operations, so I have tremendous respect for data.

But data can create its own kind of false confidence.

A spreadsheet can tell you precisely what happened without telling you why it happened.

A trend line can make two events look connected when they aren't.

A variance can point toward an explanation that feels so obvious that nobody bothers testing it.

I've seen many situations where the numbers appeared to tell one story, but a deeper look suggested another, or revealed that the presumed story simply couldn't be confirmed.

The answer isn't to trust data less.

It's to interrogate it more.

What exactly are we measuring?

What aren't we measuring?

What assumptions are we making?

What else changed?

What evidence would cause us to change our conclusion?

And perhaps most importantly:

What if we're wrong?

Those questions don't weaken analysis.

They're what make analysis useful.

The Value of a Better Question

More than fifteen years after meeting Steven Levitt, the books are still on my shelf.

I've read them, discussed them with teams, lent them out, replaced copies, and given copies to people I've worked with.

So I suppose I've followed Levitt's instruction.

But what I've tried hardest to give away isn't a copy of Freakonomics.

It's the habit the book helped reinforce.

Don't accept an explanation merely because it sounds right.

Don't assume people will interact with a system exactly as its designer intended.

Don't keep using an approach simply because it worked before.

Look at the data. Then look behind the data.

Keep asking why.

"And when everyone is busy trying to find the right answer, occasionally stop and ask whether you're answering the right question."