Is it possible that America’s most influential higher education ranking system is biased?

Those of you who have been listening to my podcast, Revisionist History, will know that recently we launched an investigation into the abomination known as the U.S. News & World Report college rankings.

We broke it all down over two episodes: First, the odd beginnings of U.S. News rankings and their dubious metrics. How exactly does their ranking algorithm work?

What is the real financial benefit — or cost — of reputation? And how is it that a promotional gimmick from the 1980s turned into a fetish object for American higher education?

In the second episode, I asked a statistics major at Reed College, Lauren Rabe, to use a statistical model to predict the most important variable that makes up U.S. News rankings — the so-called “peer assessment” score. The magazine asks every college president, provost, and admissions dean to grade the academic reputation of every one of their competitors on a scale of one to five. Nothing matters more in your eventual ranking than your performance on this metric.

I was interested in a historically black university in New Orleans called Dillard University. Dillard has a terrible peer assessment score — 2.6 — even though, on a number of very objective measures, it does an outstanding job of educating the students who go there. In theory, the reputation score reflects a school’s academic strength in the eyes of other academic professionals. I wanted to know: Do the deans and provosts who issue reputation scores ever evolve their opinions year to year? How likely is it that a school like Dillard would ever have a strong reputation score?

So I asked Lauren Rabe: What if you were to Google a couple of really simple facts about a school — how large its endowment is, how high its tuition is, what percentage of its student body is white, and what its reputation score was, say, 10 years ago. Can you accurately predict its current reputation score using those factors?

Oh yes, you can.

This is Lauren’s chart. It shows just how close you can get to a school’s actual reputation score just by looking at a combination of those four factors, along with the percentage of students who receive federal financial aid grants. Look at the final bar on the right!

And the bar second from the right is scary on its own: That’s how often you can predict a school’s current reputation score based entirely on its score from 10 years ago. 88.5 percent of the time, you can.

If you’ve applied to college recently or followed your kids through the process, I’m sure you’ve heard at some point, “Oh. That school has a great reputation.” And maybe you wondered, What does that mean? Lauren Rabe’s chart should tell you what it means. It doesn’t actually say anything about how good that school is at the job of educating young people. What it refers to is how much money the school has, how expensive it is, and how effective it is at avoiding students from non-white backgrounds. So what should you do when you hear people use that word? You should get angry.

I honestly don’t understand why there isn’t more controversy over this. By way of contrast, there has been for some time heated discussion in New Haven over whether Yale University should change its name. Yale was named after Elihu Yale, who 300 years ago gave his friend Cotton Mather 417 books, a portrait of King George I, and nine bales of goods worth about 800 pounds, in order to bail out a struggling school that Mather was affiliated with. As a thank you, Mather called the school Yale. And because Elihu Yale turns out to have been a not-very-nice person who was involved in the slave trade, some people think that Yale University should be called something else. Do I agree? Sure, why not. In the first season of Revisionist History, I wholeheartedly endorsed the movement at Princeton University to rename their Woodrow Wilson School of Public Policy, because Wilson was a big-time segregationist. (For efficiency’s sake, I suggested they pick another Wilson, like Owen or Flip or Rita.) Change the name, if you want to! But why on earth am I supposed to be more upset over a 300-year-old transaction involving a bunch of books and 800 pounds worth of dry goods than the fact that right now Yale — number 4 on the national universities list! — perpetuates its privileged position by continuing to participate in a dubious exercise like the U.S. News Rankings list? How come the morally engaged undergraduates of Yale College aren’t more upset about this? Shouldn’t they be more angry about this than the fact that Elihu Yale was a jerk, like a lot of other rich white men were in 1718? Yes, I think they should.

A few other points. Am I making a mountain out of a molehill here, with my obsession with rankings? I don’t think so. I think the closer you look at the ways higher education has changed in this country over the past generation or so, the more troubling the influence of the rankings is.

For example, what happens when hundreds of universities across the country — all with different student bodies, different professors, and different strengths — start trying to prove their merit based on the same 3 or 4 criteria?

A listener, Jonathan Garrett, sent me the following email:

I think he’s right. The University of Chicago was weird, in a wonderful way. I know lots of Chicago graduates from that era, and they talk about exactly the same idiosyncratic atmosphere that Jonathan does. And the fact that some colleges are allowed to be weird, in that way, is incredibly important. There is no single educational culture that suits all students. Chicago was a prize.

But then Jonathan goes on:

How are we better off with a University of Chicago that is exactly like every other elite college? We aren’t.

And here, again, I think Jonathan is absolutely right. The blame for Chicago’s transformation belongs to U.S. News: they have created a horse race among colleges, to see who can rank the highest. But the problem is that they don’t reward weirdness and idiosyncrasies. They only reward a single, narrow definition of what a good school looks like. They want schools with lots of money and the fanciest of programs and buildings. Is there a place in the U.S. News algorithm for the nerdy, bookish, weird culture of the old University of Chicago? No, there isn’t.

Let me give you another example of this. One of the truly weird things about American higher ed is how small American elite colleges are. (If you listen to Revisionist History, you’ll know I’ve talked about this before.)

So, I’m a Canadian. I come from a country with 38 million people. The five most prestigious colleges in Canada are probably the University of Toronto, McGill University in Montreal, the University of Western Ontario, Dalhousie University in Nova Scotia, and the University of British Columbia. Those five schools have, respectively, undergraduate student populations of 73,000, 27,500, 25,000, 15,700, and 55,000.

Now compare that to the United States, a country almost ten times the size of Canada. The top five schools in the U.S. News Best National Universities list are Princeton, Harvard, Columbia, MIT, and Yale. Those five schools have, all together, just 29,044 undergraduates. Around 6,000 each. The University of Toronto alone is bigger than all five top U.S. universities, combined and doubled. Why?

It’s not because they don’t have the money to build more dorms or hire more professors. Collectively they have $128 billion in the bank, just collecting interest. But for some strange reason, they aren’t using all that money to educate more students.

Why not? If you truly provide an excellent education, then shouldn’t you try and enroll as many students as possible?

I’m typing this newsletter on a desktop Mac, which many people believe is the best desktop computer out there. If you want to use the best desktop out there, Apple will sell you one. But Harvard? Harvard prides itself on how many prospective students it turns away. It’s not really interested in being good — in the sense of having the greatest possible impact on higher education. It’s interested in being exclusive. Which is why it is, for all intents and purposes, not a whole lot bigger than it was 100 years ago.

So why don’t American elite schools want to be bigger? In part because the U.S. News rankings encourage them to be small. I touched on this a little bit in part two of my podcast series on college rankings, “Project Dillard.” But it’s worth going over in more detail. I was trying to figure out what it would take to move Dillard up in the U.S. News rankings. The first thing they needed to do was to get richer: to have more money in the bank. That would help a lot, because many of the most important variables used by U.S. News to measure “excellence” are really just proxies for wealth. But the other obvious route to the top was for Dillard to get smaller.

This is a school whose student population comes primarily from low income communities. That’s the school’s mission — to reach those students and give them an excellent college education. But since growing up with fewer resources usually means you don’t get super-high test scores, Dillard isn’t getting any points from U.S. News for having an academically flawless incoming class. (The U.S. News rankings, of course, reward you for how gifted your incoming class is. Which is weird, right? It’s like rewarding a hospital for how healthy its patients are when they first enter the ER.)

So what’s the next best thing Dillard could do to up its rankings, aside from raising a ton of money? Send the bottom 75 percent of its student body home. Just be a really, really small school, which caters only to the incoming freshman who have really good scores.

I asked Kelly McConville, one of the statisticians from Reed College who helped me with the U.S. News analysis, to analyze that hypothetical: What would happen to Dillard’s ranking if it just cut its student body by 75 percent? Here’s what she said:

Yes, that’s the game they’re playing. In the name of academic excellence, let’s encourage our best schools to educate as few students as possible.

There’s a great adage that economists like to say: whatever you encourage, you get more of, and whatever you tax, you get less of. We have chosen in the United States, for whatever ridiculous reason, to live with a system of higher education in which U.S. News gets to write the rules.

And what do those rules say? That we’re going to encourage privilege and exclusivity. And we’re going to tax anything that differs from that model.

It’s time for the U.S. News reign to end.