Nate Silver: Big data’s biggest figure

FT

August 2, 2013 2:02 pm

By Andrew Edgecliffe-Johnson

Nate Silver has turned number-crunching into a glamour profession. Andrew Edgecliffe-Johnson talks to him

Nate Silver was down on Anthony Weiner’s chances long before the selfie-snapping former congressman’s campaign to become New York’s mayor had to contend with the publication of a second wave of X-rated messages and priapic self-portraits.

“I think his favourables were low enough that he had a cap on his support from the get-go,” says the 35-year-old data-blogger as he perches, straight-backed, on the edge of a black leather couch in his Manhattan loft. Huma Abedin, Weiner’s wife and an aide to Hillary Clinton, would have won in a landslide, he adds. The unfortunately named candidate has been a gift to the city’s tabloids but a metropolitan mayoral race feels a little small for Silver: the man who predicted how 49 of America’s 50 states would vote in 2008 and then swept the board in 2012.

The one-time economics student and KPMG consultant looks every inch the nervy nerd in glasses, brown suit trousers and pale blue shirt, but the age of “big data” has made numeracy hip. Statisticians have become stars, from the authors of Freakonomics to Billy Beane, who applied data to baseball at the Oakland A’s and ended up being portrayed by Brad Pitt in Moneyball.

Nearly a year after publishing The Signal and the Noise, his bestseller on how human foibles make most of us poor predictors of anything from card games to climate change, Silver finds people staring at him in airports, “not because your zipper’s undone, but because they saw you on TV”. 

Characteristically, he has reduced fame to an equation: “You can almost do a mathematical function, where it’s based on when were you last on TV and how high-profile is the outlet. The half-life is actually pretty short,” he laughs.

He is still invited to retell the story of how he consistently foresaw Barack Obama’s victory over Mitt Romney even as professional pundits weakly dubbed the race too close to call. Yet the biennial rhythms of US general elections and midterms have left Silver with itchy feet in the lull of an odd-numbered year. “It’s hard to tell the same story 30 or 40 times,” he says. “I want to have movement and vitality and challenge myself a little bit.”

Silver correctly called the winner in 49 of 50 states in the 2008 US presidential elections – the prediction that skyrocketed him to fame. Four years later he managed 50 out of 50 states.

This summer has provided that opportunity. In 2010, he licensed FiveThirtyEight.com – his political data blog named after the number of votes in the US electoral college – to The New York Times for three years. As the clock ticked on the deal, he was courted by other news organisations, Wall Street firms, sports teams and Hollywood studios. The NYT wheeled out editors and executives to persuade him to stay but in July he defied expectations by selling FiveThirtyEight to ESPN. Walt Disney’s deep-pocketed sports network “was kind of a 9.5 out of 10 or a 9.8 out of 10 or a 10 out of 10,” Silver explained on a conference call announcing his decision, with only slightly less precision than his readers have come to expect. The deal, “web-focused” at first but likely to include on-air appearances for coverage of major sporting events and Disney’s ABC News broadcasts, gives Silver a chance to mix politics with his original passion for applying overlooked statistics to forecast baseball players’ performances.

He is still a Detroit Tigers fan, able to enjoy games over a beer with friends rather than analysing them against his model, but he admits to having become “disengaged” from baseball as electoral calendars clashed with the October World Series season climax.

More than a return to sport, though, the former full-time poker player now plans to create a much more ambitious web property. ESPN, the wealthiest network on US television, has the budget for him to hire a few dozen journalists, editors and analysts who can use data to shed new light on anything from tennis and football to the economy, the weather, teacher performance, college choices, local restaurants, the Oscars or even foreign policy.

Silver tackled some of these subjects in his book, which he stresses had only one chapter on politics. The one-time high-school debate champion, who makes his points with focused confidence, says it is time to branch out because he has proved his point in politics “so there are diminishing returns in terms of intellectual interest for me”.

. . .

Despite his mild-mannered reporter look, Silver has become a punchy voice in debates over the future of a profession shaken by disrupted business models and new competitors. Some political journalists may dismiss him as a mere blogger whose work could not be compared with theirs and Silver’s sources may be numbers rather than nominees, but he says what he does is “almost certainly journalism”. Bald men and combs come to mind when shrinking newsrooms argue over who can define themselves as a journalist, and press-pack conformity is not for Silver, who told Out magazine last year that being gay had encouraged his independent-mindedness. (He described himself as “sexually gay but ethnically straight”.)

Silver predicted four out of the five main category winners at the 2011 Oscars, including Best Actress (Natalie Portman). This year, three out of four of his predictions were correct.

“I’m not in the business to make friends,” he says coolly, explaining that he thinks reporters overrate the value of “quote-unquote inside information” from their political sources. “You’re being spun by someone. These people are professional liars, basically,” he says, sounding affronted on the truth’s behalf. He is equally aggravated by Washington pundits who routinely get their predictions wrong. Recalling one Twitter feud with a journalist he called out for his inaccurate projection on the eve of the 2012 vote, he says: “If you don’t have that instinct for accountability, I don’t know if you can really think of yourself as a journalist, or at least not a good journalist.”

Reporting both sides’ spin does not make you objective, Silver adds, but “I think you have a lot of people in DC who are very detached from reality”. He sees that most clearly in the media’s tendency to focus on outlying polls and anecdotal pointers. It is one thing to note the number of Romney signs in front of houses in a swing state, he argues, but quite another to take them as evidence of the race being closer than most polls indicate.

People are usually too sure of themselves when they make predictions, his book notes, but general elections are a rare exception. A desire to be even-handed may help explain why political reporters seem to preserve the fiction of a close race until the end. “It’s very easy to say it’s too close to call. You won’t get yourself in trouble for saying that.”

Silver has caused plenty of trouble by stating the opposite. Republican supporters accused him of bias, with Dean Chambers of Unskewed Polls calling him “effeminate”. (That prompted Gawker founder Nick Denton, a friend, to publish a picture of Silver with his ex-boyfriend the day after November’s election, saying “I hope it drives the homophobes crazy.”) Silver is dismissive of such attacks, saying: “There are some people who think that if you have a model that says Obama has an 80 per cent chance of winning, it’s a partisan forecast.”

Politico, the Washington news outlet, has harrumphed loudest, calling Silver overrated. “Some of his stuff goes on and on, trying to use numbers to prove stuff that I don’t think can be proved by numbers alone,” its executive editor said in June. There was similar pushback in parts of The New York Times. David Brooks, one of its columnists, argued that when polls move from offering snapshots of the current mood to projecting, “they’re getting into silly land”. Margaret Sullivan, the NYT public editor, said last month that some colleagues had found him “disruptive”. Silver says that claims of tension in a newsroom he calls “a team of rivals” were “a little overblown”.

“It’s not our intention – and when I say ‘our’ I mean me and my counsel – it’s not our intention to rehash the past,” he says carefully. He is frustrated by inaccurate accounts of his exit from the NYT, but won’t set the record straight: “There’s this dilemma where you’re being professional by being careful and confidential and other people are being less careful and confidential,” he says.

Capitol Hill

Silver predicted the winner in 35 out of 35 races for the US Senate in 2008. Four years later, he got 31 out of 33.

The public editor acknowledged Silver’s “unmatched” ability to bring traffic to the NYT’s website, particularly from elusive young readers. At the election season’s height, when traffic to FiveThirtyEight peaked at 10 times normal levels, one in five NYT digital readers was visiting the blog. Silver notes the jump in digital subscriptions the NYT enjoyed that quarter, saying pointedly: “I can’t say that was all FiveThirtyEight but I think it brought a lot of value to the paper.” Post-election, his non-politics posts attracted more traffic than his political pieces, he adds.

He won’t say what ESPN paid for FiveThirtyEight. He could perhaps have made more by taking his algorithmic aptitude to a hedge fund, but the option did not appeal: “It’s not like I’m struggling by any means but I would find it much more satisfying to make a good to very good income, where you’re building out the brand in the public eye, than to go and work for a hedge fund.”

. . .

His 12th-storey apartment, close to Penn Station, is calm and sparsely decorated but not hedge-fund showy. The self-confessed design snob has broken up the white walls with a few well-picked pieces of contemporary art. The windows offer a view of the spires of a Catholic church, a few rooftop wooden water towers and Madison Square Garden, home of the New York Knicks.

But he is more interested in data than the view. As a child in Lansing, Michigan – the son of a political scientist father who would analyse stadium capacity figures for fun and an activist mother with a PhD in French history – Silver could multiply two-digit numbers in his head: taking 48 and 54 and coming up with 2,592.

An algorithm brought him his first fame. His Pecota model, which stands for “Player Empirical Comparison and Optimisation Test Algorithm”, compared baseball players with other similar individuals in major-league history. Silver’s model looked past the commonly watched stats, such as batting average, and assigned greater weight to less-quoted ones, such as how often a batter gets on base, which correlated more to teams winning games.

Similarly, FiveThirtyEight’s model weighs up factors, from pollsters’ past accuracy to the religious and economic make-up of each state, then simulates the election 10,000 times to provide a probabilistic assessment of likely outcomes, based on polls going back to 1952. “We know that we’re going to get some of them wrong,” Silver cautions. The probability of any election victory is almost never 100 per cent. “You have a 70-30 bet, you’re supposed to get that wrong 30 per cent of the time.”

He is frustrated by people who prefer simple blue-or-red forecasts to such numerate nuance. “In baseball, it’s reached a healthy equilibrium where numbers-driven analysis is used in an appropriate way.” But politics is still far behind, he says: “I feel like we’re still fighting the Moneyball wars of 2003 and it might take another 10 years, if at all.”

The problem with political commentary is that it favours ideologues. CNN, stuck between Fox News on the right and MSNBC on the left, is struggling to revive its ratings because “the energy in politics is on the extremes,” Silver observes.

If most people struggle to interpret simple polls, will companies fare any better with the “big data” they are excitedly crunching? “I don’t think we’re on the edge of a singularity in terms of people becoming much more productive,” he says. In many cases, when he explores a new field and discovers how people misread the data about it, “it just kind of becomes depressing”.

Silver says he does not get on well with political reporters but is friends with media entrepreneurs such as Gawker’s Denton and Andrew Sullivan, the prominent blogger. His generation shares that entrepreneurial ambition, he says. “It used to be that you would idolise the guy who graduated at the top of his class from Harvard, and now you idolise the guy who drops out of Harvard to run a business,” he smiles. “I think these newspapers have a lot in common with Ivy League universities.” There is the prestige and the bright people “but there’s lots of internal politics. There are pockets of amazing things that are happening, but also pudgy bureaucratic cultures in other respects.”

Silver is not exactly dropping out; ESPN is a corporate giant with the resources to commercialise FiveThirtyEight more than ever. “I’m still pretty hungry,” he says, explaining that moving to ESPN was a decision to work “really, really hard for four more years” instead of coasting. “You could take a more relaxed route and kind of write now and then and travel a lot. That’s great, but I can do that in my fifties or sixties or seventies. Right now I want to build something while I’m still young.”

-------------------------------------------

Andrew Edgecliffe-Johnson is the FT’s media editor

Nate Silver will be at the Edinburgh International Book Festival on August 13 to talk about ‘The Signal and the Noise: The Art and Science of Prediction’ (Penguin). www.edbookfest.co.uk

Share this post