Data's dividends

  • Comments (0)
  • Email
  • ShareThis
  • Print
  • Text Size A A A

Illustration by Camille Chisholm c/o

They are questions so big, so incredibly complex, that until now scholars had not even attempted to answer them: What historical event of the past 250 years most profoundly shaped US political philosophy? Is democracy good for economic growth? Can we predict outbreaks of gang violence? At Columbia’s Data Institute, teams of sociologists, economists, social workers, and data scientists, using computational methods of their own design, have tackled these questions. What they have discovered may overturn long-held academic dogmas, and even save lives.

A New Take on the Great War

No historical record may capture America’s changing political consciousness better than the president’s State of the Union address. These documents, issued annually since 1790 with little variation in form, have been studied extensively by scholars. But this year, a team of researchers led by Columbia sociologist Peter Bearman, the director of the Interdisciplinary Center for Innovative Theories and Empirics, gained a valuable new perspective on the archive. Berman, along with sociology PhD student Alix Rule '13GSAS and physicist Jean-Philippe Cointet of the University of Paris, developed an innovative text-analysis program that examined all 1.8 million words in the 227 speeches, identifying the specific nouns that US presidents have used when discussing topics like foreign policy, trade, taxes, employment, immigration, and military budgets. The researchers then looked to see when new ways of framing these issues gained influence, as evidenced by a president’s lexicon being adopted by his successors.

No matter how the researchers sliced the data, 1917 stood out as the point at which the most pronounced long-term changes in political discourse occurred. Woodrow Wilson’s speech to Congress in December of that year, delivered eight months after the US declared war on Germany, was more focused than previous State of the Union addresses on promoting an interventionist foreign policy, industry regulation, and the financing of large-scale public programs — ideas that US historians today consider core tenets of American political thought. Wilson’s shift away from the more isolationist and laissez-faire policies of his predecessors was clearly precipitated by World War I; he spoke at length about the need for Americans to aid their European allies and to grant the federal government the authority to regulate agriculture prices and control the nation’s railroads and waterways in support of the war effort. Bearman’s and his colleagues’ insight is to show that large numbers of words and concepts from Wilson’s speech crop up in the addresses of subsequent US presidents. (The words “democracy,” “united,” and “peace,” become mainstays of foreign-policy discussions beginning with Wilson, for example.)

“We know what constitutes modern political thinking, but until now have been unable to say exactly when it originated,” says Bearman.

The new study is important because historians have long debated whether World War I left an indelible mark on American political philosophy; some scholars have argued that it wasn’t until the aftermath of the Second World War, when the US proposed its Marshall Plan to rebuild Europe, that Americans became believers in big government and nation-building. In fact, Bearman and his colleagues’ analysis shows that the 1950 State of the Union address, in which Harry Truman trumpeted the success of the Marshall Plan and warned of the impending threat of Communism, was the second most influential in US history.

“When you look at historical texts as one long, continuous river of words and ideas, you’re able to spot patterns that you wouldn’t see otherwise,” Bearman says. “We believe that our methods could be useful in analyzing other large historical corpuses, whether the Atlantic slave-trade records or the Proceedings of the Old Bailey.

  • Email
  • ShareThis
  • Print
  • Recommend (53)
Log in with your UNI to post a comment

The best stories wherever you go on the Columbia Magazine App

Maybe next time