What Declassified Government Documents Reveal About America’s Dark Secrets

Columbia history professor Matthew Connelly ’90CC, codirector of the Institute for Social and Economic Research and Policy, discusses the research behind his explosive new book, The Declassification Engine, and the shocking things he learned along the way.

You are the principal investigator at the History Lab, a project that, as its Web page proclaims, approaches “history as data science.” You and a team of social scientists, statisticians, and computer scientists have created a searchable database of nearly five million declassified government records — the largest of its kind in the world. How did the History Lab, and particularly its chief project, the “declassification engine,” come about?

As a historian I’ve been working with declassified government documents for thirty years now, ever since grad school. And it’s striking how much information, even as far back as the 1950s and ’60s, remains classified. Even the documents that have been released to the public are full of redactions. But one fairly new development is that a lot of declassified documents now come to us in digital form. And some of those collections, like diplomatic cables from the State Department (which in the 1970s was an early adopter of electronic data systems), are rich with metadata — everything from the classification level to handling instructions. There are subject tags indicating if the document is about international trade or covert operations or some other aspect of government. When you have that much data, and it is what we call machine-readable, it’s a bonanza for data scientists. You can do all kinds of cutting-edge research. So I and my colleagues in Columbia’s computer-science and statistics departments started looking at what subjects and which individuals tend to be most redacted. Who’s in the room when the topic under discussion is still a secret seventy years later? You find, for example, different versions of the same document with different redactions. You can learn a tremendous amount by aggregating this information, bringing it together in one place. You simply can’t do that kind of research with traditional archival methods.

In your new book, you posit the existence of what you call the “dark state,” as distinct from the “deep state” that we all heard about during the Trump administration. Can you define these concepts? Do they intersect?

The deep state is to some extent a conspiracy theory — the idea that there are all these people in government working in secret, doing things that even the president doesn’t know about and can’t stop. I use the term “dark state” because a lot of the hidden history I study is dark, very dark; some of the secret activities pursued by government officials in the past are depraved, like subjecting children and the elderly and servicemen to various experiments and then lying about it again and again. But I don’t consider this a deep state, because the secrecy goes all the way to the top. Ultimately, it’s the president who is sovereign over secrecy, and it’s the president who sets the rules. And if presidents really wanted to do something about this explosion of government secrecy, they have the authority to change the rules about what gets classified and how. But in fact, presidents — including Donald Trump — are delighted that they have this power to decide what the rest of the world is allowed to know. They get to define what national-security information is.

You point out that nearly every president comes into office professing high ideals and vowing transparency, and then before long, each gets swept up in the culture of secrecy. Is this just the nature of modern statecraft? Why is transparency so hard?

I would argue that it is not just hard but impossible, because transparency means taking on thousands of people within government who also like the power they have to work in secret. Right now there are around three thousand people in the federal government, many of them political appointees, who have what is called original classification authority. These are the people who can determine what programs are classified. So if a new covert operation or capability is launched by the National Security Agency, say, these people with original classification authority can decide from the outset that everything related to this program will be secret or top secret. And they can also define the project as a “compartment” for sensitive, “compartmentalized” information. For someone to access compartmentalized information, it is not enough to have a security clearance at the sufficient level of sensitivity. That person must also be given authorization to see the information that lies within those compartments, and such authorization is granted on a need-to-know basis. There is a federal agency, the Information Security Oversight Office, that is supposed to exercise oversight over the classifying of government information, but it’s tiny, underfunded, and effectively powerless.

Do you see any way to reverse this trend toward more and more secrecy?

Right now, all the incentives are to classify as much as possible. To reverse this trend, the American public would need to start insisting that Congress and the courts do their jobs. It is within their power to rein in this excessive and unaccountable power that presidents have to define what the rest of us are allowed to know. Unfortunately, both Congress and the courts have largely abdicated their constitutional responsibilities in this regard. What the government cares about is reflected in its budgeting. More than $18 billion a year is allocated to protect national-security information. Out of that $18 billion, about half of 1 percent goes to costs related to declassifying information or deciding what the public should know. Whenever public officials, including the president, discuss national security and the need to protect information — and this need is often legitimate, to be sure — they invariably say they “strike a balance” between protecting national security and honoring their duty to ensure that the American public knows what the government is doing in their name. But I’m sorry: $18 billion for secrecy and less than half of 1 percent for transparency? There is no balance.

Secrecy in government might seem as American as apple pie, but you argue the opposite — that for much of its peacetime history the US embodied a kind of “radical transparency” envisioned by its founders. Can you elaborate on that? And when and why did that transparency morph into opacity?

For more than a century and a half after its founding, our government was remarkably transparent. The era of secrecy began during World War II, though secrecy in wartime is not unusual. In every American war up to that point, the government created intelligence agencies and new ways of keeping sensitive information secret. But those agencies and procedures were always dismantled as soon as the war was over. When World War II ended in 1945, though, all those wartime measures were kept in place. Only by maintaining this massive security apparatus, the argument went, could the government keep its citizens safe. But when you analyze what information tends to get classified, and what takes the longest time to be revealed, you can’t help but conclude that we paid a price for all that secrecy. Far from keeping us safe, the secret activities of government officials, the incredible risks they took, put us all in danger. Moreover, all that secrecy has made it increasingly difficult, if not impossible, even to reconstruct the history of what really happened.

The dark state does seem to kick in with FDR, who, you note, was the first president to impose a classification system for government information and who established the Office of Strategic Services, which evolved into the CIA. Your research also led you to some unsettling revelations about his administration and the secrecy around the “surprise” attack on Pearl Harbor.

I thought I knew the story of Pearl Harbor. But in recent years, historians have gone back through this documentary record, working in archives not just in the US but also in Japan. And when we start to assemble all the pieces, what’s crystal clear is that though Roosevelt claimed his administration was caught by surprise, he and his closest advisers had known for weeks that an attack was coming; indeed, they had provoked it. They just didn’t know where the attack would be. Why would they do this? Because Roosevelt had been trying to get America into World War II for months. Much has been written about his strategy in the Atlantic Ocean, where American vessels were chasing German submarines in the hope that an incident would tip into war with Germany. But nothing was working. Finally, Roosevelt came to believe that the only way to get Congress to go along with a war was to get Japan to attack. He wasn’t wrong: the US did have a vital interest in the outcome of the war, and I for one believe we should have engaged, in 1941 if not sooner. But large majorities of Americans opposed what they thought of as another war of choice.

Where I don’t go, and the conspiracy theories do, is believing that Roosevelt knew the attack targeted Pearl Harbor. There’s no evidence of that, and it makes no sense. Why would he allow US forces to be caught unawares, at the cost of 2,400 American lives? All he needed was an attack; he didn’t need a massacre.

Pearl Harbor is the original secret of the dark state. The way FDR maneuvered the country into war set the stage for the many, many secrets that followed.

One of the book’s running narratives, and probably its most dumbfounding, concerns the information you have unearthed about the bellicosity and power of the US military in the postwar era, particularly during the most volatile years of the Cold War — and about the chronic power struggle between the president and the Pentagon in those years, with the latter usually on top.

Oh, yes, the majority of Americans at that time, when surveyed, said they expected nuclear war in their lifetime and typically believed it would be a war of aggression begun by the Soviets. But when I researched back to the late 1940s, but especially the 1950s and early ’60s, I discovered that it was the American military, not the Soviets, who were intent on starting a war. They believed they had the upper hand, nukes-wise, and that the sooner war started, the better. They knew that, in time, the Soviets would build up their own arsenal, and then it would be too late for the US to destroy the Soviet Union and escape unscathed.

Your book discloses that the military’s “strategy,” which they seemingly were free to pursue unilaterally without consulting the executive branch, was to annihilate the entire communist world with nukes in what RAND Corporation strategist Herman Kahn dubbed a “wargasm,” thereby killing not only half a billion people in communist nations but another 100 million in neighboring countries allied with the US. Thoughts?

In terms of the Pentagon’s ability to act unilaterally, we’re seeing more evidence that US military officials believed in the months and years leading up to the Cuban Missile Crisis of 1962 that they had “pre-delegated authority” to start a nuclear war if they thought we were under attack. This began in the Eisenhower administration, because Eisenhower thought this was the only way to reassure both the military and American allies that the US could fight back if the Soviets attacked — and that we would not be caught unprepared again, this time with a nuclear Pearl Harbor. Only the ability of the American forces to go nuclear immediately could fend off that danger. So the administration issued these pre-delegation orders. And when JFK became president and was (rightly) alarmed by this authority, the military denied that they had any such orders. This is a case in which military brass clearly lied to protect their autonomous power to start a nuclear war.

The Cuban Missile Crisis seems to have shifted the power dynamic and set the current template for military operations being run mostly out of the White House. You suggest that the distrust between the president and the Pentagon probably led to the disaster in Vietnam but that, paradoxically, the disaster may have saved the US from a host of other military misadventures in places like Indonesia, Panama, and Morocco. How so?

This distrust between the White House and the senior military leadership grew, flourished, and peaked by the early 1960s. What strikes historians who are looking back on this period is how rare it is to find senior leaders, either in the White House or the military, who were optimistic that the US would triumph in Vietnam. But when I say that if it hadn’t been for Vietnam, we may well have had other wars, it’s because when we review many of the original redactions in declassified documents, we find a lot of war plans for the US to send forces into many different countries. Now, I don’t know if any of those wars would have materialized, but certainly the Vietnam War had a disciplining effect; afterward, Americans were more dubious about these sorts of military actions. But it’s true that foreign wars are still run largely out of the Oval Office, because Congress is unwilling to exercise its constitutional responsibility. They’ve allowed presidents to start one war after another. Even today. The legal authorization the US president draws on to conduct drone strikes and counter-insurgency campaigns in dozens of countries around the world was passed after the attacks on 9/11. That was more than twenty years ago. Congress’s unwillingness to challenge the president’s power to keep information from the public is just a symptom of a much larger abdication.

This article appears in the Winter 2022-23 print edition of Columbia Magazine with the title "Uncovering America's Dark Secrets."