The Codebreakers — Volume 12 — Why We Still Study Them — From Military Cryptanalysis to Modern Ciphers and NSA

12.1 About This Volume

The machines in this series are museum pieces. PURPLE survives as three stepping switches in a glass case; SIGABA was withdrawn from service decades ago; nobody enciphers anything with a rotor. Yet the people of the previous eleven volumes are still studied — in university courses on statistics and cryptography, in the training of intelligence analysts, in histories of computing and of women’s work, and in every inquiry into an intelligence failure. This closing volume asks why.

It gives six answers. The school’s statistical methods passed out of cryptology into the foundations of modern statistics, information theory and data science. Its design lessons are still the rules of sound cryptography. Its textbooks stayed in use for half a century and are still in print. Its archives, closed for decades, have only recently been opened, and are still being read. Its history is still being rewritten, as the women and the forgotten men are restored to it. And its greatest failure — or rather the failure of the system it served, at Pearl Harbor — is still the standard case for how intelligence should and should not be organised. The volume ends where the series began, with a school.

12.2 Statistics: From the Index of Coincidence to Information Theory

The deepest legacy of the school is a way of thinking: that the analyst’s question — is this ciphertext what I think it is? — is a question in statistical inference, to be answered by counting, comparing with expected values and estimating the chance of being wrong. Volume 2 traced that idea to Friedman’s Index of Coincidence (written 1920, printed 1922); Volume 6 works its methods in code. Its later history runs through three of the people in this series and beyond them.

12.2.1 Kullback and the measure of information

Solomon Kullback wrote the SIS’s statistics text, Statistical Methods in Cryptanalysis, in the 1930s (revised edition 1938), formalising Friedman’s coincidence counts as tests — the phi test, the chi test and the kappa values for plain and random text — with expected values and variances. After the war he left neither the agency nor the subject. In 1951 he and Richard Leibler, another mathematician who later worked for NSA, published “On Information and Sufficiency” in the Annals of Mathematical Statistics, defining a measure of how far one probability distribution diverges from another — the quantity now universally called the Kullback–Leibler divergence, or relative entropy. Kullback’s book Information Theory and Statistics (1959; Dover reprint 1968) built a whole approach to statistical inference on it.

The connection to codebreaking is direct. The analyst’s basic task, from the index of coincidence onwards, is to decide whether observed letter counts come from one distribution (plain text, or text enciphered with the right key) or another (random text, or the wrong key), and the Kullback–Leibler divergence is the natural measure of how easily the two can be told apart from a given amount of data. Today it appears in textbooks of statistics and machine learning, in the training of neural networks and in the analysis of language models, usually without any mention of where its authors learned their trade. Both Kullback (1999) and Leibler (2002) are in NSA’s Cryptologic Hall of Honor.

12.2.2 Bletchley Park and the weight of evidence

The Americans were not alone. At Bletchley Park, Alan Turing and his assistant I. J. Good developed, for the attack on the naval Enigma, a systematic way of adding up the evidence for a hypothesis in logarithmic units — Turing’s “ban” and “deciban” — and of deciding when enough had been accumulated. Good published the ideas after the war in Probability and the Weighing of Evidence (1950) and spent a long career establishing them in statistics; Turing’s own wartime paper on the applications of probability to cryptography was released to the British National Archives only in 2012. The American and British lines met during the war, when Kullback, Sinkov, Rosen and others went to Bletchley Park (Volume 3; the PURPLE unit’s Volume 6). The common point is that wartime cryptanalysis was one of the first places where statistical decision-making was done at industrial scale, against deadlines, with consequences.

12.2.3 Shannon

The third strand is Claude Shannon’s. His paper “Communication Theory of Secrecy Systems”, published in the Bell System Technical Journal in 1949 from a classified report of 1945, gave cryptography its first mathematical theory: it defined perfect secrecy, proved that the one-time pad achieves it, and measured how much ciphertext an analyst needs before a cipher’s key is, in principle, determined. Shannon worked on wartime cryptography at Bell Laboratories, not at the SIS, but his paper made rigorous what the school had practised: that a cipher’s security is a statistical property, and that redundancy in language is the analyst’s raw material. The phrase usually attached to his paper — that one should assume “the enemy knows the system” — is Kerckhoffs’s principle in a mathematician’s words.

12.2.4 Data science

There is a modern echo that would have amused the Friedmans. Much of what is now called data science — classifying documents by their statistics, detecting which language a text is in, finding duplicates and near-duplicates in large collections, deciding whether a pattern is real or chance — is recognisably the work of the SIS’s clerks and analysts, done by machine. The index of coincidence itself is still taught, often as the first example of a test statistic with a known distribution, in courses on cryptography and on statistics. Volume 11 traced how the agency’s need to do this work faster drove the early computer industry.

12.3 Design: The Rules They Wrote Down

The second legacy is a set of design rules. They were not invented by the school — Kerckhoffs had stated the most important of them in 1883 — but the school tested them against real machines and wrote them into its texts.

The security is in the key, not the machine. Friedman’s argument to the Patent Office in 1934, that “the inherent weakness of all such devices” lay in having “the keying mechanism … a part of the device itself” (Volume 9), is the principle that every published modern cipher rests on: assume the design is public, and make the secret something small, changeable and independent of the mechanism.

Never trust regularity. Every machine the school broke — Hebern’s, RED, PURPLE, the Enigma in British hands — gave way at its regularities: odometer stepping, fixed motion orders, stereotyped message forms. SIGABA was designed to have none in its stepping, and it held. The general lesson — that any predictable structure in a cipher’s operation is an opening — is the lesson of modern cryptanalysis too, from differential and linear cryptanalysis to attacks on random-number generators. Modern analyses of SIGABA itself, such as Mark Stamp and Wing On Chan’s 2007 study of its full key space, are described in the SIGABA unit and in Volume 10.

Key management is part of the cipher. Rowlett’s year of making key tape (Volume 9) is a parable for every system whose mathematics is sound and whose keys are handled badly. The school’s successes against others were overwhelmingly due to key-handling errors — PURPLE’s keying instructions sent in PURPLE itself, reused indicators, paraphrased messages sent in two systems (Volume 5; the PURPLE unit’s Volume 5) — and its own best machine was designed around making key handling simple.

Attack your own systems. Both services handed every proposed system to their cryptanalysts before adopting it, and both kept testing their systems in service — the Navy through its fleet exercises (Volume 4), the Army through the Second Signal Service Battalion’s monitoring of American radio traffic for poor practice. That habit is now an industry, under names like penetration testing and red-teaming.

Usability is security. The Navy rejected plugboards for SIGABA because plugging errors compromised keys; Rowlett rejected tape because tired operators would reuse it. A cipher that is used wrongly is broken by its users.

12.4 Texts: Still in Print

The third legacy is literal: the books. The school’s texts had an extraordinarily long working life.

Friedman’s Military Cryptanalysis (Parts I–IV, 1938–1941) taught the wartime Army — Callimahos remembered them as “our Old and New Testaments combined” at the Fort Monmouth school (Volume 3). After the war Friedman and Callimahos expanded them into Military Cryptanalytics, written in 1956 and 1959 and used at NSA for decades; Callimahos taught the agency’s advanced course, CA-400, thirty-two times to more than 270 students before his death in 1977 (The Friedman Legacy; Library copy). NSA released Parts I and II to the National Archives in 1984 as SRH-273 and SRH-274; a third part was declassified in part in December 2020. Both series, and Friedman’s Riverbank pamphlets, have been reprinted commercially and are still used by students of classical cryptanalysis.

Abraham Sinkov’s Elementary Cryptanalysis: A Mathematical Approach (1966), written after his retirement to teach university students, is still used as an introduction. Kullback’s statistics text and Friedman’s 1963 lectures — published by NSA in 1992 as The Friedman Legacy — are standard reading in the history of the subject.

The institution that grew out of Friedman’s classes also survives. The National Cryptologic School, of which Rowlett became commandant in 1965, is NSA’s training college. The line of teaching that runs through this series — Hitt’s manual, the Riverbank classes, Friedman’s vault, Driscoll’s officer trainees, the Arlington Hall courses, CA-400 — continues there.

12.5 Archives: Opened Late

The fourth legacy is an archive, and part of why the school is still studied is simply that its records have only recently become available.

The people in this series worked under secrecy rules that prevented them from publishing, from correcting public accounts, and sometimes from keeping their own papers. Friedman, who had collected cryptologic literature all his life, had a difficult relationship with NSA after his retirement in 1955; in 1958 agency employees removed papers from his house in Washington, and some material at the Marshall Foundation was later withdrawn and reclassified. After his death in 1969, Elizebeth spent years cataloguing what was left — the “most extensive private collection of cryptologic material in the world”, in the NSA’s description — and it went to the George C. Marshall Foundation in Lexington, Virginia.

The official record came out in stages. From the late 1970s the National Archives opened NSA’s historical files as Record Group 457, the “SRH” series that includes Safford’s histories used in this series. The Congressional Pearl Harbor documents had been public since 1946. And in April 2015 NSA released the William F. Friedman Collection of Official Papers — more than 7,600 documents and 52,000 pages — online, while the Marshall Foundation opened an exhibition on Friedman’s life. Many of the documents quoted in this series — Friedman’s 1943 credit memorandum, Safford’s history of the Mark II ECM, the Friedman–Rowlett patent assignment — were declassified in 2013–2014 for that release. The NSA’s Center for Cryptologic History continues to publish monographs on the period; two of the most important for this series, on Driscoll and on Ferner, appeared in 2015 and 2023.

12.6 History: Recovering the Missing

The fifth reason the school is studied is that its history is still being corrected, and the corrections have become a case study in their own right in how organisations record — and lose — credit.

The first public accounts were built around one man. Friedman was the only member of the PURPLE team whose name was known, and when the break became public through the Pearl Harbor hearings, it was his. Ronald Clark’s 1977 biography was titled The Man Who Broke Purple. Rowlett’s recollections, published in 1998, restored the team and named Genevieve Grotjan; the NSA historian Jennifer Wilcox’s booklet Sharing the Burden (1998) opened its account of women in cryptology with her.

Since then the recovery has accelerated:

• Elizebeth Friedman, overshadowed for half a century by her husband, was the subject of Jason Fagone’s The Woman Who Smashed Codes (2017) and of a PBS American Experience documentary, The Codebreaker, first broadcast in January 2021. In 2020 the Coast Guard announced that a national security cutter would bear her name.
• The women of Arlington Hall and the Naval Communications Annex — thousands of them — were the subject of Liza Mundy’s Code Girls (2017), built on interviews with surviving veterans who had kept their oath of silence for seventy years. In 2022 the U.S. Postal Service issued a stamp honouring the women cryptologists of the Second World War.
• Agnes Meyer Driscoll was the subject of Kevin Johnson’s NSA study The Neglected Giant (2015), whose title states its argument.
• Robert Ferner, co-director of the PURPLE attack, was inducted into the Hall of Honor in 2022 and restored in Brenda McIntire’s “Give to Ferner” (2023).
• Mary Jo Dunning, Mary Louise Prather and Delia Taylor Sinkov, whose contributions are recorded in Friedman’s own memoranda (Volume 5), still have no published biographies.

The pattern is instructive for historians of science and technology generally. Credit in team work drifts towards the senior person and the person who writes the account; secrecy freezes that drift for decades, because the people who could correct it are forbidden to speak; and the original documents, when they are finally released, are often more generous than the published histories. Friedman’s 1940 report deliberately spread the PURPLE credit across a team; his 1943 memo named the women who kept PURPLE readable; Safford’s wartime history gave Rowlett “full credit” for the stepping maze. The record was there. It was the telling that was lost.

12.7 Organisation: What Pearl Harbor Taught

The sixth reason is the most sober. The school’s greatest achievement before the war, the reading of PURPLE, did not prevent the attack on Pearl Harbor, and the reasons why are still taught as the classic case in how intelligence fails.

The PURPLE unit’s Volume 7 sets out the record: PURPLE never carried Japan’s military plans, and no decrypt named Pearl Harbor as a target. But the investigations that followed found faults in how the intelligence that did exist was handled, and almost all of them were organisational:

• Fragmentation. The Army and Navy divided Japanese diplomatic traffic by day of the month and alternated delivery by month (Volume 4). Nobody in Washington was responsible for reading the whole flow and saying what it meant.
• Too few linguists. For most of the 1930s John Hurt was the SIS’s only full-time Japanese translator (Volume 3). Translation, not cryptanalysis, was the bottleneck in 1941 — and later in the war, as Kullback said, the Americans read “more than the translators could go through”.
• Distribution without analysis. MAGIC decrypts went, raw and selected by a duty officer, to ten readers who had to hand them back. There was no one whose job was assessment, and nothing went to the commanders in Hawaii.
• Rivalry. The Army and Navy competed over ownership of the product and withheld their military successes from each other (Volume 4).

The historian Roberta Wohlstetter’s Pearl Harbor: Warning and Decision (1962) turned these findings into a general argument — that intelligence failures come less from lack of information than from the difficulty of separating “signals” from “noise” in an organisation not built to do it — and it remains a standard text in the study of intelligence. The practical response was institutional, and the people in this series carried it out: a single Army codebreaking organisation at Arlington Hall and a single Navy one at Nebraska Avenue during the war; the Armed Forces Security Agency in 1949; and the National Security Agency in 1952, merging the two schools that had shared a pedestrian bridge on the Mall. Rowlett, Sinkov, Kullback, Driscoll, Wenger, Dyer and Rosen all served in it.

12.8 Where to Find Them

The codebreakers of this series can still be visited, in person and on paper.

• The National Cryptologic Museum, beside NSA’s headquarters at Fort Meade, Maryland, holds the surviving fragment of the Japanese PURPLE machine, the improved American PURPLE analog of 1944, a SIGABA, the Hall of Honor with the portraits of most of the people in this series, and a bronze bust of William Friedman labelled “Dean of American Cryptology”. Visiting arrangements change; check the museum’s own pages before travelling.
• Arlington National Cemetery holds the graves of William and Elizebeth Friedman and of Agnes Meyer Driscoll.
• The George C. Marshall Foundation in Lexington, Virginia, holds the Friedmans’ own collection of books and papers, and has published the Riverbank Publications online.
• NSA’s website hosts the William F. Friedman Collection of Official Papers, the Center for Cryptologic History’s monographs (including most of the sources of this series) and the agency’s oral histories.
• The U.S. National Archives, Record Group 457, holds the SRH histories and the technical records of the Signal Intelligence Service and OP-20-G.

12.9 A School, Still

Every volume of this series has come back to the same observation: the codebreakers who broke PURPLE and built SIGABA were a school before they were anything else. They were selected for aptitude rather than experience; trained for years before they were given real work; taught from written texts and graded problems that their teachers had written; organised in teams in which records were kept so that anyone’s work could be checked by anyone else; and expected to apply what they learned against the enemy to the protection of their own side. When the war came, that school could be scaled from a few dozen people to tens of thousands in three years, because its knowledge had been written down and its methods could be taught.

That is the deepest reason they are still studied. The specific machines are obsolete; the specific methods have been superseded by computers. But the idea that a hard intellectual craft can be turned into a teachable science, and that a small group which writes down everything it learns can found an institution that outlives every one of its members, is as relevant as it was in a dusty vault in the Munitions Building in June 1930.

The NSA’s own memorial is the building it named in 2002 for two people who met at a millionaire’s estate in Illinois in 1916: the William and Elizebeth Friedman Building. Their headstone at Arlington says it more briefly. Knowledge is power — and, the school would have added, knowledge is only power if it is taught.

---

Back to the beginning — Volume 1: The School on Constitution Avenue. The photograph in front of the vault, the cast of the series, the two services, and the three ideas — cryptanalysis as science, the duty to teach, the duty to protect one’s own — from which everything in these twelve volumes grew.