Lorenz and Colossus — Volume 14 — A Colossus on the Bench — Running Tutte's Statistic

14.1 About This Volume

Volumes 12 and 13 built the German half of the story, an SZ42 that enciphers. This volume builds the British half in miniature: a machine that takes a Tunny cipher text and the day’s chi wheel patterns, and finds where the chi wheels started — the job Colossus was built for.

It is not a replica of Colossus. The real machine filled a room, had some 1,600 valves in the Mark 1 and 2,400 in the Mark 2, and read a loop of paper tape at 5,000 characters a second (Copeland, Rutherford Journal 3, 2010); Volumes 8 and 9 describe it and Volume 11 the full-size rebuild at Bletchley Park. The bench machine keeps the parts of Colossus that carry its idea — a cipher-text tape read optically and clocked by its own sprocket holes; wheel patterns generated electronically rather than read from a second tape; counters; a threshold, the “set total”; a printed list of the settings that beat it — and runs them on a microcontroller, with an FPGA option that is closer to Colossus’s own architecture.

The volume ends with a demonstration. The count is run on a real SZ42 message, one of the three sent from Paderborn in the 2007 Cipher Event that marked the completion of the Colossus rebuild, and recovers all five chi start positions from nothing but the cipher tape and the wheel patterns. It is then run on a long message enciphered on the replica of Volumes 12–13.

As in the other build volumes, nothing here has been built yet. The design is worked out, and every number in the demonstration was computed by the program that the firmware ports, 03-outputs/tools/colossus_count.py, whose self-test is described below.

14.2 The Statistic

Volume 9 explains Tutte’s 1+2 break-in in full; the bench machine needs only its arithmetic. Bletchley’s delta of a stream is each character added to the next: ΔZ = Z + Z one later. The cipher text is Z = P + χ + ψ′ (Volume 3). Differencing, and adding the first two impulses together, gives

ΔZ1 + ΔZ2 + Δχ1 + Δχ2 = (ΔP1 + ΔP2) + (Δψ′1 + Δψ′2).

The left side can be computed for any trial start of χ1 and χ2. The right side does not depend on the chi wheels at all, and it is a dot more often than a cross: Δψ′1 + Δψ′2 is a dot whenever the psis stood still, and ΔP1 + ΔP2 is a dot more often than not in German teleprinter text, with its doubled letters and its habit of sending shift characters in pairs. Tutte reckoned on “about 70%” dot from the psi part and about 60% from the plain text, giving about 55% dot on the right (Copeland 2010). At the right start of χ1 and χ2 the left side is that right side, and so shows about 55% dots; at any wrong start it is a mixture with a 50% stream and shows about 50%.

With χ1 of 41 positions and χ2 of 31, there are 41 × 31 = 1,271 possible starts. For each one, the machine counts the positions in the message where the left side is a dot. If the message has N deltas, a wrong start’s count is close to N/2 with a standard deviation of √N / 2, and the score of a count is how many standard deviations it stands above N/2 — Bletchley’s sigma. Colossus printed only the starts whose count passed a threshold, set on decade switches as the set total.

The remaining chi wheels are set the same way, one at a time, with χ1 and χ2 now known. The wartime Newmanry used a family of runs for this stage (General Report on Tunny, 1945; Volume 9). The bench machine uses a simple pair: for each start of χk (k = 3, 4, 5) it adds the counts of ΔZ1 + ΔZk + Δχ1 + Δχk = dot and ΔZ2 + ΔZk + Δχ2 + Δχk = dot. The principle — a count that peaks at the right start — is the same.

14.3 The Brief

The bench Colossus must:

1. Read a cipher-text tape of five-hole teleprinter tape — or, by default, take the cipher text over USB as Bletchley letters.
2. Hold the chi patterns of the day, entered as a key file in the format the replica uses (Volume 12).
3. Run the 1+2 count over all 1,271 χ1/χ2 starts, and the follow-on counts for χ3, χ4 and χ5.
4. Report the counts that pass a set total, with their sigma, and the best start for each wheel.
5. Agree exactly with colossus_count.py, count for count, on every setting — so that any discrepancy is a fault in the machine, not a matter of interpretation.

14.4 Architecture

The machine has four parts, and each corresponds to a part of Colossus.

Table 1 — Architecture

Colossus	Bench machine
”Bedstead” tape transport, message tape glued into a loop	Cipher text held in RAM and read round and round
Photoelectric reader, clock from the sprocket holes	Optical reader with a sprocket channel that latches each character
Thyratron rings generating the wheel patterns	Chi patterns from the key file, expanded into Δχ streams in memory
Counters, set total on decade switches, electric typewriter	Software or FPGA counters, set total in the menu, printout over USB

14.4.1 The tape reader

Five-hole teleprinter tape is 11/16 inch (17.5 mm) wide, with characters at a pitch of 0.1 inch (2.54 mm), five code tracks across it and a row of smaller feed holes that the punch’s sprocket drives — on standard five-hole tape, between the second and third code tracks. A hole is a cross.

The reader has six channels, one for each code track and one for the feed holes. Each channel is an infrared LED above the tape and a phototransistor below it, in a 3D-printed block that guides the tape over a flat bed; the phototransistor’s output goes to one comparator of an LM339 quad, with a trimmer to set the threshold, and on to a Pico GPIO pin. The feed-hole channel is the clock: its edge latches the five code channels. That is exactly how Colossus solved the problem of synchronising fast electronics with a mechanically driven tape — it “generat[ed] a clock signal from reading its sprocket holes” (Wikipedia, “Colossus computer”, citing Flowers 1983) — and it means the bench reader needs no motor speed control at all. The tape can be pulled through by hand or by a slow motor; each character is read once, into memory.

There the bench machine departs from Colossus deliberately. Colossus had no store for the message: its tape ran round as a loop, and each run was another trip round the loop. During development its reader was tested at 9,700 characters a second “before the tape disintegrated”, and 5,000 a second was settled on for regular use (Wikipedia, citing Flowers). The bench machine reads the tape once and then loops over a copy in RAM, millions of characters a second, without wearing anything out.

A vintage alternative is a mechanical five-hole tape reader of the period, such as the Creed 6S series, which senses holes with pins and sends each character as a start-stop signal on a current loop. Such a reader can be connected through the loop interface of Volume 13 instead of the optical head.

Tape needs punching. A teleprinter with a perforator attached to the replica of Volume 13 will punch the replica’s cipher output directly, which is the most satisfying route and closest to the wartime one, where intercepted signals were punched onto tape at the intercept station. Without a punch the machine is used over USB, which is the default.

14.4.2 The counting engine — microcontroller

The default engine is a Raspberry Pi Pico 2, the same board as the replica. The method is the one colossus_count.py uses, and it suits a 32-bit processor well:

1. Compute ΔZ1 + ΔZ2 for the whole message and pack it into bits, 32 to a word. A 6,000-character message is 188 words.
2. For each of the 41 starts of χ1, compute the Δχ1 stream of the same length and pack it the same way; likewise the 31 starts of χ2. That is 72 streams, about 54 KB — a tenth of the RP2350’s 520 KB of RAM.
3. For each of the 1,271 pairs, XOR the three bit streams word by word and count the ones. The count of crosses subtracted from N gives the count of dots.

Each setting costs a few thousand processor cycles, so the whole 1+2 run takes a few tens of milliseconds at 150 MHz; the χ3–χ5 runs are smaller still. For comparison, a Colossus that tested one setting per trip round a 6,000-character loop at 5,000 characters a second would have needed about 1.2 seconds per setting and some 25 minutes for the 1,271; the Mark 2’s five parallel processors cut that by five (Wikipedia, “Colossus computer”; Volume 9 describes how runs were actually organised). The speed is not the point of the bench machine, but the comparison is part of what it teaches.

14.4.3 The counting engine — FPGA option

The FPGA option rebuilds the computation in Colossus’s own form. Colossus held each wheel pattern in a ring of thyratrons, with one stage “on” at a time, stepped by the tape clock; its counters were electronic; its Mark 2 ran five settings at once in parallel processors fed from a short shift register (Wikipedia; Volume 8). An FPGA does all of that naturally:

• Rings. Each chi wheel is a circular shift register of its own length (41, 31, 29, 26 and 23 bits), loaded with the cam pattern and rotated one place per character.
• Delta. A one-character delay and an XOR give Δχ and ΔZ, as Colossus’s own shift register did.
• Parallel processors. Instead of five, the FPGA can run all 31 starts of χ2 at once against each start of χ1 — 31 copies of the Δχ2 ring, each offset by one, feeding 31 counters. Then the whole 1+2 run is 41 trips round the message.
• Counters. Thirty-one 14-bit counters, enough for messages up to 16,000 characters.

The Sipeed Tang Nano 9K, built round a Gowin GW1NR-9 with 8,640 LUT4 cells, 6,480 registers and 468 kbit of block RAM (Sipeed wiki), has room for this many times over. The message sits in block RAM, written by the Pico over a simple serial link; the FPGA streams it at its own clock rate and hands back 1,271 counts. Whether it runs at 27 MHz or at 5,000 characters a second — the FPGA can be slowed to Colossus’s own rate for display — its counts must equal the microcontroller’s exactly.

14.4.4 Output and the set total

Colossus printed its results on an electric typewriter, and only the settings whose count passed the set total (Wikipedia). The bench machine prints the same thing over USB — and optionally to a teleprinter through the loop interface, which is closer in spirit. Its set total is entered as a count or as a sigma; a sensible default for the 1+2 run is 3σ, which on 1,271 settings lets through a handful of chance results with the true one among them.

14.5 Firmware Outline

1. Load the key: chi patterns from a key file (the motor and psi lines are ignored).
2. Load the message: from the reader, or as Bletchley letters over USB, or as a Cipher Event tape file in Schüth’s format.
3. Prepare: compute ΔZ for all five impulses; build the packed Δχ streams.
4. Run 1+2: count all 1,271; sort; print those above the set total, with sigma.
5. Run χ3, χ4, χ5: with χ1 and χ2 at the best 1+2 start, run the paired counts for each; print the best three starts of each with sigma.
6. De-chi: with all five chi starts, print the first line of D = Z + χ. This is what the Newmanry handed to the Testery (Volume 6): plain text plus extended psi, in which the psis’ repeats can be seen.
7. Dump: on request, write all 1,271 counts as text for comparison with the simulator.

14.6 Verification

The bench machine is correct if, and only if, it gives the same counts as colossus_count.py, and colossus_count.py is checked against published material. Its self-test (python3 colossus_count.py --selftest) does three things:

1. Real cipher text. The three messages of the TNMOC Cipher Event of 15–16 November 2007 were enciphered on the Lorenz SZ42 with the SZ42A limitation and transmitted from the Heinz Nixdorf MuseumsForum in Paderborn. Joachim Schüth received them, broke them, and published his demodulated cipher tapes and recovered keys (schlaupelz.de). Given only each cipher tape and the chi patterns, the runs must recover the five chi start positions of Schüth’s published keys. They do, for all three messages.
2. Tutte’s proportions. At the true settings, the proportions of dots in ΔP1+ΔP2, Δψ′1+Δψ′2 and their sum must lie near Tutte’s 60%, 70% and 55%. On the three 2007 messages they are 64.6–67.5% (plain text), 64.3–64.4% (psi) and 53.3–55.9% (sum). The psi figure is lower than Tutte’s, as it should be under the SZ42A limitation, which releases the psis more often than the SZ40 motor did (Volume 3).
3. A simulated message of 4,000 characters, enciphered on the Bream wheels of 14 April 1944, must give back its five chi starts.

The machine is then checked against the program:

• Count for count. Dump the 1,271 counts of the 1+2 run on the 1600 UTC tape and compare with the simulator’s. Every count must agree exactly. The top five are given below.
• All three tapes, all five wheels. The recovered starts must be 1, 7, 8, 4, 5 (1200 UTC); 36, 27, 2, 1, 23 (1600 UTC); and 38, 4, 3, 26, 1 (1700 UTC).
• The reader. Punch the 1600 UTC tape on a perforator from the replica (Volume 13), read it back optically, and compare the characters read with the file. Any difference is a reader fault; the counts will show it as a fall in the true setting’s sigma.

What is not validated: no complete wartime Colossus print-out with its matching tape is available in the sources used, so the bench machine reproduces Tutte’s statistic, not any particular wartime run; and its χ3–χ5 runs are a simplification of the Newmanry’s procedure.

14.7 The Demonstration

14.7.1 A real SZ42 message

The 1600 UTC message is 5,880 characters long, so the run has N = 5,879 deltas. At a wrong setting a count should be near 2,940 with a standard deviation of 38.3. The five highest counts were:

Table 2 — The 1600 UTC message is 5,880 characters long, so the run has N = 5,879 deltas. At a wrong setting a count should be near 2,940 with a standard deviation of 38.3. The five highest counts were

χ1	χ2	Count	Sigma
36	27	3,286	9.0
36	19	3,110	4.4
36	4	3,087	3.8
26	27	3,068	3.4
35	31	3,062	3.2

The true setting, χ1 = 36 and χ2 = 27, stood at 9.0 sigma — 55.9% dots — with the next best at 4.4. Several of the runners-up share one wheel with it (χ1 = 36, or χ2 = 27), which is expected: a start with one wheel right is half right, and carries some of the signal.

With χ1 and χ2 set, the follow-on runs found χ3 at 2 (4.4σ, next best 2.6σ), χ4 at 1 (6.8σ, next 3.0σ) and χ5 at 23 (10.7σ, next 4.9σ). All five match Schüth’s published key. The other two 2007 messages behave the same way: 1200 UTC gave χ1 = 1, χ2 = 7 at 5.0σ (next best 2.9σ), with χ3–χ5 at 8, 4, 5; 1700 UTC gave χ1 = 38, χ2 = 4 at 8.1σ (next 4.5σ), with χ3–χ5 at 3, 26, 1. The 1200 UTC message is the instructive one. Its true setting stood at only 5.0σ, on a message of the same length as the others, because its plain text and motor happened to give a weaker bias. A set total of 3σ would have printed it with a handful of others; a set total of 6σ would have missed it. That is the judgement the Newmanry’s cryptographers made every day.

In the 2007 event itself, the rebuilt Colossus team, who chose to receive the signals on wartime radio equipment, were delayed by a day by poor reception, and Schüth won the race; his own program, running on a laptop, took less than a minute (Wikipedia, “Colossus computer”, citing BBC News, 16 November 2007). The bench machine sits between: it uses a 1940s method on 2020s hardware.

14.7.2 A message from the replica

The replica’s test message in Volume 13 is only 63 characters long — far too short for any statistic. For the demonstration the replica enciphers a long message: 4,000 characters of real German teleprinter plain text (a stretch of the deciphered 1600 UTC message), on the Bream wheels of 14 April 1944 with the SZ42A limitation, from start positions chosen at random — χ 21, 31, 5, 13, 21; μ 4, 5; ψ 35, 7, 24, 38, 4. The plain text begins E++988VERMITTLUNG++988DER++988AUSSTELLUNGSINHALTE, the doubled shift characters typical of German operators; the cipher text begins EHQAFEPQU4+YKVV3ZE8YSB9QOBN9UTXUP/VMRI4RC38Z3YACF.

Given the cipher text and the Bream chi patterns, the bench machine’s 1+2 run puts χ1 = 21, χ2 = 31 on top with a count of 2,159 out of 3,999 — 54.0%, 5.0σ — against a next best of 2,105 at 3.3σ. The follow-on runs give χ3 = 5 (3.0σ, only just ahead of the next at 2.5σ), χ4 = 13 (8.2σ) and χ5 = 21 (6.9σ). All five are right. The χ3 result is the warning: a margin of half a sigma is not one to trust without a confirming run, and a Newmanry cryptographer would have run another.

Removing the chi gives the de-chi, which begins MIZCYAZMQEE34LF3VNN4BBODDOSZEVQWZ+CO/MD4D8ADDA8X/Y34YL/SSPSM. It is still unreadable — it is plain text plus extended psi — but it has footholds. Of its six doubled letters in the first sixty, three (NN, BB and SS) are places where the psis stood still while the operator sent a doubled shift sign, ++ or 88; the other three (EE and both DDs) are coincidences. Telling the two apart is the hand work Volume 6 describes. In this message the psis stood still on 22.3% of letters.

14.7.3 How long must a message be?

The signal grows with the length of the message and the noise with its square root, so short messages are hard. Twenty messages were simulated at each length, each from a different stretch of plain text and different random start positions, and counted:

Table 3 — The signal grows with the length of the message and the noise with its square root, so short messages are hard. Twenty messages were simulated at each length, each from a different stretch of plain text and different random start positions, and counted

Length	1+2 puts true χ1, χ2 on top	All five chi starts right
1,000	3 of 20	2 of 20
1,500	11	6
2,000	14	10
2,500	18	11
3,000	19	18
4,000	20	20
5,000	20	20
6,000	20	20

On these wheels and this plain text, a message needs about 3,000 characters for the method to be dependable, and 4,000 to be safe. That is a property of this simulation, not a wartime rule; but it shows why long messages were the ones Colossus could set, and why the Germans’ limitations, which weakened the psi part of the bias, made the Newmanry’s life harder.

14.8 Parts

Table 4 — Parts

Qty	Part	Purpose	Approx. cost (Oct 2026)
1	Raspberry Pi Pico 2	counting engine, USB	$5 (Raspberry Pi)
6	940 nm IR LEDs, 6 phototransistors	reader channels	$5–10
1	LM339 quad comparator (×2 for six channels), 6 trimmers	reader	$3–6
1	3D-printed reader block and tape guide	reader	material only
1	Sipeed Tang Nano 9K (optional)	FPGA engine	about $15–25 (typical retail; check when ordering)
—	Five-hole teleprinter tape	tapes	varies
—	Loop interface from Volume 13 (optional)	teleprinter printout or mechanical reader	about $10

Prices other than the Pico 2’s are typical catalogue figures, not read from a live listing.

14.9 What the Bench Colossus Teaches

The 1+2 run is the cleanest piece of reasoning in the whole Tunny story. Nothing in it is clever once it has been seen: two impulses added, a difference taken, a count. What made it hard was everything around it — the eighteen months it took to see it, the scale of counting it needed, the electronics Flowers had to invent to count at 5,000 characters a second, and the human judgement at the end about which bulge on the print-out was real. The bench machine does the counting in milliseconds and leaves the judgement exactly where it was: with a person looking at a list of sigmas and deciding whether 5.0 is enough.

The series began, in Volume 1, with a machine nobody was told about. It ends with both halves on one bench: a Lorenz that enciphers and a Colossus that finds its chi wheels, each checked against the other and against the surviving record — Tiltman’s worked example of 1961 and the three messages that flew from Paderborn to Bletchley in 2007.

---

Back to the beginning — Volume 1: Tunny and Colossus — The Codebreaking Machine Nobody Was Told About. The orientation volume: the machine, the break, the computers, and the thirty years of silence.