Simple Ciphers are Safe from AI
Is Silicon Valley the New Bletchley? Not so Far.
There exists uncertainty regarding whether a very large pattern recognition engine like ChatGPT or other “AI” systems can attain, or even approach, human-like cognitive abilities. Various attempts to resolve this uncertainty have been made, most using tests designed to assess human capabilities in the cognitive realm. One class of test seems to have been overlooked, and that is cryptanalysis, the art and science of breaking ciphers.
Cryptanalysis requires the exercise of logical thought, intuition, the ability to make the “golden guess1,” and an internal world model to guide those individual skills toward success. So I thought it would be worthwhile to see how the “big three” AI systems (ChatGPT, Gemini and Claude) would do when presented with the rock-bottom basic cipher of monalphabetic substitution with word boundaries. Any competent human puzzler should be able to crack one of these in minutes, and if you do a lot of them you can often sight-read the plaintext. In order for the test to be truly valid I used the simplest possible prompt, so that there would not be an information transfer from a human doing step-by-step coaching. I picked a cryptogram from a source I personally use to keep my mind active. It is a bit on the difficult end of the elementary spectrum and contains a somewhat lame pun. (Spoiler in the footnote2, but I encourage you to take a crack at it).
First up was ChatGPT:
This response fits Pauli’s category of “Not even wrong.” The most basic characteristics of this most basic cipher is that the spaces in a correct solution must line up with the spaces in the ciphertext. So as best as I can tell, ChatGPT is emulating a doctor with a flashlight.
Next I visited Gemini:
Thanks for the tip about paper and pencil, but that wasn’t the question. Let’s try again:
Again, not even wrong, but neatly accompanied by “corroborative detail, intended to give artistic verisimilitude to an otherwise bald and unconvincing narrative.3”
Finally, off to Claude:
You know, you really need to keep an eye on these things.
Thus endeth my little adventure. A couple of billion ALGC (Artificial Little Grey Cells) were exercised to no effect whatever. I apologize for the waste of electricity and cooling water.
From Tennyson, “The Golden Guess/Is Morning-Star to the full round of Truth.” Epigraph to Friedman and Callimahos, Military Cryptanalytics Part I
IF ONE WANTS TO GET OUT AND STAY OUT OF DEBT HE SHOULD ACT HIS WAGE
W.S. Gilbert, The Mikado.
Not to excuse the (un)performance of the LLMs here but the cryptogram provided is, strictly speaking, wrong.
"ACT" in the cypher text cannot be substituted by "act", since we already need cyphertext "T" to be replaced by plaintext "s". The other two substitutions (cyphertext "A" for plaintext "a", cyphertext "C" for plaintext "c") are also dubious as the typical monoalphabetic substitution cypher puzzle can typically be assumed to substitute one to one, and we already have cyphertext "M" for plaintext "a".
Boo, sir. Boo.