Poly-alphabetic substitution ciphers 1. Each character can be represented by more than one symbol 2. Each ciphertext symbol can represent more than one plaintext character Simple example: Shift back and forth between two (or more) substitution alphabets Proposed as early as 1460 by Florentine Leon Battista Alberti, idea not really developed for another 100+ years The Vigenere Cipher * Proposed by Blaise de Vigenere, Treatise around 1586 * Not used much for another 200 years * Believed for a time to be unbreakable * Cracked by Charles Babbage in about 1854, but not publicized. Crimean War secrecy? * Independently by Friedrich Wilhelm Kasiski in 1863 Basic Idea * Write a key string (e.g. word or phrase) repeatedly underneath the plaintext, and encrypt by adding (mod 26) these key characters to corresponding plaintext characters. Text : twas the night before christmas and all through the house + Key: dash erd asher dasher dasherdas her das herdash erd asher ------------------------------------------------------------------ WWSZ XYH NSNLK EEXVVV FHJPWKPAK... E(x_i) = (x_i + k_(i % n) ) % 26 where k_1, k_2, ..., k_m is the key. D(x_i) = (x_i - k_(i % n) ) % 26 * Can be done by hand using a "Vigenere square" - essentially a mod 26 addition table * The same letter may be encrypted differently depending on its position * The same ciphertext character means different things depending on position * Note that arbitrarily long key phrases can be used (and if longer than the message, we have a one-time pad) * Overall effect is that groups of letters having indices equal to j % m are encrypted with different Caesar shifts (rotation by k_j). * Multiple encryption with keys of length m and n is equivalent to Vigenere encryption with a string of length lcm(m,n) * Chosen plaintext attack is trivial (feed it aaaaaaa ...) to yield key * Known plaintext is almost as trivial - subtract plaintext from encrypted text (mod 26) to produce key. * "Beaufort" Cipher. Variation in which E(x_i) = (k_i - x_i) % 26 Decryption is same as encryption. Ciphertext only attack * considerably more difficult. * Main exploitable weaknesses are periodicity, and the fact that the periodic ciphers are weak Caesar shifts. * First order of business is to find out the key length. Kasiski attack on key length. (Also Babbage) * Note that trigrams (e.g. "the") that are a multiple of the key-length apart will be encrypted identically. (So will sequences of other lengths) 1. Tabulate all identical trigrams, (or groups of 4) and the distances between them. * If any of these trigrams matches are due to the same plaintext trigram, the distance between them must be a multiple of the keylength m. 2. (Garret) For each trigram occurring more than once, find the gcd of the separation distances. We speculate that these may be a multiple of m. * Of course, some of these may also be accidents. * This gcd operation is not really needed - it provides a "sharpening" effect that can sometimes reduce the amount of work. 3. Look for recurring factors in the different distances or their gcds from step 2. If any factor jumps out as occurring in most of the gcds, guess that it is the key length. * Note that just taking the gcd of ALL the lengths is not likely to work, as there is probably an accidental duplication somewhere. * Also, we need to be careful as the length of the message increases, as we get more accidents, and even valid sets of trigrams may have an accident thrown in, destroying the "sharpening" effect of the gcd operation. Use groups of 4, or leave out the gcd step, or restrict it to smaller groups. Why does this work? * Attack would not work if all trigrams were equally probable in English, as most trigram matches would then be due to accidents, and no accumulation of evidence for any particular factor would take place. (Argument that SOME of the matches are still good, so why can't we use them fails for the same reason that a random guess at text has 1/26 characters correct, but still gives us no information.) * We can use this to get a ball-park estimate of the longest keys we could easily detect by comparing the likelihood of the most frequent trigrams (1%-2%) to 1/(26^3) = .0057%, a factor of a few hundred. So key lengths of less than 100 or so, should really jump out with trigrams, given enough text. * Of course, for long key lengths, the table size and work goes up (approximately as n^2), so things do start to get tough... Finding the Key Once we know the length of the key, we can exploit the weakness of the Caesar shift to attempt to determine the key itself. 1. Break the ciphertext into groups of characters whose indices are the same mod m. e.g. for key length 5, one group is characters 0, 5, 10, 15... the next is 1, 6, 11, 16, ... etc. * We know that each of these groups is enciphered with a Caesar shift. * We don't have adjacency information to help us, but we do have the fact that there are only 26 possible shifts, and that for the correct one, the character frequencies should be consistent with those of plaintext English. * Rather than guessing each substitution independently, which is unreliable with small number of characters, we can use all the evidence at once by trying to match the peaks and valleys of the standard alphabet distribution with a shifted version of the statistics from the ciphertext. A B C D E F G ... T U V W X Y X 8.2 1.5 2.8 4.3 12.7 2.2 2.0 9.1 2.8 1.0 2.4 0.2 2.0 0.1 ^ peaks 4 apart ^ *----- Big long hole -----* * This "histogram matching" allows even fairly sparse statistics to drive a high-confidence match. * Matching process can be formalized using standard pattern recognition methods. Specifically, find the shift that either 1. Maximizes correlation = Sum_{i=1}^{n} x_i * y_i 2. Or minimizes sum of absolute differences (SAD) * If the cryptographer has foolishly used a phrase or word as a key, we may not even have to do this for all the groups - we can guess at the key after we get some of its pieces. The Friedman attack * Kasiski method of finding key length is a bit disorganized (choices for whether to use trigrams or groups of 4, whether to employ lcd sharpening etc.) * "Index of Coincidence" discussed by William Friedman, 1925 provides a more systematic statistical treatment that greatly reduces this ad-hoc component. * This index is defined for any two strings y and z of the same length N, and is effectively the proportion of characters that match. I(y,z) = (1/N) sum delta(y_i, z_i) where delta(c1, c2) is 1 if c1 = c2, and 0 otherwise * If both (or even one) of the strings is random with equal probability for all characters, then we would expect the IOC would have expected value = 1/26 = .0385 = 3.8% * For strings of English text, because of non-randomness, the IOC is significantly higher, and can be determined empirically to have expected value of about .067 = 6.7% * Note that this depends not only on single character frequencies, but bigrams, trigrams, and statistical correlations of all orders. * However, we can approximate the IOC to first order from the single character probabilities py_i and pz_i I_1st-order = sum py_i * pz_i And we can apply this to specific strings by using the observed frequencies as estimates of character probabilities. In fact, we can apply this average definition to a single string to obtain an index that could be taken as "typical" for "strings of this type" I_avg(y) = sum (y_i)^2 These can be interpreted as the inner product and norm in the 26-dimensional vector space of character frequencies. * I_avg for English prose is about .064 = 6.4%, which shows (when compared to the full IOC of .067) that a large amount of the structure is in the first-order character frequencies. * If a (long enough) string does not have I_avg close to .064, we might doubt it was English. ==> Interesting question. Is it possible for there to be a (large, e.g. exponential size) language with an IOC of LESS than the expected value for random strings over the alphabet? Observation 1: * If two strings are encrypted with the same substitution key, then the IOC is unchanged. This is true even for poly-alphabetic substitution as long as the encryption key depends only on position. Observation 2: * If two strings are encrypted by the same transposition cipher, the IOC is unchanged. * If a string is encrypted by ANY transposition (substitution) cipher, the first order IOC of the string with itself (Garret's I_avg) is unchanged. ==> We can use Observation 1 to look for the key length in a Vigenere encrypted piece of ciphertext by taking the IOC of the string with shifted versions of itself. * When the shift is a multiple of the key length, the strings are English encrypted with the same key, and so the IOC should be close to .067 * Otherwise the IOC should be closer to the average IOC for Vigenere encrypted English, which is about .047 for short keys or ones that are English phrases. (To get down to the .038 theoretical value, we need to use long, random keys) * Need to avoid small shifts (0, 1,2,3) since the local statistics of English obscure the structure we are after. If we suspect a key length of (say) 3 it will show up via high IOC at 6, 9, etc. * May still need to verify by checking multiples of proposed key length, as occasional anomalous values will occur. (However, structure is much sharper than tables of factors with Kasiski attack) Obtaining the key using the IOC * As before, break the encrypted text into subgroups or substrings corresponding to indices that are equivalent mod m where m is the key length. Recall that each of these groups is encrypted with a different Caesar shift cipher * Garret suggests taking pairs (i,j) of these groups, and looking for the difference (j - i) between their shifts by re-encrypting i with different caesar shifts, and then looking for high IOC against j (either using I_avg, or a physically shifted version to avoid issues with local correlation). Trial and error to find CONSISTENT sets of relative shifts, followed by final search to get global offset. * Checking the consistence can be cast as the problem of ensuring that for all cycles in the fully connected, labeled, directed graph corresponding to the shift differences between groups, the edge values sum to 0. If condition holds for all (2) and 3 cycles, then it holds for cycles of any order. Short induction proof. Basic idea is to show that the sum for an n+1 cycle is the same as the n-cycle where we have taken a shortcut across one corner. This follows from the 0 sum of 3-cycles. Algorithm for checking all consistencies. 1. Set up a table where we keep the possibilities for the the rotation of each group (relative to group 0) Call these possibilities the "hypothesis set" for the the group. 2. Initialize the table by filling in possibilities derived by comparing group 0 to all the other groups. (all the (0, i) edges. 3. Compute the shift possibilities between group 1 and groups k = 2 and higher. For each k, add each of these to each of the potential shifts for group 1 (mod 26) and intersect the resulting set with what is already in the hypothesis set. This brings in all the (1, i) edges that were not already there. 4. Repeat for j = 2 and k = 3 and higher, on up to j = m - 2 where m is the key length. This works because as we explore each new group, we guarantee the consistency of all cycles involving all edges that have been examined to this point. * Simpler approach (not mentioned by Garret) is to compare each group against average frequency vector (This is the same as our previous correlation approach). Since Garret carefully eliminates local structure, all the available information is in the average IOC anyway, and we avoid the noise on one side of the equation. This yields (hypotheses for) the key character directly (no final step). Local structure in emerging plaintext can be used to resolve ambiguities. A modification of the Vigenere cipher where a sequence of general substitutions rather than Caesar rotations is used for encryption is much more difficult to break, though the method for determining the key length still works. Of course, the key is correpondingly longer, as several substitution alphabets need to be provided. Another variation are so called "autokey" systems. Basic idea is to use initial key on first m characters of message, then start using offset version of the plaintext itself. One big exploitable weakness is that key is English prose. Reading Assignment: Garret, Chapter 5, Singh, Chapters 4, 5 Encryption assignment Encrypt two pieces of English prose of at least 1000 characters using a Vigenere cipher with a key between 10 and 20 characters in length. Output text in groups of 8, 8 groups to a line. Due Thursday, Sept 19 2002 Decryption assignment Decrypt the transposition cipher given to you in class You will probably need to write or get hold of some tools that allow you to propose a trial (partial) transposition in one group, and automatically see the result of that transposition in all the other groups. Due Tuesday, Sept 24 2002.