Implementing AES

I've been working for the last 10 days or so on cryptography.  Some time ago I came across the cryptopals website, but I quit after 10 challenges or so.  I stopped about the time that they asked me to implement AES (wikipedia).

In the last week or so I've written Python code to do both DES and AES.  I'm not going to post any of it here, though the repo is here if you're interested.  Also posted there are a number of write-ups about various aspects of the project.  It's still very much a work in progress.

What I really do think will be useful for anyone interested in this topic are links to some resources I found on the web.

Prof. Kak has lectures (here).  Lecture 8 describes implementation of AES and the ones before that describe the peculiar math of Galois fields that is central to AES.

When I was getting started with this problem I found the tables showing multiplication in the field by 2,3,9,11,13,and 14 at wikipedia here.  These copy and paste nicely.  Eventually I learned enough to be able to generate them myself from first principles.

I still cannot say exactly why multiplying a word (4 bytes) by this matrix

[[2, 3, 1, 1],
 [1, 2, 3, 1],
 [1, 1, 2, 3],
 [3, 1, 1, 2]]

can be inverted by multiplying by this matrix:

[[14, 11, 13,  9],
 [ 9, 14, 11, 13],
 [13,  9, 14, 11],
 [11, 13,  9, 14]]

but I am starting to understand how irreducible polynomials work.

A source for the S-boxes that is well-behaved for copying is the official U.S. document here, which also describes the algorithm in complete detail.

Another especially useful reference is a detailed description of what data to expect as AES runs.   They provide intermediate snapshots of the state for every step, including generation of the round keys.  That was a huge help in debugging my code.

My results match theirs, and the end result is provably correct:

> openssl aes-128-ecb -e -nopad -K "5468617473206d79204b756e67204675" -in msg.txt | xxd -p
29c3505f571420f6402299b31a02d73a
>

A last great resource is a discussion here about the process of multiplication.  I found the discussion of the field generator 0x03 tremendously enlightening.  The page includes a table of exponentials and logarithms with 0x03 as the base.  That table alone would allow you to do any multiplication you need, including finding multiplicative inverses, for which another table is provided as well.

I haven't really looked at the whole book yet, but it also promises to be great.

CommonCrypto5

I came across another introductory article about how to import CommonCrypto from Swift.

Recall what we did before following this post:

Method 1:

- obtain a bridging header by adding a dummy Objective-C file in Xcode
- in the header, do #import <CommonCrypto/CommonCrypto.h>

The library functions will be available from an Xcode Swift Cocoa app project.

Method 2:

Make CommonCrypto quack like a Framework by putting a file module.map with appropriate code, inside the directory that holds OS X SDK frameworks:


/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform\
/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks


With that single change:

- we no longer need the bridging header from an Xcode project
- we can use CommonCrypto in a framework
- we can use it in an Xcode Playground

- we can also do this:


test.swift:
import CommonCrypto
print(CC_SHA1_DIGEST_LENGTH)

> swift test.swift
20
>


The referenced article shows a different way to use CommonCrypto inside a framework (where the bridging header trick won't work).

Put the module.map

module.map:
module CommonCrypto [system] {
header "/usr/include/CommonCrypto/CommonCrypto.h"
export *
}


inside your project directory (not necessary to use Xcode). Following the instructions:

Now add the new module to Import Paths under Swift Compiler – Search Paths in your project settings. Use ${SRCROOT} in the module path (e.g. ${SRCROOT}/CommonCrypto) to insure that the project works no matter where it’s checked out.

Then you can just use import CommonCrypto.

Other notable items:

- use of NSData and NSMutableData types for the buffers
- size_t(key.length) and size_t(data.length), in calling the functions

CommonCrypto4

This is the last post about CommonCrypto. I just wanted to mention that I converted Encryptor into a framework. It is still on github.

I also wrotesome command line utilities in Swift that use it. That project is here.

You can checkout the README which explains usage.

CommonCrypto3

This is a brief note about a big project (at least for a one-day thing). I finished a Cocoa app that implements basic AES-mode encryption and decryption from the CommonCrypto library. It does not do any I/O.

For now, you can just watch it run in the debugger, using pre-formed messages. But it can handle messages that are larger than a single block by breaking the data into blocks and encrypting/decrypting each one.

One nice thing is the BinaryData class, which implements some functionality around a [UInt8] including the Indexable and CustomStringConvertible protocols, and a convenience initializer that takes a String representing binary data.



BinaryData is a class because it has a derived class Key, which implements key "stretching" from CommonCrypto.

At the very end, I ran into trouble because I did not really understand the CBC protocol.

When encrypting, the output ciphertext becomes the initialization vector for the next round. But when decrypting, the input (which is also the ciphertext) becomes the initialization vector for the next round.

Here is the output for a test:


pw: my secret
salt: 3356ec169bb6
msgText: a much longer and still really big secret
61206d756368206c6f6e67657220616e64207374696c6c207265616c6c792062696720736563726574
encryptMany
encrypt round: 1
encryptOneChunk
msgLen: 16
msg:
61206d756368206c6f6e67657220616e
keyLen: 16
iv:
39131435ae3d5bbf2e300ab5edddc8c9
status: 0
result:
55bd582296f708842e5d0833e3673a99

encrypt round: 2
encryptOneChunk
msgLen: 16
msg:
64207374696c6c207265616c6c792062
keyLen: 16
iv:
55bd582296f708842e5d0833e3673a99
status: 0
result:
ded208434eb0a4e753e6cdf1c41da54c

encrypt round: 3
encryptOneChunk
msgLen: 16
msg:
69672073656372657400000000000000
keyLen: 16
iv:
ded208434eb0a4e753e6cdf1c41da54c
status: 0
result:
66e7cb3ba1b51a0d657edad13de97b37

cipherData: 55bd582296f708842e5d0833e3673a99ded208434eb0a4e753e6cdf1c41da54c66e7cb3ba1b51a0d657edad13de97b37

decryptMany
decryptOneChunk
data:
55bd582296f708842e5d0833e3673a99
keyLen: 16
iv:
39131435ae3d5bbf2e300ab5edddc8c9
status: 0
result:
61206d756368206c6f6e67657220616e

decryptOneChunk
data:
ded208434eb0a4e753e6cdf1c41da54c
keyLen: 16
iv:
55bd582296f708842e5d0833e3673a99
status: 0
result:
64207374696c6c207265616c6c792062

decryptOneChunk
data:
66e7cb3ba1b51a0d657edad13de97b37
keyLen: 16
iv:
ded208434eb0a4e753e6cdf1c41da54c
status: 0
result:
69672073656372657400000000000000

decryptedData: 61206d756368206c6f6e67657220616e64207374696c6c207265616c6c79206269672073656372657400000000000000
a much longer and still really big secret

Whoa... there were some bytes in the cut and pasted text that did not appear in the debug console nor in the editor here on blogger,, but did appear in the Preview. They are the null bytes used to pad the plaintext. They do not appear in the final product. Curious...

The Xcode project is on github here.

It was a world of fun, but I think I am done with CommonCrypto for now.

CommonCrypto2

I've been exploring how to use CommonCrypto on OS X (docs). As described in a previous post, I found an article on the web which describes how to make the CommonCrypto library available to Swift playgrounds as well as Swift Cocoa applications.

I decided to look into it a bit more. I had help from Mike Ash, as well as another knowledgable guy. Looking at the header helped a lot as well. It is extremely detailed. Since the project was somewhat challenging, I thought I would sketch out what I learned and post the code.

CommonCrypto is a C library. So, for example, the function that does one-step encryption or decryption, CCCrypt, is declared like this in CommonCrypto.h



The first two arguments can be defined in Swift like this:

let operation = CCOperation(kCCEncrypt)
let algorithm = CCAlgorithm(kCCAlgorithmAES)

We're first going to encrypt, and the encryption scheme will be AES.

If we look in the header, it tells us that the block size for AES is 16 bytes or 128 bits.

kCCAlgorithmAES128 Advanced Encryption Standard, 128-bit block

Reading further

One option for block ciphers is padding, as defined in PKCS7; when padding is enabled, the total amount of data encrypted does not have to be an even multiple of the block size, and the actual length of plaintext is calculated during decryption.

Another option for block ciphers is Cipher Block Chaining, known as CBC mode. When using CBC mode, an Initialization Vector (IV) is provided along with the key when starting an encrypt or decrypt operation. If CBC mode is selected and no IV is provided, an IV of all zeroes will be used.

To begin with we'll use ECB with PKCS7 padding because it seems simpler.


let options = CCOptions(kCCOptionPKCS7Padding | kCCOptionECBMode)


CCCrypt takes three arguments of type const void * and one of type void *. The three const arguments are the key, the initialization vector iv, and the plaintext or data (dataIn). These can be provided as Swift arrays: [UInt8], or even (for the key and plaintext) as Swift Strings. No need for NSData or UnsafePointer or whatever.

The return type for this function is CCCryptorStatus, which is 0 for success, and something else for an error. The codes are also shown in the header:



So, when I received -4301 as the result, which at first I thought was garbage, CCCrypt was actually telling me "insufficient buffer provided for the specified operation."

Of special note: I found that although the message does not have to be 128 bits or a multiple, the key does. If it is not, the encrypt operation returns 0 for success and some encrypted data, but when decrypted we don't get our plaintext back!

A buffer is needed, into which the encrypted data will be written.

let bufferSize = 128
var cipherData = [UInt8](count: bufferSize, repeatedValue: 0)
var resultLen = 0
var status: Int32 = 0

Another thing that confused me was that these sizes (except for resultLen and status) are in bits, not bytes.

When the buffer is passed into CCCrypt we need a cast:

UnsafeMutablePointer<Void>(cipherData)

We provide the address of the Int variable &resultLen, and after the function returns, that value tells how much data was written.

Finally, the other argument is the initialization vector iv.



In our first pass at this, we specify ECB mode and PKCS7Padding, so no IV is needed, and we just pass nil for this argument.

It works. The first 9 bytes of the output at the end are the same as what we put in. I put the playground on github here.

The only thing I haven't figured out with this one is how to know the size of the message when decrypting.

The second approach uses CBC. It's the same as the first, except we change the options to the default

let options = CCOptions()

define an initialization vector, and then fill it with random bytes.

var iv = [UInt8](count: blockSize, repeatedValue: 0)
SecRandomCopyBytes(kSecRandomDefault, blockSize, &iv)


Other than that, the only thing is to be sure and pad the message to 16 bytes. It works. The playground is here.

UPDATE: I implemented the other code sketched out in Mike's article: encrypting in steps for a longer message (playground), and key stretching (playground).

CommonCrypto

So Apple has a cryptography library called CommonCrypto (docs). Importantly, it is not a framework.

Here is a quote from the older documentation (last year):

Common Crypto
In OS X v10.5 and later and iOS 5.0 and later, Common Crypto provides low-level C support for encryption and decryption. Common Crypto is not as straightforward as Security Transforms, but provides a wider range of features, including additional hashing schemes, cipher modes, and so on.

Normally, I would have passed on this in favor of, say, trying to build openssl and then access that. (I recall seeing something from tptacek about it along the lines of use something more standard, don't trust that Apple will implement the basics correctly). Judging by history, that would be pretty sage advice.

But ... I came across a wonderful blog post here, which describes not only how to use CommonCrypto from Swift, and not only shows an in-progress library wrapping CommonCrypto, but also describes a hack that allows Swift playgrounds to have access to CommonCrypto. Too cool for words!

The basics: open a new Xcode project in Swift and call it MyApp. We need a "bridging header", we get this by adding a new Objective C class, then Xcode will ask if we want this header, and we say yes. Ungratefully, we promptly delete the dummy Objective C class. In the header, add:

#import <CommonCrypto/CommonCrypto.h>


That's it. For example, I put this code into crypto.swift



and call it from the AppDelegate in applicationDidFinishLaunching. The debugger prints:

e4d909c290d0fb1ca068ffaddf22cbd0

If I put the same text in a file and do

> md5 msg.txt
MD5 (msg.txt) = e4d909c290d0fb1ca068ffaddf22cbd0
>


Of course, what we've done here is the usual C hack of allocating a buffer and passing a pointer to the buffer into the program that will write into it. Except that in Swift these are not your father's pointers. You can't do arithmetic. And they have that label "Unsafe". Reminds me of Maverick.

Now for the cool part. Deep within Xcode there are one (or more for some people) SDKs. Software Development Kits.

> xcrun --show-sdk-path --sdk macosx
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform\
/Developer/SDKs/MacOSX10.11.sdk

and deep within that is:

> ls /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform\
/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks
AGL.framework
AVFoundation.framework
AVKit.framework
Accelerate.framework
...

As I said, CommonCrypto is not a framework. But we can fake things like it is one. We make a directory in that place, call it CommonCrypto.framework and inside that put module.map with some paths:

module CommonCrypto [system] {
header "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform\
/Developer/SDKs/MacOSX10.11.sdk/usr/include/CommonCrypto/CommonCrypto.h"
header "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform\
/Developer/SDKs/MacOSX10.11.sdk/usr/include/CommonCrypto/CommonRandom.h"
export *
}

(Those line breaks should work, but I don't have them in my original).

I did this:

cd /Applications/Xcode.app/Contents/Developer\
/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk\
/System/Library/Frameworks
> sudo mkdir CommonCrypto.framework
Password:
> sudo cp ~/Desktop/module.map CommonCrypto.framework
>
> cat CommonCrypto.framework/module.map
module CommonCrypto [system] {
...
>


Having "corrupted" the SDK in this way, we can now do things like this: paste that same code into an Xcode playground. I had to change the last line, not sure why just at the moment..



And with this setup, we no longer need the bridging header. I tried just deleting it but Xcode knows. So delete the whole project and make a new one with the same name. Add the code in crypto.swift and put import CommonCrypto in that same file. Then from the AppDelegate, call doIt.

It works!



It's worth pointing out that simply substituting SHA256 for MD5 in the code above works. To check the digest on the command line do:

> openssl dgst -sha256 -hex msg.txt
SHA256(msg.txt)= ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c


Encoder

I'm fooling around with a new Swift Cocoa project. The Encoder app does encryption and decryption. I use the phrase "fooling around" deliberately because an important rule of cryptography is to use standard libraries. If you don't really know what you're doing, you certainly shouldn't "roll your own."

Here is a screenshot of the current version of the app.

At the heart of it, we are doing something trivial:

return Zip2Sequence(a1,a2).map { $0^$1 }

where a1 and a2 are binary data, i.e. arrays of integers with values between 0 and 255 (UInt8). The xor operator ^ is just what we want for both encoding and decoding.

The trick, of course, is to generate a keystream of pseudo-random numbers given a key, which is a String.

Foundation provides standard functions for obtaining random numbers:
rand, random, arc4random, arc4random_uniform.

arc4random and arc4random_uniform are preferred for really random PRNG, but lack the ability to be seeded (by the caller). Seeding is needed so that the same pseudo-random keystream sequence can be regenerated, given a relatively short key.

The rand function returns an Int32 (try rand() is Int32 in a Swift playground. So in theory it might return both positive and negative values. However, it only seems to return positive numbers, which is good, because, say -3 % 256 is equal to -3, which would cause trouble when trying to convert to UInt8. According to this (very old) file, the OS X rand uses unsigned values.

The maximum value returned is RAND_MAX, which is equal to 2147483647 = 2^31 - 1, which is also equal to Int32.max. What we want are integers in the interval [0,255], so I just do:

Int(rand()) % 256


I'm not an expert, of course, but I looked at the distribution of output of rand and it seemed sufficiently random for my purposes.

The next question is that of seeding the PRNG. You might do:

srand(UInt32(time(nil)))

The seed function srand takes a UInt32. The question is, how do we turn our key into a UInt32? What I've done so far is to use the String property hashValue / This gives an Int, which is 64 bits, and of course, may also be negative.

Here is my approach:

Having initialized everything, we just call

func next() -> UInt8 {
  return UInt8( Int(rand()) % 256 )
}

One issue I'm still working on is that of loading binary data. I know how to turn an [UInt8] into data:

let a: [UInt8] = Array(0..<4)

let data = NSData(bytes: a, length: 4)

and load data from a file with NSData(contentsOfFile:fn), but I haven't been able to figure out how to turn that back into an array of UInt8 in Swift.

I puzzled out a way around the problem, however. String will take NSData in an initializer:


Given that, I get the individual bytes from the string and use a Dictionary



As usual these days, the project is on github here. Thanks to samol for explaining how to do the gist thing.

UPDATE:

I found a way to do the data conversion mentioned above. It looks like it is probably the natural way to do this in Swift.



We use NSInputStream initialized with the NSData object. We allocate the necessary space:

var buffer = Array(count: n, repeatedValue: 0)

and then read the data into the buffer using a reference.

stream.read(&buffer, maxLength: n)

The last call returns a result, which is the number of bytes read. Alternatively, one could read byte by byte until stream.hasBytesAvailable returns false. The only thing that confused me for a while was that I had Array(0..<n) which gave Int values of 64 bits each. Without the map on line 4 the data looks like:

[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2,

which I interpret as 00000000, 10000000, 20000000 and so on. These are the decimal values for each byte, where the ordering is little-endian, with the low value byte having the first memory address.

Intel x86 processors store a two-byte integer with the least significant byte first, followed by the most significant byte. This is called little-endian byte ordering.

Apple docs. If I write the 64-bit data to a file and examine it with hexdump:

> hexdump x.bin
0000000 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00


gpg

This post is about the Gnu Privacy Guard (gpg). A link from the main page leads to a download of an installer for OS X here.

I followed the Quick start tutorial here, and sent an encrypted email between two of my accounts using Mail after just a few minutes. Nice!

The lock and signature icons in the lower right-hand corner are only active if the recipient's public key is available.

Let's explore a bit in Terminal
generate a new key with a spectacularly weak passphrase: abc

> gpg --gen-key gpg (GnuPG/MacGPG2) 2.0.18; Copyright (C) 2011 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire = key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 4y Key expires at Thu Apr 28 10:11:26 2016 EDT Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: Alice Email address: alice@email.com Comment: You selected this USER-ID: "Alice " Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O You need a Passphrase to protect your secret key. We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. gpg: key 6CC18DC3 marked as ultimately trusted public and secret key created and signed. gpg: checking the trustdb gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 4 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 4u gpg: next trustdb check due at 2015-08-18 pub 2048R/6CC18DC3 2012-04-29 [expires: 2016-04-28] Key fingerprint = A84B 4F1D A439 849F 47F1 55F9 BF7C FB80 6CC1 8DC3 uid Alice sub 2048R/DC9C92B3 2012-04-29 [expires: 2016-04-28]

I have a feeling that this fake user's data may have been sent to the key server, and that's not nice. But I expected to get a prompt related to that and I didn't get one.

Let's take a quick look at what we have:

> gpg -k Alice pub 2048R/6CC18DC3 2012-04-29 [expires: 2016-04-28] uid Alice sub 2048R/DC9C92B3 2012-04-29 [expires: 2016-04-28] > gpg -k --fingerprint Alice pub 2048R/6CC18DC3 2012-04-29 [expires: 2016-04-28] Key fingerprint = A84B 4F1D A439 849F 47F1 55F9 BF7C FB80 6CC1 8DC3 uid Alice sub 2048R/DC9C92B3 2012-04-29 [expires: 2016-04-28] >

-a is "ascii armored" base64 output:

> gpg -a --export 6CC18DC3 -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org mQENBE+dTEUBCADWaa0ikPdHmp7OONsuhUJeIIr.. .. -----END PGP PUBLIC KEY BLOCK----- > > gpg -a --export-secret-key 6CC18DC3 -----BEGIN PGP PRIVATE KEY BLOCK----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org lQO+BE+dTEUBCADWaa0ikPdHmp7OONsuhUJeIIr.. .. -----END PGP PRIVATE KEY BLOCK-----

---

While this looks quite similar to RSA (my posts), the Python rsa module won't handle it as is. I didn't dissect it by hand yet. There's something called pgpdump that is recommended on the web.

It builds easily and then I can do:

gpg --export 6CC18DC3 | pgpdump -i gpg --export 6CC18DC3-secret-key | pgpdump -i

but I didn't find the output all that useful yet. It shows multiple values for n and e but doesn't show d or p and q.

---

Try some encryption:

> gpg -e -r 6CC18DC3 m.txt > hexdump -C m.txt 00000000 68 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 0a |hello, world!.| 0000000e > hexdump -C m.txt.gpg 00000000 85 01 0c 03 a9 47 9a 63 dc 9c 92 b3 01 07 ff 57 |.....G.c.......W| 00000010 86 9b 5d a0 40 d1 f0 ef 5a 6f dc eb 19 a9 eb 8c |..].@...Zo......| 00000020 ee 66 a7 34 84 e4 47 5b 6c 48 9f 9e 89 13 4c 2a |.f.4..G[lH....L*| 00000030 71 6a 31 b7 27 23 9d 56 a7 c2 ad fd db 47 57 68 |qj1.'#.V.....GWh| 00000040 da 75 9a 2d 2f f6 46 60 16 84 b6 17 bf e7 b7 5c |.u.-/.F`.......\| 00000050 36 fd d1 e2 22 ee 93 dc ad 82 f5 f1 46 99 12 f3 |6...".......F...| 00000060 fe 25 a1 b3 01 8c 37 a0 59 da ac 39 90 a4 1c ba |.%....7.Y..9....| 00000070 a0 4f 1e b6 da d5 36 55 b1 17 d6 c4 5a 28 de b4 |.O....6U....Z(..| 00000080 47 b2 af 8a c8 9c 58 85 44 f8 08 fe a1 47 c3 8f |G.....X.D....G..| 00000090 4d b1 78 50 87 dc a7 7f 55 89 f2 6e 7f 75 ae a0 |M.xP....U..n.u..| 000000a0 69 68 46 5a 64 1e b4 6e c7 ee 84 77 8d a4 ce 14 |ihFZd..n...w....| 000000b0 72 45 13 be d0 33 5c d6 23 6f 2d b2 84 2f d9 55 |rE...3\.#o-../.U| 000000c0 f7 de d2 8f b6 20 5b 71 4e 31 ae b8 d7 1b 09 bf |..... [qN1......| 000000d0 80 9e e0 1f 47 cb 73 a1 59 42 81 24 1f 2b de 4b |....G.s.YB.$.+.K| 000000e0 0d 23 fc c6 a2 83 5e c2 b3 e5 9f 1f 32 ae 75 07 |.#....^.....2.u.| 000000f0 79 7f 51 49 02 80 a8 47 c4 5c b6 6f aa ac d4 5c |y.QI...G.\.o...\| 00000100 e7 c9 b6 1f d2 c1 7e 03 45 34 59 85 d1 63 01 d2 |......~.E4Y..c..| 00000110 4e 01 27 2e e9 09 aa 82 5d 77 56 82 22 4e 2e 67 |N.'.....]wV."N.g| 00000120 1c 4a bc ba c1 43 d6 f0 86 02 5d e7 b3 58 74 79 |.J...C....]..Xty| 00000130 bc 15 69 d4 44 ba f6 76 0c a7 a1 d5 9b 1b e0 8b |..i.D..v........| 00000140 b6 7b df db b0 5f e6 34 0b 36 14 0b fd c6 62 f3 |.{..._.4.6....b.| 00000150 16 a8 97 ad 92 e7 4e a4 ee ab 59 53 91 c6 52 |......N...YS..R| 0000015f > gpg -d -o p.txt m.txt.gpg You need a passphrase to unlock the secret key for user: "Alice " 2048-bit RSA key, ID DC9C92B3, created 2012-04-29 (main key ID 6CC18DC3) gpg: encrypted with 2048-bit RSA key, ID DC9C92B3, created 2012-04-29 "Alice " > cat p.txt hello, world!

I found what looks to be an excellent tutorial on the web. I still need to work through it.

pycrypto

Here's a Quick Python post about encryption using the pycrypto module. Much of it is derived from the excellent overview here.

I tried easy_install first, but it hung for some reason, so I just did:

git clone https://github.com/dlitz/pycrypto.git cd pycrypto python setup.py build sudo python setup.py install

Let's try a little DES:

>>> from Crypto.Cipher import DES >>> des = DES.new('01234567',DES.MODE_ECB) >>> text = 'hello, world' >>> c = des.encrypt(text) Traceback (most recent call last): File "", line 1, in ValueError: Input strings must be a multiple of 8 in length >>> len(text) 12 >>> text += '.' * (8 - len(text) % 8) >>> c = des.encrypt(text) >>> c '\xc4\xf56\xad\xbb\xae\xb8\x84\xbf\xb3\xfaH^\xe3\x10\x16' >>> des.decrypt(c) 'hello, world....' >>>

Here is a different mode of DES called CFB (Cipher Feedback). It requires some random bytes, and two instantiations of the object:

>>> from Crypto.Cipher import DES >>> from Crypto import Random >>> iv = Random.get_random_bytes(8) >>> des1 = DES.new('01234567', DES.MODE_CFB, iv) >>> des2 = DES.new('01234567', DES.MODE_CFB, iv) >>> text = 'hello, world!' >>> c = des1.encrypt(text) >>> c '\x92\xc6\xc7K=\xb0\xf4\x83A\xfd\xa4\x13e' >>> des2.decrypt(c) 'hello, world!'

The venerable xor method is also present:

>>> from Crypto.Cipher import XOR >>> key = '\x00'*4 + '\x01'*4 + '\x10'*4 + '\x11'*4 >>> xor = XOR.new(key) >>> text = '\x00\x01\x10\x11' * 4 >>> c = xor.encrypt(text) >>> p = xor.decrypt(c) >>> text '\x00\x01\x10\x11\x00\x01\x10\x11\x00\x01\x10\x11\x00\x01\x10\x11' >>> key '\x00\x00\x00\x00\x01\x01\x01\x01\x10\x10\x10\x10\x11\x11\x11\x11' >>> c '\x00\x01\x10\x11\x01\x00\x11\x10\x10\x11\x00\x01\x11\x10\x01\x00' >>> p '\x00\x01\x10\x11\x00\x01\x10\x11\x00\x01\x10\x11\x00\x01\x10\x11'

I adapted one of the scripts from Laurent Luce's post to do encryption on a file. Here is the output (the hex values of the key we generated), and some display of the data using hexdump:

> python script.py 21 31 e4 25 b1 ab be fb 3d 35 95 f7 8b 67 ba 24 > hexdump -C m.txt 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 0a |Hello, world!.| 0000000e > hexdump -C c.txt 00000000 25 bc 08 25 ae 7c fb c3 3a a3 91 83 6a 34 7c 06 |%..%.|..:...j4|.| 00000010 > hexdump -C p.txt 00000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 0a 20 20 |Hello, world!. | 00000010

script.py

from Crypto import Random from Crypto.Cipher import DES3 import struct def encrypt_file(ifn, ofn, chunk_size, key, iv): des3 = DES3.new(key, DES3.MODE_CFB, iv) with open(ifn, 'r') as in_file: with open(ofn, 'w') as out_file: while True: chunk = in_file.read(chunk_size) if len(chunk) == 0: break elif len(chunk) % 16 != 0: chunk += ' ' * (16 - len(chunk) % 16) out_file.write(des3.encrypt(chunk)) def decrypt_file(ifn, ofn, chunk_size, key, iv): des3 = DES3.new(key, DES3.MODE_CFB, iv) with open(ifn, 'r') as in_file: with open(ofn, 'w') as out_file: while True: chunk = in_file.read(chunk_size) if len(chunk) == 0: break out_file.write(des3.decrypt(chunk)) SZ = 100 ifn = 'm.txt' ofn = 'c.txt' key = Random.get_random_bytes(16) iv = Random.get_random_bytes(8) L = [struct.unpack('B',k) for k in key] L = [hex(t[0])[2:] for t in L] print ' '.join(L) encrypt_file(ifn, ofn, SZ, key, iv) decrypt_file(ofn, 'p.txt', SZ, key, iv)

bytearray

I came across a post by Python guru David Beazley introducing the bytearray data type. Here is one of his examples, modified slightly:

>>> s = "Hello World" >>> b = bytearray(s) >>> b bytearray(b'Hello World') >>> b[:5] = "Cruel" >>> b bytearray(b'Cruel World') >>> for i in b: ... print i ... 67 114 117 101 108 32 87 111 114 108 100 >>> b.append(33) >>> b bytearray(b'Cruel World!') >>> print b Cruel World!

The usage b"mystring" was new to me as well. According to this SO answer:

the b prefix for string literals is ineffective in 2.6, but it is a useful marker in the program, which flags explicitly the intent of the programmer to have the string as a data string rather than a text string. This info can then be used by the 2to3 converter or similar utilities when the program is ported to Py3k.

More info here (thanks, Ned).

A bytearray has the methods of lists including __getitem__ and __setitem__. Unusually for Python, one can use arguments of both int and str types. The actual type of a bytearray item is < type 'int'>.

>>> b = bytearray('Hello World!') >>> b[-1] = '*' >>> b bytearray(b'Hello World*') >>> ord('!') 33 >>> b[-1] = 33 >>> b bytearray(b'Hello World!')

We can use a bytearray to do a small job of encryption in a very convenient way:

>>> import numpy as np >>> msg = bytearray("Hello World") >>> n = len(msg) >>> L = np.random.randint(0,256,n) >>> key = bytearray(list(L)) >>> len(key) 11 >>> key bytearray(b'N\x963\x8a\x06\xf2\xe9\x92\xb9\xf4L') >>> >>> >>> ctext = bytearray() >>> for c,k in zip(msg,key): ... ctext.append(c ^ k) ... >>> ctext bytearray(b'\x06\xf3_\xe6i\xd2\xbe\xfd\xcb\x98(') >>> ptext = bytearray() >>> for c,k in zip(ctext,key): ... ptext.append(c ^ k) ... >>> ptext bytearray(b'Hello World')