CommonCrypto4

This is the last post about CommonCrypto. I just wanted to mention that I converted Encryptor into a framework. It is still on github.

I also wrotesome command line utilities in Swift that use it. That project is here.

You can checkout the README which explains usage.

CommonCrypto3

This is a brief note about a big project (at least for a one-day thing). I finished a Cocoa app that implements basic AES-mode encryption and decryption from the CommonCrypto library. It does not do any I/O.

For now, you can just watch it run in the debugger, using pre-formed messages. But it can handle messages that are larger than a single block by breaking the data into blocks and encrypting/decrypting each one.

One nice thing is the BinaryData class, which implements some functionality around a [UInt8] including the Indexable and CustomStringConvertible protocols, and a convenience initializer that takes a String representing binary data.



BinaryData is a class because it has a derived class Key, which implements key "stretching" from CommonCrypto.

At the very end, I ran into trouble because I did not really understand the CBC protocol.

When encrypting, the output ciphertext becomes the initialization vector for the next round. But when decrypting, the input (which is also the ciphertext) becomes the initialization vector for the next round.

Here is the output for a test:


pw: my secret
salt: 3356ec169bb6
msgText: a much longer and still really big secret
61206d756368206c6f6e67657220616e64207374696c6c207265616c6c792062696720736563726574
encryptMany
encrypt round: 1
encryptOneChunk
msgLen: 16
msg:
61206d756368206c6f6e67657220616e
keyLen: 16
iv:
39131435ae3d5bbf2e300ab5edddc8c9
status: 0
result:
55bd582296f708842e5d0833e3673a99

encrypt round: 2
encryptOneChunk
msgLen: 16
msg:
64207374696c6c207265616c6c792062
keyLen: 16
iv:
55bd582296f708842e5d0833e3673a99
status: 0
result:
ded208434eb0a4e753e6cdf1c41da54c

encrypt round: 3
encryptOneChunk
msgLen: 16
msg:
69672073656372657400000000000000
keyLen: 16
iv:
ded208434eb0a4e753e6cdf1c41da54c
status: 0
result:
66e7cb3ba1b51a0d657edad13de97b37

cipherData: 55bd582296f708842e5d0833e3673a99ded208434eb0a4e753e6cdf1c41da54c66e7cb3ba1b51a0d657edad13de97b37

decryptMany
decryptOneChunk
data:
55bd582296f708842e5d0833e3673a99
keyLen: 16
iv:
39131435ae3d5bbf2e300ab5edddc8c9
status: 0
result:
61206d756368206c6f6e67657220616e

decryptOneChunk
data:
ded208434eb0a4e753e6cdf1c41da54c
keyLen: 16
iv:
55bd582296f708842e5d0833e3673a99
status: 0
result:
64207374696c6c207265616c6c792062

decryptOneChunk
data:
66e7cb3ba1b51a0d657edad13de97b37
keyLen: 16
iv:
ded208434eb0a4e753e6cdf1c41da54c
status: 0
result:
69672073656372657400000000000000

decryptedData: 61206d756368206c6f6e67657220616e64207374696c6c207265616c6c79206269672073656372657400000000000000
a much longer and still really big secret

Whoa... there were some bytes in the cut and pasted text that did not appear in the debug console nor in the editor here on blogger,, but did appear in the Preview. They are the null bytes used to pad the plaintext. They do not appear in the final product. Curious...

The Xcode project is on github here.

It was a world of fun, but I think I am done with CommonCrypto for now.

CommonCrypto2

I've been exploring how to use CommonCrypto on OS X (docs). As described in a previous post, I found an article on the web which describes how to make the CommonCrypto library available to Swift playgrounds as well as Swift Cocoa applications.

I decided to look into it a bit more. I had help from Mike Ash, as well as another knowledgable guy. Looking at the header helped a lot as well. It is extremely detailed. Since the project was somewhat challenging, I thought I would sketch out what I learned and post the code.

CommonCrypto is a C library. So, for example, the function that does one-step encryption or decryption, CCCrypt, is declared like this in CommonCrypto.h



The first two arguments can be defined in Swift like this:

let operation = CCOperation(kCCEncrypt)
let algorithm = CCAlgorithm(kCCAlgorithmAES)

We're first going to encrypt, and the encryption scheme will be AES.

If we look in the header, it tells us that the block size for AES is 16 bytes or 128 bits.

kCCAlgorithmAES128 Advanced Encryption Standard, 128-bit block

Reading further

One option for block ciphers is padding, as defined in PKCS7; when padding is enabled, the total amount of data encrypted does not have to be an even multiple of the block size, and the actual length of plaintext is calculated during decryption.

Another option for block ciphers is Cipher Block Chaining, known as CBC mode. When using CBC mode, an Initialization Vector (IV) is provided along with the key when starting an encrypt or decrypt operation. If CBC mode is selected and no IV is provided, an IV of all zeroes will be used.

To begin with we'll use ECB with PKCS7 padding because it seems simpler.


let options = CCOptions(kCCOptionPKCS7Padding | kCCOptionECBMode)


CCCrypt takes three arguments of type const void * and one of type void *. The three const arguments are the key, the initialization vector iv, and the plaintext or data (dataIn). These can be provided as Swift arrays: [UInt8], or even (for the key and plaintext) as Swift Strings. No need for NSData or UnsafePointer or whatever.

The return type for this function is CCCryptorStatus, which is 0 for success, and something else for an error. The codes are also shown in the header:



So, when I received -4301 as the result, which at first I thought was garbage, CCCrypt was actually telling me "insufficient buffer provided for the specified operation."

Of special note: I found that although the message does not have to be 128 bits or a multiple, the key does. If it is not, the encrypt operation returns 0 for success and some encrypted data, but when decrypted we don't get our plaintext back!

A buffer is needed, into which the encrypted data will be written.

let bufferSize = 128
var cipherData = [UInt8](count: bufferSize, repeatedValue: 0)
var resultLen = 0
var status: Int32 = 0

Another thing that confused me was that these sizes (except for resultLen and status) are in bits, not bytes.

When the buffer is passed into CCCrypt we need a cast:

UnsafeMutablePointer<Void>(cipherData)

We provide the address of the Int variable &resultLen, and after the function returns, that value tells how much data was written.

Finally, the other argument is the initialization vector iv.



In our first pass at this, we specify ECB mode and PKCS7Padding, so no IV is needed, and we just pass nil for this argument.

It works. The first 9 bytes of the output at the end are the same as what we put in. I put the playground on github here.

The only thing I haven't figured out with this one is how to know the size of the message when decrypting.

The second approach uses CBC. It's the same as the first, except we change the options to the default

let options = CCOptions()

define an initialization vector, and then fill it with random bytes.

var iv = [UInt8](count: blockSize, repeatedValue: 0)
SecRandomCopyBytes(kSecRandomDefault, blockSize, &iv)


Other than that, the only thing is to be sure and pad the message to 16 bytes. It works. The playground is here.

UPDATE: I implemented the other code sketched out in Mike's article: encrypting in steps for a longer message (playground), and key stretching (playground).

CommonCrypto

So Apple has a cryptography library called CommonCrypto (docs). Importantly, it is not a framework.

Here is a quote from the older documentation (last year):

Common Crypto
In OS X v10.5 and later and iOS 5.0 and later, Common Crypto provides low-level C support for encryption and decryption. Common Crypto is not as straightforward as Security Transforms, but provides a wider range of features, including additional hashing schemes, cipher modes, and so on.

Normally, I would have passed on this in favor of, say, trying to build openssl and then access that. (I recall seeing something from tptacek about it along the lines of use something more standard, don't trust that Apple will implement the basics correctly). Judging by history, that would be pretty sage advice.

But ... I came across a wonderful blog post here, which describes not only how to use CommonCrypto from Swift, and not only shows an in-progress library wrapping CommonCrypto, but also describes a hack that allows Swift playgrounds to have access to CommonCrypto. Too cool for words!

The basics: open a new Xcode project in Swift and call it MyApp. We need a "bridging header", we get this by adding a new Objective C class, then Xcode will ask if we want this header, and we say yes. Ungratefully, we promptly delete the dummy Objective C class. In the header, add:

#import <CommonCrypto/CommonCrypto.h>


That's it. For example, I put this code into crypto.swift



and call it from the AppDelegate in applicationDidFinishLaunching. The debugger prints:

e4d909c290d0fb1ca068ffaddf22cbd0

If I put the same text in a file and do

> md5 msg.txt
MD5 (msg.txt) = e4d909c290d0fb1ca068ffaddf22cbd0
>


Of course, what we've done here is the usual C hack of allocating a buffer and passing a pointer to the buffer into the program that will write into it. Except that in Swift these are not your father's pointers. You can't do arithmetic. And they have that label "Unsafe". Reminds me of Maverick.

Now for the cool part. Deep within Xcode there are one (or more for some people) SDKs. Software Development Kits.

> xcrun --show-sdk-path --sdk macosx
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform\
/Developer/SDKs/MacOSX10.11.sdk

and deep within that is:

> ls /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform\
/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks
AGL.framework
AVFoundation.framework
AVKit.framework
Accelerate.framework
...

As I said, CommonCrypto is not a framework. But we can fake things like it is one. We make a directory in that place, call it CommonCrypto.framework and inside that put module.map with some paths:

module CommonCrypto [system] {
header "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform\
/Developer/SDKs/MacOSX10.11.sdk/usr/include/CommonCrypto/CommonCrypto.h"
header "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform\
/Developer/SDKs/MacOSX10.11.sdk/usr/include/CommonCrypto/CommonRandom.h"
export *
}

(Those line breaks should work, but I don't have them in my original).

I did this:

cd /Applications/Xcode.app/Contents/Developer\
/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk\
/System/Library/Frameworks
> sudo mkdir CommonCrypto.framework
Password:
> sudo cp ~/Desktop/module.map CommonCrypto.framework
>
> cat CommonCrypto.framework/module.map
module CommonCrypto [system] {
...
>


Having "corrupted" the SDK in this way, we can now do things like this: paste that same code into an Xcode playground. I had to change the last line, not sure why just at the moment..



And with this setup, we no longer need the bridging header. I tried just deleting it but Xcode knows. So delete the whole project and make a new one with the same name. Add the code in crypto.swift and put import CommonCrypto in that same file. Then from the AppDelegate, call doIt.

It works!



It's worth pointing out that simply substituting SHA256 for MD5 in the code above works. To check the digest on the command line do:

> openssl dgst -sha256 -hex msg.txt
SHA256(msg.txt)= ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c


Swift using a C framework

Recently I explored how to write a framework in Swift and then import and use it in my Swift app. The key elements are to find a good place to stash the framework (I used ~/Library/Frameworks), and to properly let Xcode know that it should link that framework into the app.

The objective for this post is to learn how to import and use a framework not written in Swift. Before I use a real one (like this) for today I am going to use a very simple example written in C, following this post from a few years ago.

Here is the code, which provides amazing functionality:

There is a slight wrinkle. In the original version, we didn't use a header file but instead declared the functions as extern like this:

extern int f1(int x);

within the file that uses them.

As an aside, note that I should probably have used #import rather than #include. See for example, here. However, I had already gone through twice re-testing the steps for today's post, and I didn't want to do it all again.

Now we do:

> clang -g -Wall -c add*.c
>

which generates add1.o and add2.o. Then

> clang -g -Wall useadd.c add1.o add2.o -o useadd
>


> ./useadd
f1: 1; main 2
f2: 10; main 12
>

Although we didn't explicitly tell clang about add.h (i.e. in the command line invocation), it is needed for this to work. I suppose clang finds the header file in the build directory.

Now, let's make an old-fashioned library. Delete useadd. Do:

> libtool -static add*.o -o libadd.a
>

We have libadd.a. Let's use it:

> clang -g -Wall -o useadd useadd.c -L. -ladd
>

And it works:

> ./useadd
f1: 1; main 2
f2: 10; main 12
>

Here we have explicitly directed clang with -L. to the build directory, where we want it to look for -ladd (short for libadd).

Now the goal is to make a Cocoa app that uses f1 and f2... whether written in Objective C or in Swift. I thought at first we would need a framework, but we don't. Just copy libadd.a into ~/Library/Frameworks. Make a new Xcode project Myapp, a Cocoa app in Objective C. Add the library to the project as described (by clicking + on Linked Frameworks and Libraries, etc.)

We still have the header issue. For this version using libadd.a I just dragged the header into the project, with copy files, and then did the import in either AppDelegate.h or AppDelegate.m.

Now, for a framework. Make a new Xcode framework in Objective C, called Adder. Drag in add1.c and add2.c and do copy files. Put the declarations from add.h into Adder.h which Xcode provided for us.

Build it. Use the Show in Finder trick to find and then drag the framework to the Desktop and then to ~/Library/Frameworks.

Try to use the framework from the command line.

I specified the path to find the header folder which is in the framework.

And it works:


> clang -g -o useadd -F ~/Library/Frameworks/ -framework Adder useadd.c\
-I~/Library/Frameworks/Adder.framework/Headers

> ./useadd
f1: 1; main 2
f2: 10; main 12
>

We move away from the command line. Make a new Xcode Project for a Cocoa app in Objective C. Call it MyApp.

In the AppDelegate do #import <Adder/Adder.h>. Fix the error that this introduces by adding the linked binary, as usual. In applicationDidFinishLaunching: do:

int n = f1(10);
NSLog(@"%d", n);

Build and run. It works. The debugger prints:

f1: 10;2015-12-10 08:33:28.318 MyApp[6168:235988] 11

Now for the last step. Make a new Xcode Project for a Cocoa app in Swift. Call it MySwiftApp. In the AppDelegate do import Adder. Fix the error by adding the framework. In applicationDidFinishLaunching add:

let result = f1(10)
Swift.print("\nresult: \(result)")

And it works! The debugger prints:

f1: 10;
result: 11

The print from C didn't include a newline. Oops.

It works without a "bridging header". If you do have to generate one of those, just add a dummy Objective C module to your project. Xcode will generate the bridging header for you.

At this point, I have the confidence to move ahead with a real project.

[UPDATE: Repeating it all again for the third or fourth time, I am unable to get the last two steps to work smoothly. The Objective C one builds with a warning "implicit declaration of function 'f1' is invalid in C99", but it runs and prints what we expect. The Swift example allows me to "import Adder" but doesn't recognize the symbol "f1".

Since it really did work before (I didn't just imagine it), there is something that I "got for free" before that I am missing now. More exploration is needed.]

[UPDATE 2: I went through every step again from scratch. It works now. I wrote it up for the book. I guess I just have to wait and see if it fails again. ]

Swift framework from the command line

Following up on the previous post, I want to import my new Swift framework when building, or running Swift code from the command line.

The example above shows building. It works!

I struggled with this.. and now that I know, it just seems so silly. When providing the path to the framework, don't provide the full path, just give the path to the folder that contains it ... of course. We also need the path to the SDK, which Xcode will prompt you about, should you leave that part out.

For the other method (xcrun swift file.swift) this doesn't work. That's probably not too surprising, but maybe there will be a way.

Building and using a framework in Swift

In the past, I've looked into the theory and some of the complications of building libraries and frameworks on OS X, much of it written up on this blog. I got interested today yesterday in the problem of building and using a pure Swift framework. I tried some of what the top hits on Google said to do, but it wasn't very helpful (because it's all iOS, and because it didn't seem to work), however, eventually I noodled out something that I think does work. I have yet to do more than skim the official Apple documentation.

According to this

Most public frameworks should be installed at the local level in /Library/Frameworks

That's our goal here. Any app that launches and then needs to find a framework should find it by searching "the usual suspects". According to the docs, the best place is /Library/Frameworks, but here I will use ~/Library/Frameworks.

Such a framework would be a dynamic framework, not static, or baked into the app. If you were to write an installer for such an app, you should take care to install its frameworks in the right place. R is a great example of how to do it right.

---

So in Xcode:

OS X > New Project > Framework & Library > Cocoa Framework > Swift




I named it SpeakerFramework.framework. Add to the framework a new Swift file

speaker.swift:



Having the init and speak functions both public (as well as the class itself) is required for external visibility (ref).

Build the framework.

Under Products, find SpeakerFramework.framework.



Control-click and show in Finder



then drag it to the Desktop.

For what we will do in a minute, we need the Finder to show ~/Library in an Open File dialog. With your home folder selected in Finder, do CMD-J (or View > Show View Options) and select Show Library Folder. (If you are just on the Desktop, then Show View Options shows something different).



Now for the app. In Xcode:

New Project > OS X > Application > Cocoa App > Swift




I named it MyApp.

In AppDelegate.swift add

import SpeakerFramework


This import statement gives an error and the project will not build (No such module 'SpeakerFramework').



To fix this, open ~/Library/Frameworks and drag the framework we just copied to the Desktop into that folder.



Alternatively, just do this in Terminal:

cp -r SpeakerFramework.framework ~/Library/Frameworks


Now, back in Xcode, select the project in the Project Navigator. In the tab view, select General and scroll to the bottom where it says Linked Frameworks and Libraries.



Click the + symbol below that line (not shown in the screenshot, because I only took it after I added the framework to this project).



Click Add Other... Then navigate to the framework in ~/Library/Frameworks and select it.

The MyApp project shows the framework in the General tab



Go back to the AppDelegate. The warning should be gone.



MyApp will build now. So let's use it. Edit the AppDelegate to call the speak method:



In the Debug window, we can see the expected output.



It would be nice to make a bigger statement.

I'll just outline the steps briefly. Delete the default window in MainMenu.xib. Add a new Cocoa class, subclassing NSWindowController. Have Xcode make the xib file too. Drag a label onto that window. Make it really, really big.



Hook it up to the new class (MainWindowController) as an IBOutlet, with this code for the AppDelegate:



and for MainWindowController:



Pretty impressive:



If we select Products > MyApp.app, then show in the Finder, and drag it to the Desktop, and delete everything else, except the framework in ~/Library/Frameworks, it still works. If we then do

> mv ~/Library/Frameworks/SpeakerFramework.framework/ old
> ls ~/Library/Frameworks
>




So it looks like everything is working as expected.

I thought I would snoop on the Loader as I did long ago:

> export DYLD_PRINT_LIBRARIES=1
> open -a MyApp.app

But it gives nothing. I am not sure why, yet.

Encoder

I'm fooling around with a new Swift Cocoa project. The Encoder app does encryption and decryption. I use the phrase "fooling around" deliberately because an important rule of cryptography is to use standard libraries. If you don't really know what you're doing, you certainly shouldn't "roll your own."

Here is a screenshot of the current version of the app.

At the heart of it, we are doing something trivial:

return Zip2Sequence(a1,a2).map { $0^$1 }

where a1 and a2 are binary data, i.e. arrays of integers with values between 0 and 255 (UInt8). The xor operator ^ is just what we want for both encoding and decoding.

The trick, of course, is to generate a keystream of pseudo-random numbers given a key, which is a String.

Foundation provides standard functions for obtaining random numbers:
rand, random, arc4random, arc4random_uniform.

arc4random and arc4random_uniform are preferred for really random PRNG, but lack the ability to be seeded (by the caller). Seeding is needed so that the same pseudo-random keystream sequence can be regenerated, given a relatively short key.

The rand function returns an Int32 (try rand() is Int32 in a Swift playground. So in theory it might return both positive and negative values. However, it only seems to return positive numbers, which is good, because, say -3 % 256 is equal to -3, which would cause trouble when trying to convert to UInt8. According to this (very old) file, the OS X rand uses unsigned values.

The maximum value returned is RAND_MAX, which is equal to 2147483647 = 2^31 - 1, which is also equal to Int32.max. What we want are integers in the interval [0,255], so I just do:

Int(rand()) % 256


I'm not an expert, of course, but I looked at the distribution of output of rand and it seemed sufficiently random for my purposes.

The next question is that of seeding the PRNG. You might do:

srand(UInt32(time(nil)))

The seed function srand takes a UInt32. The question is, how do we turn our key into a UInt32? What I've done so far is to use the String property hashValue / This gives an Int, which is 64 bits, and of course, may also be negative.

Here is my approach:

Having initialized everything, we just call

func next() -> UInt8 {
  return UInt8( Int(rand()) % 256 )
}

One issue I'm still working on is that of loading binary data. I know how to turn an [UInt8] into data:

let a: [UInt8] = Array(0..<4)

let data = NSData(bytes: a, length: 4)

and load data from a file with NSData(contentsOfFile:fn), but I haven't been able to figure out how to turn that back into an array of UInt8 in Swift.

I puzzled out a way around the problem, however. String will take NSData in an initializer:


Given that, I get the individual bytes from the string and use a Dictionary



As usual these days, the project is on github here. Thanks to samol for explaining how to do the gist thing.

UPDATE:

I found a way to do the data conversion mentioned above. It looks like it is probably the natural way to do this in Swift.



We use NSInputStream initialized with the NSData object. We allocate the necessary space:

var buffer = Array(count: n, repeatedValue: 0)

and then read the data into the buffer using a reference.

stream.read(&buffer, maxLength: n)

The last call returns a result, which is the number of bytes read. Alternatively, one could read byte by byte until stream.hasBytesAvailable returns false. The only thing that confused me for a while was that I had Array(0..<n) which gave Int values of 64 bits each. Without the map on line 4 the data looks like:

[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2,

which I interpret as 00000000, 10000000, 20000000 and so on. These are the decimal values for each byte, where the ordering is little-endian, with the low value byte having the first memory address.

Intel x86 processors store a two-byte integer with the least significant byte first, followed by the most significant byte. This is called little-endian byte ordering.

Apple docs. If I write the 64-bit data to a file and examine it with hexdump:

> hexdump x.bin
0000000 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00


Images and Icons

As I mentioned previously, I've been programming again for OS X and trying to learn Swift. Although I really like Xcode, I also seek the simplicity of running Swift programs from the command line. For reasons I will explain in a minute, I was thinking about images and drawing. I remembered an example with something similar (here, with references in the link), where we had an NSView that could be drawn without any window on screen---it was not an app with a GUI. NSView's dataWithPDFInsideRect can construct a PDF and then save it to a file. The first step was to repeat this approach in Swift (although I didn't get around to repeating the actual drawing that we had before).

Here is a screenshot from the playground:

The playground itself is on github here.

This actually works! One indicator which can be seen in the screenshot is the result returned by the last call: true, shown in the results panel on the right. The file we just wrote cannot itself be found in the playground (Show Project Navigator will not help). However, Spotlight does find it, and we can then paste the path to Terminal and view it in Preview:

> open -a Preview /Users/telliott/Library/Containers/com.apple.dt.playground.stub.OSX.pdfs-D6033292-DB23-46A3-AA4D-7C17D756519F/

This behavior is likely due to sandboxing of Playgrounds. To get around it, paste the same code into a file, my example is pdf.test.swift, and then do

> cd Desktop
> xcrun swift pdf.test.swift
>

The pdf then appears on the Desktop (or whatever the current directory is).

One reason for thinking about images was that I would like an icon for my SudokuBlocks app. According to the docs, Apple requires that the developer provide a bunch of icons (10) in different sizes.

It seemed a bit much for hobbyist programming. So what I might have done at this point was to use Preview's Tool -> Adjust Size ..

Instead, I wondered if I could write a Swift command line program (again not using Xcode) that would read an image file, resize it, and write the result to disk.

Reading the file is easy. In a playground, all you need to do is to show Project Navigator, and drag the file to the Resources folder. Then call NSImage(named:"x.png"). (This Playground and all the code and other resources for this post are on github here). The result is an optional.

At this point, I got help (once again) from Mike Ash here.

As he describes, although NSImage is a container that may have one or more representations of a particular image, including an NSBitmapImageRep, he recommends that the correct way to get at the data reliably is to first draw the image into a new NSBitmapImageRep and then look at that.

The initializer is crazy, with 10 different arguments (ref).

The result is again an optional.

The next step is to go through a vaguely recalled dance with NSGraphicsContext, saving the current context, setting up a new one using the NSBitmapImageRep.

Now, any drawing that takes place is done in the image rep. And that is where we do some resizing, calling:

img.drawInRect(dst, fromRect: src, operation: op, fraction: f)

The source src is a CGRect of the size of the image we just loaded (unless you want to clip or something), The destination dst is a rect of the size of image we want to produce. By changing dst we change the reduction or magnification of the image produced. The other two values are:

let op = NSCompositingOperation.CompositeCopy
let f = CGFloat(1.0)

There are a lot of choices for operation, e.g., see here

After that we just grab the data:

let data = imgRep.representationUsingType(.NSPNGFileType, properties: [:])

(notice the empty Dictionary [:]) and write it to disk:

data!.writeToFile("out.png", atomically: true)

The code is in images.swift in the github rep. It's a bit rough. I trimmed it down and added some command line parsing, and used it to resize square png images. That code is in resizer.swift

Usage:

xcrun swift resizer.swift filename sz
xcrun swift resizer.swift x.png 256

The last thing here is about the icon for SudokuBlocks. The official docs say that you should provide 10 different images. But it turns out that it is possible to copy a single image into the project and be done with it. Click on Assets.xcassets and then double click on Appicon and there will be a place to add the files. As a test I made a new project and dragged in a 256 x 256 image. When I build and run it the image will appear in the dock. Then I went to Products and found the app, did control click and view in Finder, and dragged it to the Desktop. It looks like this:

.

And here is a screenshot with the latest version of SudokuBlocks (you can even see it in the Dock):

SudokuBlocks

Recently I've come back to application programming for OS X after a long time away.  I reviewed my notes on Swift from last year (on github here, not updated for Swift 2 yet).  (I was a little dismayed at how little I remembered, but it mostly came back after 8 or 10 hours).

I bought a copy of Hillegass et al. new edition of their great book to write Cocoa applications, which now uses Swift.  I haven't actually done that much with the book yet, though it was a good reference on setting up an Xcode project and hooking up a custom NSView.

For my first project I re-wrote my Color Sudoku program (post about the old one here).  At the top is a screenshot from the new one.  I notice there are similar things out there on the web now, I don't know if they were invented independently or what, but I built my first one in 2006.  In any event, it isn't suitable for mobile because of the tiny squares, so it could probably never be a commercial success.

I've been consistently amazed at how easy programming in Swift has been.  When there was a programming error it was easy to diagnose and fix.  I particularly love the ability to break out function definitions to a new file on a whim (no header files really helps here).  The fact that function names are visible in all the files of a project is a little scary, but it makes this kind of refactoring easy.

I put the new project up on github here, and also put a copy on Dropbox here.  I included a built version of the app in the project folder.  You will have to temporarily "allow apps downloaded from anywhere" or alternatively, install Xcode and build it yourself from the source.

I'm sure there are bugs.  Let me know if you find one.

Stuff on GitHub

This is my first post for a while.  I just wanted to let you know that I have been learning how to use git.  My github repo is here.

I have placed a number of "books" there which I have worked on in the last year including

MyJava
MyUnix
MyCrypto (just beginning)
PyBioinformatics

Also, there is a collection of writeups on more than 60 topics in mathematics.  The tex files are on github here and the pdfs are on Dropbox here (48 MB).

Swift talks to Objective C

I am back brushing up on Objective C and Cocoa, and now working on Swift.  It is supposed to be easy to integrate Swift code with an Objective C project, but I found it difficult.  So as I've done before, I've cooked up a working example that is as simple as it could possibly get.  I thought I would post a writeup on Dropbox.

It is here.  (It's a pdf printed from Sphinx html).  The original will be on github soon.  Good as long as Dropbox yet lives.

Still kicking

I have an online persona on another site.  Looks like this:

You can find the attribution here.

Anyway, that version of me shows a quote (maybe Descartes?):  "if I can still compute, I'm not dead yet!"  So, anyway I'm still computing and I'm not dead yet.  I have various projects on github which you can find under my username (telliott99), including a basic review for Java, a little bit of Crypto, and my old book on Python.

Simple server

I've been working on a new demonstration project that I'd like to tell you about.

When I want to run a bioinformatics script on a sequence file, I just go to the command line and do something like:

python myprog.py infilename arg1 arg2 > results.txt

But suppose instead the scenario is that I have co-workers who just will not do this, and they pester me until I run an analysis for them.  I'd like to explore an alternative approach:  set up a simple web server that will guide the command-line-phobic user through the process of choosing a script and a sequence file and also entering the appropriate options for that script.  We can use html forms to do the work.

I've written a skeleton web server (using the Flask framework and the development server that comes with it) as a proof of concept.  It is not a web server in the usual sense and it's not actually exposed to the web.  It is designed to be run in the background on a single machine, with the permissions of the user, and can read and write files with those permissions.  The "server" could be configured to startup on Launch, if desired, so the user would never even be aware of that.

I've put what I have so far on github:

git clone git://github.com/telliott99/scripter.git

As currently configured, the python scripts don't actually do anything except read the options data and return a nice image.  However, the web pages are wired up together. The command line output shows that all the data is flowing through as desired.  Here is a screenshot of the index page:

What you would need to do to run the demo is to use pip to install the Flask framework.  Many people use virtualenv to set up something like this, and I did that for my first run through with this flask tutorial.

However, my setup is a separate python from the System python, installed with Homebrew, and I can easily replace it at any time.  I don't worry about the risks of experimentation, and I don't have so much stuff there that I worry about conflicts.  The new Python comes first in my $PATH:

> which python
/usr/local/bin/python
> which pip
/usr/local/bin/pip
> pip install flask

I'm not actually using the WTForms in the current version of the project, but if you did want them you would need flask-wtf.  I did not install any of the other extensions for this project.  With flask available from python, and the project downloaded, just cd into the project directory and do

> ./run.py
 * Running on http://127.0.0.1:5000/
 * Restarting with reloader

Point your browser at localhost:5000 and you should see the index page.

The python code to render the form is simple:



The form itself is built from two templates, a base template that provides the header, and this:



The form data is routed back to the correct url/route/function to start the next steps (by "action=/prog_request").

The main issues are:  enforcing that the user provide a filename and make a choice of script, showing an options page with appropriate options for that script, and then dispatching correctly.  I'll have something to say about those in another post.

I am also just getting started with git, so for reference here is what I did to get this up on github.  I got a free account (and Homebrew installed git), and configured my editor.  That's important because the git commit step requires a commit message, and the default editor that comes up is vim, which I don't know how to use.  (Yes, I know "everyone" loves it).

git config --global core.editor TextMate

then from the scripter directory

git init
git add .
git commit
git remote add origin https://github.com/telliott99/scripter.git
git push -u origin master

github prompts for my credentials and then says the magic words which mean success.  One more task for the future:  setup ssh to do the push.

matplotlib followup for Python 3 on OS X 10.8

In my last post, I reported success in installing matplotlib on OS X. I realize now that this is not anything new to report, since the currently recommended install process (which I read belatedly today) in the README is to use Homebrew (or MacPorts), which is what I did. I'm just happy to know that it works.

I thought I'd say a word about Python 3 here.

I did

$ brew install python3 --framework

which first installed readline and sqlite (linked into /usr/local/opt), as well as gdbm and xz, and also Distribute and pip.

So now we have:

$ ls /usr/local/lib/python3.3/site-packages/ __pycache__ setuptools-0.6c11-py3.3.egg-info distribute-0.6.45-py3.3.egg setuptools.pth easy-install.pth site.py pip-1.3.1-py3.3.egg sitecustomize.py

$ python3 Python 3.3.2 (default, Jun 19 2013, 15:41:32) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from test import autotest == CPython 3.3.2 (default, Jun 19 2013, 15:41:32) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] == Darwin-12.4.0-x86_64-i386-64bit little-endian == /usr/local/bin Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1) [ 1/372] test_grammar [ 2/372] test_opcodes [ 3/372] test_dict [ 4/372] test_builtin

The tests hang at this point, and I haven't figured out why yet. They do fine with my system Python. Then I did:

$ pip3 install numpy $ pip3 install nose

since we need numpy for matplotlib (and don't get it for free as with System Python).

Following this advice, I did:

$ python3 .. >>> import numpy >>> numpy.test('full') Ran 4808 tests in 64.611s OK (KNOWNFAIL=6, SKIP=6) <nose.result.TextTestResult run=4808 errors=0 failures=0>

Now to matplotlib

$ git clone git://github.com/matplotlib/matplotlib.git $ cd matplotlib $ python3 setup.py build $ sudo python3 setup.py install

(Grabs pyparsing, dateutil, tornado).

$ cd.. $ python3 .. >>> import matplotlib.pyplot as plt >>> Y = [1,4,9,16] >>> plt.scatter(range(len(Y)),Y,s=250,color='r') <matplotlib.collections.PathCollection object at 0x1094e4650> >>> plt.savefig('example.png')

works!

$ pip3 install cython

had a permissions problem, so I did

$ sudo chmod -R 755 /usr/local/lib/python3.3/site-packages/ $ pip3 install cython

Now, for scipy

$ git clone git://github.com/scipy/scipy.git scipy $ cd scipy $ python3 setup.py build $ sudo python3 setup.py install

$ cd .. $ python3 .. >>> from scipy.stats import norm >>> norm.cdf(2) 0.97724986805182079

>>> import scipy >>> scipy.test('full') .. Ran 7486 tests in 725.812s FAILED (KNOWNFAIL=42, SKIP=296, failures=82)

Some failures, but all in all it looks good, and a very easy install.

Matplotlib (and SciPy) on OS X Mountain Lion

Yesterday I installed matplotlib on my MacBook---for about the 10th time. This can be a complex undertaking, but a basic installation is relatively easy, so I thought I would outline it here.

Some things I did that made it easier:

• I used a clean install of OS X. I did this to make sure cruft (links, alternative versions of libraries from old installs) does not interfere. Also, this way I can be sure that things work for the reasons I have written about here.

To do the install I just made a second partition on my hard drive and installed OS X on it from a USB installer. If you've never done this before, good instructions for the USB part are here.

It's quite straightforward. Use the App store to update when the install is complete. I now have OS X 10.8.4.

Although it's not necessarily good practice, I used the same username and password for my accounts on OS X on both partitions, this allows me to read and write files in both accounts easily using the Finder.

• I used the System Python. This doesn't appear to be a very popular choice, but it avoids the headache of having multiple Pythons and various $PATH issues, etc. If you do install a second Python, make sure it's a framework version, since even saving a plot as a png using matplotlib will fail otherwise. I may do some more experiments later, but I think this is the right choice

My Python is

$ which python /usr/bin/python

(well, it's not really the whole of Python on OS X, but that is a whole 'nother story).

$ python Python 2.7.2 (default, Oct 11 2012, 20:14:37)

A bonus is that we already have numpy

>>> import numpy >>> numpy.__version__ '1.6.1'

• I used Homebrew to get the most important matplotlib prerequisites, libpng and freetype. zlib is also a prerequisite but comes with OS X.

The instructions for Homebrew are here. Weirdly, I got a prompt for an admin password, which happened because the directory where Homebrew puts stuff, /usr/local, did not exist yet. So I bailed from the process and first did:

sudo mkdir /usr/local

Now, look for issues revealed by brew doctor and fix them if needed. Finally:

$ brew doctor Your system is ready to brew. $ brew update Already up-to-date. $ brew install pkgconfig libpng freetype .. $ brew list freetype libpng pkg-config

the "Bottles" mentioned in the output I cut means these are pre-built.

I got pkgconfig because it helped with building/linking problems previously, but should not really be necessary when the libraries are in /usr/local. Not sure why it is some times pkg-config and other times pkgconfig, but OK:

$ ls /usr/local/bin/pkg-config /usr/local/bin/pkg-config $ ls /usr/local/lib/pkgconfig/ freetype2.pc libpng.pc libpng15.pc

(If you use e.g. the XQuartz versions of these two libraries, you can set the PKG_CONFIG_PATH environment variable to point to the correct directory, and pkg-config will do the rest).

Now we're ready for matplotlib.

$ git clone git://github.com/matplotlib/matplotlib.git .. $ cd matplotlib $ python setup.py build $ sudo python setup.py install

And test it:

cd .. python >>> import matplotlib.pyplot as plt >>> Y = [1,4,9,16] >>> plt.scatter(range(len(Y)),Y,s=250,color='r') <matplotlib.collections.PathCollection object at 0x103963110> >>> plt.savefig('example.png')

Works!

>>> import matplotlib >>> matplotlib.test() ..KK.KK.....KK.KK..KK.KK.KK.KK.KK etc.

I didn't test extensively but there don't seem to be massive failures. And matplotlib has lots of optional prerequisites I didn't install that would explain some failures.

It's worth pointing out that the matplotlib install process grabs several additional packages:

$ ls /Library/Python/2.7/site-packages README distribute-0.6.28-py2.7.egg easy-install.pth matplotlib-1.4.x-py2.7-macosx-10.8-intel.egg nose-1.3.0-py2.7.egg pyparsing-1.5.7-py2.7.egg setuptools.pth tornado-3.1-py2.7.egg $

Let's get a few more things while we're at it:

$ sudo easy_install pip Password: **** Searching for pip Reading http://pypi.python.org/simple/pip/ Best match: pip 1.3.1 .. Installing pip script to /usr/local/bin Installing pip-2.7 script to /usr/local/bin ..

Come to think of it, SciPy would be a great addition. For that we need gfortran (Fortran compiler) and Cython:

$ brew install gfortran (takes a while, several dependencies)..

Homebrew doesn't have Cython so:

$ sudo pip install cython

And now for SciPy:

$ git clone git://github.com/scipy/scipy.git scipy $ cd scipy $ python setup.py build $ sudo python setup.py install $ cd .. $ python .. >>> from scipy.stats import norm >>> norm.cdf(2) 0.97724986805182079

If you check for the libraries loaded (I used export DYLD_PRINT_LIBRARIES=1), you'll see we're using Accelerate.

Finally, get PyCogent too..

$ git clone git://github.com/pycogent/pycogent.git $ cd pycogent $ python setup.py build $ sudo python setup.py install

That was pretty easy!

See ya

Posting has become much rarer here recently, as my energy has flagged.

And this morning I have a rather preemptory request that I fix the old links that were broken as a result of Apple's ditching iDisk. While I still have the files, somewhere, and I could theoretically put them on Dropbox, I'm really not ready to fix what must be several hundred links.

So I thought I would post this warning as what will probably be my last post. It's been fun, and I'm grateful to my readers for having taken an interest. We had more than 50,000 unique visitors and over 150 flags. I just wish someone from Tonga had come by.

If you really must have an old file, I can probably find it.

[ UPDATE: I made a zip of the directory tree which was used to be on iDisk under
http://web.mac.com/telliott99/Python/

It is on Dropbox here and weighs in at 17.8 MB. If you check the old links that don't work, the file should be in that archive. Let me know if you can't find something you'd like.

More fun with geometry

I ran into a fun problem with parallelograms the other day.

The graphic shows a parallelogram broken up into pieces by its diagonals.

Since the opposing sides of a parallelogram are parallel and equal, it's easy to show that the top and bottom triangles are congruent, just rotated, likewise with the other two.

The problem posed was to express the total area in terms of the obtuse angle at the center, the one which is > 90 degrees. Let's call that angle A, and its supplementary angle (the acute angle at the center) B.

The reason why I've drawn the figure in this way is that, as shown below, the triangles can be rearranged to give a different paralellogram with the same area.

x = length of long diagonal (orange + maroon)
y = length of short diagonal
y/2 = 1/2 length of short diagonal (blue)

Area = x (y/2) sin B

The sine values for supplementary angles are equal.

sin (π/2 - θ) = sin θ (ref)

So:

Area = x (y/2) sin A

Nice! Much better than using the law of sines, and for the vectorists out there, expressed even more compactly as the cross-product.

variadic arguments

I got curious about functions with variadic arguments like in NSArray:

+ (id)arrayWithObjects:(id)firstObj, ...


Here's are two C examples. In either case, it's required to have a named argument that precedes the variable part. In the first method, that variable gives a count of the arguments. In the second, it's not used for anything except to show where to start. A sentinel value shows the end of the variable length argument list. I also incorporated __func__, which I came across in my reading and was new to me.

// clang variadic.c -o prog #include <stdio.h> #include <stdarg.h> int f(int count, ...) { printf("%s%s", __func__, "\n"); int n, sum; va_list ap; va_start(ap, count); sum = 0; for ( ; count > 0; count--) { n = va_arg(ap, int); printf("n = %3d\n", n); sum += n; } va_end(ap); return sum; } int g(int x, ...) { printf("%s%s", __func__, "\n"); fflush(stdout); int n, sum; va_list ap; va_start(ap, x); n = x; sum = 0; while (!(0)) { printf("n = %3d\n", n); sum += n; n = va_arg(ap, int); if (n == 0) { break; } } va_end(ap); return sum; } int main(int argc, char * argv[]) { int result = f(3,1,2,3); printf ("sum = %d\n", result); result = g(1,2,3,0); printf ("sum = %d\n", result); return 0; }

> ./prog f n = 1 n = 2 n = 3 sum = 6 g n = 1 n = 2 n = 3 sum = 6

NSString (1)

Test harness:

// clang strings1.m -o prog -framework Foundation -fobjc-gc-only #include <Foundation/Foundation.h> int main(int argc, char * argv[]) { // ..code here.. return 0; }

NSString *s; NSMutableArray *ma = [NSMutableArray arrayWithCapacity:10]; char *cstr = "abc"; // real C string w/ "\0" also works uint ascii = NSASCIIStringEncoding; uint utf8 = NSUTF8StringEncoding; [ma addObject:[NSString stringWithCString:cstr encoding:ascii]]; [ma addObject:[NSString stringWithCString:cstr encoding:utf8]]; [ma addObject:[NSString stringWithUTF8String:cstr]]; NSMutableString *ms = [[NSMutableString alloc] init]; [ms appendString:@"abde"]; [ms insertString:@"c" atIndex:2]; [ma addObject:[ms copy]]; // otherwise code below will alter [ms appendString:@"fghij"]; [ms replaceOccurrencesOfString:@"B" withString:@"*" options:NSCaseInsensitiveSearch range:NSMakeRange(0,2) ]; [ma addObject:ms]; for (id obj in ma) { NSLog(@"%@ %@", obj, [obj class]); } [ma removeAllObjects]; printf("-----------------\n");

> ./prog 2012-05-29 16:45:03.624 prog[444:707] abc __NSCFString 2012-05-29 16:45:03.626 prog[444:707] abc __NSCFString 2012-05-29 16:45:03.627 prog[444:707] abc __NSCFString 2012-05-29 16:45:03.627 prog[444:707] abcde __NSCFString 2012-05-29 16:45:03.628 prog[444:707] a*cdefghij __NSCFString -----------------

[ma addObject:[NSString stringWithFormat:@"abcd"]]; [ma addObject:[NSString stringWithFormat:@"ab%@", @"cd" @"ef"]]; [ma addObject:[NSString stringWithFormat:@"a%C%C", 0x62, 0x63]]; char b[] = { 0x61, 0x62, 0x63, 0x64, 0x65 }; NSData *d = [NSData dataWithBytes:b length:5]; s = [[NSString alloc] initWithData:d encoding:utf8]; [ma addObject:s]; for (id obj in ma) { NSLog(@"%@ %@", obj, [obj class]); } printf("-----------------\n");

2012-05-29 16:45:03.629 prog[444:707] abcd __NSCFString 2012-05-29 16:45:03.629 prog[444:707] abcdef __NSCFString 2012-05-29 16:45:03.630 prog[444:707] abc __NSCFString 2012-05-29 16:45:03.630 prog[444:707] abcde __NSCFString -----------------

d = [NSData dataWithBytes:b length:5]; [d writeToFile:@"x" atomically:YES]; NSError *err = nil; NSStringEncoding enc = NSUTF8StringEncoding; s = [NSString stringWithContentsOfFile:@"x" usedEncoding:&enc error:&err]; if (err != nil) { NSLog(@"%@", [err userInfo]); } else { NSLog(@"%@", s ); } b[4] = 0xff; d = [NSData dataWithBytes:b length:5]; [d writeToFile:@"x" atomically:YES]; s = [NSString stringWithContentsOfFile:@"x" usedEncoding:&enc error:&err]; if (err != nil) { NSLog(@"%@", [err localizedDescription]); } else { NSLog(@"%@", s ); }

2012-05-29 16:45:03.641 prog[444:707] abcde 2012-05-29 16:45:03.658 prog[444:707] The file “x” couldn’t be opened because the text encoding of its contents can’t be determined.