matplotlib on OS X Lion--revised

In the previous post, I repeated an installation of matplotlib (now for OS X Lion) following the same method that I've linked to a number of times on the blog. It worked this time as well, but lead me to consideration of other possible methods. I found one on the web here. So that's what this post is about.

BTW, whatever you do, do not follow the instructions that the matplotlib developers provide. You do not need another Python, including MacPython or the Endthought distribution or anything else..

To recap, matplotlib lists dependencies (in the make.osx file) of: libpng, libfreetype, and zlib. I read somewhere on the matplotlib site today (but can't find the link now) that zlib is not a required dependency. These days, the other two are actually provided by Apple:

> ls -al /usr/X11/lib/libpng* -rwxr-xr-x 1 root wheel 296864 Jul 29 16:11 /usr/X11/lib/libpng.3.dylib lrwxr-xr-x 1 root wheel 14 Jul 29 16:11 /usr/X11/lib/libpng.dylib -> libpng15.dylib -rwxr-xr-x 1 root wheel 294160 Jul 29 16:11 /usr/X11/lib/libpng12.0.dylib -rwxr-xr-x 1 root wheel 305008 Jul 29 16:11 /usr/X11/lib/libpng15.15.dylib lrwxr-xr-x 1 root wheel 17 Jul 29 16:11 /usr/X11/lib/libpng15.dylib -> libpng15.15.dylib > ls -al /usr/X11/lib/libfreetype* -rwxr-xr-x 1 root wheel 1060416 Jul 29 16:11 /usr/X11/lib/libfreetype.6.dylib lrwxr-xr-x 1 root wheel 19 Jul 29 16:11 /usr/X11/lib/libfreetype.dylib -> libfreetype.6.dylib

I'm not sure at the moment whether X11 came with Xcode (as it used to) or was present in the original Lion install.

In any case, I spent an hour or two trying to figure out how to use the build commands from the makefile that comes with matplotlib, but point at these libraries. In the process I found what appears to be a bug:

/usr/X11/include/png.h:666: error: forward declaration of ‘struct png_info_def’

(and see Prashant's answer here), but we're going to use a different approach so it doesn't matter. I'm not sure how our method solved this in the end. As mentioned, the approach is to use Homebrew, in particular, something called pkgconfig. I used a Ruby script that downloads and installs Homebrew, as described here. The install gave this warning:

Warning: The following *evil* dylibs exist in /usr/local/lib They may break builds or worse. You should consider deleting them: /usr/local/lib/libfreetype.6.dylib /usr/local/lib/libpng.3.dylib /usr/local/lib/libpng12.0.dylib /usr/local/lib/libz.1.2.5.dylib

These are, of course, the libraries we just installed in order to get matplotlib to build.

That's what got me working on the other method more seriously. I don't think the danger is all that great, but clearly it is better to use libraries that are (i) still available from the maintainers and (ii) have been vetted at least to some extent by other folks including Apple. AFAIK there aren't any known security issues with the versions we installed previously. I ripped them out anyway (they seemed to confuse pkgconfig).

If we ask Homebrew for the libraries, it just points us to the ones we saw in /usr/X11:

> brew search libpng Apple distributes libpng with OS X, you can find it in /usr/X11/lib. However not all build scripts look here, so you may need to call ENV.x11 in your formula's install function. > brew search freetype Apple distributes freetype with OS X, you can find it in /usr/X11/lib. However not all build scripts look here, so you may need to call ENV.x11 in your formula's install function.

Following the instructions in the blog post I first installed pkgconfig:

> sudo brew install pkgconfig Cowardly refusing to `sudo brew' > brew install pkgconfig

!!

That's all the Homebrew we need. This is followed by:

cd ~/Desktop git clone git://github.com/matplotlib/matplotlib.git cd matplotlib/ python setup.py build sudo python setup.py install

And we can see that I really did overwrite the first matplotlib install, and that we're actually using the libraries from /usr/X11, by first doing

export DYLD_PRINT_LIBRARIES=1

Then when we run a script that imports matplotlib.pyplot, the Terminal shows (among much else) this:

dyld: loaded: /usr/X11/lib/libfreetype.6.dylib .. dyld: loaded: /usr/X11/lib/libpng15.15.dylib

So that's what I'd recommend and it seems to be working fine. This simple script works exactly as you'd expect.

import matplotlib.pyplot as plt Y = [1,4,9,16] plt.scatter(range(len(Y)),Y,s=250,color='r') plt.savefig('example.png')

So the next thing to do is to figure out pkgconfig and Homebrew work their magic!

P.S. You will still need to make and edit ~/.matplotlibas discussed last time.

matplotlib on OS X Lion--old

[ UPDATE: I'm going to leave this post up, for the record, but I found a better way, and that's in the following post. ]

Just a note to say that I got matplotlib installed with only a few issues to solve. As before, I relied on Gavin Huttley's instructions (here). Although the matplotlib instructions say you should install a different Python, I've found that just leads to confusion down the road, and I want to try to use the system version only if I can get away with it. YMMV.

The sticking points for the install always have to do with the basic dependencies, which are listed at the top of the file make.osx in the matplotlib source (from here). The source is the last link at the bottom.

ZLIBVERSION=1.2.3 PNGVERSION=1.2.39 FREETYPEVERSION=2.3.11

As silly as it seems, although OS X provides excellent functionality for dealing with compression, png images and fonts, we need to install these libraries for matplotlib to use. Unfortunately, I'm not expert enough to alter matplotlib to get rid of these dependencies, so we're stuck with it.

---

libpng

I got libpng-1.2.39 from here. I used this version b/c that's what matplotlib asks for, despite the fact that the current release is 1.5.4 (here). Standard magic incantations:

./configure make sudo make install

---

freetype

curl -O http://ftp.twaren.net/Unix/NonGNU/freetype/freetype-2.3.7.tar.bz2

According to Gavin's notes, we need these compiler flags for the other two (but they interfere with the libpng build):

export MACOSX_DEPLOYMENT_TARGET=10.7 export CFLAGS="-arch i386 -arch x86_64" export FFLAGS="-arch i386 -arch x86_64" ./configure make sudo make install

---

zlib

zlib is a compression library. The current version is 1.2.5. I tried this:

curl -O http://www.zlib.net/zlib-1.2.3.tar.gz

but as I've seen before, although you get a (small) file back that has the right icon, it is really a 404 Page Not Found, which doesn't do me any good. It seems the older versions have been disappeared off the web and I couldn't find them poking around on the site. What I should've done at this point is try the version they do offer, but instead I went to my other machine and grabbed 1.2.3. I guess that since I'd used it before to build zlib, I found I needed to clean it first:

make clean make sudo make install

---

matplotlib

Finally, I got the matplotlib source. I altered the file make.osx:

PREFIX=/usr/local PYVERSION=2.7 PYTHON=python${PYVERSION} ZLIBVERSION=1.2.3 PNGVERSION=1.2.39 FREETYPEVERSION=2.3.7 MACOSX_DEPLOYMENT_TARGET=10.7 OSX_SDK_VER=10.7 ARCH_FLAGS="-arch i386-arch x86_64"

make -f make.osx mpl_build sudo python setup.py install

mkdir ~/.matplotlib cp matplotlibrc.template ~/.matplotlib/matplotlibrc

Edit the above file to have:

backend : MacOSX

> python Python 2.7.1 (r271:86832, Jun 16 2011, 16:59:05) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from matplotlib import * >>>

Yes!

[ UPDATE: I see I used freetype 2.3.7 rather than 2.3.11, by mistake.

And if you don't like my approach, you could use Homebrew and follow this post. I don't have any experience with this yet. ]

OS X Lion

OS X Lion has been released for a week now, so I thought I'd try it out on one of my machines. A definitive review is here. I'll be too busy to really play with it for a while, or to do much programming, but I thought I should at least try. An additional motivation is that my laptop (an aluminum MacBook from about December 08), had developed an issue. It completely drains the battery (starting from full charge) in sleep mode, even when no apps are running. The battery hasn't been particularly stressed:

Health Information:
Cycle Count: 568
Condition: Normal
Battery Installed: Yes
Amperage (mA): 0
Voltage (mV): 12546

And, it doesn't drain when powered off. This problem appeared recently, and I am interested to see if installing Lion might fix it, but that hasn't been tested yet. I'll update when I have those results.

This post is just to document that I got Lion onto a USB flash drive and used it to do the install and it worked fine. Instructions from here. I did this because it's the only way to do a clean install. One odd thing: if the machine on which you are attempting to do the download can't run Lion, you won't be allowed to do the download at all! I had to bring my laptop to work (and get the IT guys to reauthorize it for our network).

Also, I have two Apple IDs---an old one and one that goes with my MobileMe account. When I set up Mail, the ID for the computer was automatically set to be the latter, but I originally did the Lion download with the former. We'll see if and when that causes a problem.

It seems clear that the Mac OS will (in time) become more iOS-like (locked down, user not allowed to risk hurting himself, even if he wants to) and less easy to hack around with. But for now, it's still a win over Linux, for me.

Python is 2.7:

> python Python 2.7.1 (r271:86832, Jun 16 2011, 16:59:05) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin Type "help", "copyright", "credits" or "license" for more information.

The default scrolling is like that on the iPhone---as if you're moving the content, rather than a scrollbar. After 2 or 3 hours, I changed the setting back to the old method. Maybe I will try to retrain my brain at a later date.

I also downloaded Xcode 4. It quit in the middle for no reason, but completed on the second try. Weirdly, in the middle of the install I got this alert panel:



I say "weirdly" because iTunes is not running. Starting and quitting iTunes had no effect, either. Finally I went in Terminal and looked for an iTunes-related process:

163 ?? 0:00.03 /Applications/iTunes.app/Contents/MacOS/iTunesHelper.app/Contents/MacOS/iTunesHelper -psn_0_57358

after I killed it, the install finished fine. There is a mention in the big Ars Technica review that apps don't actually quit (or perhaps not always), and may not show the little dot for an active app, but there was no process for iTunes so I don't think so. It's weird that a bug like that should still be present at this stage of the game.

And I like the new looks in Terminal:



[ UPDATE: Yep, the battery does not drain on sleep anymore. (Well, 2% in 2 hr). That's the good news. The bad news is that the extension I had to help autofill passwords when forms contain autocomplete="off" doesn't work with Safari any more (post here, extension here). I sure hope Apple didn't do that on purpose. ]

UPDATE 2: I just needed to check to the right box in the Prefs.

Verification

I posted two derivations for Euler's famous equation (here and here):

eix = cos x + i sin x

This can be verified in a particularly simple way.
The series representation of the exponential function:

ex = 1 + x + x2/2! + x3/3! + x4/4! + ..

is especially neat because each term in the series is the derivative of the term following, and the result of that is:

d/dx ex = ex

Which is, indeed, one definition of this function.
Substitution of ix leads to a simple shift in the pattern:

eix = 1 + ix - x2/2! - ix3/3! + x4/4! + ..

repeating with a period of 4. Powers of x like:

4n + 1 are multiplied by i 4n + 2 -1 4n + 3 -i 4n 1

But remembering the series for sine and cosine, and multiplying the former by i:

sin x = x - x3/3! + x5/5! - x7/7! + .. i sin x = ix - ix3/3! + .. cos x = 1 - x2/2! + x4/4! - x6/6! + ..

Adding:

cos x + i sin x = 1 + ix - x2/2! - ix3/3!+ x4/4! + .. = eix

Note on trig substitution

A fact we needed in a recent post deriving Euler's formula is the value of a particular integral:

∫ dx / √(x2 + 1) = ln (x + √(x2 + 1))

This gave me a lot more trouble than expected, so I thought I'd work through it here for reference. The easiest way is to differentiate the answer. Let:

u = x + √(x2 + 1) du = [1 + 2x / 2 √(x2 + 1)] dx = [1 + x / √(x2 + 1)] dx y = ln(u) dy = 1/u du = [1 / (x + √(x2 + 1)) ] [ 1 + x / (√(x2 + 1)] dx

The trick is to notice that when we find the common denominator for the part at the far right and then add the terms, we generate the denominator of the part to the left.

dy = [1 / (x + √(x2 + 1) ] [ (√(x2 + 1) + x) / (√(x2 + 1)] dx = 1 / (√(x2 + 1)

The integrand is normally written more generally as:

∫ 1 / √(x2 + a2)

and that doesn't change anything.

The forward version starts with a trigonometric substitution.

x = a tan y y = tan-1 (x/a)

Remembering that

(u/v)' = (vu' - uv')/v2 tan' = (cos2 + sin2)/cos2 = 1/cos2 = sec2

So

dx = a sec2 y dy

The other part of the integral is:

1 / √(x2 + a2) x = a tan y x2 = a2 tan2 y x2 + a2 = a2 tan2 y + a2 = a2 ( 1 + tan2 y)

If we start with the standard identity:

sin2 y + cos2 y = 1

and divide by cos2 y:

tan2 y + 1 = sec2 y x2 = a2 ( 1 + tan2 y) = a2 sec2 y

Thus the integral reduces to:

∫ (a sec2 y) / (a sec y) dy = ∫ sec y dy

This small integral is itself a bit tricky. The answer, which I found on a really nice site (here), is to multiply top and bottom by:

sec y + tan y sec y ------------- sec y + tan y Let u = sec y + tan y

If you work through it you'll see that

du = (sec y tan y + sec2 y) dy

Thus, this is

∫ du/u = ln(u) = ln (sec y + tan y) = ln [ ((√(x2 + a2) + x ) / a ]

As I said, not so easy as other trig substitutions.

Euler's Gem 2

Here is a sketch of a second derivation of Euler's famous formula:

eiθ = cosθ + i sinθ

as presented by William Dunham in his book Euler, The Master of Us All. First post here.

The first step is to recall a standard trigonometric substitution in calculus:

y = sin x x = sin-1 y √(1 - y2) = cos x

We're interested in the integral:

∫ dy / √(1 - y2)

Substituting with x we see that:

dy = cos x dx

And the integral is

∫ (1/cos x) cos x dx = ∫ dx = x x = ∫ dy / √(1 - y2)

Now Euler makes a complex change of variable:

y = iz x = ∫ dy / √(1 - y2) = ∫ i dz / √(1-(iz)2) = i ∫ dz / √(1 + z2) = i ln [√(1 + z2) + z]

The last step is another standard result from calculus which I will assume for the time being (more here).

Undo the substitution:

z = y/i = sin x / i z2 = -sin2 x √(1 + z2) = √(1 - sin2 x) = cos x x = i ln (cos x + sin x / i)

We will use two identities involving i:

u / i = - i u 1 / (cos u - i sin u) = (cos u + i sin u)

(For the second one, see the previous post). Now:

x = i ln (cos x + sin x / i) x = i ln (cos x - i sin x) ix = - ln (cos x - i sin x) = ln [ 1 / (cos x - i sin x) ] = ln (cos x + i sin x)

Just eponentiate:

eix = cos x + i sin x

Wow, again!

Euler's gem

Here is a sketch of the derivation of Euler's famous formula:

eiθ = cosθ + i sinθ

as presented by William Dunham in his book Euler, The Master of Us All.

The first part of the proof is similar to when we used Euler's formula to derive other formulas for trig functions of sums and differences of angles (post), only backward. Start from the definition of i:

i = √-1

To begin with, having i allows us to factor new expressions:

1 = cos2 s + sin2 s = (cos s + i sin s)(cos s - i sin s)

(I'm going to use s and t, as before, rather than θ and φ).

This shows where the original idea of cos + i sin comes from. (Of course, we could just as well do sin + i cos, that would result in a different convention for the orientation of the complex plane).

Suppose we have two angles s and t, we can multiply and then use the formulas from before (obtained by the geometric proof):

(cos s + i sin s)(cos t + i sin t) = = (cos s cos t - sin s sin t) + i(sin s cos t + cos s sin t) = cos(s + t) + i sin(s + t)

Set s = t:

(cos s + i sin s)2 = cos(2s) + i sin(2s)

In fact Euler showed it works for fractional n but I'll assume that part:

[1] (cos s + i sin s)n = cos(ns) + i sin(ns) n >= 1

If we multiply the difference rather than the sum:

(cos s - i sin s)(cos t - i sin t) = = (cos s cos t - sin s sin t) - i (sin s cos t + cos s sin t) = cos(s + t) - i sin(s + t)

Again, with s = t we have:

(cos s - i sin s)2 = cos(2s) - i sin(2s) [2] (cos s - i sin s)n = cos(ns) - i sin(ns)

Adding [1] and [2] we have:

2 cos(ns) = (cos s + i sin s)n + (cos s - i sin s)n

---

The middle part of the proof is where the magic happens. Let:

s = x/n

As

n -> ∞ s -> 0 cos s -> 1 sin s -> s

So..

cos x = cos ns = 1/2 [(cos s + i sin s)n + (cos s - i sin s)n] cos x = 1/2 [(1 + is)n + (1 - is)n] cos x = 1/2 [(1 + ix/n)n + (1 - ix/n)n]

But..

eix = (1 + ix/n)n as n -> ∞

So

cos x = 1/2 [eix + e-ix]

---

By very similar manipulation to what's in the first part we can also handle the sine:

2i sin(ns) = (cos s + i sin s)n - (cos s - i sin s)n

We will obtain:

sin x = 1/(2i) [eix - e-ix]

Now it's just a matter of addition:

cos x + i sin x = 1/2 [eix + e-ix + eix - e-ix] = 1/2 [eix + eix] = eix

Wow!