Showing posts with label matplotlib. Show all posts
Showing posts with label matplotlib. Show all posts

Sunday, March 25, 2012

matplotlib on 10.7.3

I upgraded my home "Desktop" (a Mac mini that had been running OS X Server 10.6) to Lion (10.7.3) yesterday. This post is just to note that the pkgconfig method still works for installing matplotlib. To recap, I used Homebrew to install pkgconfig, which can then be used by the matplotlib build process to find zlib, libpng and libfreetype. It's as simple as:

git clone git://github.com/matplotlib/matplotlib.git
cd matplotlib/
python setup.py build
sudo python setup.py install

There were a few hiccups, of course. The motivation for changing to 10.7 was that my MobileMe email (and iDisk storage) are going away at then end of June. So I "upgraded" to iCloud, after installing 10.7.3 on my laptop. However, this made Mail extremely sluggish on the machines with 10.6. Mail showed the little triangle (which I think means it's having trouble) for 15 minutes at a time and wouldn't get email from the server.

I had various troubles with the install that I believe relate to having bought Lion using my first Apple ID (which used my work email) rather than my second (which uses my .mac email). It irritated me, but seemed that the simplest solution was to just buy Lion again. So I did that, and everything started working fine.

Another issue was to get Xcode (Developers Tools) from the App Store. This is free, but involves various hoops to jump through, the weirdest of which is that gcc is no longer a part of the default install. You have to start Xcode and go to Preferences > Downloads > Command Line Tools. Why would they do that? Anyway, having gcc

> ls -al /usr/bin/gcc
lrwxr-xr-x  1 root  wheel  12 Mar 24 23:14 /usr/bin/gcc -> llvm-gcc-4.2

I used Homebrew to get pkgconfig, and then built matplotlib as shown above. Simple scripts work. I used easy_install nose (with sudo) to install nose, which is needed by matplotlib's tests, then did

python tests.py -v -d

from within the matplotlib directory. The result looks good:

Ran 1086 tests in 290.786s

OK (KNOWNFAIL=546)

Now I just have to re-install my favorite Safari extensions (forgot to save them), and find the license key for TextMate. I'm making it work, but all these hoops to jump through, and Apple's Big Brother mentality, are making me think seriously about switching to Linux for my hobbyist programming.

Sunday, July 31, 2011

matplotlib on OS X Lion--revised

In the previous post, I repeated an installation of matplotlib (now for OS X Lion) following the same method that I've linked to a number of times on the blog. It worked this time as well, but lead me to consideration of other possible methods. I found one on the web here. So that's what this post is about.

BTW, whatever you do, do not follow the instructions that the matplotlib developers provide. You do not need another Python, including MacPython or the Endthought distribution or anything else..

To recap, matplotlib lists dependencies (in the make.osx file) of: libpng, libfreetype, and zlib. I read somewhere on the matplotlib site today (but can't find the link now) that zlib is not a required dependency. These days, the other two are actually provided by Apple:


> ls -al /usr/X11/lib/libpng*
-rwxr-xr-x 1 root wheel 296864 Jul 29 16:11 /usr/X11/lib/libpng.3.dylib
lrwxr-xr-x 1 root wheel 14 Jul 29 16:11 /usr/X11/lib/libpng.dylib -> libpng15.dylib
-rwxr-xr-x 1 root wheel 294160 Jul 29 16:11 /usr/X11/lib/libpng12.0.dylib
-rwxr-xr-x 1 root wheel 305008 Jul 29 16:11 /usr/X11/lib/libpng15.15.dylib
lrwxr-xr-x 1 root wheel 17 Jul 29 16:11 /usr/X11/lib/libpng15.dylib -> libpng15.15.dylib

> ls -al /usr/X11/lib/libfreetype*
-rwxr-xr-x 1 root wheel 1060416 Jul 29 16:11 /usr/X11/lib/libfreetype.6.dylib
lrwxr-xr-x 1 root wheel 19 Jul 29 16:11 /usr/X11/lib/libfreetype.dylib -> libfreetype.6.dylib


I'm not sure at the moment whether X11 came with Xcode (as it used to) or was present in the original Lion install.

In any case, I spent an hour or two trying to figure out how to use the build commands from the makefile that comes with matplotlib, but point at these libraries. In the process I found what appears to be a bug:


/usr/X11/include/png.h:666: error: forward declaration of ‘struct png_info_def’


(and see Prashant's answer here), but we're going to use a different approach so it doesn't matter. I'm not sure how our method solved this in the end. As mentioned, the approach is to use Homebrew, in particular, something called pkgconfig. I used a Ruby script that downloads and installs Homebrew, as described here. The install gave this warning:


Warning: The following *evil* dylibs exist in /usr/local/lib
They may break builds or worse. You should consider deleting them:
/usr/local/lib/libfreetype.6.dylib
/usr/local/lib/libpng.3.dylib
/usr/local/lib/libpng12.0.dylib
/usr/local/lib/libz.1.2.5.dylib


These are, of course, the libraries we just installed in order to get matplotlib to build.

That's what got me working on the other method more seriously. I don't think the danger is all that great, but clearly it is better to use libraries that are (i) still available from the maintainers and (ii) have been vetted at least to some extent by other folks including Apple. AFAIK there aren't any known security issues with the versions we installed previously. I ripped them out anyway (they seemed to confuse pkgconfig).

If we ask Homebrew for the libraries, it just points us to the ones we saw in /usr/X11:


> brew search libpng
Apple distributes libpng with OS X, you can find it in /usr/X11/lib.
However not all build scripts look here, so you may need to call ENV.x11
in your formula's install function.

> brew search freetype
Apple distributes freetype with OS X, you can find it in /usr/X11/lib.
However not all build scripts look here, so you may need to call ENV.x11
in your formula's install function.


Following the instructions in the blog post I first installed pkgconfig:


> sudo brew install pkgconfig
Cowardly refusing to `sudo brew'
> brew install pkgconfig

!!

That's all the Homebrew we need. This is followed by:


cd ~/Desktop
git clone git://github.com/matplotlib/matplotlib.git
cd matplotlib/
python setup.py build
sudo python setup.py install


And we can see that I really did overwrite the first matplotlib install, and that we're actually using the libraries from /usr/X11, by first doing


export DYLD_PRINT_LIBRARIES=1


Then when we run a script that imports matplotlib.pyplot, the Terminal shows (among much else) this:


dyld: loaded: /usr/X11/lib/libfreetype.6.dylib
..
dyld: loaded: /usr/X11/lib/libpng15.15.dylib


So that's what I'd recommend and it seems to be working fine. This simple script works exactly as you'd expect.


import matplotlib.pyplot as plt
Y = [1,4,9,16]
plt.scatter(range(len(Y)),Y,s=250,color='r')
plt.savefig('example.png')


So the next thing to do is to figure out pkgconfig and Homebrew work their magic!

P.S. You will still need to make and edit ~/.matplotlibas discussed last time.

matplotlib on OS X Lion--old

[ UPDATE: I'm going to leave this post up, for the record, but I found a better way, and that's in the following post. ]

Just a note to say that I got matplotlib installed with only a few issues to solve. As before, I relied on Gavin Huttley's instructions (here). Although the matplotlib instructions say you should install a different Python, I've found that just leads to confusion down the road, and I want to try to use the system version only if I can get away with it. YMMV.

The sticking points for the install always have to do with the basic dependencies, which are listed at the top of the file make.osx in the matplotlib source (from here). The source is the last link at the bottom.


ZLIBVERSION=1.2.3
PNGVERSION=1.2.39
FREETYPEVERSION=2.3.11

As silly as it seems, although OS X provides excellent functionality for dealing with compression, png images and fonts, we need to install these libraries for matplotlib to use. Unfortunately, I'm not expert enough to alter matplotlib to get rid of these dependencies, so we're stuck with it.



libpng

I got libpng-1.2.39 from here. I used this version b/c that's what matplotlib asks for, despite the fact that the current release is 1.5.4 (here). Standard magic incantations:


./configure
make
sudo make install




freetype


curl -O http://ftp.twaren.net/Unix/NonGNU/freetype/freetype-2.3.7.tar.bz2


According to Gavin's notes, we need these compiler flags for the other two (but they interfere with the libpng build):


export MACOSX_DEPLOYMENT_TARGET=10.7
export CFLAGS="-arch i386 -arch x86_64"
export FFLAGS="-arch i386 -arch x86_64"

./configure
make
sudo make install




zlib

zlib is a compression library. The current version is 1.2.5. I tried this:


curl -O http://www.zlib.net/zlib-1.2.3.tar.gz


but as I've seen before, although you get a (small) file back that has the right icon, it is really a 404 Page Not Found, which doesn't do me any good. It seems the older versions have been disappeared off the web and I couldn't find them poking around on the site. What I should've done at this point is try the version they do offer, but instead I went to my other machine and grabbed 1.2.3. I guess that since I'd used it before to build zlib, I found I needed to clean it first:


make clean
make
sudo make install




matplotlib

Finally, I got the matplotlib source. I altered the file make.osx:


PREFIX=/usr/local
PYVERSION=2.7
PYTHON=python${PYVERSION}
ZLIBVERSION=1.2.3
PNGVERSION=1.2.39
FREETYPEVERSION=2.3.7
MACOSX_DEPLOYMENT_TARGET=10.7
OSX_SDK_VER=10.7
ARCH_FLAGS="-arch i386-arch x86_64"



make -f make.osx mpl_build
sudo python setup.py install



mkdir ~/.matplotlib
cp matplotlibrc.template ~/.matplotlib/matplotlibrc


Edit the above file to have:


backend      : MacOSX



> python
Python 2.7.1 (r271:86832, Jun 16 2011, 16:59:05)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from matplotlib import *
>>>


Yes!

[ UPDATE: I see I used freetype 2.3.7 rather than 2.3.11, by mistake.

And if you don't like my approach, you could use Homebrew and follow this post. I don't have any experience with this yet. ]

Saturday, March 12, 2011

Dental project (5)



This post is one of a series (see dental project here or in the sidebar).

Last time I said I would show you how I make heatmaps these days. I've approached it several different ways over the past few years (R, Cocoa, matplotlib), but I think now that matplotlib is best, at least for me. Ultimately what I want is flexibility, and if you're a Python coder and you have matplotlib installed (as we've also discussed many times), then you'll have that. But I don't want to get into the technical details---and actually the script is a bit long, so I just put it (Heatmapper.py and its helper Preprocessor.py) into the zipped project files on Dropbox (here). The output from two different modes is at the bottom of the post. You just need a file data.csv in the same directory. It looks a little fuzzy and not as clean as I would like, but that's because there are so many samples, and partly because of the italic font. If you do savefig to a pdf file, and then blow it up, it looks great.

In this post I want to talk in a general way about the project and what I think it means. It began about four years ago, when we became aware that some folks in Dentistry at our school (WVU) were involved in a huge study of people from Appalachia (it's called COHRA). Poor oral health is a particular problem in West Virginia, and this study had collected thousands of samples along with patient histories and lots of clinical data. My belief is that the important thing about these samples is that the patients are young yet have serious periodontal issues. In any event, we convinced the people who actually run the project (based elsewhere) to let us have (a small part of) 8 samples out of all their thousands sitting in the freezers down the hall.

We did PCR with "universal" primers for the bacterial 16S rRNA gene, and cloned and sequenced the numbers you see in the table. It's not a big study (we don't have much money anyway), but we saw something which I think is truly significant. In high disease individuals, a broad group of microbes from the Clostridiales including an unusual clade called the Veillonellaceae are increased in abundance, whereas the sequences from control individuals in this clade were all very closely related to Veillonella parvula.

One reason this observation may be important is that the so-called "red complex", which is thought to be associated with serious periodontal disease, can only be recovered in about half the individuals with this diagnosis (not even considering abundance).

That story is in the modified version of the map above, where I drew a red box around the region of interest for the three controls, or "low disease" samples. Time went on, and another set of samples was added to the study from a different group, and we were able to get the work published. So that's why the study looks so old-fashioned, in an era of millions of reads, we've got about a thousand.

My role in all this was to actually do the analysis. I remember "we" wrote a grant (actually, someone else did!) and listed me as a technical expert in bioinformatics. Of course, the reviews were scathing. Dr. E doesn't have a degree in bioinfomatics. How could he know anything?

Well, I've learned a few things over the years. Rule one is, never make your own database: let someone else do it. That's why HOMD (and Greengenes and RDP) are so great. I particularly like the tools at the RDP site. It is very nice software.

And rule two is, if you live long enough, you will see work that took you months or years to accomplish be achieved using new tools in mere seconds or hours. Sequencing is a great example of this. When I was young I spent most of three months getting 500 bp; when I was a bit older I invested six months for 3.5 kb; still later it was a year for 20 kb.

This project is another example. I spent a year and more writing some 50 or so Python scripts (and rewriting them), and now QIIME does the whole thing in mere seconds.

Well, not quite the whole thing. I have a bit more to do with this project. I want to show you the UniFrac analysis of beta diversity, and show how to make what I think is a nicer plot of the PCoA results. Also, I want to show some phylogenetic trees detailing the increased diversity (species richness, really) in the Veillonellaceae that I mentioned.

And I should say: it's been fun. Even if I don't have that degree, or any papers with Rob Knight, I think I've learned something about Bioinformatics in the last 5 years.




Thursday, February 10, 2011

A plot of the quadratic in Python


I have another simple matplotlib example, based on the parabola post from the other day (here). The conclusion from that post was that any function:


y(x) = ax2 + bx + c


can be manipulated to the form:


y - y0 = a(x - x0)2


That is, the parabola is just y = a(x)2 translated to have a different vertex. I didn't do a systematic examination or anything, but here is one example in Python.

The function quadratic takes arguments a, b, and c and returns the vertex (x0, y0) as well as a numpy array containing x such that |x - x0| <= 4, and a second one containing f(x) for each value of x.

We plot two sets of three parabolas, each set has one for each of a = 1, 2 and 3. One set is at the origin (cyan, blue and purple). The second set (magenta, salmon and red) has c = 3 and:


a = 1;  b = 2
a = 2; b = 4
a = 3; b = 6


Since x0 = -b/2a is the same for each parabola in the second set, they all have the same axis of symmetry. The only difference (besides the shape parameter a) is y0, which can be calculated either from plugging a, b, c and x0 into the standard form, or by using the fact that


y0 = b2/4a - b2/2a + c
= -b2/4a + c


Using the second method, I get:


a = 1;  b = 2;  y0 = -1 + 3 = 2
a = 2; b = 4; y0 = -2 + 3 = 1
a = 3; b = 6; y0 = -3 + 3 = 0


We ought to be able to solve for b and c to put the parabola anywhere on the x,y-plane..

UPDATE:


x0 = -b/2a
b = -2ax0

y0 = -b2/4a + c
c = y0 + b2/4a
= y0 + ax02

check:

y = ax2 + bx + c
= ax2 -2ax0x + y0 + ax02
y - y0 = a(x2 -2x0x + x02)
= a(x - x0)2




from __future__ import division
import matplotlib.pyplot as plt
import numpy as np

def func(a,b,c):
@np.vectorize
def f(x):
return a*x**2 + b*x + c
return f

def quadratic(a,b=0,c=0):
f = func(a,b,c)
x0 = -b/(2*a)
X = np.arange(x0-4,x0+4,0.01)
return x0,f(x0),X,f(X)

blues = ['cyan', 'blue', 'purple']
reds = ['magenta', 'salmon', 'red']

for i,(a,b) in enumerate(zip((1,2,3),(2,4,6))):
x0,y0,X,Y = quadratic(a,b,c=3)
c = reds[i]
plt.scatter(X,Y,s=2,color=c)
plt.scatter(x0,y0,s=100,color=c)
print 'x0',x0,'y0',y0

x0,y0,X,Y = quadratic(a)
c = blues[i]
plt.scatter(X,Y,s=2,color=c)
plt.scatter(x0,y0,s=100,color='k')

ax = plt.axes()
ax.set_xlim(-3,3)
ax.set_ylim(-1,5)
plt.grid(True)
plt.savefig('example.png')

Tuesday, February 8, 2011

Plotting the Taylor Series for sine

One day about a year ago I was in a chain bookstore killing time, not really expecting to find much, and I came across a Calculus textbook which has become one of my favorites. The author is Morris Kline, and the title, Calculus, An intuitive and Physical Approach, tells most of the story. It's a minor miracle that the book was in the store---it had obviously been there for years, but was too big for them to discard, or something.

Anyway, I was browsing Kline's section on Taylor Series last night and it made me think of a little project. Before I get to that, if you don't know, the Taylor Series for sin(x), cos(x) and ex can be used to derive Euler's famous formula, which we touched on briefly a while back (here and here).

The project: there is a terrific graphic in the wikipedia article showing what a good approximation one can get with only a few terms from the series. So that's our goal for today, to reproduce the plot using matplotlib.

The Taylor series for f(x) when x is near a:


f(a) + f1(a)/1!(x-a) + f2(a)/2!(x-a)2 + .. 


where fn is the nth derivative of f

If a = 0 this is the Maclaurin series


f(0) + f1(0)/1!(x) + f2(0)/2!(x)2 + ..


For sin(0) and cos(0) the sequential derivatives form a repeating series (with period equal to 4) and every other term equal to 0. If f(x) is sin(x):


sin(0) + cos(0)/1!(x) - sin(0)/2!(x)2 - cos(0)/3!(x)3
+ sin(0)4!(x)4 + ..


All the sin(0) terms are equal to 0, and cos(0) is equal to 1:


  = x - x3/3! + x5/5! - x7/7! ..


To make the plot, we define a "factory" function that returns functions for the various degrees:


def func(n):
@np.vectorize
def f(x):
v = x**n / factorial(n)
if n % 4 == 3:
v *= -1
return v
return f


which we will call with odd values of n. Then the only trick to the plot is to accumulate the sum. We plot the sine curve last so that it lies on top of the others. Sure enough, with 8 terms we can cover roughly -2π < x < 2π.




from __future__ import division
from math import pi, factorial
import matplotlib.pyplot as plt
import numpy as np

X = np.arange(-3*pi, 3*pi, 0.001)

def func(n):
@np.vectorize
def f(x):
v = x**n / factorial(n)
if n % 4 == 3:
v *= -1
return v
return f

cL = ['r','g','b','purple','cyan',
'magenta','steelblue','0.6']
D = dict()

for i in range(1,16,2):
c = cL.pop(0)
D[i] = func(i)(X)
j = i-2
if j > 0:
D[i] += D[j]
plt.scatter(X,D[i],color=c,s=2)

plt.scatter(X,np.sin(X),color='k',s=2)

ax = plt.axes()
ax.set_xlim(-10,10)
ax.set_ylim(-10,10)
plt.grid(True)
plt.savefig('taylor.png')

Sunday, January 9, 2011

Python for PyObjC

This is a short post to document that I was able to download the Python.org's Framework build of Python 2.7 and then install matplotlib in it, as well as modify a Cocoa-Python application to link against that Framework. The only thing that looks a little shaky is the PyObjC install, but it still seems like it's working. Not every detail is given, but I hope it's enough.

The installer comes from here for python-2.7.1-macosx10.6.dmg (Mac Installer disk image (2.7.1) for OS X 10.6 and later).

I did a stock install except I unchecked Shell profile updater under Custom.. Not sure why, except I didn't want to spend time figuring out what it does.

Python.org installs stuff in


/Library/Frameworks/Pythonframework
/Applications
/usr/local/bin


I added this to .bash_profile


> export PATH=/$PATH:/Library/Frameworks/Python.framework/Versions/2.7/bin


I anticipated that easy_install would be difficult, but I got (something like) it from distribute:


> curl -O http://python-distribute.org/distribute_setup.py
> /usr/local/bin/python distribute_setup.py


I couldn't find any decent docs, though. It looks like they've just copied the easy_install documentation which I found impossible to figure out so far. But, poking around, I found easy_install and easy_install-2.7 in bin:


> which easy_install-2.7
/Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install-2.7


The way my $PATH is set up, easy_install runs the wrong Python. That's OK, we just do:


> easy_install-2.7 -U numpy
> easy_install-2.7 pyobjc==2.2
> easy_install-2.7 -U pip


pyobjc has lots of warnings but looks OK


> /usr/local/bin/python 
Python 2.7.1 (r271:86882M, Nov 30 2010, 10:35:34)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.5.1'
>>> import objc
>>> objc.__version__
'2.2'
>>> from Foundation import *
>>>
[1]+ Stopped /usr/local/bin/python


Next, I installed matplotlib using instructions from Gavin Huttley (here, here). Since matplotlib-1.0.0 is available, I went for it. In the make.osx file:


PYVERSION=2.7
PYTHON=python${PYVERSION}
ZLIBVERSION=1.2.3
PNGVERSION=1.2.39
FREETYPEVERSION=2.3.11
MACOSX_DEPLOYMENT_TARGET=10.6
OSX_SDK_VER=10.6
ARCH_FLAGS="-arch i386-arch x86_64"


I changed it to Python 2.7.

I also needed to update to libpng-1.2.39 and libfreetype-2.3.11 (although these are not the latest versions.. In fact, I had a bit of trouble with libpng since I got back a file that was really html, though titled as .gz or .bz2 or .xz. I finally found it here). For the other one:


> FREETYPEVERSION=2.3.11
> curl -O http://ftp.twaren.net/Unix/NonGNU/freetype/freetype-{$FREETYPEVERSION}.tar.bz2


As I said, the build instructions were as given at the link.

[UPDATE: Poking around in libpng file INSTALL, I notice that it says: "Before installing libpng, you must first install zlib, if it is not already on your system." So I'll do it in that order next time.]

I tested matplotlib by running the script from here.

Then, I got PyCogent (download link):


> cd PyCogent-1.5
> /usr/local/bin/python setup.py build
> sudo /usr/local/bin/python setup.py install
>>> from cogent import *
>>>
[2]+ Stopped /usr/local/bin/python



> cd tests
> /usr/local/bin/python alltests.py "$@"


I had a few failures:


Ran 3602 tests in 184.766s

FAILED (failures=12, errors=4)


Some of these are related to executables not present. Looks pretty good. Finally, let's setup a new Xcode project:

Xcode > Cocoa-Python application

change SDK to 10.6
delete Python.framework
drag in new Python.framework
under Targets > X > Link Binary ..
drag in new Python.framework


Since our Python is a universal binary with 64-bit:


> /usr/local/bin/python
Python 2.7.1 (r271:86882M, Nov 30 2010, 10:35:34)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; print hex(sys.maxint)
0x7fffffffffffffff
[2]+ Stopped /usr/local/bin/python
> /usr/local/bin/python -c "import struct; print struct.calcsize('P')"
8
> /usr/local/bin/python -c "import platform; print platform.architecture()"
('64bit', '')


Build the Xcode project as 64-bit as well. Add this to main.py:


import sys
print sys.version
print hex(sys.maxint)


And run it from the Console:


2.7.1 (r271:86882M, Nov 30 2010, 10:35:34) 
[GCC 4.2.1 (Apple Inc. build 5664)]
0x7fffffffffffffff

It works!

Sunday, December 19, 2010

Cycle fun


 157 156 155 154 153 152 151 150 149 148 147 146 145
158 111 110 109 108 107 106 105 104 103 102 101 144
159 112 73 72 71 70 69 68 67 66 65 100 143
160 113 74 43 42 41 40 39 38 37 64 99 142
161 114 75 44 21 20 19 18 17 36 63 98 141
162 115 76 45 22 7 6 5 16 35 62 97 140
163 116 77 46 23 8 1 4 15 34 61 96 139
164 117 78 47 24 9 2 3 14 33 60 95 138
165 118 79 48 25 10 11 12 13 32 59 94 137
166 119 80 49 26 27 28 29 30 31 58 93 136
167 120 81 50 51 52 53 54 55 56 57 92 135
168 121 82 83 84 85 86 87 88 89 90 91 134
169 122 123 124 125 126 127 128 129 130 131 132 133


I've forgotten where I saw this---perhaps it was a Project Euler problem. In any case, you can see the pattern. It's a nice (fairly simple) exercise in Python to construct a list of lists and the functions to modify it. The great thing is that once you have the inner 3 x 3 circle correct, the rest is automatic. The code for this is in cycle.py.

The real reason I'm posting about it is that it makes a fun plotting project for matplotlib, especially in combination with the prime number generator that we worked up the other day (in primes.py).

The graphic at the top of the post was obtained by modifying the array to hold n % 10 for each value, and using the matplotlib color map called "winter." I think for that one I had n = 50, so that's a 101 x 101 grid.

The prime example :) has n = 40 and uses the 'autumn' colormap. That's the one that script.py is set up to generate at the moment. Is there a pattern or is it just random?

[UPDATE: I did a bit of testing. In tracing the results for the prime plot using a small n, I discovered that matplotlib has shown the matrix in a different way than I printed it at the top of the post. Their "heatmap" goes vertically for the first step and cycles clockwise; it is both a mirror image and rotated 180 degrees from the way I did it. Sorry. But, now that I'm done, I think I can just leave it that way... ]

Zipped project files on Dropbox (here).

Thursday, December 16, 2010

Exploring mitochondrial genomes 2



Continuing with the problem from the last post (here), we want to construct a plot similar to what the ape book has on p. 60 (Fig 3.5), which shows the nucleotide composition of the mitochondrial genes from the species we're looking at. The code shown below is in the same script with the listing from last time, and depends on it.

In the first part, we organize the sequence data into a dict keyed by gene name, where the value is a list of all the sequences for that gene in the data. In the second part we go through that dict, pull out the sequences for each of our target genes and count all the A, C, G, T. The data structure is a nested one: a dict of dicts. The third part is somewhat more complicated. We want to plot not the total counts for each nt, but the fraction of each. The first line does the calculation:

    values = [nD[nt]*1.0/S for nt in 'atcg']
L = [sum(values[:j]) for j in range(1,5)]
for z,c,f in zip([4,3,2,1],colors,L):
b = plt.bar(i+1,f,width=1,color=c,zorder=z)

I chose the ATCG because then we can see at a glance what the GC content is for each gene. If we wanted to get slightly fancy we could sort on that. We're going to do a bar plot, so the value for each bar is not its fraction, but its fraction plus the sum of the fractions of all the nucleotides plotted previously to it. The plt.bar call looks a little convoluted: the purpose is to plot the bars in the correct z-order. In reality, we plot the bars one on top of the other, but we don't see what's underneath. We also tweak matplotlib into plotting the gene names as the tick labels.

The figure could use a key relating the colors to the nucleotides, but I'll leave that as an exercise for the reader.

Code listing:

D = dict()
targets = ['ATP6','ATP8','COX1','COX2','COX3',
'CYTB','ND1','ND2','ND3','ND4','ND4L',
'ND5','ND6']
for gene in targets: D[gene] = []

for item in rest.strip().split('\n\n'):
if 'Sequence does not exist' in item:
continue
title,seq = item.strip().split('\n',1)
i = title.find('(')
j = title.find(')')
gene = title[i+1:j]
D[gene].append(seq)
#---------------------------------

rD = dict()
for k in targets:
nD = dict(zip('acgt',[0]*4))
for seq in D[k]:
for nt in 'acgt':
nD[nt] += seq.count(nt)
rD[k] = nD
#---------------------------------

colors = ['0.2','0.4','0.6','0.8']
iL = list()
for i,k in enumerate(targets):
iL.append(i+1)
nD = rD[k]
S = sum(nD.values())
values = [nD[nt]*1.0/S for nt in 'acgt']
L = [sum(values[:j]) for j in range(1,5)]
for z,c,f in zip([4,3,2,1],colors,L):
b = plt.bar(i+1,f,width=1,color=c,zorder=z)

ax = plt.axes()
ax.set_xlim(1,len(targets)+1 )
ax.xaxis.set_ticks([i+0.5 for i in iL])
ax.xaxis.set_ticklabels(targets)
plt.savefig('example.png')

Saturday, December 4, 2010

Color lists, again


It's common to want lists of colors for plotting, for heat maps or the RNA secondary structure plot from the other day (here). If you're doing this for a real project, I'd recommend matplotlib's LinearSegmentedColormap, which looks complicated but isn't hard to understand---I posted about it here.

Another option is to get the colors from R, either by getting R to write them to a file on disk, or using RPy. I posted about the former method before too (here). RColorBrewer has some great color schemes.

Finally, we can make our own list. It seems like a nice problem that exercises simple Python skills. The chief goal is flexibility. The core function in the code below assembles a list of tuples of rgb values, where each value is either a constant for the current series, or drawn from the hex values in order (either 00 to FF or reversed FF to 00).

The function's signature is:
series(r=None,g=None,b=None,rev='')

If a value is given for r, g, or b, that's the constant to use. Otherwise, we grab the list. For each of 'r','g', or 'b' that is contained in rev, we reverse the list. The second function simply assembles the color list. To show it in action, we make a list with 5 components: purple to blue, blue to cyan, cyan to green, green to yellow and yellow to red. The results are plotted with matplotlib.

code listing:

import matplotlib.pyplot as plt
import numpy as np

values = '0123456789ABCDEF'
hL = [m+n for m in values for n in values]
rhL = hL[:]
rhL.reverse()
N = len(hL)

# if r,g, or b are given, they are constants
# otherwise we copy hL or rhL
def series(r=None,g=None,b=None,rev=''):
L = list()
for name,c in zip('rgb',[r,g,b]):
if c:
L.append([c] * N)
else:
if name in rev:
temp = rhL[:]
else:
temp = hL[:]
L.append(temp)
return compose_rgb(L)

def compose_rgb(L):
rL = list()
for r,g,b in zip(L[0],L[1],L[2]):
rL.append(''.join(['#',r,g,b]))
return rL

# r decreasing -g +b
cL = series(g='00',b='FF',rev='r')
# -r g increasing +b
cL.extend(series(r='00',b='FF'))
# -r +g b decreasing
cL.extend(series(r='00',g='FF',rev='b'))
# r increasing +g -b
cL.extend(series(g='FF',b='00'))
# +r g decreasing -b
cL.extend(series(r='FF',b='00',rev='g'))

# take only every 8th value, 5 x 32
L = [cL[i] for i in range(0,1280,8)]
xL = range(1,33) * 5
yL = list()
for j in range(5):
yL.extend([j]*32)
plt.scatter(xL,yL,c=L,marker='s',s=200)
plt.savefig('example.png')

Sunday, November 28, 2010

Plotting the transition bias in evolution


I saw this graphic in Page & Holmes and thought it was really nice (kind of reminds me of the old letter counts script in the analysis of Ulysses (here), and I thought it would be fun to try to recreate it. Today we're going to do the left-hand plot, the one that originates from the alignment of the human and chimpanzee mitochondrial DNA sequences.

First we need the sequences of the human and chimpanzee mitochondrial genomes. Go to the NCBI page and get: Homo sapiens (NC_012920 - 16569 nt) and Pan troglodytes (NC_001643 - 16554 nt).

For reasons that surpass understanding, the two are not in synch. This can be seen from a pairwise BLAST, or by looking at the gene layouts after following the links from the main mitochondrial genomes page:

Homo    ND1     3307-4262
Pan ND1 2725-3681

Homo looks like it has about 582 nt extra on the front side, which matches reasonably well with the extra '-' inserted to the Pan sequence observed from a preliminary alignment run with MUSCLE. I saved that run of '-' to a file and counted them:

>>> FH = open('extra.txt','r')
>>> data = FH.read()
>>> data.count('-')
576

So then I looked at the very beginning of the Pan sequence in the alignment, and searched for the same 10-mer sequence (GTTTATGTAG) in human at about the right place. To check, I pasted the preceeding human sequence into a file and measured its length:

>>> FH = open('extra.txt','r')
>>> data = FH.read()
>>> data = data.replace('\n','')
>>> data
'GATCACAGGTCTATCA...
>>> len(data)
576

Looks good. Just cut that from the front of human and paste it at the end of the original sequence. Then rerun the alignment:

$ muscle -in homo.pan.mito.txt -out align.txt

No extra '-' to be seen on the front or back of the sequence after alignment. Overall, there are a few gaps, not many.

>>> FH = open('align.txt','r')
>>> data = FH.read()
>>> homo,pan = data.split('>Pan')
>>> homo.count('-')
6
>>> pan.count('-')
21

The next step is to count the number and kind of the differences. We save a count for each pair in a dict. In the run shown here, I consolidated 'A','C' and 'C','A' (for example) as one entry. However, I used the actual values (that is A-C different than C-A) for the plot.

('-', 'A') 6
('-', 'C') 9
('-', 'G') 2
('-', 'N') 1
('-', 'T') 9
('A', 'A') 4871
('A', 'C') 63
('A', 'G') 420
('A', 'T') 47
('C', 'C') 4645
('C', 'G') 14
('C', 'T') 904
('G', 'G') 1930
('G', 'T') 6
('T', 'T') 3648
--------------------
16575 total
15094 identical
1324 transition
130 transversion
27 gap
0.0879
TC.CT.CT.TC.TC.TC.AG.CT.CT.AG.TC.CT.CT.CT.GA.TC.CT.TC.CT.GA
AC.TA.AC.AC.AT.CA.AC.TA.AT.GC.CA.CA.TA.GT.AT.TA.CA.TA.AT.TA

I printed a few of the transitions and transversions as a sanity check.

The plot uses scatter as we've done many times before, and I added the labels by hand. The values are log-scaled to make the smaller ones more visible. The file is on Dropbox (here).

We don't really match the original very well. It shows either C-T or T-C as larger than C-C. The plot doesn't show that, and neither do the actual counts. I don't think it can be a problem with the alignment, since the sequences are so close. Guess I'll have to track down the original reference to know why.


Wednesday, November 17, 2010

simple text in matplotlib



This post is a "note to myself" about fonts and text using matplotlib. It can be hard to remember things like what is a FontProperties object and what kind of object is a font_dict, so here is a little example. It shows how to get (1) the default font with italics, (2) a default font in a particular "family"---sans-serif, (3) and (4) two named fonts and (5) a font loaded from a particular .ttf file. Notice that in (4) we do not observe the desired response to changing the 'weight'---I'm not sure why yet.

The second figure shows a change in the font using rc. It looks nice, takes several seconds to draw, and does not give the desired italics. They're all here as a reference. Perhaps they'll be of some use to you as well.


output:


:family=Arial:style=normal:variant=normal:weight=normal:stretch=normal:file=/Library/Fonts/Arial Italic.ttf:size=12.0



import matplotlib.pyplot as plt
import matplotlib.font_manager as fm

# can take a 'fontdict', Python dictionary
x,y = 0.1,0.1
fd = { 'fontsize':24, 'style':'italic',
'va':'center', 'ha':'left' }
plt.text(x,y,'1. text ' + 'default',**fd)

fd['family'] = 'sans-serif'
y += 0.1
plt.text(x,y,'1. text ' + 'sans-serif',**fd)

L = ['Arial','Helvetica']
for i,n in enumerate(L):
fd['fontname'] = n
y += 0.1
plt.text(x,y,str(i+2) + '. text ' + n,**fd)

fd['weight'] = 1000
y += 0.1
plt.text(x,y,'4. text--not properly bolded',**fd)

fp = fm.FontProperties(family='Arial',
fname='/Library/Fonts/Arial Italic.ttf')
print fp
y += 0.1
plt.text(x,y,'5. text Arial--named',
fontsize=24,
fontproperties=fp)

ax = plt.axes()
ax.xaxis.set_visible(False)
ax.yaxis.set_visible(False)
plt.savefig('example.png')



from matplotlib import rc
import matplotlib.pyplot as plt

x,y = 0.1,0.1

rc('font',**{'family':'sans-serif',
'sans-serif':['Helvetica'],
'style':'italic',
'size':24 })
rc('text', usetex=True)
y += 0.1
plt.text(x,y,'text using rc--not properly italic')


ax = plt.axes()
ax.xaxis.set_visible(False)
ax.yaxis.set_visible(False)
plt.savefig('example.png')

Sunday, November 14, 2010

New plotter for phylogenetic trees: version 0.1


This is the last post on this project. The previous posts should be listed in the archive in the sidebar.

I have a phylogenetics project for which I constructed a boutique database. The definition is in the file db.groups.txt, one section of which looks like this:

aeromonas
X60415.1 Aeromonas_trota_ATCC49657
X74677.1 Aeromonas_hydrophila_ATCC7966T
EU770300.1 Aeromonas_enteropelogenes_MS12

set off from surrounding entries by double newlines. The "group" aeromonas has three sequences. There are also super-groups that correspond to bacterial Phyla or sub-Phyla, e.g., the gamma-Proteobacteria. These are defined in the file big.groups.txt like this:

gamma
aeromonas
cardio
pseudo
moraxella
entero
haemo1
haemo2
steno
xantho

Normally the sequences would be fetched from Genbank by a slightly complicated script, but I'd like you to be able to follow along if you want, so we'll do it almost manually. The file fetchSeqs.py contains a function that parses these two files and grabs the info we need for the "gamma" super-group. It also does a manual fetch from Genbank. We save the data to disk like so:

python fetchSeqs.py > seqs.txt

The beginning of one of the entries looks like this:

>gi|312967|emb|X60415.1| Aeromonas trota 16S rRNA gene, strain ATCC 49657
GAGTTTGATCATGGCTCAGATTGAACGCTGGCGGCAGGCCTAACACATGCAAGTCGAGCGGCAGCGGGAA

As you can see, the title line of this FASTA-formatted text is really long. So in the second script, analyzeSeqs.py, we use the same info from before to replace the long title with something shorter. This script also uses MUSCLE to align the sequences. You need to have it installed for this to work. Or you could use Clustal. I used the ape library in R to make a neighbor-joining tree and write this to disk as 'tree.txt'. I show a plot of what R gave me (actually the real process for this one was more complicated, resulting the pretty colors). But at least you can see what the tree is supposed to look like.

The last module is test_plotter.py It shows how to make a couple different kinds of plots with our plotter. All of them have colored node labels as defined in the script. The first plot is the standard one, the graphic is at the top of the post. In the next, we plot with the internal node labels showing, so that we can identify the name of a node to re-root the tree, if wanted.

In the third example, I got a balanced tree from PyCogent and plot that.

You might notice that the font used for the external node labels is now italic as it should be. It doesn't look that hot, I think that matplotlib is not using the OS X system fonts. Not sure why that is, yet.

I put a zip of all the files up on Dropbox (here).

New plotter for phylogenetic trees: customization


Continuing with a tree-plotter in Python. Previous posts here, here, and here. These examples depend on PyCogent and matplotlib.

I've got a somewhat bigger tree here, using sequences from a project we're working on. I used clustal and R to make a neighbor-joining tree. As you can see in the code, there is a lot of flexibility in terms of customized colors, etc.

code listing:


import tree_utils as TU
import tree_plotter as TP

fn = 'veillonella.tree.txt'
D = TU.make_tree_dict(fn=fn)
tr = D['meta']['tree']

attr = TP.get_default_attr()
attr['e_node_dot_size'] = 35
attr['figure_type'] = 'png'
attr['e_node_label_size'] = 10
attr['e_node_bar_color'] = 'k'

attr['using_alternate_names'] = True
e_node_names = D['meta']['e_node_names']
#L = [e.replace('_',' ') for e in e_node_names]
L = [e.split('_')[-1] for e in e_node_names]
# lose the version numbers on Genbank seqs
L = [e.split('.')[0] for e in e_node_names]
alt_name_dict = dict(zip(e_node_names,L))
attr['alternate_name_dict'] = alt_name_dict

attr['using_e_node_specific_colors'] = True
def f(name):
if 'DB' in name: return 'b'
return 'r'
L = [f(e) for e in e_node_names]
node_label_color_dict = dict(zip(e_node_names,L))
attr['node_label_color_dict'] = node_label_color_dict

def f(name):
if 'DB' in name: return 'b'
return 'r'
L = [f(e) for e in e_node_names]
dot_color_dict = dict(zip(e_node_names,L))
attr['dot_color_dict'] = dot_color_dict

TP.plot(D,attr=attr)

'''
Rcode:
library(ape)
setwd('Desktop')
dna = read.dna('veillonella.fasta',format='fasta')
tree = nj(dist.dna(dna))
plot(tree,cex=1)
axisPhylo()
write.tree(tree,'veillonella.tree.txt')
'''

New plotter for phylogenetic trees: re-rooting


Continuing with a tree-plotter in Python. Previous posts here and here. These examples depend on PyCogent and matplotlib.

One of the great things about building on PyCogent is we can use its re-rooting routine. In the output for this example we print the same tree as before, rooted at all possible internal nodes.

I've made a few tweaks to the code. make_tree_dict will take either a tree_string, a filename or a PyCogent tree. For the graphic above, we grab the default plot attributes, make the internal nodes visible, and then call plot. The code for this example is at the bottom. (I'm not going to post modifications to the other code until it's finalized. If you want it, shoot me an email).

To do list:
• learn matplotlib typography (e.g. italics)
• adjust for label sizes in a smart way
• provide for printing bootstrap values or internal node names

output:

$ python test_plotter.py 
original root
/-Stenotrophomonas_maltophilia
/edge.0--|
| \-Kingella_oralis
|
-root----|--Pseudomonas_aeruginosa
|
| /-Salmonella_typhi
| /edge.1--|
\edge.2--| \-Escherichia_coli
|
\-Haemophilus_parainfluenzae
------------------------------------------------------------
edge.0
/-Stenotrophomonas_maltophilia
|
|--Kingella_oralis
-root----|
| /-Pseudomonas_aeruginosa
| |
\edge.0--| /-Salmonella_typhi
| /edge.1--|
\edge.2--| \-Escherichia_coli
|
\-Haemophilus_parainfluenzae
------------------------------------------------------------
edge.2
/-Salmonella_typhi
/edge.1--|
| \-Escherichia_coli
|
-root----|--Haemophilus_parainfluenzae
|
| /-Stenotrophomonas_maltophilia
| /edge.0--|
\edge.2--| \-Kingella_oralis
|
\-Pseudomonas_aeruginosa
------------------------------------------------------------
edge.1
/-Salmonella_typhi
|
|--Escherichia_coli
-root----|
| /-Haemophilus_parainfluenzae
| |
\edge.1--| /-Stenotrophomonas_maltophilia
| /edge.0--|
\edge.2--| \-Kingella_oralis
|
\-Pseudomonas_aeruginosa
------------------------------------------------------------
balanced tree
/-Stenotrophomonas_maltophilia
/edge.0--|
| \-Kingella_oralis
|
-root----|--Pseudomonas_aeruginosa
|
| /-Salmonella_typhi
| /edge.1--|
\edge.2--| \-Escherichia_coli
|
\-Haemophilus_parainfluenzae


code listing:

import tree_utils as TU
import tree_plotter as TP

fn = 'tree.txt'
D = TU.make_tree_dict(fn=fn)
tr = D['meta']['tree']
print 'original root'
print tr.asciiArt()
print '-'*60

L = D['meta']['i_node_names']
for name in L[1:]:
print name
print tr.rootedAt(name).asciiArt()
D = TU.make_tree_dict(tr=tr)
print '-'*60

print 'balanced tree'
print tr.balanced().asciiArt()

tr = tr.rootedAt('edge.2')
D = TU.make_tree_dict(tr=tr)
attr = TP.get_default_attr()
attr['i_node_dots_visible'] = True
TP.plot(D,attr=attr,ofn='balanced')

Saturday, November 13, 2010

New plotter for phylogenetic trees: plotting


Here is a first pass at a plotter, following up on the last post.

It's ugly code yet, but I thought I would show you what I have. What I need to do is really think harder about how this module should be organized. But you can compare the result with what we got here.

[UPDATE: Modified the code to be easily customizable. Next up: more extensive testing.]
UPDATE2: Modified the code yet again.]

output:

         alternate name dict   None
default e node dot color k
dot color dict None
e node bar color r
e node bar color list None
e node dot size 75
e node dots visible True
e node label default color k
e node label size 14
e node labels visible True
figure type png
horizontal axis visible True
i node bar color k
i node dot color magenta
i node dots visible False
i node vertical bar color k
label width factor 1.1
line width 2
node label color dict None
r node dot color orange
using alternate names False
using e node specific colors False
vertical axis visible False


code listing:

import matplotlib.pyplot as plt
import tree_utils as tu

def plot(D,attr=None,ofn=None):
if not attr:
attr = get_default_attr()
SZ = attr['e_node_dot_size']
lw = attr['line_width']
maxx, maxy = D['meta']['max_xy']

all_node_names = D['meta']['all_node_names']
e_node_names = D['meta']['e_node_names']
i_node_names = D['meta']['i_node_names']

# actual node dictionaries
e_node_dicts = [D[n] for n in e_node_names]
i_node_dicts = [D[n] for n in i_node_names]

# extract the values we need
e_node_positions = [(nD['x'],nD['y']) for nD in e_node_dicts]
e_node_lengths = [(nD['dist_to_parent']) for nD in e_node_dicts]
i_node_positions = [(nD['x'],nD['y']) for nD in i_node_dicts]
i_node_lengths = [(nD['dist_to_parent']) for nD in i_node_dicts]
i_node_verticals = [(nD['y_bott'], nD['y_top']) for nD in i_node_dicts]

# external nodes
for i in range(len(e_node_names)):
x,y = e_node_positions[i]
d = e_node_lengths[i]
xleft = x - d
# bars
L = attr['e_node_bar_color_list']
if L:
c = L[i]
else:
c = attr['e_node_bar_color']
plt.plot([xleft,x],[y,y],color=c,lw=lw,zorder=1)
# dots
name = e_node_names[i]
if attr['e_node_dots_visible']:
if attr['using_e_node_specific_colors']:
c = attr['dot_color_dict'][name]
else:
c = attr['default_e_node_dot_color']
plt.scatter(x,y,color=c,s=SZ,zorder=2)

# internal nodes, root is i = 0
for i in range(len(i_node_names))[1:]:
x,y = i_node_positions[i]
d = i_node_lengths[i]
y_bott, y_top = i_node_verticals[i]
x0 = x - d
# bars
c = attr['i_node_bar_color']
plt.plot([x0,x],[y,y],color=c,lw=lw,zorder=1)
# verticals
c = attr['i_node_vertical_bar_color']
plt.plot([x,x],[y_bott, y_top],color=c,lw=lw,zorder=1)
# dots
if attr['i_node_dots_visible']:
c = attr['i_node_dot_color']
plt.scatter(x,y,color=c,s=SZ,zorder=2)

# root
i = 0
x,y = i_node_positions[i]
y_bott, y_top = i_node_verticals[i]
# verticals
c = attr['i_node_vertical_bar_color']
plt.plot([x,x],[y_bott, y_top],color=c,lw=lw,zorder=1)
# dots
if attr['i_node_dots_visible']:
c = attr['r_node_dot_color']
plt.scatter(x,y,color=c,s=SZ,zorder=2)

L = [len(name) for name in e_node_names]
max_label_width = maxx/100.0 * max(L)
max_label_width *= attr['label_width_factor']

# external node labels
if attr['e_node_labels_visible']:
for i in range(len(e_node_names)):
x,y = e_node_positions[i]
dx = maxx/100.0 * 4
name = e_node_names[i]
if attr['using_alternate_names']:
s = attr['alternate_name_dict'][name]
else:
s = name
if attr['using_e_node_specific_colors']:
c = attr['node_label_color_dict'][name]
else:
c = attr['e_node_label_default_color']
plt.text(x + dx, y, s,
fontname = 'Helvetica',
fontsize = attr['e_node_label_size'],
color = c,
ha = 'left',va = 'center')

ax = plt.axes()
if not attr['vertical_axis_visible']:
ax.yaxis.set_visible(False)
if not attr['horizontal_axis_visible']:
ax.xaxis.set_visible(False)
ax.set_xlim(-maxx/10.0,maxx*1.1 + max_label_width)
ax.set_ylim(-maxy/10.0,maxy*1.1)

if ofn:
plt.savefig(ofn + '.' + attr['figure_type'])
else:
plt.savefig('example.' + attr['figure_type'])

def get_default_attr():
attr = dict()
attr['e_node_dot_size'] = 75
attr['line_width'] = 2
attr['figure_type'] = 'png'
attr['horizontal_axis_visible'] = True
attr['vertical_axis_visible'] = False

attr['e_node_bar_color'] = 'r'
attr['e_node_bar_color_list'] = None
attr['e_node_dots_visible'] = True
attr['e_node_labels_visible'] = True
attr['default_e_node_dot_color'] = 'k'
attr['e_node_label_size'] = 14
attr['e_node_label_default_color'] = 'k'
attr['label_width_factor'] = 1.1

attr['i_node_bar_color'] = 'k'
attr['i_node_vertical_bar_color'] = 'k'
attr['i_node_dots_visible'] = False
attr['i_node_dot_color'] = 'magenta'
attr['r_node_dot_color'] = 'orange'

attr['using_alternate_names'] = False
attr['alternate_name_dict'] = None
attr['using_e_node_specific_colors'] = False
attr['node_label_color_dict'] = None
attr['dot_color_dict'] = None
return attr

def print_defaults():
attr = get_default_attr()
N = max([len(k) for k in attr.keys()])
for k in sorted(attr.keys()):
v = attr[k]
k = k.replace('_',' ')
print k.rjust(N), ' ', v

if __name__ == '__main__':
fn = 'tree.txt'
file_data = tu.load_data(fn)
D = tu.make_tree_dict(ts=file_data)
plot(D)
print_defaults()

New plotter for phylogenetic trees: parsing

In July of last year, I had a series of posts about drawing phylogenetic trees (here, here, here, here and here). Some very early stuff is here and here.

I want to revisit this topic to try to get a tree plotter in python that is really flexible. The important thing to remember here is that we do not want to "roll our own" phylogenetics software. We should let the experts do that for us (and debug it!). Now that we know about PyCogent (web page, links in the sidebar), I say let them do the heavy lifting.

So I wrote a module today that starts with a Newick format phylogenetic tree (the one we used before here), and gets PyCogent to figure out all its connections. You can see how versatile the PyCogent code for dealing with trees is in the docs.

This is the tree:

((Stenotrophomonas_maltophilia:0.07574,
Kingella_oralis:0.08026)
:0.00827,
Pseudomonas_aeruginosa:0.05950,
((Salmonella_typhi:0.01297,
Escherichia_coli:0.01491)
:0.03356,
Haemophilus_parainfluenzae:0.06113)
:0.03863);


Anyway, here is the output from the module. It's not very well tested yet. So I hope you find a bug and let me know. But I'm not Donald Knuth so I'm not going to pay you :)

[UPDATE: I replaced the code with a later version]

output:

$ python tree_utils.py
making default tree

finding children
edge.0
['Stenotrophomonas_maltophilia', 'Kingella_oralis']
edge.1
['Salmonella_typhi', 'Escherichia_coli']
edge.2
['edge.1', 'Haemophilus_parainfluenzae']
root
['edge.0', 'Pseudomonas_aeruginosa', 'edge.2']

Stenotrophomonas_maltophilia
node 'Stenotrophomonas_maltophilia':0.07574;
ancestors ['edge.0', 'root']
name Stenotrophomonas_maltophilia
is_root False
y 0.0
x 0.08401
dist_to_parent 0.07574
is_external True

Kingella_oralis
node 'Kingella_oralis':0.08026;
ancestors ['edge.0', 'root']
name Kingella_oralis
is_root False
y 0.019952
x 0.08853
dist_to_parent 0.08026
is_external True

Pseudomonas_aeruginosa
node 'Pseudomonas_aeruginosa':0.0595;
ancestors ['root']
name Pseudomonas_aeruginosa
is_root False
y 0.039904
x 0.0595
dist_to_parent 0.0595
is_external True

Salmonella_typhi
node 'Salmonella_typhi':0.01297;
ancestors ['edge.1', 'edge.2', 'root']
name Salmonella_typhi
is_root False
y 0.059856
x 0.08516
dist_to_parent 0.01297
is_external True

Escherichia_coli
node 'Escherichia_coli':0.01491;
ancestors ['edge.1', 'edge.2', 'root']
name Escherichia_coli
is_root False
y 0.079808
x 0.0871
dist_to_parent 0.01491
is_external True

Haemophilus_parainfluenzae
node 'Haemophilus_parainfluenzae':0.06113;
ancestors ['edge.2', 'root']
name Haemophilus_parainfluenzae
is_root False
y 0.09976
x 0.09976
dist_to_parent 0.06113
is_external True

root
y_top 0.084796
ancestors []
name root
is_root True
immediate_children ['edge.0', 'Pseudomonas_aeruginosa', 'edge.2']
y_bott 0.009976
y 0.039904
x 0
dist_to_parent 0
is_external False

edge.0
y_top 0.019952
ancestors ['root']
name edge.0
is_root False
immediate_children ['Stenotrophomonas_maltophilia', 'Kingella_oralis']
y_bott 0.0
y 0.009976
x 0.00827
dist_to_parent 0.00827
is_external False

edge.2
y_top 0.09976
ancestors ['root']
name edge.2
is_root False
immediate_children ['edge.1', 'Haemophilus_parainfluenzae']
y_bott 0.069832
y 0.084796
x 0.03863
dist_to_parent 0.03863
is_external False

edge.1
y_top 0.079808
ancestors ['edge.2', 'root']
name edge.1
is_root False
immediate_children ['Salmonella_typhi', 'Escherichia_coli']
y_bott 0.059856
y 0.069832
x 0.07219
dist_to_parent 0.03356
is_external False

code listing for tree_utils.py:

import sys
from cogent import LoadTree

s = '''
((Stenotrophomonas_maltophilia:0.07574,
Kingella_oralis:0.08026)
:0.00827,
Pseudomonas_aeruginosa:0.05950,
((Salmonella_typhi:0.01297,
Escherichia_coli:0.01491)
:0.03356,
Haemophilus_parainfluenzae:0.06113)
:0.03863);'''

def load_data(fn):
FH = open(fn,'r')
data = FH.read().strip()
FH.close()
return data

# for flexibility, we can this function with
# a tree_string
# a filename
# or a PyCogent tree
def make_tree_dict(ts=None,fn=None,tr=None,debug=False):
if tr:
pass
elif fn:
tr = LoadTree(filename=fn)
else:
tr = LoadTree(treestring=ts)
nodes = tr.getNodesDict()

# and their names
all_node_names = tr.getNodeNames()
# PyCogent organizes the external node names
# in the correct order, so the rest is easy

# rearrange the data my way
D = dict()
D['meta'] = { 'all_node_names':all_node_names }
D['meta']['tree'] = tr
e_node_names = list()
i_node_names = list()

for name in all_node_names:
node = nodes[name]
external = node.isTip()
if external: e_node_names.append(name)
else: i_node_names.append(name)

# in order up the tree
aL = [a.Name for a in node.ancestors()]
dist_to_root = node.distance(nodes['root'])

nD = {'name':name,
'node':node,
'is_external':external,
'is_root': name == 'root',
'x':dist_to_root,
'ancestors':aL }

if not nD['is_root']:
dist = node.distance(nodes[aL[0]])
nD['dist_to_parent'] = dist
else:
nD['dist_to_parent'] = 0
D[name] = nD

D['meta']['e_node_names'] = e_node_names
D['meta']['i_node_names'] = i_node_names
D = compute_y_pos(D,debug=debug)
if debug: show(D)
return D

def compute_y_pos(D,debug=False):
all_node_names = D['meta']['all_node_names']
e_node_names = D['meta']['e_node_names']
i_node_names = D['meta']['i_node_names']

# actual node dictionaries
e_node_dicts = [D[n] for n in e_node_names]
i_node_dicts = [D[n] for n in i_node_names]

# just make a square plot
xmax = max([nD['x'] for nD in e_node_dicts])
ymax = xmax
D['meta']['max_xy'] = xmax, ymax

# compute y for external nodes
N = 1.0*(len(e_node_names)-1)
for nD in e_node_dicts:
i = e_node_names.index(nD['name'])
f = i/N
nD['y'] = f*ymax

def mean(L): return sum(L)*1.0/len(L)

def get_child_node_names(parent):
if debug: print parent
cL = list()
for name in all_node_names:
if name == 'root': continue
nD = D[name]
if parent == nD['ancestors'][0]:
cL.append(name)
if debug: print cL
return cL

# compute y for internal nodes
# must fill these from the bottom up
L = i_node_names[:]
#L.sort() # sorts 0,1,2 etc. then root
def f(name): # sort by x-value !
nD = D[name]
return nD['x']
L = sorted(L,key=f,reverse=True)

if debug: print 'finding children'
for name in L:
nD = D[name]
cL = get_child_node_names(parent=name)
nD['immediate_children'] = cL
yL = [D[child]['y'] for child in cL]
y0 = min(yL)
y1 = max(yL)
if len(yL) == 3:
y = sorted(yL)[1]
else:
y = mean([y0,y1])
nD = D[name]
nD['y'] = y
nD['y_top'] = y1
nD['y_bott'] = y0
if debug: print

# sanity check for ancestor lists
for name in all_node_names:
if name == 'root': continue
assert D[name]['ancestors'][-1] == 'root'
return D

def show(D):
e_node_names = D['meta']['e_node_names']
i_node_names = D['meta']['i_node_names']
for name in e_node_names + i_node_names:
print name
nD = D[name]
for k in nD:
if k == 'node' and not nD['is_external']:
continue
print k, nD[k]
print

if __name__ == '__main__':
tree_string = None
if len(sys.argv) > 1:
fn = sys.argv[1]
try:
tree_string = load_data(fn)
except IOError:
print 'could not open file:', fn
sys.exit()
if not tree_string:
print 'making default tree', '\n'
tree_string = s.strip()
D = make_tree_dict(ts=tree_string,debug=True)