Qiime (9) shell scripts

This is the README.txt file of a zipped directory on Dropbox (here). It's been tested on OS X and on a Virtual Box running Linux with QIIME (my post here).

The scripts in this directory are intended to simplify the process followed in the Qiime overview tutorial:

http://qiime.sourceforge.net/tutorials/tutorial.html

What they do:

download the tutorial files

configure the tutorial directory with short paths and file names

carry out each step of the workflow scripts individually

organize things the way I like it


They do require that Qiime is already installed, and that the files core.txt and mask.txt be available in ~/data.

To use this, copy this directory (qiime_shellscripts) to the Desktop.
Fire up the Terminal and do:

~/Desktop/qiime_shellscripts/download.sh

This script will download the Qiime overview tutorial and unpack it to a new directory qiime_tutorial on your Desktop. On Linux, you will need to substitute wget for curl (or install curl). As things progress, it is likely that you will have to edit the version number in the script. The zipped download remains on the Desktop, so if things get messed up, you won't have to repeat yourself. Then do:

~/Desktop/qiime_shellscripts/preliminary.sh

This will do step 0, splitting the sequences. Next do:

~/Desktop/qiime_shellscripts/run_all.sh

If you wish to re-run the workflow analyses individually, you can first remove results of the previous work with:

~/Desktop/qiime_shellscripts/clean.sh

This last script does not reverse the split step.

If you need a new copy of this directory, it should still be on Dropbox:

http://dl.dropbox.com/u/3534458/qiime_shellscripts.zip

Qiime (8) In the Virtual Box

We still have more to do in the tutorial, but this post is about the virtual machine with QIIME installed. The website discussion for this is here. My thinking is that for a real 454 or Illumina data set (which I should have at some point in the future), I will want to run "in the cloud." It's all over the web:

If you need milk, would you buy a cow ?'.

To do that I will need to get comfortable with Linux and Qiime in this environment. The Qiime website has discussion about the EC2 "image" (here), but for today's example I used the Virtual Box (VB) with Qiime.

A couple of issues had to be addressed first. There is something called "Guest Additions" that needs to be installed. The way I did it was to do:

cd /media/VBOXADDITIONS_4.0.4_70112/ ./autorun.sh

and then restart the Qiime VB. The second is to train my brain to use

command (Apple symbol) on OS X control when not in the terminal on the Virtual Box control-shift when in terminal on the Virtual Box

And finally, I found the password (qiime---had to search the QIIME Forum for that). The Linux install doesn't have curl so I used:

wget http://bmf.colorado.edu/QIIME/qiime_tutorial-v1.2.1.zip unzip qiime_tutorial-v1.2.1.zip rm -r qiime_tutorial-v1.2.1.zip

Then I just followed my own instructions from here and here. I got a failure with:

plot_taxa_summary.py: command not found

which I fixed by:

svn co https://qiime.svn.sourceforge.net/svnroot/qiime/trunk Qiime cd Qiime python setup.py install --install-scripts=/software/qiime-1.2.0-release/bin cd tests ./all_tests.py

Failed the following unit tests. /home/qiime/Qiime/tests/test_pycogent_backports/test_uclust.py /home/qiime/Qiime/tests/test_pyronoise.py Failed the following unit tests, in part or whole due to missing external applications. Depending on the QIIME features you plan to use, this may not be critical. /home/qiime/Qiime/tests/test_identify_chimeric_seqs.py /home/qiime/Qiime/tests/test_make_per_library_sff.py /home/qiime/Qiime/tests/test_trim_sff_primers.py /home/qiime/Qiime/tests/test_workflow.py Failed the following script tests. /software/qiime-1.2.0-release/bin/process_sra_submission.py

I'm not too worried about the failures. I'll look into it.
There's a screenshot at the top of the post. Looking good.

[ UPDATE: Instructions for shared folders with the host OS here. ]

Qiime (7) Area Plot

We were plotting a summary chart in QIIME (here). Turns out I just skipped a step:

summarize_taxa.py -i figs/otu_table.txt -o figs/otu_table_Level3.txt -L 3 plot_taxa_summary.py -i figs/otu_table_Level3.txt -l Phylum -o figs -k white

Qiime (6) quick version

This example is for the impatient (like me): running part 1 of the QIIME tutorial--checklist version.

[ UPDATE Note: the "workflow script" in the tutorial is actually easier to run than what I have here. It needs only three changes: paths to the Greengenes files (below) and changing the rep set picking method to most_abundant, if desired. This post is about doing the workflow in such a way that the scripts are invoked individually. ]

Prerequisites (as discussed here and here)
Install and test:

Python 2.6

Numpy > 1.3.0

PyCogent

uclust

PyNAST

FastTree

Java RE (for RDP)

RDP Classifier

QIIME


Make sure you have the equivalent of this (some scripts don't seem to look at $PATH):

> cat ~/.qiime_config qiime_scripts_dir /Users/telliott/bin/qiime/bin

and that ~/bin/qiime/bin is on your $PATH

> pick_otus.py -h | head -n Usage: pick_otus.py [options] {-i/--input_seqs_filepath INPUT_SEQS_FILEPATH}

---

From the Desktop, get the Greengenes core set data file (38 MB download link). Also get the Greengenes alignment lanemask file (download link). And get the tutorial files (download link), or do:

curl -O http://bmf.colorado.edu/QIIME/qiime_tutorial-v1.2.0.zip tar -xf qiime_tutorial-v1.2.0.zip rm -r qiime_tutorial-v1.2.0.zip

Make some directories:

mkdir qiime_tutorial/gg mkdir qiime_tutorial/figs mkdir qiime_tutorial/other

For the next two steps, if you get an error, recheck the file extensions:

mv core_set_aligned.fasta.imputed qiime_tutorial/gg/core.txt mv lanemask_in_1s_and_0s.txt qiime_tutorial/gg/mask.txt

I prefer short names, so these directory and file names are different than those in the tutorial.

cd qiime_tutorial mv Fasting_Example.fna data.fna mv Fasting_Example.qual data.qual mv Fasting_Map.txt map.txt mv Fasting_Example.sff.txt other/Fasting_Example.sff.txt mv Fasting_Example.sff other/Fasting_Example.sff mv qiime_tutorial_commands_parallel.sh other/parallel.sh mv qiime_tutorial_commands_serial.sh other/serial.sh mv README other/README mv custom_parameters.txt other/custom.txt > ls data.fna figs map.txt data.qual gg other

Step 0:

split_libraries.py -f data.fna -m map.txt -q data.qual -o split mv data.fna other/data.fna mv data.qual other/data.qual cp split/seqs.fna seqs.fna

At this point, you could make a backup (including the huge core.txt):

cp -r ../qiime_tutorial ../qiime_tutorial_backup

---

Analysis:

pick_otus.py -i seqs.fna -m uclust -s 0.97 -o otus pick_rep_set.py -f seqs.fna -i otus/seqs_otus.txt -m most_abundant -o otus/reps.txt align_seqs.py -i otus/reps.txt -m pynast -t gg/core.txt -o aln assign_taxonomy.py -i otus/reps.txt -m rdp -o tax filter_alignment.py -i aln/reps_aligned.txt -m gg/mask.txt -o aln2 make_phylogeny.py -i aln2/reps_aligned_pfiltered.fasta -o tree.tre make_otu_table.py -i otus/seqs_otus.txt -t tax/reps_tax_assignments.txt -o figs/otu_table.txt

Rather than do another heatmap, I thought I'd show a summary area chart

plot_taxa_summary.py -i figs/otu_table.txt -l Phylum -o figs -k white

Unfortunately, it's not what we want (the graphic at top seems to be plotting the individual OTUs rather than Phyla). The example in the tutorial uses otu_table_Level3.txt

I don't see an option for controlling the level of output in make_otu_table.py. I'll have to investigate how to do that. Perhaps if we follow the workflow script it will turn out correctly.

Qiime (5) alpha diversity

Continuing with the exploration of QIIME (earlier posts: one two three four).

We're following the overview tuturial (here), which has four main parts. There are the three big sections of custom_parameters.txt:

picking and analysis of species/OTUs

analysis of alpha diversity

analysis of beta diversity

plus

a suite of data visualization programs


Today we're thinking about alpha diversity (within sample diversity). Goals of diversity analysis (using 16S rRNA sequences in bacteria) include measurement of:

observed species/OTU richness

population distribution among species/OTUs

phylogenetic diversity among OTUs


where OTUs can be defined at different thresholds of sequence identity.

For this post, I've got screenshots of two graphics produced from the Qiime tutorial analysis of alpha diversity. Consider the first graphic, which plots observed species in two different samples as a function of the number of individual sequences examined. Because chance influences the order in which samples are obtained, resampling techniques are used to generate a rarefaction curve that averages the results of many random samplings of the observed data. Perhaps more important, it allows normalization for the number of samples observed, allowing comparison of samples with different sizes.

These plots are rarefaction curves.



The most important questions are probably these two:

(1) If we could sample exhaustively, what would be the final species count (or phylogenetic distribution or.. ). In other words, does the curve level off, and what is the asymptotic value?

(2) Can we decide whether two different populations differ significantly even without exhaustive sampling?

I do not know much about this area, so I should probably just be quiet at this point, but I have to say that I am suspicious that the first question does not always have a good answer (even if people would wish for one). The problem is that the shape of the curve as it goes out into high number of samples does not necessarily depend on the shape at lower numbers. It might do so, if the population structure is not too skewed. But no one can say in advance whether that is true or not.

Anyway, we follow the tutorial. First remake (and this time, save) the otu_table:

make_otu_table.py -i otus/seqs_otus.txt -t tax/reps_tax_assignments.txt -o figs/otu_table.txt

We'll use the workflow script (I don't see too much of interest after a cursory look at the intermediate steps). We can get the required options with -h (help):

> alpha_rarefaction.py -h .. REQUIRED options: The following options must be provided under all circumstances. -i OTU_TABLE_FP, --otu_table_fp=OTU_TABLE_FP the input otu table [REQUIRED] -m MAPPING_FP, --mapping_fp=MAPPING_FP path to the mapping file [REQUIRED] -o OUTPUT_DIR, --output_dir=OUTPUT_DIR the output directory [REQUIRED] -p PARAMETER_FP, --parameter_fp=PARAMETER_FP path to the parameter file [REQUIRED]

What we actually run is:

command:

alpha_rarefaction.py -i figs/otu_table.txt -m map.txt -o rare/ -p custom_parameters.txt -t figs/tree.tre -f

The default settings in custom_parameters.txt are fine for this.

If you do want to run each of the steps individually, I suggest you run the workflow script, then the log file in will contain a history of the commands that were executed:

# Alpha rarefaction command python /Users/telliott/bin/qiime/bin/multiple_rarefactions.py -i figs/otu_table.txt -m 10 -x 148 -s 13 -o rare//rarefaction/ --num-reps 5 # Alpha diversity on rarefied OTU tables command python /Users/telliott/bin/qiime/bin/alpha_diversity.py -i rare//rarefaction/ -o rare//alpha_div/ -t figs/tree.tre --metrics chao1,observed_species,PD_whole_tree # Collate alpha command python /Users/telliott/bin/qiime/bin/collate_alpha.py -i rare//alpha_div/ -o rare//alpha_div_collated/ # Rarefaction plot: All metrics command python /Users/telliott/bin/qiime/bin/make_rarefaction_plots.py -i rare//alpha_div_collated/ -m map.txt -o rare//alpha_rarefaction_plots/ --background_color white --resolution 75 --imagetype png

The second graphic shows that considering phylogenetic diversity may reveal significant differences that are not seen when just counting OTUs.

Qiime (4) PyNAST

Today I've been on a side-track from the QIIME pipeline (or toolkit), related to the construction of multiple sequence alignments (MSA). The favored approach in QIIME is that of NAST (DeSantis et al. 2006 PMID 16845035) as reimplemented in PyNAST using Python (Caporaso et al. 2010 PMID 19914921).

The fundamental problem with MSA is that it scales very poorly. Analysis of the complexity is itself complex (see the discussion in Edgar 2004 PMID 15318951), but Edgar gives it as O(N⁴ + L²) overall (N sequences of typical length L). Though of similar complexity, MUSCLE is significantly faster. Still, it grinds quite slowly when you have a lot of sequences (say, > 500).

From the first MUSCLE paper:

In agreement with other studies, [e.g. Katoh et al. (8)], we found that T-Coffee was unable to align more than approximately 10² sequences of typical length on a current desktop computer. CLUSTALW was able to align a few hundred sequences, with a practical limit around N = 10³ where CPU time begins to scale approximately as N⁴. The largest set had 5000 sequences of average length 350. MUSCLE-p completed this test in 7 min, compared with 10 min for FFTNS1; we estimate that CLUSTALW would need approximately 1 year.

NAST (Nearest Alignment Space Termination) is one solution to this problem. The basic idea is simple: use an existing multiple sequence alignment as a framework against which to align new sequences. The trick is that addition of a new sequence will not change the length of the MSA. Besides the algorithm, which we'll talk about, a significant part of the effort going into NAST has been the construction of a large, curated database of high quality bacterial and archaeal 16S rRNA sequences.

The implementation has these steps:

• identify the best matching sequence to the candidate in the MSA database
• trim the candidate and do a global alignment to this best match
• resolve positions where the alignment introduces a gap into the template

The example from DeSantis et al is below (T = template; C = candidate). I've shown the template spaces as underlines to distinguish them from alignment gaps:

T: ATAC_____GTA_AC____GTA___C___G_T_AC_GG

Collapse the template spacing:

T: ATACGTAACGTACGTACGG C: CACGTTAAACGTCGTACCCGG

Construct the pairwise alignment

T: ATACGT-A-ACGTACGTAC--GG C: C-ACGTTAAACGT-CGTACCCGG

Re-introduce template spacing

* * T: ATAC_____GT-A-AC____GTA___C___G_T_AC--GG C: C-AC_____GTTAAAC____GT-___C___G_T_ACCCGG

If you look at the template in the pairwise alignment you will see four positions containing introduced gap characters. Two of these can be accomodated by gaps in the original template spacing, and two cannot (marked with *). These will have to go.

The * positions must be collapsed to maintain the constant length MSA. In the case of the first one, the aligned GT to its left is displaced. In the second one, the gap between T_A in the candidate is consumed. The final result is:

T: ATAC_____GT-AAC____GTA___C___G_T_AC_GG C: C-AC____GTTAAAC____GT-___C___G_TACCCGG

Note that in both cases, a mismatch was accepted in order to repair the gap. This is the weakness of the NAST approach.

The template may not look exactly the same as when it started, but this difference is not propagated to the database.

We obtained two files from Greengenes following the Qiime tutorial (my post here). The first is a "core_set":

>>> FH = open('core_set_aligned.fasta.imputed.txt') >>> data = FH.read().strip() >>> FH.close() >>> data.count('>') 4938 >>> first = data.split('>')[1].strip().split('\n',1)[1] >>> len(first) 7682

It contains nearly five thousand high quality 16S ribosomal RNA gene sequences that have been aligned. Each one of those sequences consists mostly of '-' gap characters. Almost all of those positions are gaps in every single sequence. These are indexed using the "lane mask":

>>> FH = open('lanemask_in_1s_and_0s.txt') >>> mask = FH.read().strip() >>> FH.close() >>> len(mask) 7682 >>> mask.count('0') 6395 >>> mask.count('1') 1287 >>> L = [c for c,m in zip(first,mask) if m == '1'] >>> print ''.join(L) >>> print ''.join(L)[:50] ATTGAACGCTGGCGGTATGCTTAACACATGCAAGTCGAACG-AGTGGCGG >>> len(L) 1287

Naturally, the "core_set" is a large file

>>> data.count('>')*len(mask) 37933716 > ls -l core_set_aligned.fasta.imputed.txt -rw-r--r--@ 1 telliott staff 37975992 Feb 19 06:07 core_set_aligned.fasta.imputed.txt

I made some fake data to run through pynast:

>0 A GAGTTTGATCCTGGCTCAGATTGAACGCTGGCGGTATGCTT >1 B AGTTTGATCCTGGCTCAGATTGAACGCTGGCGGTATGCTT >2 C GAGGATCCTGGCTCAGATTGAACGCTGGCGGTATGCTT >3 D CCCGAGGATCCTGGCTCAGATTGAACGCTGGCGGTATGCTT

Sequence B is the same as A, but missing the first nt, C is missing the first 6 nt, and D has a substitution for the first 6 nt.

The template looks like this:

>Y GAGTTT-GA--T-CC-T-G-GCTC-AG-AT-TGAA-C-GC--TGG-C--G-GT-A-TG--C----T-T >Z gagttt-ga--t-cc-t-g-gctc-ag-at-tgaa-c-gc--tgg-c--g-gc-a-gg--c----c-t

> pynast -i data.txt -t template.txt -l 25 /Library/Python/2.6/site-packages/cogent/evolve/likelihood_tree.py:6: UserWarning: Not using MPI as mpi4py not found from cogent.util.parallel import MPI

There's a warning about not having code for parallel processing. And here's the alignment:

>0 A 1..41 GAGTTT-GA--T-CC-T-G-GCTC-AG-AT-TGAA-C-GC--TGG-C--G-GT-A-TG--C----T-T >1 B 1..40 -AGTTT-GA--T-CC-T-G-GCTC-AG-AT-TGAA-C-GC--TGG-C--G-GT-A-TG--C----T-T >2 C 1..38 ---GAG-GA--T-CC-T-G-GCTC-AG-AT-TGAA-C-GC--TGG-C--G-GT-A-TG--C----T-T >3 D 1..40 -CCGAG-GA--T-CC-T-G-GCTC-AG-AT-TGAA-C-GC--TGG-C--G-GT-A-TG--C----T-T

One of the things that's been overlooked in the present exploration is the crucial contribution of another Robert Edgar program, uclust. That's used both for MSA and at least one other step in the QIIME pipeline. I need to read the paper and we'll look at it another day.

Qiime (3) Heatmap

This example should really be part of the previous post (here), about the first segment of the overview tutorial for Qiime (here). One script in Qiime makes a heatmap, and it is quite something. To do this example, I modified #7 from last time to save otu_table.txt in a sub-directory rather than write to the screen. The second command makes the heat map.

Here are the commands:

make_otu_table.py -i otus/seqs_otus.txt -t tax/reps_tax_assignments.txt -o figs/otu_table.txt make_otu_heatmap_html.py -i figs/otu_table.txt -o figs

The first thing is that the output is html and can be displayed in the browser (great idea). The second is that it's quite complicated, with more than 5000 lines of javascript required for display. It allows drag and drop of the table rows, and custom mapping files to control the sorted order of the table rows. There are "tool tips" to display detail for each cell.

The primary author, Jesse Stombaugh, and also the library authors Erik Bosrup (overLIB) and John Resig (jQuery) obviously put a huge amount of effort into this.

I am not sure why the default output from this example (by Taxonomy) looks so weird. Anybody know why there are so many duplicate row names?

I would probably make the red used for the highest counts a brighter color.

Put this on the list for more investigation.

Qiime (2)

I'm continuing with the QIIME overview tutorial (project link, tutorial link, first post here).

These scripts were all installed in a directory which is on my $PATH, they have the executable permission set, and they all have the shebang thing so they can be run using the script name only. Extensive help is available by running with -h only. Many more options are usually available. There is also extensive online help. An index page for all the scripts is here.

I renamed the example data files, which were originally Fasting_Example.*, to data.*, where * is 'fna' or 'qual' just to make them shorter, except the map file (mapping barcodes and primers to samples) is map.txt.

The zeroth step is to organize the sequences by sample, using the bar code. This identifier is the first few nt (here n=12), followed immediately by the primer (both are removed).

Step 0: split_libraries.py

required options: -f sequence filename -m map filename -q quality scores filename -o output sub-directory (will be created) some default options: -l MAX_SEQ_LEN [default:200] -s MIN_QUAL_SCORE [default:25] -M MAX_PRIMER_MM [default:0] command: split_libraries.py -f data.fna -m map.txt -q data.qual -o split

According to the log file in split/ we have 1333 sequences split among 9 samples, after filtering out 6 sequences for one of: >1 primer mismatch, > 0 ambiguous bases, quality score < 25. The sequences are in seqs.fna.

Step 1: pick_otus.py

required options: -i input filename some default options: -m uclust -c CLUSTERING_ALGORITHM [default:furthest] # for mothur -s similarity [default:0.97] command: pick_otus.py -i split/seqs.fna -m uclust -s 0.97 -o otus

According to the log file in otus/ we have 417 otus (at 97% identity). These are in seqs_otus.txt identified by the names assigned in step 0.

Step 2: pick_rep_set.py

Pick representative sequences for each otu, to be used for taxonomy assignment and sequence alignment. There's no point in assigning or aligning a large number of essentially identical sequences (and would be impossible with a realistic data set).

required options: -i input filename (otu mapping file from #1) -f sequence filename -m most_abundant some default options: -s SORT_BY [default: otu] command: pick_rep_set.py -f split/seqs.fna -i otus/seqs_otus.txt -m most_abundant -o otus/reps.txt

Step 3: align_seqs.py

Align representative otus using MUSCLE (for example) or PyNAST. PyNAST requires a template alignment, which for the time being I will just put into the qiime_tutorial directory, in gg/core.txt (gg stands for Greengenes).

required options: -i input filename (representative FASTA seqs from #2) some default options: -m ALIGNMENT_METHOD [default: pynast] -a PAIRWISE_ALIGNMENT_METHOD [default: uclust] -d BLAST_DB [set using environment variable] -t TEMPLATE_FP command: align_seqs.py -i otus/reps.txt -m pynast -t gg/core.txt -o aln

The output alignment is in aln/reps_aligned.txt. Here is the strangely sparse Greengenes type of alignment. You could use MUSCLE, but it takes a while.. Although for this relatively small example, it was just a few minutes.

command: align_seqs.py -i otus/reps.txt -m muscle -o aln_muscle

Step 4: assign_taxonomy.py

Assign taxonomy using the RDP classifier.

required options: -i input filename (representative FASTA seqs from #2) some default options: -m ASSIGNMENT_METHOD [default: rdp] -c CONFIDENCE [default: 0.8] command: assign_taxonomy.py -i otus/reps.txt -m rdp -o tax

Step 5: filter_alignment.py

The Greengenes method results in many columns having all gaps. These should be removed using the lanemask file we downloaded from them. (I don't really understand all this yet---it's my next project).

required options: -i input filename (representative FASTA seqs from #2) some default options: -m LANE_MASK_FP [default: none] -g ALLOWED_GAP_FRAC [default: 0.999999] -t THRESHOLD [default: 3.0] command: filter_alignment.py -i aln/reps_aligned.txt -m gg/mask.txt -o aln2

Step 6: make_phylogeny.py

Just make a phylogenetic tree. Here, we use FastTree.

required options: -i input filename (from filtered alignment, step 5) some default options: -t fasttree -r ROOT_METHOD [default: tree_method_default] command: make_phylogeny.py -i aln2/reps_aligned_pfiltered.fasta -o tree.tre

Step 7: make_otu_table.py

Does what it says

required options: -i input filename (from picking otus, step 1) some default options: -t TAXONOMY_FNAME [default: none] (step 4) -e EXCLUDE_OTUS_FP (e.g. chimeras) -o OUTPUT_FP [default: otu_table.txt] command: make_otu_table.py -i otus/seqs_otus.txt -t tax/reps_tax_assignments.txt

[UPDATE: -o default is actually to print to the screen]

Of course, you can run the whole "workflow" at once with (docs link):

pick_otus_through_otu_table.py

But first, you need to make a few changes to the file custom_parameters.txt as described in the tutorial. It provides options for all of the scripts in the sequence.

I also decided to download FigTree and try it out. The graphic at the top of the post is our tree output from step 6.

There is so much more to explore here:

we've just started with Qiime
we need to understand how PyNAST works
and how the RDP classifier works

And it would be nice to color the tree by Phylum or sample, or .. even better---plot a heat map! That's for next time.

QIIME (1)

Yesterday and this morning, I've been working on installing QIIME and doing the tutorial. It stands for: Quantitative Insights into Microbial Ecology (link). It's yet another major software project out of Rob Knight's lab.

I'd like to post on the issues/questions/answers I run into as I work through it. It'll also give me an opportunity to get back to Bioinformatics (e.g. multiple sequence alignments).

The first thing is that they recommend you use a virtual machine (Virtual Box) and install the QIIME Virtual Box on it, which is a huge download that packages Ubuntu Linux with all of the QIIME dependencies, correctly configured. Sounds like a great idea.

I started by trying this. I had to download overnight on my machine at work (> 2 GB), then buy a new thumb drive to bring the file home, since I don't have the recommended amount of memory for the VM (1 GB) on that machine. But I ran into trouble---basically much aggravation dealing with the different keys on Linux, but also the Virtual Box and extra work trying to figure out copy/paste, moving files over, etc. The killer was when the VM prompted for an administrator's password, and of course I don't have it. There shouldn't be one needed..

[ After a search of the Forum the mystery word is revealed to be: qiime. I shoulda guessed it. ]

So I decided instead to deal with the dependencies for QIIME. I got almost all of them (like 24 or so), but for the "default pipeline" you really don't need so many. Particularly if you already have PyCogent and matplotlib installed, as I do, I would recommend just going down the list. It's not difficult, and it's better to be in an environment where you're comfortable working.

There was really no trouble configuring them (at least the essentials).

#1: Python 2.6

I have it, since I'm on OS X. I ran into a little trouble because (from playing with MacPorts) I still had /opt/local/bin on my path as changed in .bash_profile. The QIIME install checks this variable and modifies its scripts (the she-bang thing), so I had to fix that later on.

#2: Numpy 1.3.0

I have this too.

>>> import numpy >>> print numpy.__version__ 1.3.0

Not sure what the default OS X version is, but easy_install will fix you up quickly.

#3: PyCogent

Of course, this has its own dependencies. But we've been through that before (here, here, here).

#4: MUSCLE

This is not one of the primary dependencies, but it's an old friend. I upgraded to the latest and greatest version, but it caused an error in the QIIME tests. Luckily I still had muscle3.6_src around. As usual, I put it in ~/Software and do:

ln -s ~/Software/muscle3.6_src/muscle ~/bin/muscle

and I put ~/bin on my $PATH in .bash_profile:

export PATH=$HOME/bin/:$PATH

#5: MAFFT

It's an installer. Piece of cake. Check it:

> mafft

#6 uclust

See the QIIME install notes for this. This is the work of Robert Edgar (also the creator of MUSCLE), who is a talented programmer, but also a businessman. Hence, no source code, and no 64-bit for some things. But there's a download link on the QIIME site. The only problem is that Safari put a .txt extension on the download and I thought there was a problem, went off on a wild goose chase, and got a version that is not new enough. Enough said. Put it in ~/Software and link as usual

> uclust uclust v1.2.21q

#7 PyNAST

(download)

unpack and move to Software

python setup.py install cd tests python all_tests.py

After installing MUSCLE and MAFFT passes all tests except:

AssertionError: Got (DnaSequence(ACGTACG... 23), DnaSequence(ACGTACG... 23)), but expected (DnaSequence(ACGTACG... 23), DnaSequence(ACGTACG... 23))

That's pretty silly.

#8 Greengenes files

greengenes core set data file (fasta)
greengenes alignment lanemask file (txt)

Where to put them? I put them in qiime_tutorial, see below, but will eventually want them in Qiime somewhere.

#9 FastTree

fasttree 2.1.0 (src)

gcc -Wall -O3 -finline-functions -funroll-loops -o FastTree -lm FastTree-2.1.0.c

move to Software and link

#10 Java

I don't need to add the Java runtime. I got it from Apple a while ago (here):

> java -version java version "1.6.0_22" Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261) Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)

#11 RDP Classifier

rdp_classifier-2.0.1 (src)

in .bash_profile:

export RDP_JAR_PATH=$HOME/Software/rdp_classifier/rdp_classifier-2.0.jar

Check for mismatch between the name of the classifier file. What I actually got was named 2.0 not 2.0.1.

#12 QIIME

That's it for the default pipeline..

svn co https://qiime.svn.sourceforge.net/svnroot/qiime/trunk

I followed their advice and put the install scripts in a special place (but I wouldn't do that again).

python setup.py install --install-scripts=~/bin/qiime/bin/ cd tests python all_tests.py

The problem with the special directory is you will need a .qiime_config file (see the docs)

Make sure to put it in ~/.qiime_config.

Also not needed yet but I'll list them here anyway:

#13 BLAST

blast-2.2.22

We've done that one before. The BLASTMAT variable must point to the NCBI data directory. In .bash_profile:

export BLASTMAT=$HOME/Software/blast-2.2.22/data

#14 Infernal

(download)

./configure make make check sudo make install

#15 R

Run R and do:

install.packages('randomForest')

I upgraded to R 2.12.1 from 2.10.0, but it has some issue. Trying to get the ape package from CRAN hangs the app. I backed off to 2.11.1 (see here).

Still to come:

10 more bits of software.

But they're not needed for the first part of the tutorial, so we should just do that first. Oh..I also have Sphinx installed (from the PyCogent instructions), so the documentation got built. But I've just been working from the web version anyway. On to the fun stuff.