Unix spoken here (5)

A link to a great review of Unix text processing tools.

Unix spoken here (4)

There are a just few more Unix commands I should mention before we exhaust everyone's patience. (Three posts here, here, and here).

I'm sure you're familiar with sudo, which stands for superuser, do once. If not, you can see wikipedia. Normally you will use this to write (install) or even delete files in /usr/bin and the like.

> mv /usr/bin/znew ~/Desktop/znew mv: rename /usr/bin/znew to /Users/telliott_admin/Desktop/znew: Permission denied

Don't forget to read the man page on this one. There is always something interesting in the manual.

Many Unix commands appear simple but have numerous options and arguments that make them more like a Swiss Army Knife than a pocketknife. Output on the command line using man can be broken up into chunks or streamed using the down arrow, but maybe you want to study it or keep it as a reference.

For some reason you need to format the output from man in a special way before directing it to a file, for example:

man ls | col -bx > results.txt head -n 8 results.txt .. SYNOPSIS ls [-ABCFGHLOPRSTUW@abcdefghiklmnopqrstuwx1] [file ...]

Nearly the whole alphabet! But doing this without the col leads to selective stuttering:

LS(1) BSD General Commands Manual LS(1) N NA AM ME E l ls s -- list directory contents S SY YN NO OP PS SI IS S l ls s [- -A AB BC CF FG GH HL LO OP PR RS ST TU UW W@ @a ab bc cd de ef fg gh hi ik kl lm mn no op pq qr rs st tu uw wx x1 1] [_ f_ i_ l_ e _ ._ ._ .] D DE ES SC CR RI IP PT TI IO ON N For each operand that names a _ f_ i_ l_ e of a type other than directory, l ls s

head and tail are useful for looking at the first few lines or last few lines of a file. less is like what you get when you just do man some_command ---it gives chunks or streams.

Try man cat to see what else that can do.

There is also split but I couldn't figure out how to make it take my favorite separator ('\n\n').

diff does what you might expect. The use case might be that you have some good edits but also some bad edits to a file, so you want to revert to the old one, and then try the edits one by one.

> diff old.txt new.txt > results.txt > cat results.txt 28a29 > my edit 50,51c51 < < --- > deleted a line here

Or drop results.txt onto TextMate for a little color (graphic).



There's a command to find out information about different processes with finer grained control than Activity Viewer. For example:

ps -u username ps -U userid ps -p processid

Say:

> ps -p 48068 -O %mem PID %MEM TT STAT TIME COMMAND 48068 1.1 ?? S 0:00.91 /Applications/R64.app/Contents/MacOS/R -psn_0_2404939

The man page on ps is extensive.

find is tricky. I need a lot more practice to understand it. Here's a simple one, and then a fun one though:

> find . -name "old.txt" -print ./old.txt > find -f ~/* | wc 118791 235608 11536780

I "find" all the files under my home directory and feed the results to wc (word count), discovering that there are 118,791 lines in the output---that's the number of files! This link seems to have good info on find.

Sometimes I just want to use Python as a calculator or get an ascii value, but without calling up the interpreter (and having to dismiss it):

> python -c 'print 3.0/17' 0.176470588235 > python -c 'print ord("a")' 97 > python -c 'print chr(101)' e > python -c 'import math; print math.e' 2.71828182846

> hexdump -C post.txt 00000000 4d 61 6e 79 20 55 6e 69 78 20 63 6f 6d 6d 61 6e |Many Unix comman| 00000010 64 73 20 61 70 70 65 61 72 20 74 6f 20 62 65 20 |ds appear to be | 00000020 73 69 6d 70 6c 65 20 62 75 74 20 68 61 76 65 20 |simple but have |

Finally, here are a few more that either I've found useful or I'm sure they will be, but I don't feel expert enough to try to explain them. Have fun exploring:

alias awk curl hash md5 openssl sha1 set tar top touch unset

If you get bored here; or try to check out how Greg's email address gets generated!

And there's always ssh. But that really does deserve a post of its own.

Unix spoken here (3)

Let's do some more with Unix commands. Previous posts here and here.

You might have noticed that I have a very short prompt in Terminal, just the character `>` followed by a space. That's because I have an (invisible) file in my home directory named .bash_profile, containing this line:

> cat ~/.bash_profile PS1="> "; export PS1

We can change back to the original behavior by changing the name of the file or moving it, and then re-starting Terminal:

mv ~/.bash_profile ~/Desktop/x.txt

c-98-236-78-154:Desktop telliott$

We could comment out that line of the file (now that it's visible with no leading `.` in the filename, we can easily open it in TextEdit).

#PS1="> "; export PS1

But I like it, so I'll revert to the original situation.

I think there is a way to get TextEdit to show hidden files in the Open dialog, but I forget how at the moment. In any event, it's interesting that having opened and closed the file while it was visible on the Desktop, Open Recent now shows .bash_profile as an option.

Another way to edit the file is to use pico (a command line editor):

> pico ~/.bash_profile

I'm sure you can figure out how to use pico.

One more useful thing in .bash_profile is modification to the $PATH variable. Let's look at that from the command line:

> echo $PATH /opt/local/bin:/opt/local/sbin:/Users/telliott/bin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/ncbi/blast/bin:/usr/texbin:/usr/X11/bin

When I call ls, for example, these directories are searched in turn until the program is discovered finally. It is in:

> whereis ls /bin/ls

If I wanted to modify the $PATH variable I could do:

> PATH=~/Desktop:$PATH; export PATH > echo $PATH /Users/telliott/Desktop:/opt/local/bin:/opt/local/sbin:/Users/telliott/bin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/ncbi/blast/bin:/usr/texbin:/usr/X11/bin

My $PATH is pretty complicated for several reasons: use of MacPorts, some things related to ncbi and X11, etc. It would take too long to explain.

The shell has command history. If you want to repeat a command, just press the up arrow key. To repeat a command further back, keep pushing until you see the one you want. Then press Return. Suppose you have a series of commands (say 3 in a row), and the first of these is 6 commands ago. Press the arrow 6 times, then return. Press the arrow 6 times, then return. Press the arrow 6 times, then return. Because the command history is updated each time, 6 arrow presses gets you back exactly where you need to be for each of the three commands in the sequence.

Alternatively you could do history:

> history 501 cd Desktop 502 pico ~/bash_profile 503 echo $PATH 504 file ls 505 whereis ls 506 PATH=~/Desktop:$PATH; export $PATH 507 PATH=~/Desktop:$PATH; export PATH 508 echo $PATH 509 history

which lists each command I've entered since the last time I booted my machine (even saves between Terminal sessions). One can select a command from the history by number or by partial name match:

> !505 whereis ls /bin/ls > !wh whereis ls /bin/ls

First it prints the command, and then the result. You can combine commands on a single line with `;`

> !504; !512 whereis ls; echo abc /bin/ls abc > history | tail -2 515 whereis ls; echo abc 516 history > !515 whereis ls; echo abc /bin/ls abc

We're getting pretty good with this stuff! We don't really want to list all 500+ commands in the history, just the last two. So we feed the results of history to the Unix tool called tail through a `|`, a Unix "pipe." tail takes an option for how many lines at the end of the "file" to display.

The Unix command grep searches lines of text for those matching a pattern. It's particularly useful in combination with `|`.

> history | grep pico 40 pico ~/.hgrc 298 pico 502 pico ~/bash_profile 515 history | grep pico

Here, I edited some file with pico but I can't remember which one it was. So the entire output of the history is fed through the pipe to grep, searching for lines containing the pattern. Now I can just do:

> !40 pico ~/.hgrc

I won't show you the file.

The last command for this post is ln with the option -s which makes a "soft link" to a file. I can never remember, but you want to put the existing file first (that's the opposite of the way I would design it):

> cat temp/y.txt echo "abc" > ln -s foo temp/y.txt ln: temp/y.txt: File exists > ln -s temp/y.txt foo > ls foo temp > cat foo echo "abc" > ls -l total 8 lrwxr-xr-x 1 telliott staff 10 Jan 31 07:53 foo -> temp/y.txt drwxr-xr-x 5 telliott staff 170 Jan 31 07:53 temp

Now the listing for foo shows an `l` as the first character in the file mode, and it also shows -> temp/y.txt, where it links to.

Unix spoken here (2)

Let's continue with an introduction to Unix commands. First post here. And here are two more good links (one, two).

Let's start by putting some data into a file. We can use the command echo and the `>` redirection operator. Take a look at what's there with cat:

> cd > cd Desktop > echo "abc" > x.txt > cat x.txt abc

Suppose we wish to move this file to a new sub-directory. First we construct the new directory and then we move the file with mv:

> mkdir temp > mv x.txt temp/x.txt > ls temp x.txt > ls temp

Let's make the file `x.txt` actually do something, e.g.:

> echo "echo xyz" > temp/x.txt > cat temp/x.txt echo xyz > sh temp/x.txt xyz

Note that we wrote over the previous contents of temp/x.txt and the file system did not warn us first. There is no way to recover from this if it was a mistake!

The sh command executes the text in the file you give it; in this case the output is familiar. Normally, we'd probably use the file extension .sh for such a script, but it's not required. (Even with Python scripts, the .py extension is only required if you want to import a module).

We can also use the she-bang thing (#!) to make the code in a file executable. Use TextEdit to save the following code in a file y.txt on the Desktop:

#! sh echo abc

Then we do the following. An executable can be executed just by entering its path, but.. there is a restriction that you can't do that for a program in the current directory. You must specify it as the relative path from the directory where we are currently./executable where the `.` means this directory (the restriction has never made a lot of sense to me either, but it's apparently a security hole):

> cat y.txt #! sh echo abc > ./y.txt -bash: ./y.txt: Permission denied

What has happened? We need to make the file "executable" by setting the "permission bits." Without getting into too much detail, these are output using a new option with the ls command:

> ls -l y.txt -rw-r--r-- 1 telliott staff 19 Jan 30 20:08 y.txt > chmod 711 y.txt > ls -l y.txt -rwx--x--x 1 telliott staff 19 Jan 30 20:08 y.txt > ./y.txt abc

The three permissions are r (read), w (write) and x (execute), listed in order for the user, her group, and the world. 7 is all of them. `x` is 1, `w` is 2, `r` is 4, and the combinations result from addition.

There are various shorthands for this. First reverting the previous change, then we could do this:

> chmod 644 y.txt > chmod u+x y.txt > ls -l y.txt -rwxr--r-- 1 telliott staff 19 Jan 30 20:08 y.txt

We have given the user (only) permission to execute the file.

> ls -l ~/Desktop total 1 drwxr-xr-x 2 telliott staff 102 Jan 30 20:08 temp -rw-r--r-- 1 telliott staff 15 Jan 30 20:08 y.txt

When we do ls on the Desktop directory (above), we get a listing for both the temp sub-directory and the file. Note that the first character in the "file mode" is d for the directory. staff is my group name, and after that comes the number of bytes in each "file", including directories.

From man ls

The Long Format
If the -l option is given, the following informa-
tion is displayed for each file: file mode, num-
ber of links, owner name, group name, number of
bytes in the file, abbreviated month, day-of-
month file was last modified, hour file last mod-
ified, minute file last modified, and the path-
name. In addition, for each directory whose con-
tents are displayed, the total number of 512-byte
blocks used by the files in the directory is dis-
played on a line by itself, immediately before
the information for the files in the directory.
If the file or directory has extended attributes,
the permissions field printed by the -l option is
followed by a '@' character. Otherwise, if the
file or directory has extended security informa-
tion (such as an access control list), the per-
missions field printed by the -l option is fol-
lowed by a '+' character.

And finally, ls also has a recursive option, which can be grouped into -lR:

> ls -lR total 8 drwxr-xr-x 3 telliott_admin staff 102 Jan 31 06:59 temp -rw-r--r-- 1 telliott_admin staff 15 Jan 31 07:00 y.txt ./temp: total 8 -rw-r--r-- 1 telliott_admin staff 4 Jan 31 06:59 x.txt

Let's continue by copying this file into our temp directory:

> cp y.txt temp > ls temp y.txt > ls temp x.txt y.txt

On the other hand, mv actually "moves" the file:

> mv y.txt temp > ls temp > ls temp x.txt y.txt

Once again, note the lack of warnings. The old copies of y.txt on the Desktop (and in temp) are gone forever.

Eventually, we tire of this. First we try to delete a single file:

> rm temp/y.txt > ls temp x.txt

But maybe temp has lots of files, so we try:

> rmdir temp rmdir: temp: Directory not empty > rm -r temp > ls >

The `-r` option is really dangerous. The classic Unix "joke" is to tell someone to do some combination of rm and -r and *. Don't fall for it.

You might think about looking at the man page for rm and maybe using the -i option.

Unix spoken here (1)

There are probably hundreds of pages on the web with good info about basic Unix commands (e.g. here, here). I thought I'd give some of the commands I think are most useful, but I have to start at the beginning. So, here goes with Unix in 5 minutes, Unleashed, For Dummies..

> cd > pwd /Users/telliott > cd Desktop > pwd /Users/telliott/Desktop > cd ~ > pwd /Users/telliott > cd Desktop > ls x.txt > cat x.txt some data

The first command, cd, says to change directory; with no other argument given, we go to our home directory. To see where we are in the file system we can do pwd: print working directory. Also, the ~ symbol is a shorthand for the user's home directory.

The "shell" is smart and has "command-completion." For example, from my home directory I enter cd De and then press tab. The shell fills out "Desktop."

Another very common command is ls: list the contents of the current directory (since there is no argument). And to see the contents of a file you might do cat with the results shown.

> ls -a . .DS_Store .temp .. .localized x.txt > ls /usr/bin/pyth* /usr/bin/python /usr/bin/python2.6-config /usr/bin/python-config /usr/bin/pythonw /usr/bin/python2.5 /usr/bin/pythonw2.5 /usr/bin/python2.5-config /usr/bin/pythonw2.6 /usr/bin/python2.6

Any directory may have "hidden" files whose names are prefaced by a dot `.`. These can be shown with ls -a. There are many (many) other options with results cataloged under man some_command.

The shell understands paths to files in both an absolute and relative sense. A path starting with / is absolute, otherwise it's relative. Hence /usr/bin is a directory starting from the root of the file system, while Desktop is only understood as a valid destination if I'm in my home directory (or some other directory with Desktop as a sub-directory).

The character * is a wild card and matches anything; ls /usr/bin/pyth* lists all the entries matching pyth- + something in the given directory.

> pushd /usr/bin /usr/bin ~/Desktop > pwd /usr/bin > popd ~/Desktop > pwd /Users/telliott/Desktop > cat -n ~/Desktop/x.txt 1 some data > cal January 2011 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 > cd .. > pwd /Users/telliott_admin

Sometimes we want to make a jump in the file system but intend to come right back. pushd and popd are useful here, with results as shown.

As one more example of using options or arguments to a command, cat takes the option -n, which numbers the output lines. cal is probably faster than finding the Calendar.app, since I don't usually have it in the Dock.

Finally, to move toward the tips of the file system tree, you specify the directory at the next level so it has to be explicit, but there is only one parent, which we can reach implicitly with cd .., meaning "go up."

User's environment in OS X

The question arises: how to get access to the user's path on OS X. Stack Overflow does not disappoint (here):

// gcc -o test test.m -framework Foundation -fobjc-gc-only // .test #import <Foundation/Foundation.h> int main (int argc, const char* argv[]) { NSDictionary *eD = [[NSProcessInfo processInfo] environment]; NSLog(@"%@",[eD description]); return 0; }

$ ./test 2010-12-14 11:56:23.002 test[650:903] { "Apple_PubSub_Socket_Render" = "/tmp/launch-bvNjPH/Render"; "COMMAND_MODE" = unix2003; DISPLAY = "/tmp/launch-cWN8zD/org.x:0"; HOME = "/Users/telliott_admin"; LANG = "en_US.UTF-8"; LOGNAME = "telliott_admin"; OLDPWD = "/Users/telliott_admin"; PATH = "/Users/telliott_admin/bin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin"; PWD = "/Users/telliott_admin/Desktop"; SHELL = "/bin/bash"; SHLVL = 1; TERM = "xterm-color"; "TERM_PROGRAM" = "Apple_Terminal"; "TERM_PROGRAM_VERSION" = 273; TMPDIR = "/var/folders/rK/rK4-yxtZHUGADBsC5YPRk++++TI/-Tmp-/"; USER = "telliott_admin"; "_" = "./test"; "__CF_USER_TEXT_ENCODING" = "0x1F5:0:0"; }

Timeit from the command line in Python

Python has a module called timeit, whose function should be obvious. It can be used simply from the command line.

python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...]

The -s flags an argument that should be executed only once. Multiple -s [setup code] lines can be included. There can also be multiple statements which is, or are, executed many times and the timing reported. I was curious about the relative speed of two approaches to grouping list elements (post here). I put the code for the two functions into a module test.py (the listing is at the end of the post).

Here is what I got:

$ python -m timeit -s 'import test' 'list(test.grouper(2,"abcdef"))' 100000 loops, best of 3: 5.34 usec per loop $ python -m timeit -s 'import test' 'test.grouper(2,"abcdef")' 100000 loops, best of 3: 2.19 usec per loop $ python -m timeit -s 'import test' 'list(test.chunks(2,"abcdef"))' 100000 loops, best of 3: 2.85 usec per loop $ python -m timeit -s 'import test' 'test.chunks(2,"abcdef")' 1000000 loops, best of 3: 0.685 usec per loop

The itertools solution takes 2-3 times as long, and getting the actual list rather than a generator or iterator has overhead as well.

from itertools import izip_longest def grouper(n, iterable, fillvalue=None): global itertools "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx" args = [iter(iterable)] * n return izip_longest(fillvalue=fillvalue, *args) def chunks(n, L): """ Yield successive n-sized chunks from L. """ for i in xrange(0, len(L), n): yield L[i:i+n]