OS X Library basics 2

Here is a second example, of a dynamic library (adapted from a Stack Overflow answer here). The same .c files are used from the first example (here).

step 1, create libadd.dylib

$ gcc -g -Wall -c add*.c $ file add1.o add1.o: Mach-O 64-bit object x86_64 $ gcc -dynamiclib -current_version 1.0 add*.o -o libadd.dylib $ file libadd.dylib libadd.dylib: Mach-O 64-bit dynamically linked shared library x86_64 $ otool -L libadd.dylib libadd.dylib: libadd.dylib (compatibility version 0.0.0, current version 1.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.1)

Find out more about otool by doing

man otool | col -b > result.txt

col helps to format the output from man properly. I learned we can do this:

$ otool -tvV "libadd.a(add1.o)" libadd.a(add1.o): (__TEXT,__text) section _f1: 00000000 pushl %ebp 00000001 movl %esp,%ebp 00000003 pushl %ebx 00000004 subl $0x14,%esp 00000007 calll 0x0000000c 0000000c popl %ebx 0000000d movl 0x08(%ebp),%eax 00000010 movl %eax,0x04(%esp) 00000014 leal 0x2fd-0xc(%ebx),%eax 0000001a movl %eax,(%esp) 0000001d calll _printf+0x100000000 00000022 movl 0x08(%ebp),%eax 00000025 incl %eax 00000026 addl $0x14,%esp 00000029 popl %ebx 0000002a leave 0000002b ret

I don't speak assembly, but I can get the general idea.
Step 2, create useadd

$ gcc -c useadd.c $ gcc -v useadd.o ./libadd.dylib -o useadd Using built-in specs. Target: i686-apple-darwin10 Configured with: /var/tmp/gcc/gcc-5664~89/src/configure --disable-checking --enable-werror --prefix=/usr --mandir=/usr/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin10 --with-gxx-include-dir=/usr/include/c++/4.2.1 --host=i686-apple-darwin10 --target=i686-apple-darwin10 Thread model: posix gcc version 4.2.1 (Apple Inc. build 5664) /usr/libexec/gcc/i686-apple-darwin10/4.2.1/collect2 -dynamic -arch i386 -macosx_version_min 10.6.5 -weak_reference_mismatches non-weak -o useadd -lcrt1.10.6.o -L/usr/lib/i686-apple-darwin10/4.2.1 -L/usr/lib/gcc/i686-apple-darwin10/4.2.1 -L/usr/lib/gcc/i686-apple-darwin10/4.2.1 -L/usr/lib/gcc/i686-apple-darwin10/4.2.1/../../../i686-apple-darwin10/4.2.1 -L/usr/lib/gcc/i686-apple-darwin10/4.2.1/../../.. useadd.o ./libadd.dylib -lSystem -lgcc -lSystem $ nm -gpv useadd 0000202c D _NXArgc 00002030 D _NXArgv 00002038 D ___progname 00001000 A __mh_execute_header 00002034 D _environ 00001f02 T _main 00001ec4 T start U _exit U _f1 U _f2 U _printf U dyld_stub_binder $ ./useadd f1: 1; main 2 f2: 10; main 12

Step 3. Use an environment variable to give output when libraries are loaded:

$ export DYLD_PRINT_LIBRARIES=1 $ ./useadd dyld: loaded: /Users/telliott_admin/Desktop/add/./useadd dyld: loaded: /Users/telliott_admin/Desktop/add/libadd.dylib dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /usr/lib/system/libmathCommon.A.dylib f1: 1; main 2 f2: 10; main 12 $ mv libadd.dylib /tmp dyld: loaded: /bin/mv dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /usr/lib/system/libmathCommon.A.dylib $ export DYLD_LIBRARY_PATH=/tmp $ ./useadd dyld: loaded: /Users/telliott_admin/Desktop/add/./useadd dyld: loaded: /tmp/libadd.dylib dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /usr/lib/system/libmathCommon.A.dylib f1: 1; main 2 f2: 10; main 12 $ unset DYLD_LIBRARY_PATH

Notice in the second run of ./useadd, we loaded /tmp/libadd.dylib.
Finally, let's snoop on Python:

$ python dyld: loaded: /usr/bin/python dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /usr/lib/libauto.dylib dyld: loaded: /usr/lib/libicucore.A.dylib dyld: loaded: /usr/lib/libobjc.A.dylib dyld: loaded: /usr/lib/libz.1.dylib dyld: loaded: /usr/lib/libstdc++.6.dylib dyld: loaded: /usr/lib/system/libmathCommon.A.dylib dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.6/Python dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation dyld: loaded: /usr/lib/libauto.dylib dyld: loaded: /usr/lib/libicucore.A.dylib dyld: loaded: /usr/lib/libobjc.A.dylib dyld: loaded: /usr/lib/libz.1.dylib dyld: loaded: /usr/lib/libstdc++.6.dylib dyld: loaded: /usr/lib/system/libmathCommon.A.dylib Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/readline.so dyld: loaded: /usr/lib/libedit.2.dylib dyld: loaded: /usr/lib/libncurses.5.4.dylib >>> import string dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/strop.so >>> import sys >>> import math dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/math.so >>> [1]+ Stopped python $ unset DYLD_PRINT_LIBRARIES

We can see that some .so libraries can be loaded. However, there are some fundamental differences between the Unix method of shared object .so libraries and the OS X / Mach model of bundles and frameworks.
That's for the future!

OS X Library basics

This is an elementary post about libraries on OS X. As in other areas, I'm just getting started with this, so if you spot an error, please let me know. The primary purpose is to orient myself to help in troubleshooting when a software install at the command line runs into a problem.

If we have our code split up into a bunch of different files, compiled separately, we can link them into a single executable as discussed in a previous post (here), which uses the make tool.

It also useful to define libraries that may contain a number of different modules which are already pre-compiled and linked. A primary distinction is between static libraries, which are copied into the target application, and dynamic libraries which are not.

We'll start with the static version in this post. The example is adapted from Dalrymple & Hillegass (Advanced Mac OS X Programming) which is still a valuable resource even though it's a bit dated. It looks like there may be a new edition later this year. Here is a simple file add1.c with a nice function:

#include <stdio.h> int f1(int x) { printf( "f1: %d;", x ); return x+1; }

A similar function f2 is defined in add2.c We do the following:

gcc -g -Wall -c add*.c

(-g adds debugging symbols, -Wall shows all warnings). The result is familiar: two object files add1.o and add2.o.

These functions are used by your program, defined in useadd.c:

#include <stdio.h> extern int f1(int x); extern int f2(int x); int main(int argc, char** argv){ printf(" main %d\n", f1(1)); printf(" main %d\n", f2(10)); return 0; }

Rather than referring to a header file for the definitions of f1 and f2, we make a declaration of extern. Now the book says to do:

ar crl libadd.a add*.o gcc -g -Wall -o useadd useadd.c -L. -ladd

with the flags to ar:

c create if needed r replace / add l next token is name of file to generate

and flags passed through gcc to ld are a search path and file description as described here.

It works:

$ ./useadd f1: 1; main 2 f2: 10; main 12

The ar tool is Unix archive utility, which has for the most part been superseded by tar. Here is what the output file looks like for a run of ar on two simple text files. I think you can guess what they contain:

!<arch> file1.txt 1294193315 501 20 100644 9 ` my data1 file2.txt 1294193320 501 20 100644 9 ` my data2

Now, Apple suggests that you use libtool:

libtool -static add*.o -o libadd.a

[UPDATE: The docs (here) say that everything must be compiled as well as linked with -static for this to work]

Also, for this example, you don't need the search paths, but could just do:

$ gcc -g -Wall -o useadd useadd.c libadd.a

It's important to list the library after useadd.c, otherwise the linker will discard the library symbols as unneeded. We can explore the symbols defined in our library (or any object file for that matter):

$ nm libadd.a libadd.a(add1.o): 00000000 T _f1 U _printf libadd.a(add2.o): 00000000 T _f2 U _printf

The function names are preceded by an underscore. The U means that printf is as yet undefined, and the T stands for (Text)---not sure what that refers to.

$ gcc -g -Wall -c useadd.c $ nm useadd.o U _f1 U _f2 00000000 T _main U _printf

Here we see that in useadd.o the functions f1 and f2 are as yet undefined. Another very interesting thing is to set the environment variable:

$ export DYLD_PRINT_LIBRARIES=1 $ ./useadd dyld: loaded: /Users/telliott_admin/Desktop/add/./useadd dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /usr/lib/system/libmathCommon.A.dylib f1: 1; main 2 f2: 10; main 12

We can watch as the libraries are loaded. There are a bunch of other things here.

simple C example 4: file read

This post describes basic use of fscanf to read data in from a file. A detailed manual page about the function is here.

The usual approach is that we must first pre-allocate storage of the appropriate type for the data we expect (at least, if we wish to save it for further manipulation). So far I've used char, int and double data.

A difficulty is that we usually don't know how much data we will read from the file. For the sites program (here), when we read scores or counts, the file contains an integer as the first value and that specifies how much data is present. But in general we don't know, so we'll have to allocate storage as we go.

fscanf takes a pointer to the storage as an argument. We can either do this directly, or pass the address of a variable of the correct type. There are lots of options for fscanf, including the ability to read data in larger chunks, to read data of different types (in a specified order), or to skip certain characters, but I'm not going to worry about those complications here.

In the first example, we don't save the data, just read it and echo to stdout. The file 'lorem.txt' is the default source, but an alternate can be specified on the command line.

example1.c:

#include <stdio.h> #include <stdlib.h> int main(int argc, const char* argv[]) { const char *ifn; if (argc > 1) { ifn = argv[1]; } else { ifn = "lorem.txt"; } FILE *ifp = fopen(ifn,"r"); if (ifp == NULL) { printf("open file failed with: %s\n", ifn); exit(EXIT_FAILURE); } char c; int result; result = fscanf(ifp,"%c",&c); if (result == -1) { printf("file read failed with: %s\n", ifn); exit(EXIT_FAILURE); } int count = 0; while (result != -1) { if ((count > 40) && (c==' ')) { printf("\n"); count = 0; } else { printf("%c", c); count++; } result = fscanf(ifp,"%c",&c); } printf("\n"); return 0; }

output:

$ gcc example1.c -o test $ ./test Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

We read integers (or floats) by appropriate changes to the arguments to fscanf:

int value; int result; result = fscanf(ifp,"%d",&value);

In the third example, we're reading the DNA sequence of E. coli (more than 4 million nucleotides). A rather crude approach is to just allocate an array of sufficient size (N = 5000000 works). A more flexible and less wasteful implementation would appropriately scale up the memory usage as needed. We substitute the following for the second half of the code above (starting at int count = 0;), reading just the first 240 nt:

int N = 240; char *buffer = (char *) malloc (N+1); if (buffer == NULL) { printf("not enough memory\n"); exit(EXIT_FAILURE); } int i=0; char *p = buffer; *p = c; while ((result != -1) && (i < N)) { p++; i++; result = fscanf(ifp,"%c", p); } int count = i; for (i=0; i < count; i++) { if ((i) && (!(i%60))) { printf("\n"); } printf("%c", buffer[i]); } printf("\n"); return 0; }

Output:

$ gcc example3.c -o test $ ./test ECsequence.txt agcttttcattctgactgcaacgggcaatatgtctctgtgtggattaaaaaaagagtgtc tgatagcagcttctgaactggttacctgccgtgagtaaattaaaattttattgacttagg tcactaaatactttaaccaatataggcatagcgcacagacagataaaaattacagagtac acaacatccatgaaacgcattagcaccaccattaccaccaccatcaccattaccacaggt

Of course, the appropriate source file must exist for this to work.

Zipped project files on Dropbox (here)

simple C example 3: structs

Here's a simple struct example, containing two elements, a char * and an int. We typedef it to be an "A" and then construct one in four different ways: (i) in main, (ii) by calling an auxiliary function, (iii) by assigning the address of the A from (i), and by using malloc in an auxialiary function.

The syntax for access to the member elements is different in the case of a pointer to the struct variable and a struct variable itself.

As you can see from the output, all four approaches give valid A's but the last one has a quite different address. It is on the "heap" rather than on the "stack."

One thing I don't understand is whether examples 2 and 4 are dangerous. Under what circumstances do objects that we obtain from a function go away when those functions return? If you know, I'd be grateful for your insight. [UPDATE: I'm pretty sure example 2 is not something you should do, because after the function returns that memory will be reallocated. I guess we got away with it here because no further allocations were made.]

Output:

I'm an A: A 1 0x7fff5fbff9b0 me too : A 2 0x7fff5fbff9a0 me three: A 1 0x7fff5fbff9b0 me four : A 3 0x100100080

Code listing:

#include <stdio.h> #include <stdlib.h> struct myStruct { char *s; int x; }; typedef struct myStruct A; A stack_A(char *, int); A * heap_A(char *, int); // bad bad bad A stack_A(char *s, int x) { A a; a.x = x; a.s = s; return a; } A * heap_A(char *s, int x) { A * p = (A *) malloc(sizeof(A)); p->x = x; p->s = s; return p; } int main() { A a1; a1.x = 1; a1.s = "A\0"; printf("I'm an A: %s %d %p\n", a1.s, a1.x, &a1); A a2 = stack_A("A\0",2); printf("me too : %s %d %p\n", a2.s, a2.x, &a2); A *p = &a1; printf("me three: %s %d %p\n", p->s, p->x, p); A *p2 = heap_A("A\0",3); printf("me four : %s %d %p\n", p2->s, p2->x, p2); return 0; }

DNA binding sites 7

Continuing in the same vein as a number of recent posts (some links here), the Dropbox link below leads to my version of a site analysis program written in C. It's not very user friendly, for example, paths to the input files are hard-coded. A substantial part of the effort required went into ScoreList.c, which implements a function for constructing a list of scores from a list of nucleotide counts for each position, and a second function which reads such a score list in from disk.

The logic of the program was explained in the previous post. I won't spend much time explaining the details here.

Let's just say that if this blog has a motto it is: "learn by doing."

So... get in there and try writing your own version. If you get stuck, you can see how (or if) I handled the problem.

A few other notes: I tested the program by using a list of counts for crp sites obtained as described (here). The output shows the position, score and sequence of crp sites in the E. coli genome. The first few (threshold = 12) are:

$ ./test 18968 14.33 tattgtgaactatcgcaaagaa 42067 19.16 ttctgtgattggtatcacattt 42187 12.44 attggtgatccataaaacaata 49633 14.48 aagagtgacgtaaatcacactt 70157 15.02 aagtgtgacgccgtgcaaataa 141283 18.95 atgtgtgatcgtcatcacaatt 141663 12.43 tgatgtgaaaatcctcaaagat 184120 13.67 aattgtgcttattttagcattt 223495 13.60 ggatgtgaatcacttcacacaa 243785 12.34 atttctgacgttagtcatattt 268498 13.40 taatgtgaacatgatcaacgaa 281362 18.40 atatgtgatccagcttaaattt 304272 12.45 tgttgttattcactacacgttt 312539 17.25 ttttttgacatgtatcacaaat 365617 14.65 atgagtgagctaactcacatta ..

After implementing the algorithm (which handles the sequence one character at a time) I realized that it would be good to have the site sequence available at the time of printing (as shown above). This required remembering the current site's sequence, which I grafted onto the program at the very end. Luckily it didn't change the running time by much. I think now that while the circular linked list seemed like an elegant approach, it caused as many problems as it solved, and I probably wouldn't do it again.

You can tell just from looking at the patterns that we are getting good matches to the crp consensus [T/A]₃TGTGAN₆TCACA[T/A]₃. Also, the last site is the lac 1 (lacZ) site---reversed. So I think it's working OK.

Finally, it runs in less than 2 seconds! That's a huge improvement on the previous time. Zipped files on Dropbox (here).

DNA binding sites 6

A few weeks ago I had some posts about finding binding sites in DNA using a simple PSSM (position specific scoring matrix; last post here). The time required to evaluate all the potential sites in a bacterial genome of ≈ 5 million bp is prohibitively long (> 1 min on my machines), so I wanted to explore ways to do it faster. I started looking at Cython (here), but then realized that I need to brush up my C skills first (here).

I also had an idea that I thought at first would make the code faster, though probably it doesn't. Then I thought it would make the code clearer, though I realized after actual implementation that it doesn't do that either. What it does do is make our pass through the sequence simpler in logical terms, and it's the basis of my C-program to evaluate sites or motifs in a DNA sequence. We construct a circularly linked list, as illustrated schematically in the graphic. Each item (node) in the list holds the current value for an accumulating score that will become the score for an individual site in the sequence.



Probably it's best to illustrate with an example. Let's say we have a scoring system used to evaluate potential sites. For example, a site with a in position 1 (we'll use 1-based indexing here) contributes score a1, c in position 2 adds c2, etc.

Suppose we're looking for sites of length N = 4, and we have this sequence: acgtg. We construct a list like the following (shown here in rows to make the layout simple). After the initial priming, we end up with this:

-> 1 2 g1 3 c1 g2 4 a1 c2 g3

I set up a toy example with scores of
0.01 .. 0.04 for a1 .. a4; 0.11 to 0.14 for c1 .. c4, etc.
Output from the priming phase looks like this:

nt = a item = 4 before: 0.00, add: 0.01 after: 0.01 nt = c item = 3 before: 0.00, add: 0.11 after: 0.11 item = 4 before: 0.01, add: 0.12 after: 0.13 nt = g item = 2 before: 0.00, add: 0.21 after: 0.21 item = 3 before: 0.11, add: 0.22 after: 0.33 item = 4 before: 0.13, add: 0.23 after: 0.36 1 0.000000 2 0.210000 3 0.330000 4 0.360000 end priming

Now we consider the next nucleotide in the sequence: t. We add the appropriate scores for t to each item in the list, then read the score for the item that contains a total of N scores, and finally, zero that item. The score we read is the score for the site that has sequence acgt. Schematically:

t 1 t1 2 g1 t2 3 c1 g2 t3 -> 4 a1 c2 g3 t4 1 t1 2 g1 t2 3 c1 g2 t3 -> 4

Output looks like this:

t item = 1 before: 0.00 add: 0.31 after: 0.31 item = 2 before: 0.21 add: 0.32 after: 0.53 item = 3 before: 0.33 add: 0.33 after: 0.66 item = 4 before: 0.36 add: 0.34 after: 0.70 zeroing item = 4 final score = 0.70

The next nucleotide is g:

g 1 t1 g2 2 g1 t2 g3 -> 3 c1 g2 t3 g4 4 g1 1 t1 g2 2 g1 t2 g3 -> 3 4 g1

g item = 4 before: 0.00 add: 0.21 after: 0.21 item = 1 before: 0.31 add: 0.22 after: 0.53 item = 2 before: 0.53 add: 0.23 after: 0.76 item = 3 before: 0.66 add: 0.24 after: 0.90 zeroing item = 3 final score = 0.90

The site acgt has score:
a1 + c2 + g3 + t4 = 0.01 + 0.12 + 0.23 + 0.34 = 0.70


The site cgtg has score:
c1 + g2 + t3 + g4 = 0.11 + 0.22 + 0.33 + 0.24 = 0.90

Zipped files on Dropbox (here). These will change in the next few days as I debug and test the project more...

The C code for the linked list looks like this:

item *make_circular_linked_list(int N) { int i; item *current, *first, *previous; for (i=0; i<N; i++) { current = (item *) malloc(sizeof(item)); current->num = i+1; current->score = 0; if (i==0) { first = current; } else { previous->next = current; } previous = current; } current->next = first; return first; }

simple C example 2

Continuing from last time (here), we put all the previous code (minus the function main) into a separate file print_bits.c and then put the declarations of our two functions in a header file: print_bits.h:

void print_byte(unsigned char); void print_word(const char *);

We modify main, adding code to get information passed in from the operating system, and put that in a separate file bits.c:

#include <stdio.h> #include "print_bits.h" #include "cast.h" int main(int argc, const char* argv[]) { const char *greeting = "Hello world!\0"; printf("greeting: %s\n\n", greeting); char c; c = 'a'; printf("%c\n", c); print_word(greeting); int i; printf("%s", "\n"); if (argc > 1) { for (i=1; i < argc; i++) { print_word (argv[i]); } printf("%s", "\n"); } cast_int(); return 0; }

We need to #include <stdio.h> to use printf, but we don't need the other C library headers here; they are placed in print_bits.c. There is a second set of new files (cast.c and cast.h) that I'll explain below.

Our project isn't very complicated but it still can benefit from using the make tool. We put a file in the same directory as we're working called Makefile. Ours has this code:

bits: bits.o print_bits.o cast.o gcc -o bits bits.o print_bits.o bits.o: bits.c gcc -c bits.c print_bits.o: print_bits.c gcc -c print_bits.c cast.o: cast.c gcc -c cast.c clean: rm -f bits.o print_bits.o cast.o bits

As explained in the manual for make and Norm Matloff's introduction (here), these instructions consist of

target ... : prerequisites ... recipe ...

When invoked by itself, make will try to build an executable named bits by linking 3 object files, these in turn depend on the listed code files. The -c compiler directive says not to do the linking at that stage. And clean is self-explanatory. One odd requirement is that after the colon, and on each line that's shifted out, it must be a tab character.

The point is that make will call the compiler only if the timestamp of last modification to a .c file shows it's necessary. So in a complex project, only those files that have been changed get re-compiled in each debug cycle.

[UPDATE: As pointed out in comments, there was a problem with an earlier version of this post, because I forgot the proper syntax for our personal header files, which is #include "print_bits.h" etc. rather than #include <stdio.h>. The search path is determined by the format of the #include. See here. My makeshift solution :) was to tell make where to look for our files using -I., which says to search for them in the current directory. ]

The last part of this little project explores casting. We have two int variables j and k, and a long (int) variable m, declared and assigned in turn (in cast.c). In order to examine what these look like in memory, we do this:

char *c = (char *) &j;

which assigns the address of j to a char pointer c. We tell the compiler that yes, we really want to do this, using a cast (char *). We examine the surrounding bytes as follows:

int i; int j = 10; int k = 660; long m = 21; char *c = (char *) &j; printf("address of c: %p%s", c, "\n\n"); for (i=-16; i < 4; i++) { print_byte(c[i]); printf(" %p%s", &c[i], "\n"); } printf(" int: %ld%s", sizeof(int), "\n"); printf(" long: %ld%s", sizeof(long), "\n"); printf("size_t: %ld%s", sizeof(size_t), "\n");

$ make gcc -c bits.c -I. cc bits.o print_bits.o cast.o -o bits $ ./bits address of c: 0x7fff5fbff9d8 00010101 21 0x7fff5fbff9c8 00000000 0 0x7fff5fbff9c9 00000000 0 0x7fff5fbff9ca 00000000 0 0x7fff5fbff9cb 00000000 0 0x7fff5fbff9cc 00000000 0 0x7fff5fbff9cd 00000000 0 0x7fff5fbff9ce 00000000 0 0x7fff5fbff9cf 00000000 0 0x7fff5fbff9d0 00000000 0 0x7fff5fbff9d1 00000000 0 0x7fff5fbff9d2 00000000 0 0x7fff5fbff9d3 10010100 148 0x7fff5fbff9d4 00000010 2 0x7fff5fbff9d5 00000000 0 0x7fff5fbff9d6 00000000 0 0x7fff5fbff9d7 00001010 10 0x7fff5fbff9d8 00000000 0 0x7fff5fbff9d9 00000000 0 0x7fff5fbff9da 00000000 0 0x7fff5fbff9db int: 4 long: 8 size_t: 8

We can see a number of things from this example. The pointer to char c (assigned the address of j using &j), when dereferenced, gives decimal 10 in the first byte, followed by three empty bytes. The sizeof an int on our system is 4 bytes, even though this is a 64-bit executable:

$ file bits bits: Mach-O 64-bit executable x86_64

The Intel Core 2 Duo chip addresses memory in "litte-endian" fashion---it's the first byte that holds the value 00001010. And four bytes previous to this is the first byte corresponding to the int k, even though k was assigned after j. (k = 148 + 2*256 = 660). One other thing is that the long int m, whose size is 8 bytes, actually starts 12 bytes before k. I'm not sure why this position was chosen, but I guess it's a "boundary" or something. The fundamental unit of size:

sizeof(size_t)

is 8 bytes.

And finally, if we force the compiler to compile a 32-bit executable by using the flag m32:

gcc bits.c print_bits.c cast.c -I. -m32 -o bits

the last part will print

int: 4 long: 4 size_t: 4

Now we get 4 byte (32-bit) longs and size_t is also 4 bytes.

Zipped files on Dropbox (here).

simple C example 1

This post is in the category of notes to myself. Unless you're a complete newcomer to programming, it probably won't help you. But I hope it will help me to remember the essentials.

The smallest fundamental data type in C is the char. A char, whether signed or unsigned, occupies one byte (8 bits). We can assign to a char variable using either a single-quoted text character like 'a' or a numeric constant, whether decimal, octal or hexadecimal:

char c; c = 'a'; c = 97; c = 0141; c = 0x61; printf("%c\n", c);

The call to printf (declared in stdio.h) will print 'a' for each of the 4 assignments.

For the example I'm showing here, the #include statements we need are:

#include <stdio.h> #include <string.h> #include <math.h>

These give access to the printf, strlen and pow functions, which are declared in these .h files and defined elsewhere.

Our approach to examine the individual bits of a char is to use a combination of the bitwise AND operator (&) and a bit-shift operation, as shown in the following code fragment:

void print_byte(unsigned char x) { int i, value = 0; char A[8]; // read lowest to highest for (i = 0; i < 8; i++) { if (x & 01) { A[i] = '1'; } else { A[i] = '0'; } x >>= 1; }

We examine the least significant bit of x by doing a bitwise AND using x and the octal constant 01 (00000001 in binary). We get each bit in turn by doing a rightward bit-shift operation, and assign the result directly to x. Since this approach reads the bits from right to left, I decided to just save the characters '0' and '1' to an array and then print them in reverse order.

Another approach would be to test the high value bit and print it directly. In order to do that, you would use this test: if (x & 0x80), and do a leftward bit-shift x <<= 1.

int value = 0; // print highest to lowest for (i = 7; i > -1; i--) { printf("%c", A[i]); if (A[i] == '1') { value += pow(2,i); } } printf(" %3d ", value); }

We use the pow (power or exponentiation) function from the math library to convert the value of the byte to decimal.

A second function applies the first one successively to the chars of a C string variable, defined as shown below, in main.

The variable that we'll print is a pointer to char or char *, that is, a variable which when de-referenced gives us back a char. So the string is actually an array of chars. It is distinguished by the double quotes and a termination marker. In C you normally need to know how many values are present in an array in order not to do something stupid, but in the case of a string the convention is to terminate with the \0 character. The code for print_word is:

void print_word(const char *s) { int j; char z; printf("word: %s\n", s); for (j = 0; j < strlen(s); j++) { z = s[j]; if (z != '\n') { printf("%c ", z); print_byte(z); printf(" %p \n", &s[j]); } } } int main() { const char *greeting = "Hello world!\0"; printf("greeting: %s\n", greeting); print_word(greeting); return 0; }

We combine these four sections of code in a file test.c and do:

$ gcc test.c -o test $ ./test greeting: Hello world! word: Hello world! H 01001000 72 0x100000e96 e 01100101 101 0x100000e97 l 01101100 108 0x100000e98 l 01101100 108 0x100000e99 o 01101111 111 0x100000e9a 00100000 32 0x100000e9b w 01110111 119 0x100000e9c o 01101111 111 0x100000e9d r 01110010 114 0x100000e9e l 01101100 108 0x100000e9f d 01100100 100 0x100000ea0 ! 00100001 33 0x100000ea1

For each character of the string we print the binary representation, the decimal equivalent, and the position of that char in memory. That last comes from here:

printf(" %p \n", &s[j]);

So we can see that the individual chars of the string are laid out in successive bytes in memory.

This is all quite fundamental. The parts I have trouble remembering when I come back to C after a long absence are:

- the distinction between single and double quotes
- one can access the individual chars of a string and assign them to chars
- the formatting codes (% plus c, s, i, f, p etc.)
- remembering to #include
- remembering to add parentheses around this test: if (A[i] == '1')
- remembering to add brackets around the entire group of conditional statements in:
if (x) { a; b; c; }

My usage of brackets is non-standard, but I think it makes sense. I always start them at the end of a line, and terminate at a position spaced out to the first character of the line where they started (except when the expression is really short).

You'll notice const and unsigned modifiers in the code above, but they're not essential.

I think that's it. In the next post, I'll develop this into a multi-file project and introduce the make tool.