Threats on OS X

It is said that if you know your enemies and know yourself, you will not be imperiled in a hundred battles; if you do not know your enemies but do know yourself, you will win one and lose one; if you do not know your enemies nor yourself, you will be imperiled in every single battle.

--Sun Tzu

Like every other Mac user who is paying attention this week, I've been reading the stories about malware on OS X (e.g. Ars Technica about Flashback here and here).

Charles Edge's Enterprise Mac Security is a good place to start learning about this stuff (although the book's pub date is 2010 and in places it looks quite a bit older). It's also probably not for the average user.

Some terminology: malware is a comprehensive term for various strategies by which bad guys attack users' machines, including (quoting Edge):

Virus: code attached to a file, which can replicate itself and spread to other files on a computer. A simple virus does not spread over a network without transfer of infected files. [Wikipedia has a somewhat different take.]

Worms: spread across networks by taking advantage of security flaws. [An example is OSX.Leap.A.]

Trojan Horse: these embed themselves in an application and activate only when the app is opened. [The app itself is bad, it disguises its actions from the user.]

Rootkit: a software program written to control an operating system.

---

I've been running OS X since the beginning (when we left System 9 behind), and I've never had anti-virus software. But recent events made me start thinking about being more proactive.

I download and install a lot of non-Apple, non-App Store software, much of it for bioinformatics. I'm well aware that if I type in my admin password for some evil app, there isn't much protection. Yet I run in admin mode all the time, because I find it a pain to be on a plain account. That's the first thing to change if you're not in the same situation.

The most important aspect of my security policy is that I don't have anything on the computer I can't afford to have stolen. I don't have any banking passwords saved. I could be undone by a Keylogger, but that's about it.

My Sharing Prefs panel consists entirely of unchecked boxes. If you want to share stuff you should definitely read Edge's book or another.

I always do a clean install when I upgrade the OS. I have most data backed up to two different hard drives. I pay attention and always update ASAP.

For my desktop, I have automatic login enabled. If someone has physical access to your machine, you're pretty much hosed anyway. (Unless you want to do FileVault).

I've never been crazy about anti-virus software:

the market share argument is MS propaganda and FUD---OS X with no sharing is hard to crack

most (all?) Pwn2Own stuff is user account-only, not root (not positive about this, how hard is privilege escalation?)

anti-virus never protects against zero-day exploits

the AV guys talk a lot to drum up business, like ambulance-chasing, bottom-feeding blood-sucking lawyers

Nevertheless, the news that the Java exploit somehow got around the requirement for an admin password really worries me. It's not easy to find out how it works (I suppose they don't want copycats). So I'm happy that I don't have Java on my machines running Lion, and I disabled it on the old one.

---

I've taken two new steps toward heightened security. I downloaded and am evaluating Little Snitch.

So far, I haven't seen any evidence of network connections that wouldn't be expected. However, I was a bit surprised to see that many apps (including all Apple apps) phone home each time they are launched (origin of the phrase).

Also, there are a lot of things (Agents and daemons) doing network stuff that I wasn't aware of---as one example, PubSubAgent.

This agent checks for updates to your RSS feeds, of which I have none (too old fashioned). At some point recently, a Safari update changed the Pref for this to "automatically update articles in Bookmarks bar". It bothered me because I've been methodical about deleting cookies and website data when I'm done. Yet PubSubAgent wanted to connect to feeds.arstechnica.com. I thought, how would they know about that? before understanding that it's a Safari RSS thing. Anyway, I turned off the reader as a test, and we'll see if PubSubAgent doesn't disappear.

I heard about Lingon in Edge's book. It's an app for managing LaunchAgents and (I thought from the book) LaunchDaemons.

Lingon is from Peter Borg, who made (still makes) Smultron. I used Smultron a lot in the days before TextMate, and loved it. So I downloaded the app from the AppStore, but I'm disappointed because it's been crippled. It only works with current user-specific processes and won't show me anything about system stuff.

One thing it did show me is an automatic Adobe Reader update job. I only installed Reader (300 MB!!) for one time when I had to use it. That job is now deleted.

The second new tool that I'm working with is ClamAV, an open source anti-virus tool. I read about it here and got it (with a Mac GUI) from here.

Apparently ClamAV has sucked for OS X in the past, but lately things have changed for the better. Anyway, the scan didn't find anything on my account's files.

What else should I worry about?

My Wi-Fi network is not "closed", the name is visible when you scan for networks. This could be a security liability. I don't know if it's possible to do a brute-force password attack on the Airport---something to read up on. But at least I have a decent password. Luckily, I live in a place where very few people drive by.

My ip address from the isp is supposed to be dynamic, but it's not---hasn't changed forever, as far as I know. Apparently one can force a change by spoofing a new MAC address for the router, but I don't want to do that. Maybe I'll go rummage in the basement for an old router and see what happens.

Rootkits certainly do exist for OS X. There have been ways to look for them (by scanning hashes of system files for alterations) like this Linux tool.

I believe that Code Signing should make standard rootkits obsolete. But maybe not, since this is not for "scripts" and such but full-fledged applications. Perhaps a question to apple.stackexchange is in order.

Finally, I think the advantage has to lie with the offense, in a situation where they cannot themselves be attacked. Structure your defense by asking yourself first: what's the worst that could happen if I did get hacked? And take steps to ameliorate that.

One last thing. The future of OS X is clearly not good for people like me: my computer will be like my iPhone. Apps can do whatever they want on the network, there's so much network activity (and push stuff), that the user can't make sense of it, but in theory an app will be so completely sandboxed it can't touch the rest of the machine. My programs won't be able to just do whatever they want anymore. Like the impact of barbed-wire fencing on the old West.

Sorry for the rambling post..

PyObjC Templates for Xcode are back!

I came across a page from a guy who went to the trouble of making and posting new templates for using PyObjC with Xcode 4.

Thanks!

They are on github here. I just followed the instructions:

Copy the File Templates and Project Templates folders to the following path in your home directory, creating any missing intermediate directories if needed:

~/Library/Developer/Xcode/Templates/

I'm a little rusty with Xcode (and it's become more and more like flying the space shuttle or something. Still, I was able to recycle this old project, while changing it just a bit.

One funny thing, I couldn't figure out how to hook up the text field outlet in the old way. I had to use bindings, and of course I did so for the popup as well (both Content and Selected Index).

I don't have a lot of time to fool with Xcode any more, but I do want to check out Greg's blog. Thanks again.

Here's the AppDelegate:

#-*- coding: utf-8 -*- from Foundation import * from AppKit import * class SpeakerAppDelegate(NSObject): TF = objc.ivar('TF') voiceL = objc.ivar('voiceL') nameL = objc.ivar('nameL') selectedIndex = objc.ivar('selectedIndex') notSpeaking = objc.ivar('notSpeaking') pretext = 'Hi, my name is ' def init(self): s = NSSpeechSynthesizer.alloc().initWithVoice_(None) self.speaker = s self.speaker.setDelegate_(self) self.setVoices() self.notSpeaking = True self.setSelectedIndex_(0) return self def setTF_(self, value): if value != self.TF: self.TF = value def setSelectedIndex_(self, value): if value != self.selectedIndex: self.selectedIndex = value name = self.nameL[self.selectedIndex] self.setTF_(self.pretext + name) def setVoices(self): L = NSSpeechSynthesizer.availableVoices() self.voiceL = L DL = [NSSpeechSynthesizer.attributesForVoice_(v) for v in L] nL = [D.objectForKey_('VoiceName') for D in DL] self.nameL = NSMutableArray.arrayWithArray_(nL) NSLog("%s" % self.nameL) @objc.IBAction def speak_(self,sender): i = self.selectedIndex NSLog("%i Say: %s" % (i, self.TF)) self.speaker.setVoice_(self.voiceL[i]) self.notSpeaking = False self.speaker.startSpeakingString_(self.TF) pass def speechSynthesizer_didFinishSpeaking_(self,speechSyn,flag): NSLog("didFinishSpeaking_") self.notSpeaking = flag

Damn you, autocorrect

Apparently, I'm not the only one who has problems with autocorrect.

At some point (Lion?), Safari acquired this ability. I keep seeing weird errors in my posts that I have no idea where they came from. So I turned it off system-wide under Language and Text Prefs. I also unchecked the box (which is checked in the screenshot) for replacing (c) with the symbol ©, etc.

Testing

It seems improbable, but it is a fact that on the first exam in some courses, a few students may score lower than expected for guessing at random. We usually see some scores of less than 20 out of 100 for an exam whose questions have five possible answers ABCDE.

We can turn this into an illustration of statistical significance. How high must a score be before one can dismiss the null hypothesis (no knowledge of the subject) with a p-value of 0.05?

We have a Bernouli trial with p = 0.2.

The expected value or mean is np. There is a beautifully simple derivation of the mean and variance in the wikipedia article, the variance is np(1-p). For our example, the mean is 20 and the variance is 100 x 0.2 x 0.8, so the standard deviation is √16 = 4.

For a large class (N = 100) of randomly answering students, the distribution appraches normal, and we expect that 2.5% of such students will score at or above two standard deviations above the mean. To make our rate for a type I error less than 0.05, we should require a score of 28 or higher.

I wrote a short script to simulate this. Each student takes a test with 100 questions, and each trial consists of a class of 100 students. We count the number of students with scores at or above a threshold T. The results for 10 trials are printed, and the results for 100 trials are averaged.

A few quick points: the mean appears to fall between 20 and 21. I'm sure there's a simple explanation but it eludes me at the moment. The score that 2.5% of students regularly exceed is 28.

This is a great example of the danger of multiple hypothesis testing!

One last thing is that the code is not very smart. What I should have done was extract the statistics for each value of T for each trial, rather than repeating the trial for each value of T. But it runs fast enough, and this way I don't need a data structure to remember the results.

> python script.py 20: 56 43 54 57 57 54 57 46 45 64 55.0 21: 39 48 42 44 39 44 48 48 50 48 43.5 22: 43 32 39 30 38 35 36 42 33 41 34.6 23: 40 29 35 31 27 19 30 20 31 25 26.0 24: 19 18 24 26 20 22 21 17 15 24 19.2 25: 11 14 8 8 13 11 12 15 17 13 12.9 26: 12 6 6 6 11 4 8 7 10 6 9.2 27: 7 4 6 5 8 7 5 7 7 7 5.9 28: 3 2 4 2 3 1 2 3 5 1 3.6 29: 4 4 2 2 2 1 3 1 0 1 2.2 30: 1 1 1 0 1 1 1 0 0 1 0.9 31: 4 0 1 0 0 0 0 0 0 0 0.6 32: 0 0 0 0 1 0 0 1 0 0 0.3 33: 0 0 0 0 0 1 0 0 0 1 0.1 34: 0 0 0 0 0 0 0 0 1 0 0.1

script.py

import random def student(N=100, p=0.2): L = [random.random() <= p for i in range(N)] return sum(L) def trial(T): L = [student() for i in range(100)] return sum([1 for n in L if n >= T]) def mean(L): return 1.0*sum(L)/len(L) for T in range(20,35): L = list() print '%2d: ' % T, for i in range(100): L.append(trial(T)) if i < 10: print '%2d' % L[-1], print '%4.1f' % mean(L)

Hex puzzle [-1]

Here is a fourth (and final) post in the saga of the "Let us have hex" puzzle. (Actual puzzle, reddit comments, and previous posts here, here and here).

Although I know the redditeers have already solved the problem, I had hoped to work my way through it more systematically. But I ran into a constraint I can't get around very easily (or perhaps at all), which is that the available data is not complete. It's not enough to yield a systematic solution.

An important feature of the puzzle is that visitors to the puzzle page get served 100 digit hex strings that are refreshed to a new value slowly (maybe every half hour), and also might (at least theoretically) be restricted by ip address, so that a social venue like reddit is critical to gathering all the data. But the reddit faction quit too soon, after they'd guessed the answer.

The total number of strings I found in the reddit comments is 340. Let me show two demonstrations that the complete data is not there. First, recall that the base64 encodings have "codons" of length 4. So two adjacent 50-character base64 strings that are not correctly aligned (because they abut without overlap, and were decoded individually), might be missing as many as 4 base64 characters at the joint. This would affect in turn at most 2 adjacent characters in the C or PHP program, since the ratio is 4:3.

A simple decoding of string i = 6 gives:

< 73; i++) { pri
{ echo $a[$i];

The first line is C and the second PHP. The problem is that there's nothing in the known data that corresponds to what likely comes next---there's no "echo $b[$i];" in the data. It's just missing. I suppose there could be a series of very short overlaps that would build it, but it seems pretty unsporting. Similarly, the formatting of the C output is not there.

More conclusive evidence comes from an analysis of the solution. The first word is "Design" and it is derived from the C program, which has this fragment:

int a[] = {4, 4, 6, 5, 7, 3, 6, 9, 6, 7..

If we combine these as hex digits and then print the ASCII equivalent we get:

>>> L = ['44','65','73','69','67'] >>> L = ['0x' + s for s in L] >>> L = [chr(int(s,base=16)) for s in L] >>> L ['D', 'e', 's', 'i', 'g']

We need the "n" in design which is '0x6e' in hex:

>>> hex(ord('n')) '0x6e'

But there is no 'e' in the data with a leading 6. The overlap is not there.

I think I'm done now. I had fun. I wrote a lot of code to try to deal with the challenge of how to assemble a bunch of strings where the candidate strings with a large overlap might be misleading. I was prepared to do the thing by hand, as shown in the screenshot. But there is not enough data, you have to guess. Here is what I was reduced to:

Hex puzzle (3)

I've been working on the hex puzzle (previous posts here and here). We have a list of 340 unqiue lines of data, each 100 characters in length. The first and last lines have been given to us as i = 137 and i = 61, respectively.

The data file ('reddit_hex.txt') is on Dropbox here.

The data seem to be hexadecimal bytes with an unusual distribution that leads to the first part of the decoding: these are bytes that represent ASCII characters in the set A..Za..z0..1. This suggests the possibility that the given data are a hex representation of base64 (although the last two characters '/' and '+' were not found).

Unfortunately, I haven't yet been able to stitch the lines together in a way that generates a 1936 hex digit mesage, with the correct ends, to be decoded. Apparently I go wrong at some points in the assembly, probably because of repeats in the data stream, although perhaps I just messed up.

It might be a disappointment to you, but I'm going to try using another critical clue gleaned from the comments to the reddit post. After converting the base64 data to actual ASCII characters, it turns out that there are two intertwined messages: the alternating bytes form a C program and a PHP program. So I wrote a script that uses all these clues to analyze each of the lines of the original data.

Let's look at some sample output, say the first entry (i = 137):

> python demo.py 137 137 497a78705032357759326873634855675a43526c595341675044317a494852685a484a70636d39684c6e6c6f4b4434354369 I z x p P 2 5 w Y 2 h s c H U g Z C R l Y S A g P D 1 z I H R h Z H J p c m 9 h L n l o K D 4 5 C i IzxpP25wY2hscHUgZCRlYSAgPD1zIHRhZHJpcm9hLnloKD45Ci 0 ('#include <stdio.h>', '<?php $a = array(9')

What's going on here is that we have 100 hexadecimal digits or 50 bytes in the original data. Each byte can be converted to its ASCII representation as shown in the second and third lines.

>>> chr(int('0x49',base=16)) 'I'

We can do the whole line at once with:

>>> import binascii >>> s = '497a78705032357759326873634855675a43526c595341675044317a494852685a484a70636d39684c6e6c6f4b4434354369' >>> binascii.unhexlify(s) 'IzxpP25wY2hscHUgZCRlYSAgPD1zIHRhZHJpcm9hLnloKD45Ci'

The base64 data decoding takes 4 characters at a time. For example:

>>> import base64 >>> d = base64.b64decode >>> d('Izxp') '#<i' >>> s = 'IzxpP25wY2hscHUgZCRlYSAgPD1zIHRhZHJpcm9hLnloKD45Ci' >>> d(s[:-2]) '#<i?npchlpu d$ea <=s tadriroa.yh(>9'

Because base64 is a 4-letter code, there are 4 possible frames, and we only know the frame for the first line of data. It turns out that for this data, for each line, only one of the frames yields just ASCII characters. In the script below, we filter out decodings like this one:

'\xd0\xb0\x80\xc8\xb1\x80\xc0\xb0\x80\xd8\xb0\x80\xc4\xb1\x80\xd8\xb0\x81'

by turning the whole line into an array of ints and then asking whether any is > 128.

def filter(line): B = [struct.unpack('B',b)[0] for b in line] return not max(B) > 128

After conversion from base64 to ASCII, we build strings from the alternate bytes, resulting in the output line:

0 ('#include <stdio.h>', '<?php $a = array(9')

This indicates that the two strings were derived from frame 0. It's obvious that the first one is the beginning of a C program, and the second certainly looks like PHP to me.

Note that we can't just use the data as is, because we have thrown away bytes at the joints. Decoding in frame 0, we lose the 'Ci' at the end of the base64 data.

Also, consider i = 8:

> python demo.py 8 8 357759326873634855675a43526c595341675044317a494852685a484a70636d39684c6e6c6f4b4434354369787049473432 5 w Y 2 h s c H U g Z C R l Y S A g P D 1 z I H R h Z H J p c m 9 h L n l o K D 4 5 C i x p I G 4 2 8 5wY2hscHUgZCRlYSAgPD1zIHRhZHJpcm9hLnloKD45CixpIG42 2 ('clude <stdio.h>\nin', 'hp $a = array(9, 6')

Notice that 'clude <stdio.h>\nin' looks like a continuation of '#include <stdio.h>' from the first line.

>>> import base64 >>> d = base64.b64decode >>> d('Y2hscHUg') 'chlpu '

The 'cl' of 'clude' are coming from the base64 'Y2hs'. It's reassuring that we can align 0 and 8:

IzxpP25wY2hscHUgZCRlYSAgPD1zIHRhZHJpcm9hLnloKD45Ci 5wY2hscHUgZCRlYSAgPD1zIHRhZHJpcm9hLnloKD45CixpIG42

So the plan is, to stitch data lines together one at a time, and examine the result to be sure it makes sense after going through base64 decoding and de-interleaving.

demo.py

import sys, binascii, base64, struct def load_data(fn): FH = open(fn,'r') data = FH.read().strip() FH.close() return data.strip().split('\n') def untangle(s): s1 = ''.join([s[i] for i in range(0,len(s),2)]) s2 = ''.join([s[i] for i in range(1,len(s),2)]) return s1,s2 def filter(line): B = [struct.unpack('B',b)[0] for b in line] return not max(B) > 128 def process(s): d = base64.b64decode rL = list() for i,j in [(0,-2),(1,-1),(2,len(s)),(3,-3)]: line = d(s[i:j]) if filter(line): rL.append((i,untangle(line))) return rL def analyze_line(i,L): D = {'i':i} s = L[i] D['hex'] = s s = binascii.unhexlify(s) D['b64'] = s result = process(s) assert len(result) == 1 j, t = result[0] D['frame'] = j D['t'] = t return D def format_dict(D): s1 = '%3d ' % D['i'] + D['hex'] s2 = ' ' + ' '.join(list(D['b64'])) s3 = '%3d ' % D['i'] + D['b64'] s4 = '%3d ' % D['frame'] + str(D['t']) return '\n'.join([s1,s2,s3,s4]) L = load_data('reddit_hex.txt') DL = [analyze_line(i,L) for i in range(len(L))] try: i = int(sys.argv[1]) text = format_dict(DL[i]) print text, '\n' except IndexError: for D in DL: text = format_dict(D) print text, '\n'

Hex puzzle (2)

Let's continue with the puzzle introduced in the last post (here).

The next step is to stitch the lines together to form the complete text. If they contained random bytes, that would be relatively easy. There are 65536 possible two-byte combinations. We have a total of 340 * 25 = 8500 two-byte (4 hex character) combinations in our data, so the expected number of times any combination should appear in random data is much less than 1.

However, the data is not random, instead, it is highly skewed. Simple modifications to count.py (from last time) show a total of only 370 two-byte combinations in the data (i.e. 65166 of all possible values are not observed), and the top 20 have counts ranging from 464 down to 72.

464 4341 CA 331 4943 IC 309 4167 Ag 281 4c43 LC 234 734c sL 174 4173 As 154 4377 Cw 140 7773 ws 139 7349 sI 139 674d gM 139 674c gL 137 4c44 LD 126 674e gN 104 6749 gI 103 7767 wg 95 4e69 Ni 89 5973 Ys 78 4944 ID 76 4947 IG 72 4e6a Nj

In the table above, the first column is the count, the second and third are the hex and ASCII representations.

That compares with a maximum count of 3-4 for random data, and an observed total of 8000 different values on an average run with random data.

This suggests that we should undertake some data exploration to see whether the repetitive character of the data will cause difficulties for the assembly.

The core operation is to take an individual entry at index i in the data (the query), and depending on a parameter (tlen) for the number of characters to choose at the very end of the query, make a substring sequence t, the target to search for. Then, we go through all the other entries looking for a match containing t, and when one is found we save the index of the query and the match as well as the position k where the target t begins in the match string.

The quality of these matches can be assessed in various ways. We expect that the sequence of each match string upstream of the target should be exactly the same as the query. We calculate the offset o where the beginning of the match string should be found in the query. Since this depends on tlen (o = len(L[i]) - k - n) we calculate it for each hit and save it with the other parameters. The offset is also useful for printing the actual alignments in show.

A second quality test is to examine the match strings downstream of the target. In the region of overlap, we expect they should be identical as well. A failure of either test would suggest that the data is causing trouble for our simple search algorithm.

---

As mentioned last time, the observed hex digits comprise all (and only) those which encode ASCII characters 0..1A..Za..z. Rather than print out strings of 100 characters, it will simplify the output to convert the data to ASCII first. We're assuming that this is what we need to do with the data, but I peeked at the answer on reddit, so it's safe to go ahead.

python convert.py > data.mod.txt

import binascii from utils import load_data L = load_data('reddit.txt') for line in L: print binascii.unhexlify(line)

Note the extremely repetitive nature of the data by comparing some adjacent output lines:

A1NiAsNiAsMyAsMSA0NiwsICAyJyxmICcwLCwgIDc2KSw7ICAx A1NywsICAyNSwsICA0NiwsICAyJyxmICcwLCwgIDc2KSw7ICAx AgLDUgLDcgLDMgLDAgLDAgLDcgLDIgLDIgLDYgLDYgLDEgNTQs AgLDUgLDcgLDMgLDAgLDAgLDcgLDYgLDIgLDYgLDYgLDEgNTQs xpIG42dCwgIG00YSxpIG42KCwpICAzeywgICAyICwgICA2ICxp xpIG43dCwgIGE3WyxdICA3PSwgIHswNCwsICA2NCwsICAxNiws

However, for a well-chosen query (one with a rare two-byte combination as the target) and a long enough target length, it's easy to find matches that appear correct. In the output below, we examine the string found at index 0 in the data as the query. The analysis finds 25 matches, data strings where the terminal 4 characters appear somewhere in the match, and all of those matches are perfect ones whether looking upstream and comparing against the query, or looking downstream in each one (beyond the target) and comparing with each other, as shown in the second half of the output.

We'll exercise these routines a bit more next time. I want to see how big to make the target size, and also whether we should exclude some strings from the preliminary analysis. But overall, it looks like we might be OK.

> python search.py i= 0 : 25 matches found 0NiwsICAyJyxmICcwLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc GExc NiwsICAyJyxmICcwLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc iwsICAyJyxmICcwLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc CAyJyxmICcwLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc AyJyxmICcwLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc xmICcwLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc mICcwLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc CcwLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc cwLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc wLCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc LCwgIDc2KSw7ICAxJCxiICA2PSwgIGExc CwgIDc2KSw7ICAxJCxiICA2PSwgIGExc wgIDc2KSw7ICAxJCxiICA2PSwgIGExc gIDc2KSw7ICAxJCxiICA2PSwgIGExc CAxJCxiICA2PSwgIGExc AxJCxiICA2PSwgIGExc xJCxiICA2PSwgIGExc JCxiICA2PSwgIGExc CxiICA2PSwgIGExc xiICA2PSwgIGExc A2PSwgIGExc 2PSwgIGExc SwgIGExc gIGExc IGExc GExc no mismatches upstream GExc j jR jRyLGE jRyLGEg jRyLGEgeTYo jRyLGEgeTYoL jRyLGEgeTYoLDc jRyLGEgeTYoLDcg jRyLGEgeTYoLDcgL jRyLGEgeTYoLDcgLD jRyLGEgeTYoLDcgLDQ jRyLGEgeTYoLDcgLDQg jRyLGEgeTYoLDcgLDQgL jRyLGEgeTYoLDcgLDQgLDkgLDIgLDY jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYg jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgL jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgLD jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgLDA jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgLDAg jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgLDAgLDUg jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgLDAgLDUgL jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgLDAgLDUgLDc jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgLDAgLDUgLDcgL jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgLDAgLDUgLDcgLD jRyLGEgeTYoLDcgLDQgLDkgLDIgLDYgLDAgLDUgLDcgLDM no mismatches downstream

search.py

from utils import load_data # tlen is minimum overlap # functions deal with one string at a time # do not assume upstream matches # e is search string, f is potential match def one_search(i,L,tlen): # search result has associated tlen oL = list() e = L[i] t = e[-tlen:] for j,f in enumerate(L): if i == j or not t in f: continue # offset is pos in e where f should start # not guaranteed to match except over t offset = len(e) - f.index(t) - tlen oL.append((j,offset)) oL.sort(key=lambda item:item[1]) return oL # mmL is a list of mismatched strings # ummL contains upstream mismatches def test_upstream(i,L,oL,ummL): e = L[i] for j,o in oL: f = L[j] if not f.startswith(e[o:]): if i < j: ummL.append((i,j)) else: ummL.append((j,i)) def trim_downstream(oL,L,tlen): pL = list() for j,o in oL: f = L[j] k = len(f) - o pL.append((j,f[k:])) return pL def test_downstream(pL,dmmL): mismatches = 0 for i,s1 in pL: for j, s2 in pL: if s1 == s2 or len(s1) < len(s2): continue if not s1.startswith(s2): if i < j: dmmL.append((i,j)) else: dmmL.append((j,i)) if __name__ == '__main__': L = load_data('data.mod.txt') tlen = 4 i = 0 e = L[i] oL = one_search(i,L,tlen) print 'i=', i, ':', len(oL), 'matches found' print e print ' ' * (len(e)-tlen) + e[-tlen:] print for j,o in oL: k = len(e) - o print ' '*o + L[j][:k] ummL = list() test_upstream(i,L,oL,ummL) if ummL: print len(ummL), 'mismatches' else: print 'no mismatches upstream\n' dmmL = list() print e[-tlen:] pL = trim_downstream(oL,L,tlen) for item in pL: print ' '*tlen + item[1] test_downstream(pL,dmmL) if dmmL: print len(dmmL), 'mismatches' else: print 'no mismatches downstream'

Hex puzzle (1)

Last Saturday, an interesting puzzle was posted on reddit/r/programming. I'm having trouble finding the original post, but the actual puzzle page is here, and the (complete) comments to the post are here.

As you can see from the screenshot or the link

the puzzle is a message consisting of 1936 hex digits, and upon further manipulation it promises to yield the answer. You can actually follow the link to see the answer if you're impatient. I wasn't able to compete (I had a party to cook for, and could not have gone as fast as these guys anyway), but I'd still like to understand how they got there.

Apparently, we need to assemble a number of overlapping 100 digit pieces of a 1936 digit message and then figure out what it means. The pieces were served up to different visitors to the site and collected as comments on reddit. The first objective is to recover the observed data. I believe reddit must have an API for this, but I don't know it. What I did was to save the source for the page with all 410 comments (in the file comments.txt) and then run scrape.py below.

I count 251 strings of 100 characters, all in "0..9a..f".

According to the instructions, the first and last strings are marked, and these were recovered and posted on reddit:

497a78705032357759326873634855675a43526c595341675044317a494852685a484a70636d39684c6e6c6f4b4434354369 //first 100
4270494630674f3349675a5831304948566c636d4e75614342764d4341374a79426c494363674f7941674944393950676f3d //last 100

[UPDATE: There is a link to the complete (known) data from reddit to github here. I count a total of 340 strings, substantially more than what I've been working with in early attempts. This should help with the assembly problem. ]

The data clearly has substantial structure, as indicated by the screenshot from a "find" search for the digit "4" in TextEdit.

Since only hexadecimal digits are present it suggests that this might actually be hexadecimal. It's interesting that not all digits are equally represented. In particular, only those bytes (two hex digits) which encode ASCII "a..zA..Z0..9" are present (although the distribution is decidedly odd). So it seems pretty likely that we just need to decode the hex to ASCII and then figure out what it means.

However, there is an assembly problem that comes first.

Output from count.py (listing at the end):

> python count.py 54 30 48 0 0 40 31 49 1 1 256 32 50 2 2 71 33 51 3 3 58 34 52 4 4 82 35 53 5 5 64 37 55 7 7 39 38 56 8 8 63 39 57 9 9 1070 41 65 A A 133 42 66 B B 1102 43 67 C C 414 44 68 D D 111 45 69 E E 38 46 70 F F 169 47 71 G G 101 48 72 H H 812 49 73 I I 151 4a 74 J J 132 4b 75 K K 667 4c 76 L L 399 4d 77 M M 532 4e 78 N N 83 4f 79 O O 82 50 80 P P 67 51 81 Q Q 65 52 82 R R 286 53 83 S S 58 54 84 T T 51 55 85 U U 12 56 86 V V 56 57 87 W W 31 58 88 X X 315 59 89 Y Y 92 5a 90 Z Z 16 61 97 a a 13 62 98 b b 278 63 99 c c 61 64 100 d d 58 65 101 e e 8 66 102 f f 938 67 103 g g 57 68 104 h h 247 69 105 i i 201 6a 106 j j 116 6b 107 k k 62 6c 108 l l 64 6d 109 m m 97 6e 110 n n 61 6f 111 o o 209 70 112 p p 896 73 115 s s 25 74 116 t t 23 75 117 u u 15 76 118 v v 532 77 119 w w 211 78 120 x x 414 79 121 y y 192 7a 122 z z

scrape.py

# python scrape.py > data.txt fn = 'comments.txt' FH = open(fn) data = FH.read() FH.close() L = list() i = 0 while True: j = data.find('>',i) j += 1 i=j k = j + 100 if k > len(data): break if not data[k] == '<': continue w = data[j:k] if ' ' in w or '>' in w or '<' in w: continue for c in w: assert c in '0123456789abcdef' L.append(w) if i == 0: break L = list(set(L)) L.sort() for w in L: print w

count.py

import binascii from collections import Counter def load_data(): fn = 'data.txt' FH = open(fn,'r') data = FH.read().strip() FH.close() L = data.split('\n') return L L = load_data() n = 2 cL = list() for line in L: for i in range(0,len(line)-n+1,2): cL.append(line[i:i+n]) C = Counter(cL) for h,n in sorted(C.most_common()): i = int(h, base=16) print '%4d %s %4d %s %s' %\ (n, h, i, chr(i), binascii.unhexlify(h))

Interesting programming links

A collection of links to tutorials, blog posts and software that look interesting, which I hope to explore soon:

Wikibooks: Reverse Engineering/Mac OS X

Reverse Engineering Mac OS X

Learn Python The Hard Way

Learn C The Hard Way

Apple docs on shell scripting

Everything Sysadmin

pymacadmin

Hetland: Instant Hacking

Hetland: Python Algorithms

Beazley: Generator Tricks

tcpdump primer

Little Snitch

MachOViewer

Commands to change a large number of OS X defaults.

mosh---ssh replacement

DYLD_INSERT_LIBRARIES

The so-called Flashback trojan is in the news the last few days. There is a breathless piece at PCWorld which compares it to Conficker, which is pretty funny (and completely stupid). FWIW, I don't have Flash Player installed on my machine, or Java either. I use a Safari Extension called YouTube5 on the very rare occasions I want to watch something requiring this.

I got interested in how it works, and a google search led me to two pages with example code using DYLD_FORCE_FLAT_NAMESPACE and DYLD_INSERT_LIBRARIES.

I shamelessly copied the code from here (also see here), with slight changes. Very similar code is in a YoLinux tutorial here.

We make a "shared library" containing a function f. We can use this function in our program (main.c below) by simply telling gcc where to find it with:

gcc -o call_f mysharedlib.dylib main.c

Here are the listings:

mysharedlib.c

#include <stdio.h> #include "mysharedlib.h" void f() { printf("hello\n"); }

mysharedlib.h

void f();

main.c

int main() { f(); return 0; }

A shell script to compile and test what we've got so far:
before.sh

#!/bin/bash gcc -dynamiclib -o mysharedlib.dylib mysharedlib.c gcc -o call_f mysharedlib.dylib main.c echo "before:" ./call_f

Output:

> ./before.sh before: hello

---

Now, here is where it gets more interesting. We have an additional file which also contains a function called f, which will be discovered first and called rather than the function we think we're calling (it "shadows" the first one). That file is
openhook.c

#include <stdio.h> #include <dlfcn.h> #include <unistd.h> #include "mysharedlib.h" typedef void (*fType)(); static void (*real_f)() = NULL; void f() { if ( ! real_f) { void* handle = dlopen("mysharedlib.dylib", RTLD_NOW); real_f = (fType)dlsym(handle, "f"); if ( ! real_f) { printf("NG\n"); return; } } printf("--------hijacked!--------\n"); real_f(); }

The if block above grabs the "real" f. We carry out our nefarious plan (printing a warning) and then call the original function.

A shell script to run the second part is:
after.sh

#!/bin/bash gcc -flat_namespace -dynamiclib -o openhook.dylib openhook.c export DYLD_FORCE_FLAT_NAMESPACE= export DYLD_INSERT_LIBRARIES=openhook.dylib echo "after export:" ./call_f echo "env:" $DYLD_INSERT_LIBRARIES unset DYLD_FORCE_FLAT_NAMESPACE unset DYLD_INSERT_LIBRARIES echo "after unset:" ./call_f echo "env:" $DYLD_INSERT_LIBRARIES

And the output:

> ./after.sh after export: --------hijacked!-------- hello env: openhook.dylib after unset: hello env:

---

We've explored dlopen and dlsym a bit in a previous post.

There is nothing very complicated happening in our code. It's just that the DYLD_FORCE_FLAT_NAMESPACE= and DYLD_INSERT_LIBRARIES=openhook.dylib load open hook.dylib before mysharedlib.dylib. This allows us to intercept the function call and execute our special code before doing what the original caller asked for and expects. If we're careful, they may not notice.

There is a bit more detail in the man page for dyld.

For extra credit, you might want to use DYLD_PRINT_LIBRARIES to snoop on the loader, as we did before. Or use otool:

> otool -L call_f dyld: loaded: /usr/bin/otool dyld: loaded: /Users/telliott_admin/Desktop/openhook.dylib dyld: loaded: /usr/lib/libstdc++.6.dylib dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /usr/lib/libc++abi.dylib dyld: loaded: /usr/lib/system/libcache.dylib dyld: loaded: /usr/lib/system/libcommonCrypto.dylib dyld: loaded: /usr/lib/system/libcompiler_rt.dylib dyld: loaded: /usr/lib/system/libcopyfile.dylib dyld: loaded: /usr/lib/system/libdispatch.dylib dyld: loaded: /usr/lib/system/libdnsinfo.dylib dyld: loaded: /usr/lib/system/libdyld.dylib dyld: loaded: /usr/lib/system/libkeymgr.dylib dyld: loaded: /usr/lib/system/liblaunch.dylib dyld: loaded: /usr/lib/system/libmacho.dylib dyld: loaded: /usr/lib/system/libmathCommon.A.dylib dyld: loaded: /usr/lib/system/libquarantine.dylib dyld: loaded: /usr/lib/system/libremovefile.dylib dyld: loaded: /usr/lib/system/libsystem_blocks.dylib dyld: loaded: /usr/lib/system/libsystem_c.dylib dyld: loaded: /usr/lib/system/libsystem_dnssd.dylib dyld: loaded: /usr/lib/system/libsystem_info.dylib dyld: loaded: /usr/lib/system/libsystem_kernel.dylib dyld: loaded: /usr/lib/system/libsystem_network.dylib dyld: loaded: /usr/lib/system/libsystem_notify.dylib dyld: loaded: /usr/lib/system/libsystem_sandbox.dylib dyld: loaded: /usr/lib/system/libunc.dylib dyld: loaded: /usr/lib/system/libunwind.dylib dyld: loaded: /usr/lib/system/libxpc.dylib call_f: mysharedlib.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

Deleting Safari data

I've been annoyed with Safari for a while now because it's so difficult to completely remove all website data. These items include not only cookies but also the Cache and Local Storage.

It's not unusual to do Preferences > Privacy > Remove All Website Data.., only to see some of the cookies come back. One solution is to zap the data, quit the application and then re-open it, and zap again. It also helps if you do a reset first: Safari > Reset Safari.. > Reset. I do this with "Remove saved names and passwords" unchecked.

One solution would be to do private browsing (Safari > Private Browsing.. > OK), but sometimes you want cookies for interaction with a few websites.

A stereotyped sequence of actions---that sounds like a candidate for a script. The default scriptable items in Safari are pretty minimal: drag-and-drop the Safari application icon onto the Applescript Editor icon in Utilities to see. (I just noticed that it looks as though these can be extended by GUI scripting. So that might be solution 2 to the problem, I'll have to check it out.)

A third solution is developed here. It's not perfect, mainly because in Lion the previous state of the browser is restored, reloading pages and their cookies. But I think it's interesting in combining some different techniques. I got a start on this from a Q & A on Stack Exchange.

The Python script (listing at the end below) consists of five functions (calling a sixth), and then a seventh that calls the others:

def reset(): close_windows() close_app('Safari') eat_cookies() p = '/Library/Caches/com.apple.Safari/Cache.db' remove_data(p) p = '/Library/Safari/LocalStorage' remove_data(p, is_dir=True) open_app('Safari') reset()

The function home returns the users home directory. We could do this just staying with Python:

>>> import os >>> os.path.expanduser(os.getcwd()) '/Users/telliott/Desktop'

but this hasn't worked quite as you'd expect. Instead, in the script we get the HOME environment variable using NSProcessInfo. We'll use this to build the paths for the Cache and Local Storage data.

In close_windows, we use Applescript. We write the script to a temporary (hidden) file, execute it from the shell, and afterwards remove the script file. This part doesn't act as I'd like, because even though it works, Safari will reload the window, its URL and their cookies when we re-start the app.

The function open_app just calls open -a Safari from the shell.

The function close_app can't do close from the shell, as there is no such command. My solution was to use ps, followed by a filter on active processes for the string "Safari" (checking that there is at most one result) and then killing that process if it's running. [ UPDATE: I modified the script to first check if Safari is running, and only do the close and open if it is. ]

The function remove_data checks to see if a path exists, and if it does, removes the file. It can be called with is_dir=True to do so recursively.

Finally, eat_cookies uses NSHTTPCookieStorage.sharedHTTPCookieStorage. I explored this in the interpreter, screening dir(store) until I came across deleteCookie_.

In summary, we use Applescript to close all Safari windows, the shell to quit and re-open the app, a Foundation API to remove the cookies, and standard file deletion functions to remove the Cache and Local Storage. The only problem is if there's a page or set of tabs open when we do this, they come back, along with their cookies.

Finally, a note about NSURLCache as mentioned in the answer on StackExchange.

c = NSURLCache.sharedURLCache c.removeAllCachedResponses()

doesn't work, because it seems that most applications construct their own NSURLCache.

import os, sys, time, subprocess from subprocess import Popen, PIPE import Foundation def home(): obj = Foundation.NSProcessInfo D = obj.processInfo().environment() return D['HOME'] def get_pid(app): cmd = 'ps -ax | grep ' + app p = Popen(cmd,shell=True, stdout=PIPE,close_fds=True) result = p.stdout.read() L = result.strip().split('\n') L = [e for e in L if app + '.app' in e] if not(L): return None assert len(L) == 1 pid = L[0].split()[0] return pid def close_windows(): s = ''' tell application "Safari" close windows end tell ''' script_path = home() + '/.x.scpt' FH = open(script_path, 'w') FH.write(s) FH.close() cmd = 'osascript ' + script_path result = subprocess.call(cmd,shell=True) cmd = 'rm ' + script_path result = subprocess.call(cmd,shell=True) def open_app(app): cmd = 'open -a ' + app result = subprocess.call(cmd,shell=True) def close_app(pid): subprocess.call('kill ' + pid,shell=True) def remove_data(path, is_dir=False): path = home() + path try: os.stat(path) except OSError: return cmd = 'rm ' if is_dir: cmd += '-r ' cmd += path result = subprocess.call(cmd,shell=True) def eat_cookies(v=True): obj = Foundation.NSHTTPCookieStorage store = obj.sharedHTTPCookieStorage() L = store.cookies() L = [store.deleteCookie_(c) for c in L] if v: print 'deleted ', len(L) - len(store.cookies()), print 'cookies' def reset(): app = 'Safari' pid = get_pid(app) if pid: close_windows() close_app(pid) eat_cookies() p = '/Library/Caches/com.apple.Safari/Cache.db' remove_data(p) p = '/Library/Safari/LocalStorage' remove_data(p, is_dir=True) if pid: open_app('Safari') reset()

ifconfig and "interfaces" (2)

I'm still puzzling about network interfaces (previous post here). I'm trying to understand why the output from ifconfig has changed to add new interfaces like gif0 and stf0, and whether it's related to something I did (and therefore a potential cause for concern).

At the time it happened, I'd have sworn it was a result of something I had just done in the last hour or two, but it was not undone by a reset of the Airport to factory default, nor by a clean install of OS X.

I'm sure many people could explain this to me in a second, but I can't find them. This morning I was googling various terms and found my own "scratching my head" post coming up as the top hit, and that's certainly not a good sign.

Anyway, I don't have a complete solution but the executive summary so far is:

- interfaces are machine-specific. It does not appear to have anything to do with the Airport (in fact, Airport has its own list of interfaces).

- it appears to be something that the System is doing on boot, but I haven't seen anything in the logs yet to tell me what that is.

---

To start from the basics, a network interface controller is a piece of hardware that connects a computer to a network, like the internet. So when I think back to that day in 1994 when the IT guys came around and added a gizmo (to my serial port?) and then installed Netscape on my machine, the gizmo or "Ethernet card" was one of these devices.

In the OSI model, the interface comprises layers 1 and 2, where the other layers might be various options like:

3 IPv4
4 TCP or UDP
5 a socket
6 SSL
7 HTTP

I haven't found a good description of how these physical devices are integrated into a modern Mac, but there are a lot of them and it seems that probably most are implemented in software rather than hardware. The man pages for gif and stf:

gif -- generic tunnel interface stf -- 6to4 tunnel interface

indicate that these two are for "tunneling" (wrapping IPv6 somehow to tunnel it over IPv4):

The stf interface supports ``6to4'' IPv6 in IPv4 encapsulation. It can tunnel IPv6 traffic over IPv4, as specified in RFC3056.. The gif interface is a generic tunneling pseudo device for IPv4 and IPv6. It can tunnel IPv[46] traffic over IPv[46].. Each gif interface is created at runtime using interface cloning..

Although these can be configured using ifconfig (if stands for interface)

The ifconfig utility is used to assign an address to a network interface and/or configure network interface parameters.

> ifconfig gif0 gif0: flags=8010 mtu 1280 > sudo ifconfig gif0 destroy Password: > ifconfig gif0 ifconfig: interface gif0 does not exist

It will come back after reboot.

Also, on my single location ("Automatic"), I deleted all but Wi-Fi in Network Prefs, but the output from ifconfig still shows fw0 etc.

I found a document at Apple called Introduction to System Configuration Programming Guidelines which seems right on point.

The CurrentSet preference always contains the internal ID that represents the currently active set in the Sets preference (in Figure 3-1, this is Setk). When a user selects a different location in Network preferences or from the Apple menu, the preferences monitor notices and updates the dynamic store to reflect the change. For more information on the preferences monitor, see “Preferences Monitor ”).

The remaining two top-level preferences, Sets and NetworkServices, contain the bulk of the information the system needs to configure network services. As you can see in Figure 3-1, a set listed in the Sets dictionary contains the internal IDs that represent individual services listed in the NetworkServices dictionary. The following sections describe the Sets and NetworkServices dictionaries in more detail.

I haven't digested all the details yet, but two key points. First, the "Persistent Store" is saved in this file:

/Library/Preferences/SystemConfiguration/preferences.plist

and if I follow the trail from CurrentSet to Network to ServiceOrder to Interface I see a single DeviceName, which is en1. (Let's not worry where en0 and lo0 come from just yet).

So, if I'm reading this right, my Prefs in the GUI sorta match what's in the file.

The second point is however:

Preferences Monitor
The preferences monitor reads the currently active configuration set specified by the user’s CurrentSet preference and loads the associated preferences into the dynamic store.
..

Kernel Event Monitor
The kernel event monitor maintains a list of all network interfaces defined in the system, the link status associated with each interface, and any assigned addresses. It monitors low-level kernel events and watches the network stacks, keeping track of the link status of each network interface. The kernel event monitor’s main job is to post the status of each network interface in the dynamic store. This frees applications from having to reach into the kernel to find out, for example, if the Ethernet cable is plugged in or if the assigned addresses have changed.

The system also modifies the network interfaces, it's not just from my Network Prefs. Probably what I will need to do is figure out how to navigate SystemConfiguration. Although, if you notice, I only retrieved the three en0, en1 and fw0 using SCNetworkInterfaceCopyAll(), plus one dealing with Bluetooth.

Finally, I made some progress with snmp and the Airport Extreme.

> sudo snmpwalk -v 2c -c public 10.0.1.1 .. IF-MIB::ifDescr.1 = STRING: mgi0 IF-MIB::ifDescr.2 = STRING: mgi1 IF-MIB::ifDescr.3 = STRING: mv0 IF-MIB::ifDescr.4 = STRING: mv1 IF-MIB::ifDescr.5 = STRING: lo0 IF-MIB::ifDescr.6 = STRING: wlan0 IF-MIB::ifDescr.7 = STRING: wlan1 IF-MIB::ifDescr.8 = STRING: pppoe0 IF-MIB::ifDescr.9 = STRING: bridge0 .. IF-MIB::ifType.1 = INTEGER: ethernetCsmacd(6) IF-MIB::ifType.2 = INTEGER: ethernetCsmacd(6) IF-MIB::ifType.3 = INTEGER: ieee80211(71) IF-MIB::ifType.4 = INTEGER: ieee80211(71) IF-MIB::ifType.5 = INTEGER: softwareLoopback(24) IF-MIB::ifType.6 = INTEGER: ethernetCsmacd(6) IF-MIB::ifType.7 = INTEGER: ethernetCsmacd(6) IF-MIB::ifType.8 = INTEGER: ppp(23) IF-MIB::ifType.9 = INTEGER: bridge(209)

(why would you label a password---that's what "public" is above---as "community"??) snmp is one of the oddest protocols I've ever seen.

Without getting into all the details, these interfaces are different than the ones I see on my machine. So this is all consistent with interfaces being, in a way, a statement of what the device is capable of doing. They are device-specific.

The Airport settings are not responsible for what I see when I do ifconfig on my machine, and my adventures in snmp land aren't likely to be responsible for the changed output I get from ifconfig.

Since I did a clean install of OS X Lion, it seems clear that I'm mistaken about when these new interfaces appeared. I have to believe now it was probably with the 10.7.3 upgrade.

[ UPDATE: Found the general issues very well explained in this book, though not this specific problem. ]

ifconfig and "interfaces"

This post is about my encounter with network "interfaces" on OS X Lion during the last few days. It's in the category of letting google "catalog my head." I haven't figured out the fundamental problem, but I learned a few things along the way and I want to record them.

Here is the text of my post to a site that will remain unnamed (due to my considerable embarassment, as I'll explain in a bit):

I started with a simple idea: do bad guys knock on the door of my LAN very often, and does the Airport Extreme keep logs that would tell me anything about it. It didn't seem like Airport Utility was going to get me very far.

In the middle of a long saga involving installation of a tool called mrtg, I bailed out from a perl script to set various configuration parameters for use of snmp because it became clear that I had no idea what I was doing.

I come back to Terminal and do ifconfig and notice three new interfaces (not there just before the previous adventure):

gif0: flags=8010 mtu 1280
stf0: flags=0<> mtu 1280
fw0: flags=8863 mtu 4078
lladdr 60:fb:42:ff:fe:f4:cd:9e
media: autoselect
status: inactive

So I have two questions: where can I find out more about these "interfaces." And even more important, after I reset my Airport to factory default, and reinstalled OS X Lion on my machine, with a wiped disk, why are they still there? Where is that info stored and how would I modify it? I tried various ifconfig commands but they're like Arnold, they always come back.

As to finding out more about the interfaces, one can simply do man gif. As often happens with man, the text leads to an ever expanding web of stuff I don't understand, but the gist is clear, these things relate to IPv6. I had already seen this page on apple.stackexchange which translates some of the man details into english. But it gives me no idea as to how these started appearing in my output from ifconfig. So I asked another question, this time in a more appropriate forum (apple.stackexchange). The answers all indicate that these things are "benign," but unfortunately give no indication of where they came from. Also, it occurs to me that my impression of when they appeared may be mistaken, hence my comment:

As to why the interfaces appeared, my best guess is that while they showed up after the snmp adventure, they weren't caused by it.

because I cannot explain things in any other way except to think that my computer sets up these things when it boots, and something in a plist somewhere has changed to modify what appears with ifconfig, and that whatever it is, it is present after a clean install of Lion. It's Apple that's messing with me here. These interfaces are clearly controlled by the OS X boot process.

Two things one might have thought would work, but do not:

Using Network Prefs to remove FireWire causes it to go away in the GUI, but it's still there in ifconfig, even with a re-boot. According to the man page for ifconfig it looks like this should work:

sudo ifconfig gif0 destroy

and it makes gif0 go away (though not stf0), but on reboot it comes back. So, things are still mysterious.

In the meantime (almost done), found something that looks really interesting (no python package required for this):

>>> from SystemConfiguration import * >>> interfaces = SCNetworkInterfaceCopyAll() >>> for interface in interfaces: ... bsd_name = SCNetworkInterfaceGetBSDName(interface) ... display_name = SCNetworkInterfaceGetBSDName(interface) ... print "BSD: %s , Display: %s" %(bsd_name, display_name) ... BSD: Bluetooth-Modem , Display: Bluetooth-Modem BSD: en0 , Display: en0 BSD: fw0 , Display: fw0 BSD: en1 , Display: en1

There seem to be a lot of SystemConfiguration APIs available from Python

>>> import SystemConfiguration >>> len(dir(SystemConfiguration)) 4338

Curious, that this does not give gif0 and stf0, nor the "loopback" interface.

And finally, what happened with the other question/answer site? The problem is that I had not realized (to quote the FAQ):

and it is not about…

Anything in a home setting

It's even in bold type in the FAQ. So I made three mistakes: I posted about a home topic and I complained when a guy with big rep down-modded me without even answering what I thought was a reasonable question. After that, they got about as mad as you can get on the web.

My bad.

launch control to monitor my server

This post integrates the previous post with the series that came just before it on a Ubuntu server running on OS X Lion under VirtualBox. I'm a complete amateur at this (really just using blogger as a scientific notebook), so if you have suggestions I'd be very happy to hear from you.

The idea is that the first thing a black hat will likely do is wipe (or sanitize) the logs, so we should move the data off the server periodically. To do this I take advantage of the ability to ssh into the server from OS X, and do a scp to Lion. After doing this, the script compares the current logfile with previous versions, and only logs the changes. The listing for the script is at the end of the post.

I changed the plist as shown by the diff. The change runs the script every 60 seconds rather than watching a path on OS X.

> diff com.TE.script.plist com.TE.script.old.plist 14,15d13 < <key>StartInterval</key> < <integer>60</integer> 19a18,21 > <key>WatchPaths</key> > <array> > <string>/Volumes/HD/Users/telliott_admin/Desktop/y.txt</string> > </array>

To make things easier, I worked on the script from the Desktop until I thought things were OK, then ran this shell script retry.sh (using ./retry.sh):

sudo cp ~/Desktop/watch.py ~/Library/Scripts/watch.py
launchctl unload com.TE.script.plist
launchctl load com.TE.script.plist

The plist contains the key RunAtLoad with a value of true, so reloading the job runs the script.

To test, I just do:

> date Sat Mar 31 16:10:25 EDT 2012 > curl http://10.0.1.2:8082 ^Z [21]+ Stopped curl http://10.0.1.2:8082

Unfortunately, marking the time doesn't help because the server and OS X are out of synch. :)

And then I watch the tmp directory for the appearance of 201126.31.log (that's hour minute second.day), which contains the forensic evidence, 7 repeats of the following entry with intervals of a second or two (reformatted):

Mar 31 16:18:06 U32 kernel: [12503.991135] [UFW BLOCK] IN=eth1 OUT= MAC=08:00:27:d7:ba:0e:00:26:b0:fa:75:7f:08:00 SRC=10.0.1.3 DST=10.0.1.2 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=50692 DF PROTO=TCP SPT=49982 DPT=8082 WINDOW=65535 RES=0x00 SYN URGP=0

I would be very interested to know of other solutions. For example, it might be cleaner to watch the log files on Ubuntu, then email them or something when things change. But I would not want to have Ubuntu able to ssh to OS X. That wouldn't be a good design for a "honeypot."

watch.py

#! /usr/bin/python import sys, os, subprocess from time import gmtime, strftime # you'd want year first to sort properly for real tm = strftime("%H%M%S.%d", gmtime()) tmpdir = '/Users/telliott_admin/Desktop/tmp/' # scp the log over to OS X in tmp/uwf.log scp = 'scp' src = 'telliott@10.0.1.2:/var/log/ufw.log' dst = tmpdir + 'uwf.log' cmd = ' '.join((scp, src, dst)) obj = subprocess.call(cmd,shell=True) # perhaps this fails (e.g. server is down) if not obj == 0: sys.exit() # check for no previous entry src = dst prev = tmpdir + 'prev.log' try: os.stat(prev) except: dst = prev cmd = ' '.join(('cp', src, dst)) obj = subprocess.call(cmd,shell=True) # write to a time-stamped file dst = tmpdir + tm + '.log' cmd = ' '.join(('cp', src, dst)) obj = subprocess.call(cmd,shell=True) sys.exit() # load data: current logfile FH = open(tmpdir + 'uwf.log','r') current = FH.read().strip().split('\n') FH.close() # load data: previous logfile FH = open(tmpdir + 'prev.log','r') previous = FH.read().strip().split('\n') FH.close() # only keep data that is new t = previous[-1].split()[2] tL = [e.split()[2] for e in current] if t in tL: i = tL.index(t) current = current[i+1:] if not current: sys.exit() print 'logging', len(current), 'items' # write to a time-stamped file s = '\n'.join(current) dst = tmpdir + tm + '.log' FH = open(dst,'w') FH.write(s) FH.close() cmd = ' '.join(('cp', dst, prev)) obj = subprocess.call(cmd,shell=True) ,

Basics of launchctl on Lion

I made a start on learning how to use launchd to run scripts. (An old) MacResearch tutorial is here. There is also a wikipedia page.

The basic usage is quite simple. We just need two files.

watch.py

#! /usr/bin/python from time import gmtime, strftime fn = '/Users/telliott_admin/Desktop/time.txt' FH = open(fn,'w') FH.write(strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime())) FH.close()

Make sure the script is executable (I did chmod 755), and copy it to ~/Library/Scripts.

The other file is a plist that shows the details for what we want launchd to do:

com.TE.script.plist

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" \ "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>com.TE.script</string> <key>LowPriorityIO</key> <true/> <key>Program</key> <string>/Users/telliott_admin/Library/Scripts/script.py</string> <key>RunAtLoad</key> <true/> <key>ProgramArguments</key> <array> <string>script.py</string> </array> <key>WatchPaths</key> <array> <string>/Volumes/HD/Users/telliott_admin/Desktop/y.txt</string> </array> </dict> </plist>

The plist goes in ~/Library/LaunchAgents.

There are many additional things you could do---the man page is here.

The script just writes the current time to a file on my Desktop. It is set to RunAtLoad (normally that would be login, but see below), and whenever the WatchPath changes. Here that is just a file which is also on my Desktop, but it could be a directory.

We can see if it's been loaded by doing:

> launchctl list | grep "com.TE"
>

Not yet.

We can manually (un)load it by doing:

launchctl load ~/Library/LaunchAgents/com.TE.script.plist
> launchctl list | grep "com.TE"
- 0 com.TE.script


When we do that we should see the file time.txt appear on the Desktop because we set RunAtLoad to true. Also, if the file y.txt is altered, say by:

> touch ~/Desktop/y.txt

The script should execute again.

Ubuntu on Lion under VirtualBox (7)

This post is mostly about logfiles for Apache and the ufw firewall.

Security is such a big topic that it's hard to know where to start. To begin with, we should probably have the document root and the script directory moved to less predictable places. But how to redirect those client requests is a goal for the future. We should also review carefully the configuration details for Apache, including which Apache modules are enabled.

There are a number of modules that are not recommended for one reason or another, as described here:

userdir – Mapping of requests to user-specific directories. i.e ~username in URL will get translated to a directory in the server autoindex – Displays directory listing when no index.html file is present status – Displays server stats env – Clearing/setting of ENV vars setenvif – Placing ENV vars on headers cgi – CGI scripts actions – Action triggering on requests negotiation – Content negotiation alias – Mapping of requests to different filesystem parts include – Server Side Includes filter – Smart filtering of request version – Handling version information in config files using IfVersion as-is – as-is filetypes

I can check whether any are in the compiled modules for Apache I'm running by doing apache2. Investigating its use with man I see:

.In general, apache2 should not be invoked directly, but rather should be invoked via /etc/init.d/apache2 or apache2ctl.

Nevertheless:

telliott@U32:/var/log$ /usr/sbin/apache2 -l Compiled in modules: core.c mod_log_config.c mod_logio.c worker.c http_core.c mod_so.c

It seems OK.

telliott@U32:~$ sudo find / -name "apachectl" [sudo] password for telliott: /usr/sbin/apachectl telliott@U32:~$ /usr/sbin/apachectl ulimit: 88: error setting limit (Operation not permitted) Usage: /usr/sbin/apachectl start|stop|restart|graceful|graceful-stop|configtest|status|fullstatus|help /usr/sbin/apachectl /usr/sbin/apachectl -h (for help on )

Never mind

The log files for Apache are in /var/log/apache2. There are three:

telliott@U32:~$ ls /var/log/apache2/
access.log error.log other_vhosts_access.log

For example, we can see who has made requests to the server recently:

telliott@U32:/var/log/apache2$ cd telliott@U32:~$ cd /var/log/apache2 telliott@U32:/var/log/apache2$ tail -1 access.log 10.0.1.3 - - [30/Mar/2012:11:36:10 -0400] "GET /cgi-bin/test.py HTTP/1.1" 200 801 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.54.16 (KHTML, like Gecko) Version/5.1.4 Safari/534.54.16"

The first field is the IP address of the request. We can use grep with the -v flag to exclude patterns (e.g. the known hosts 10.0.1.* and 127.0.0.1):

telliott@U32:/var/log/apache2$ cat access.log | grep -v "10.0.1" | grep -v "127.0.0" telliott@U32:/var/log/apache2$

There is undoubtedly a more elegant way to do what's above, but that works. No one but 10.0.1.* and 127.0.0.1 has asked for anything (yet).

I'd like to write these files to the OS X host. Since I set up ssh, I can use that to do it from a Terminal in OS X:

> scp telliott@10.0.1.2:/var/log/apache2/error.log error.log > scp telliott@10.0.1.2:/var/log/apache2/access.log access.logf

The error log shows a number of entries like this:

[Fri Mar 30 07:14:37 2012] [error] [client 127.0.0.1] File does not exist: /var/www/favicon.ico

Apparently we're supposed to have a favicon. This is annoying, but I can fix it as described here.

However, it's not just Apache we need to worry about. There are system logs like /var/log/syslog.

For the future, we should set up a way to systematically monitor the logfiles for unusual activity. I'd rather not write such tools from scratch, though it's a possibility. I'll have to look into it.

There is also the firewall (ufw), which I haven't actually set up yet. (The default state is off). It really is time to think about that. The docs for ufw are here.

telliott@U32:~$ sudo ufw default deny [sudo] password for telliott: Default incoming policy changed to 'deny' (be sure to update your rules accordingly) telliott@U32:~$ sudo ufw allow from 10.0.1.3 to any port 8080 Rule added telliott@U32:~$ sudo ufw allow from 10.0.1.3 to any port 22 Rule added telliott@U32:~$ sudo ufw status Status: active To Action From -- ------ ---- 22 ALLOW 10.0.1.3 8080 ALLOW 10.0.1.3 telliott@U32:~$ sudo ufw enable Firewall is active and enabled on system startup telliott@U32:~$ sudo ufw logging on Logging enabled

Restart Ubuntu. Check the server still works. From OS X Terminal:

> curl http://10.0.1.2:8080 <html><body><h1>It works!</h1> .. > curl http://10.0.1.2:8080/cgi-bin/test.py <head> <p> HTTP_ACCEPT */* </p> ..

Now, if an attempt is made on a different port, ufw should log it.. For example, OS X has a utility called stroke, which can carry out a port scan:

> cd "/Applications/Utilities/Network Utility.app/Contents/Resources" > ./stroke 10.0.1.2 22 22 Port Scanning host: 10.0.1.2 Open TCP Port: 22 ssh > ./stroke 10.0.1.2 8080 8080 Port Scanning host: 10.0.1.2 Open TCP Port: 8080 http-alt > ./stroke 10.0.1.2 8081 8081

The scan command hangs.. so do CTL-Z to kill it. That last scan should trigger a logfile entry on Ubuntu.

telliott@U32:~$ tail -1 /var/log/ufw.log Mar 30 14:53:20 U32 kernel: [ 528.390507] [UFW BLOCK] IN=eth1 OUT= MAC=08:00:27:d7:ba:0e:XX:XX:XX:XX:XX:XX:08:00 SRC=10.0.1.3 DST=10.0.1.2 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=40207 DF PROTO=TCP SPT=54520 DPT=8081 WINDOW=65535 RES=0x00 SYN URGP=0

The first part of that MAC address is for Ubuntu on the VM, and the second part is for my OS X airport card. I don't know about the last part (08:00). But the SRC and DST IP addresses are clear, as is the DPT=8081. Just to check, try it again with 8082..

Yes:

telliott@U32:~$ tail -1 /var/log/ufw.log Mar 30 15:05:49 U32 kernel: [ 1277.422229] [UFW BLOCK] IN=eth1 OUT= MAC=08:00:27:d7:ba:0e:XX:XX:XX:XX:XX:XX:08:00 SRC=10.0.1.3 DST=10.0.1.2 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=21050 DF PROTO=TCP SPT=54579 DPT=8082 WINDOW=65535 RES=0x00 SYN URGP=0

I have misplaced the logfile, but earlier I had an access attempt from somone unknown: 199.47.218.151. Just ask google: "who is" 199.47.218.151. It's Dropbox. Huh. Didn't know they were in Wichita.

http://www.ip-adress.com/ip_tracer/199.47.218.151

I had installed Dropbox on Ubuntu for file transfer, then deleted it. It seems that they were still trying to contact me for some time after that.

Finally, there are other tools out there. For example: nmap. This is another sophisticated tool that will require a more serious investigation. I can run nmap from Ubuntu on itself:

telliott@U32:~$ nmap -sT 10.0.1.2 Starting Nmap 5.21 ( http://nmap.org ) at 2012-03-30 15:11 EDT Nmap scan report for 10.0.1.2 Host is up (0.00076s latency). Not shown: 997 closed ports PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 8080/tcp open http-proxy Nmap done: 1 IP address (1 host up) scanned in 0.21 seconds

This does not trigger a logfile entry by ufw. It is curious that although I only set up rules for ports 22 and 8080, port 80 still is open (as I'd like for Firefox to work properly). Note that these port scanning tools shouldn't be used against other people's servers. That will be interpreted as aggressive behavior (probing for weaknesses), and will surely make folks upset. It's the sort of thing we want to be on the lookout for on our server.

Immediate goals for the future: understand the detailed setup of ufw and get a good book about Apache.