Words

I wrote an Objective-C program to parse all of my iChat log files (necessary because the format is a binary property list that I really didn't want to have to parse by hand) which go back to early 2005 (the pre-2005 logs are backed up elsewhere and in another format, so I'm ignoring them for now) - and produce some SQL insert statements; one per chat message. After much anguish regarding how to properly escape string literals (specifically backslashes; hint: double-escape the backslash and use the octal representation, ie in a C-like language, replace "\\" with "\\\\134", which is really "\\134" in the SQL statement, which the Postgres parser then turns into "\134" for bytea insertion - thank you Steven for typing a :-\ smiley face and costing me an hour of my life), the doubly-escaped SQL output was then dumped into Postgres and we finally have something to work with.

I had 291,640 total chat messages (150,091 received, 141,549 sent) comprised of 2,503,521 words (41,415 distinct words), including typos and garbage contributed by 170 "people" (ie, screen names) over 911 days between March 9, 2005 and March 31, 2009. If the set of distinct words is reduced to just those words used at least five times, it works out to 13,342 words (8,640 received, 9,374 sent). Not exactly a testament to my vocabulary. Anyway....

Then I wrote a Perlprogram to pull down each chat message, split it into words, lowercase it, create a unique word entry if the word hasn't been encountered before, and associate that word id with that chat id. An apostrophe that does not occur at the beginning or ending of a word was the only non-A-Z character I allowed in a word; everything else defined the boundary of a word. After watching four Futurama episodes it still wasn't done so I went to bed. As if by magic, it was done in the morning. (This process would go much, much faster if I put the directory recursing code in the Objective-C program instead of doing the recursing in the shell. But I really don't care.)

Then I wrote some SQL to extract the top 100 most used words by people I chat with, and the top 100 most used words by me. The results can be seen in the table that follows.

The people I talk with apparently tend to talk about themselves more than I talk about myself (I save that for my blog, I guess):

"I" is #1 vs #2
"my" is #15 vs #18
"me" is #22 vs #26

and about me more than I talk about them (with the possible exception that it could just be lots of instances of speaking of oneself in third person):

"you" is #5 vs #9
"your" is #40 for both

and apparently a lot of people send me links because "http" made it in as #90.

Update: I found this site, wordcount.org (please try to ignore its interface which tries to make the information as useless as possible), which ranks English words by popularity (of use). What's interesting is that its top-ten is very different. "I" doesn't even make it into the top-ten. I guess instant messaging isn't entirely similar to their British National Corpus data source.

Okay, here's the table. Top-100 words by use, separated into incoming vs outgoing.

Shown on a logarithmic scale, it's apparent how quickly word frequency drops off:

I have a ton more email history than chat history - I'll see about parsing through all of that when I get the time. Got to figure out how to avoid the in-line base-64 encoded attachments.