To a geeky statistician, those are dangerous words. Dangerously appealing words.

Before you continue, let me warn you: extreme geekitude follows; performing some analysis of this was like bringing an elephant gun to a squirrel hunt. A very geeky squirrel hunt (perhaps squirrel fishing). If you’d just like to see a graph of the final model, feel free to skip to the end.

So, the first thing we do is try to come up with a model for this curve. The basic idea is this: every time someone puts up a web page mentioning Kirk’s Khan scream, they have some number of a’s which they’re going to use. We consider that everyone has some number of a’s they tend to feel is appropriate, and that we are selecting from the population of people who put Khan scream references on the web. So we are modeling some underlying distribution of preference for a’s among these people.

Footnote: I also have to recognize that in addition to a distribution of preference over people, an individual person has some variation in how many a’s they actually put up; that there may be multiple populations of people; and that different kinds of people are more likely to add references to Khan on the web. Some may even post multiple times. While a more complex model which took this into account might be able to make a better fit to the data, we simply consider it all as combined into a single conditional distribution–given that the post was made, what is the probability of it including a certain number of a’s.

The first model is pretty basic: it says that after each ‘a’ is added, there’s a chance that you’ll stop, add an ‘n’, and be done. This probability is the same after each a–it’s not dependent on how many you’ve entered before. This results in the number of a’s being expected to follow a geometric distribution: each ‘a’ entered is a trial, and we continue adding a’s until we ’succeed’ and add an ‘n’. On a log scale, this model is a straight line.

After seeing this (and a few other models), and doing a little web research, we remove the two leftmost points from the data for our model. These are ‘Khan’ and ‘Khaan’ (1 and 2 a’s). They are much higher than the rest, and substantially change the model. We suspect that their references are largely due to very different sources: anyone referring to Khan Noonien Singh himself (or Gengis Khan, or any other Khan) for the first, and anyone referring to Khaan (an actual animal and also a common alternative transliteration of Khan) for the second.

After we do this, we can see an improved fit, though there are clearly still some regions of higher- or lower-than-expected occurrences.

So we now make our model a bit more complex, reflecting in part the complexity discussed above. We make a mixed model, suggesting that there are two populations posting Khan references. One follows the geometric model we used above; but the other, we will model as a negative binomial distribution: one explanation is that these are people who are aiming for a large number of a’s, and we are modeling their variation in what they think of as “a large number of a’s”. Fitting this mixed model (using maximum likelihood to determine how many people fall into each group, and the distribution parameters for each group) gives us the next graph.

A more complex model would attempt to model the conditional probability of adding another a (given how many a’s have already been added) as varying smoothly, depending on the number of a’s already added…we could, of course, model this as some sort of generalized additive model…sorry, please excuse my drool. Let’s continue.

Of course, I had to take it another couple of steps further. When I started this project, I wrote a perl script which would go to google each day and save the number of Google results for each search, stored in a file by date. Further, I extended the range to 125 a’s (anything longer than this, Google considers too long). So what we now have is a time series: for each day, we have an entire graph of values. Using this, I was hoping to see how the numbers change over time. Unfortunately, it appears that the results are not consistent over time, having significant variance up or down. Presumably, this is a result of Google trying out different variants on what results to return. But it means that rather than seeing counts increase over time, we see some variance in each count. For example, the counts for “khaaaaaaaaaaaaaaaaaaaaaaaaaan” (26 a’s) vary from around 150 to around 8000.

You can see the variance overall by looking at a boxplot of the ranges for each number. For some reason, there’s a lot of variance for 5-34 A’s, but not too much outside of that range.

So, time series analysis is pretty much out; this is a shame, because you can pretty easily make a video of the counts on each day, over time (with a fitted model for each day). The trouble is that the counts are more affected by the algorithmic decisions google is making behind the scenes than by any underlying change in the number of pages.

(click to see animation)

But we can at least try to use this variance to see if it smooths out any of our earlier outliers. Here, we’ll take the median reported values, over time, for each number of a’s (rather than the individual reported numbers on any specific day) and repeat the earlier geometric/negative binomial mixed model:

Final model: Mixed Geometric/Negative Binomial on median counts

And that, I think, actually looks like a pretty decent fit. Notice that the negative binomial portion is actually fitting the low-A section now, rather than the strange middle-A hump we saw before; this seems to give a more natural interpretation: most people will put in around 6 A’s for KHAAAAAAN!, and for people stretching longer, a geometric distribution fits pretty well for determining how long they’ll keep adding A’s.

So there you go. Proof that anything can be overanalyzed. If people like this (drop a comment here or email me), I’ll keep collecting data and will look at doing some additional analysis with more data in a few months.

You can download the perl and R code and khan data from thomaslotze.com. While this was inspired directly by Kevin Martin’s post referencing squidnews, there were also earlier graphs from drtofu, Walrus, and Jim Finnis.

Click the picture for a full-resolution version. It’s entirely edible, down to the fondant rocks and robber. Most impressive. This birthday cake (actually, 19 cupcakes) was made by housemates Cassia and Kim for our friend Tony’s birthday. I believe that the inspiration came from Cupcakes of Catan.

It sounds pretty ridiculous, doesn’t it? I thought so, too. Then I started thinking about how this is one of those amazing things about art: that it can take inspiration from anywhere. No matter the source, if you can create something which speaks to some aspect of our humanity, it doesn’t matter what inspired it. Truth can come from surprising places, and we can find important insights into the patterns of life and interaction in all sorts of human creations, even (perhaps especially) in those created primarily to entertain. We shouldn’t reject something as not being meaningful, just because its source is unexpected or unorthodox. One of the reasons people love Star Trek itself is that it frequently grapples with meaningful human issues, even as it’s surrounded by rubber suit aliens and gobbledygook science. There’s no reason Klingon opera can’t do the same.

Then I watched some. I’ll admit, while I did think their exploration of sound and its qualities was interesting, I found it more than a little bit stilted…and kind of ridiculous.

On the other hand, I feel that way about most opera. Enjoy the new movie!

Maybe this will start to change that, and push smarter board games more into the popular culture in the US. I’d certainly be excited to see more of that. But from the graph (which you’ll note is cumulative sales), the growth is relatively stable. Although it does appear to have increased recently, the per-year increase doesn’t look tremendously large–about 100,000 sold up to 2004, then 100,000 in 2004 and 2005; 150,000 sold in 2006 and 2007. 600,000 copies sounds pretty good for a single board game–but remember that with approximately 100 million households in the US, that’s still under 1% penetration. Hasbro is reticent to let out detailed data on Monopoly sales or estimated number of copies in the US (as was Mayfair, the source of the data in the article), but a New York Times article reported that they sell “several million copies” in the US each year, and they claim to have sold over 250 million copies total worldwide. So while Catan has an undeniably growing public awareness and acceptance, it would have to multiply its sales fiftyfold before it reaches Monopoly.

Finally, a couple of other interesting links:

• An academic paper analyzing musti-agent systems for Catan-playing bots
• A website analyzing Monopoly using Markov chains, showing probabilities of landing on the various spaces, expected returns, etc.

This is a most amazingly addictive game. You build machines. Machines! Simple machines that do things. Amazing things. All in service of getting an object to a target area. Once you’ve struggled your own way through the various levels (fun and frustrating at times), you can see some of the incredible things that others have created with simple tools. Eventually, you will realize that it is 4 in the morning and you should have gone to bed a long, long time ago.

First is an idea that I find quite appealing: the open company. Now, many of you will protest that open source companies are old news; heck, Red Hat has been public for almost ten years now. But this is something new: the idea of running a company not about open source products, but running the company as if it were an open source project. Essentially, this means that anyone can contribute whatever they feel motivated to, and be paid using a peer-rating system. It sounds pretty idealistic: the idea that in the future, we can literally just create whatever value is of interest to us and get paid appropriately. And it may only work when most of the output is intangible. But I think it’s a fascinating notion, and Alexander Stigson, creator of the E Text Editor, says he’s actively moving his company to become an open company. I’m really excited to see how that turns out.

Second, a link to a recent post by Tim O’Reilly, on books that have shaped the way he thinks. I think it’s interesting not just because he cites Dune as one of his influences, but also because he cites Rissa Kerguelen, by F.M. Busby, as

[a] science-fiction book I read at about the time I was starting my company, and that influenced me deeply. One key idea is the role of entrepreneurship as a “subversive force.” In a world dominated by large companies, it is the smaller companies that keep freedom alive, with economics at least one of the battlegrounds. This book gave me the courage to submerge myself in the details of a fundamentally trivial business (technical writing) and to let go of my earlier hopes of writing deep books that would change the world.

I think it’s important to remember that new ideas can be subversive; that both startups and science fiction are often about pursuing new ideas; and that new ideas have the potential to change the world.

Finally, on that note, I thought I’d add a quick link to the NationStates site, a free sort-of-game from the author of Jennifer Government. Enjoy!

As you might imagine, I’m not a huge fan of this idea. Instead, what really appeals to me is this response poster from Matt Jones: “Get Excited and Make Things”! I think that we all have some (reasonable) tendency to simply enjoy things in fandom, but I think that we do better when we take it further and expand upon what others have done, when the things we love inspire us to new things. Don’t just be a passive observer and rely on someone else to take care of everything: take an active, positive role in creating something new! Get Excited and Make Things!

Anyway, I was recently debating with a friend how Calvin and Hobbes ended, and came across this interesting collection of post-series takes on Calvin and Hobbes. The last one is the fake Calvin and Hobbes ending that broke my friend’s heart (and temporarily mine, before I came home to confirm that it was false). If you need a boost after seeing it, you can find the real final strip here.