e1ven dot com

Compress any file into a Tweet?!

This morning I launched .com, an online tool for reducing any file to a tiny, almost impossible size.

I launched a fun website for it, as a portmanteau of “Hash” and “Shrink”.

The theory behind it was one I first started playing with in 2008, which is the idea of creating every possible iteration of a file, to find all the hash-collisions.

You’d then reject any candidates that didn’t fit certain qualifications. For instance, you might reject any file that isn’t a valid MP3, or any file that doesn’t match a perceptual hash
After all the filters, you’d indicate which hash-collision generated the correct version of you file.

I wrote up two versions to play with the idea, and see what it might look like.



Writing it was a lot of fun, and I got more practice using multi-processor code in Python.
Unfortunately, it’s entirely impractical, at least without Quantum Computers.

The problem is, even a fast machine using the GPU can only do 400K hash collisions per second.

At this rate, it would take more time to re-create the file, than has so-far elapsed in the universe.
That’s a bit slow

In any event, I figured it’d be fun to make up a website for the project, and put a demo online, which calculates how-long your file would take to return.

After a bit of playing and a cheap template from ThemeForest, I put together some images for it.

The logo was a lot of fun, trying to emphasize the dual-nature of the name.

I also had fun trying to make images which ‘pop’^{I hate this word}, and get practice using some flash tools to cycle between them.

.com was born.

Comparison of Open Source Licenses

As I’ve been working on my little Forum Project over the last few months, I’ve run across a number of code snippets that I’d love to use.

Often they’re little things, posted to blogs explaining how to format a URL or whatnot, but I can’t legally use any of them.

Under the Berne Convention, every single bit of text or code you create is automatically copyrighted.
You don’t need to put a © or date on it, and no one can use it without your expressed permission.

I’ve written to each of the authors, and received releases, but one common thread I’ve heard is that people aren’t sure what license is right for them.

I’m certainly not an expert, but I wanted to put together a quick comparison of different licenses, and the implications.
I’ve included links for each to read them yourself. Please read through the license before applying it to your project. I’m just a Sysadmin, not a legal advisor.

One thing to note is that if you wrote the code, releasing it under a certain license never limits what You can do with it going forward.
You’re welcome to release the code under the GPL, then keep all your updates closed-source and for-pay, and never release them under any open source license.
So long as you’re the one who wrote all the code, releasing under an Open Source license just explains what Other people can do with the code.

For example, For a long time, Mozilla released the Firefox code under the MPL, AND the GPL, AND LGPL, letting people choose whatever license they wanted for the codebase.
Oracle releases MySQL under the GPL, but will also sell you a commercial version under other terms.
As long as all the code in the project is yours, you’re free to do whatever you want.
The only warning is that when things are open, other people often want to help write code- You’ll want to make sure they either assign the copyright to you (letting you do whatever you want, as if you wrote it).

There are few elements with different licenses differ about:

Attribution- Different licenses ask that you recognize the author in different ways. Some let you take credit for the code, others ask for a line in the About section, and others want placement on all advertisements for any product that uses any of it’s code.
Some licenses include a clause that makes it clear that you shouldn’t use the name of the original company when talking about your own product. This is referred to as an “Anti-endorsement” clause.
Copyleft- Licenses that use Copyleft are sometimes called “Viral” licenses; If you include any code that uses that license in your code, then your entire program has to be released under a compatible license.
Their goal is to stop companies from taking your code, building on it, then keeping all the changes to themselves.
If they use your code, they have to release theirs.

• A “Strong” copyleft license says that only programs which are under the same license are able to use this code.
• A “Weak” copyleft license says that all changes to this particular code must be released, but other software can be built using this as a library, and that can software can be under a different license.

Patent Protection- Traditionally, software licenses, including Open Source Software licenses don’t deal with patents at all. This has lead to difficulties wherein a company might release their software as Open Source, but years later explain they have a patent on the underlying ideas. Even though you’re allowed to use their software under Copyright law, they can still come after you under Patent law. Some licenses include a patent license, to avoid this situation.
Web Application- Another shift we’ve seen is companies move from distributing software, to running offering hosted versions of that software on their websites. Some copyleft software licenses ask that Web Applications release their code, where others only apply if you actually distribute the software to someone else.
Permissive- Sometimes people will talk about how “Permissive” a license is. The more Permissive a license is, the more you are allowed to do with that software without the license coming into play. These range from Very permissive licenses, such as the MIT/X license, to more restricted licenses such as the EULA that commercial software packages have.

One important thing to note is that OSS licenses are almost universally licenses on Distribution, NOT on use.
Unlike a EULA, you do not need to agree to the license in order to use the software.

The license only comes into play if you are distributing it, either directly (GPL/LGPL/etc) or by exposing it as a webapp (AGPL)

What a mess!
What’s even more “Fun” is trying to combine them.
Some of the licenses are compatible with one another, and others aren’t.
Still others are compatible, but only if you stand on your head and chant in latin.
I’ve included my personal understanding of how they combine below. As always, consult a lawyer, and let me know if I missed something.

I wrote this table from the perspective of, assuming I have existing codebase A, am I allowed to import code B?

Also note that I’ve simplified the result slightly, to reflect the practical effect. For example, I say that MIT/X11 + 4-Clause BSD Result = 4-clause BSD.
Realistically, the code is now combination of the two licenses, and you need to follow and include both code, but as far as effective restrictions, it now acts like 4-clause BSD.

[1]

In many cases, licenses contain a clause which allows you to upgrade to later versions of the same license.
For example, the default version of the GPLv2 includes a clause to allow users to “upgrade” to GPLv3.
If this cause was included, you can freely mix this code, with the resulting combination being covered by the newer license.
You should check to see that this clause is included. Some prominent software, such as the Linux Kernel, does not have it.

Other software may require BOTH sides to update to newer versions of the licenses
For instance, the AGPLv1 and GPLv2 are incompatible, but if both packages contain “Or later versions” clauses, they can be upgraded to AGPLv3 and GPLv3, which are compatible.

[2]

The MPL2 performs kind of a neat hat-trick
It indirectly allows compatibility with the GPL-series of licenses, by a bit of legal maneuvering.
The end result is that you can use combine them, but keep the MPL code MPL’d, and the GPL’d code GPL’d.

Thoughts on CloudFlare

Overview

Over the last few weeks, I tested CloudFlare as a low-cost CDN for RoboHash.org.

I did this partly to test the speed, and partly to reduce the bandwidth costs that I was seeing through MaxCDN.

A little background – CloudFlare works by taking over as the primary DNS provider for your domain- In order to use their service, you need to set CloudFlare as your domain’s primary name servers.
This makes some sense, since they can route to various servers on their site as necessary, but creates a bit of a headache-

I offer Robohash in both HTTP and HTTPS versions. In order to serve HTTPS through CloudFlare, you need to be a premium/paid member. That’s fine, and I don’t mind the fee, but you can’t upgrade your account until your DNS already is moved over.

This creates a bit of a headache in that you to commit to fully using CloudFlare before you get a chance to really test it. With most CDNs I’ve used, I can setup an Origin-Pull, and run both concurrently.

Speed

Subjectively, the site did feel faster. I also noticed the number of hits that were hitting my nginx instance were cut down dramatically. Within 2 days, the number of hits reaching my server were cut to 1/7th of what they had been prior.

That makes sense, since the RoboHash.org content is HIGHLY cacheable. In the typical usage, such as a forum or blog, the Robot will be generated for the first user, and then every subsequent load will be from cache.

Pingdom didn’t report any clear change in the load times.
If anything, the page load times seem far more variable after moving over.

SSL

Enabling SSL support through Cloudflare was spookily simple.
CloudFlare accepts the SSL request to their site, and proxies it back to my origin over normal HTTP.

I didn’t need to give them a copy of my certificate, or give them any additional information.

Essentially Off

CloudFlare has a built-in feature which helps to protect your sites against attack- When users who are suspected being a threat (such as infected PCs, Spammers, or would-be hackers), CloudFlare blocks their access to your site.
Optionally, you can allow these users access, if they pass a CAPTCHA.

RoboHash is a pretty simple service, and I want to make it available to as many people as possible.
Since it loads as part of OTHER people’s sites, I don’t want it blocked. I don’t care if they’re actively trying to hack ME, don’t block it. I don’t want users of Robohash to have a degraded experience.

I emailed CloudFlare, and asked that we disable the blocks entirely. They admitted that there are false positives, and there’s no way to fully disable the feature. They suggested the ‘Essentially Off’ functionality, which is supposed to only challenge the worst of the worst.
Running the service for 2 weeks gave me over 30 blocked IPs, most for being part of a botnet.

I’m not at all comfortable with the number of blocks they were performing.
Further, CloudFlare sets a cookie on every request.

I don’t want this, and it’s not fair to downstream users of RH.org

threepwood:~ e1ven$ curl -I http://robohash.org/ABCD.png
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Mon, 27 Feb 2012 20:15:10 GMT
Content-Type: image/png
Connection: keep-alive
Expires: Tue, 26 Feb 2013 20:15:10 GMT
Cache-Control: public, max-age=31536000
CF-Cache-Status: HIT
Set-Cookie: __cfduid=d9c25dc8324ac644c1ae0f980ae487c311330373710; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.robohash.org

versus the native request

threepwood:~ e1ven$ curl -I https://robohash.org/ABCD.png
HTTP/1.1 200 OK
Server: nginx/1.0.2
Date: Mon, 27 Feb 2012 20:16:43 GMT
Content-Type: image/png
Connection: keep-alive
Expires: Wed, 29 Feb 2012 20:16:43 GMT
Cache-Control: max-age=172800

As an additional point of comparison, here is the result from MaxCDN, another CDN who I’ve had good luck with.

threepwood:~ e1ven$ curl -I http://static1.robohash.org/ABCD.png
HTTP/1.1 200 OK
Server: nginx/0.8.36
Date: Mon, 27 Feb 2012 20:34:32 GMT
Content-Type: image/png
Connection: keep-alive
Expires: Wed, 29 Feb 2012 20:21:28 GMT
Cache-Control: public, max-age=172800
CF-Cache-Status: HIT
X-Cache: HIT

Missing Images

Finally, however, I started seeing an increasing number of missing images.
Every so often, maybe one in 20 page loads, I’d see a Robot not just appear at all.

That’s simply unacceptable.

Summary

Ultimately, I’ve removed CloudFlare from the DNS for Robohash.
I apologize for any inconvenience caused while I was using it.
I like the idea behind their service, and I can see how it might be useful for a large number of sites, but their offerings are tremendously invasive, give you limited control over disabling their ‘features’.

It’s a good deal if you’re looking to help protect your blog from the slashdot-effect, particularly as a solution that semi-technical users can implement. If you have the patience to learn, it seems like in most cases you’d be better off with a caching Nginx server, potentially combined with a cheap standalone CDN.

You’ll get better performance, more control, and fewer surprises.

My fun time playing Grim Fandango

My experience playing Grim Fandango, a game from 1998

I’m somewhat ashamed to admit this, especially as someone who purports to be a fan of Adventure Games, but here it goes.. I’ve never played Grim Fandango.

I know.. It’s a game which is frequently rated as one of the best adventure games, ever.

Colin Panetta’s amazing illustration of Manny Calavera

Obviously, this is a very serious situation, which required immediate rectification.

Particularly with the news that the lead developer from Grim Fandango, Tim Schafer, is at work on a new multi-million dollar adventure game.

I had some time off of work, so I decided to take the plunge. I installed and played through the game this week..
The Puzzles were generally paced well, the story was engaging, and the characters were charming.
My biggest challenge with the entire game, it turns out, was getting the game to run properly long enough to finish it

Running on my Macbook

Lately, I’m doing most of my computing on a 17″ Macbook Pro – I’ve pimped the RAM up to 16G, it’s got a speedy processor, the OS is fun to use.
Since there’s no Mac-native version, I took a look at ScummVM, which I had used for other LucasArts games in the past, such as Monkey Island, and Day of the Tentacle.
It turns out that ScummVM only supports the 2d games, but a sister-project exists, ResidualVM which supports Grim.
Their website recently shows the game as 100% completable, so I downloaded the VM, copied the files from the CDs and fired it up.

It took a tiny bit of fidgeting- I needed to manually edit

"~/Library/Preferences/Residual\ Preferences"

and add

"fullscreen=true"

to allow the game to play fullscreen, but Google was helpful enough in advising me.

The game was glorious! The game’s graphics were a little dated, but were given such a stylized look that it rarely mattered.
I was able to get a few hours into the game, before I ran into a bug in Residual- I walked into an area I wasn’t supposed to get to yet.
Never having played the game before, I had thought that I was playing normally. The tone of the game changed fairly dramatically, but I wrote it off as a mystery I was supposed to discover.
As I kept playing, however, the game was referencing things that took place earlier, which I had no recollection of!

I kept playing the game for a while, until I realized that I had found myself in an unwinnable position.
I had somehow jumped further ahead in the story than I was supposed to be, and had missed some crucial events.
(For anyone who’s played the game, the specific bug I hit was clipping into Domino’s office, immediately after first meeting Meche)

Eventually, I realized that this couldn’t possibly be expected behavior- I checked a walkthrough, and found where the VM had glitched.
Since this was my first play though, I realized that perhaps ScummVM wasn’t quite ready for me yet.

If I had known what lay ahead in trying to play other versions, I probably would have kept playing, bugs be damned. Alas, Hindsight can be cruel.

Try 2, My Trusty old PC

Grim is a PC game, after all, so I figured I’d avoid these bugs if I played it in the original engine.
Simple enough.. I had an old P4 desktop PC I built a few years back.. I just needed to install a hard drive, replace the power supply, install Windows 7, find drivers for all the various components, and I’d be off the races.

Touching these two wires together is the power switch

Eventually, I got the machine up and running- I reinstalled Grim, set the game to run in compatibility mode, and again, I was playing.
The saves weren’t compatible, so I had to start from Scratch, but that was a small price to pay for knowing that I’d not run into any VM-specific bugs.

The more I played, the more I was having difficulty controlling the game- Grim Fandango is one of those games that feels like a bad console port, even thought it was only ever released on PC.
After a few hours of playing, I realized I really ought to be playing it with a controller.
I trip to Microcenter later, and I was back, a new Xbox360 controller in my bag, and ready to start making serious progress.

The only problem was, the game crashed. Frequently.
Sometimes, the game would crash mid-save, corrupting the save file.
Othertimes, it would just grind to a halt, and become unplayable.

OK, I thought to myself- Compatibility mode doesn’t work. Let’s go back to the source.

Win98 time

Grim was originally released in 1998, so it seemed natural to try running the game under Windows 98.
I still had my original Win98 CDs in the closet, so once more, I formatted my PC, this time installing the classic OS.
The first problem I ran into was that my HD was too large- Each time I asked the Win98 setup utility to partition and format the 500G HD resulted in the machine hanging.
Despite enabling LBA mode, formatting it with a Linux boot CD, and leaving it for hours on the formatting screen, no progress was made.

I dug around, until I was able to find an older 80GB harddrive deep in a closet full of things I ought to have thrown out years ago.

Now, I was able to format successfully. The install was able to get through the text-based part of the installation.. And then crashed.

Googling a bit more, it turns out that Windows 98 won’t run with >512 of RAM.

OK..Luckily, there was a workaround. A few tweaks to system files later, and I’m up and running in glorious 16-color windows.

My Video card was released just a few year too late, it seems, to have had any Windows 98 drivers created. The oldest version of the drivers for my card run on Windows 2000.
I wasn’t stopping here, though-
After some digging, I found a generic SVGA driver that was built for Windows 98.. It wasn’t fast, but it worked.

Next up, Audio- Creative’s site doesn’t list the Win98 drivers anymore, but some clever folks at the MSFN Forum had found a working link to a similar card, built on the same chipset.

Finally, I could fire up the game!

And… It crashed. Every time. On launch.

It didn’t write a log.. Win98 doesn’t have an Event Viewer, so I didn’t have many options.

I realized that I might need some critical Win98 patch in order to play, so I fired up Windows Update.. And was greeted with a redirect loop on Microsoft’s site.
It seems that they don’t even offer the Win98 update service anymore.

Luckily, another Win98 Aficionado put together an Unofficial Win98 Service Pack 2.1, which includes all the updates ever released.

Installed, and the game started…. And the game ran!

The problem was, I was in seconds-per-frame territory.

The Generic SVGA driver does work, but it looks like it can’t take advantage of ANY 2d acceleration.. So the game was entirely unplayable.

Win98 in a VM

OK, I thought! This is solvable. VMware has tested, working drivers for Win98, so I’ll use them.

It might be a bit of CPU overhead, but the video performance should be much faster.

I dutifully reinstalled Windows7, and installed a 30-day demo of Vmware.
It turns out that I needed to use Vmware 7, since the newest Vmware 8 wouldn’t run on my P4 CPU.

I installed Win98, patched it, then loaded the Vmware drivers.
By this point, I was starting to get pretty quick at the whole process- I had all of the drivers I needed for Windows7 copied to my Dropbox, so it was rather straightforward to get running again.

VMware installed cleanly, and Win98 ran mostly without issue. The biggest problem was the Audio drivers that ship with Win98 aren’t always compatible with VMware’s virtual machine hardware – There are replacements however.

Once things were finally back up and running, I re-installed and restarted Grim. This time, I was greated with a beautiful 4-panel screen, in inverted colors, before the game crashed.

I found that the the game would run in Windowed mode, however!
The speed better than native Windows 98, but not yet playable.

Watching the system to see where the load might be, I saw that the CD was being accessed heavily.
Since the game was designed in 1998, the designers tried to minimize the amount of disk space used, preferring to seek the needed content from the CDROM wherever possible.
Luckily, there was an updated launcher for the game, which a fan had released- This program copies all the content from the game’s CDs to the local hard drive, then instructs Grim to use the local version.

The only problem was that this app was that it needed GDI+, a library Microsoft released around the time of Windows XP.
Microsoft had released a version of this library for Windows 98, but they don’t distribute it anymore.

Luckily for me, other sites still have mirrored copies.

After installing, I needed to install DirectX. Microsoft released DirectX 9.0c for Windows 98, so it should be an easy task.. But it turns out that Microsoft released 18 different versions of DirectX with the same version number!.
Once more, I was able to find them Mirrored on a third party page.

Finally, I was able to run the game, and despite it running in a Window, it ran reasonably well.
It was certainly playable, but it stuttered heavily. A bit more research showed that while Win98 would run the WDM drivers for the sound card, it suffered performance problems when using them, compared to the older VXD drivers.
Further, running in Windowed mode heavily cut into performance.

Once more, into the breach

I had read a number of reports of the game running properly in Windows XP’s compatibility mode, so this was my next step.

I created a new VM, installed XP, drivers, updates.
Now that I was on a more modern OS, I was able to use the official Xbox 360 drivers.
After patching Grim, and installing the modified launcher, I was able to run in full screen, and had the best performance of any system yet!

The only problem at this point was the 3d models were flickering in and out of existence while I played. Using DX Diag and disabling DirectDraw finally fixed this problem, allowing me to finally, after days of prep work, play the game reliably.

Conclusions

While it’s clear, in retrospect, that playing the game in Windows XP compatibility mode is the safest balance of driver-availibity to game-compatibility, that was far from obvious initially.

One might reasonably suspect (as I did) that using either the Latest OS, or the one from the time, might be the best choices.

What seems so infuriating is just how difficult it is to play a game from the not-so-distant past!

Grim Fandango was released in 1998.
For reference, That was the same year than the movie Office Space was released.

With Office Space, I can still play a DVD I bought in 1999, or buy a Blu-Ray today.
I can stream the movie from Netflix, iTunes, or Amazon.

I can watch the movie in any number of formats, quickly, easily on any number of devices.

I can still buy and watch movies from nearly 100 years ago, without much effort!

I can watch these movies more easily than I can play a game that’s younger than either of them.

But with a Video Game of the same Era, I have to clear obstacle after obstacle.

Now as I mentioned in the introduction, I really enjoy Adventure Games- I’m willing to jump through hoops to play what is considered a Great Game.. But this is far beyond what could be expected of anyone who was simply curious what this great game was about.
Further, to get the game running in each of these scenarios, I relied on a Myriad of third party sites mirroring older utilities, many which technially violate copyright laws.

For older games, DosBox provides a solution to this problem, by allowing Publishers to distribute the game running in a Virtual Machine, very similarly to my eventual solution in VMware.

This isn’t a legal option for newer games, however- If Lucasarts wanted to sell a copy of Grim Fandango this way, they’d need to include a copy of Windows with each game!

This is an instance of Copyright creating unintended consequences.

Eventually, projects such as Wine or ReactOS may allow a dosbox-like packaging, but this is always going to lag years (in this case, over a decade) behind OS development.

Outside of moving entirely to FOSS, I would propose that best solution to this puzzle might lie with Microsoft. If they were to release an embeddable version of Windows, that could be used as a wrapper for applications, they could largely solve their compatibility problem.

We already see Microsoft adopting a similar strategy to allow users to test on various versions of IE. Expanding this program to the general public would allow Microsoft to gain the benefits of having programs written to an open standard (indefinite durability), while still tightly controlling the publishing platform.

A somewhat non-technical explantion behind the creation of RoboHash

With hundreds of millions of variations, Robohash.org is the among the leading robot-based hashing tools on the web.

About

A few months ago, I released a new website that turns anything you type into Robots.
I realized I never mentioned it here, so I figured now was as good a time as any to explain how it worked.
First off, the code is available, so you’re free to take a look yourself.

Genesis

I’ve been working on a new type of forum, that’s designed to make it easier for people around the world to communicate, share ideas, and talk to one another, regardless of the potential hostility of their regime. I was inspired by the crackdown in Libya and the rise of the Arab Spring, and the later Occupy movement, to try to make communication more reliable, and harder to oppress.

In the process of making my new forum, I realized that I needed a way to identify people. Regular usernames wouldn’t work, since I needed them to be able to be unique, but work without any central infrastructure. “Ah-Ha!”, I thought to myself. “This isn’t a hard problem! Use Public Keys!”
Cool. It works, and everyone is unique everywhere… The only problem is that if everyone is using a public key, rather than a username, how would you recognize them?

I added code to let people add in a username as well, but we’re back to the uniqueness problem. How do you know that “E1ven” in Message1 is the same as “E1ven” in Message2?

I added various tricks, such as displaying the publickey on mouseover, and displaying the first part of it after the username “E1ven?AAASDASDAS…” but I knew it wasn’t going to work. It’s just too hard to keep track of public keys.

I recalled reading about some research a few years back that might help..

Faces

It turns out that people are really crappy at remembering Secret Phrases.
Outside of spies during the Cold War, the only time people really have to memorize Secret Phrases are to log into their computer systems or ATMs.
We can DO this, but we’re not very good at it, the research implied.

Instead, they developed a system that optimized for what people ARE good at – Recognition. In Particular, Recognition of faces.
This is something we do every day- When we see a co-worker, we instantly recall who they are, in a billionth of a second.
Even people we don’t see all that often are easy to recognize, even if you can’t picture what they look like.
If you ran into your Hairdresser at the Supermarket, you’d recognize it was her, even if you couldn’t think of her name right that second.

One system I thought was kinda cool actually took this idea and made it into a Password system-
You’d pick the faces you recognized out of a picture of crowd of people, and it would know you’re you.

I realized I could use this same technique to help people recognize a poster to the forum!
Rather than making people memorize a long annoying public key, they could see a picture, say “That’s the same guy”, or see one that didn’t match, and know it was an imposter.
It didn’t need to be perfect, just a gut-check “Hey, I don’t think this is the same guy.”
They could always still compare the public keys, or go on to more advanced techniques, after they suspected a problem..

The problem is, if I randomly assign people’s Public key to a face, say, out of some sort of hypothetical face-picture-database, people will associate the poster with the model for the picture.
An African dissenter might feel awkward, or be disinclined to use the service, if his avatar rendered to a girl pink hair and a ponytail.

Design

I needed something recognizable, but still generic. Something like Robots or Monsters.

I realized that if if the design were simple enough, I could assemble the pieces, like you would if you were making a cartoon.
Take mouth A, put onto body B.

I could re-use the various pieces, and combine them in a way that was quick to assemble, but still recognizable.

I talked with a number of cartoonists- A number of them had a hard time with the concept, thinking the creatures would all be separate, and I’d just have the forum choose from one of 27 pictures.
Others were totally in-line with what I was looking for, but much more than I could afford for a minor feature, on a free forum.

Eventually, I went to 99 designs.

After spelling everything out, and getting a number of high-quality submissions, I was able to narrow it down to three finalists- Zikri Kader, Hrvoje Novakovic, and Julian Peter Arias.
Each of them had Brilliant designs, full of life, with characterization and expression.
I worked with each of them to narrow things down, tweak the designs, and build expandable characters.
They were great.
At the end of the day, I had to choose one to win the contest.. I hated doing so, though, since I had been working with all three in email for days, asking them to tweak things to work for the site.
It’s something I hate about 99 designs in general. As well as it works for me as a client, it’s rough on the artists.

I wanted to do better. I talked to the artists outside of the site, and was able to negotiate a deal where I’d pay them a secondary prize, in exchange for them giving me a license to use the images.

This gave me a Wealth of Robots.. I certainly didn’t need that many, but now I had to do SOMETHING with them.

Assembly

I spent a day lining up the components- The nose should always be at these XY coords, and the mouth should always be at these..
I exported each component out into a separate PNG file lined up for assembly.

It was a pain, and rather manual, but I only had to do it once. Per set.

After I had each component exported out, I opened up Tornado.. Tornado was the programming package I had used for my Lonava project, so I knew the basics of making it dance.

I decided to make a quick web service, that would take in a string, such as the public key I needed, and spit out a robot.

I’d make it so it gave you the image directly through the HTTP request, that was I didn’t have to do any fancy coding at the forum level, just include a regular <img> tag.

The next step was to figure out what Roboparts to use.
I wanted each Robotic part to be seemingly random, but to give the SAME seemingly random value each time.

That meant, if I made a robot for the word “ANDROID”, it would always make the SAME robot for that word, no matter how many times I ran it.. That was essential, to ensure that I could use them to identify the Public Keys.

So to do this, I turn the word into numbers, and then use that number to pick the robot pieces.
Imagine that we assigned a number to each letter, where A=1, B=2, C=3, etc..
The first letter from the word “ANDROID” (an A) would turn into a 1, the second (N) would be 14, the third (D) would turn into a 4, etc.

Then, we use this number to choose which Robot Part to use.
Each letter basically chooses one part.

So If I have 6 Robot eyes to choose from, and our first number is 1 (because it starts with an A), we choose the first set of eyes in the set.

For the body, we might use the second letter.
The mouth would be the third, and so on.

Now, for the code on the site, I don’t use the letters directly like that, because then everything that started with the same letters would look the same.

Instead, I pull bits out of it’s SHA-512 hash. This ensures that similar words don’t necessarily look anything like each other.

I had python squish all the images together, and spit out a single-unified image, with all the components.

Squishing it together, essentially emulating the “merge layers” command in photoshop, was easy, since I had spent the time to make each picture the single-component, rendered against transparent.

Let’s walk through the assembly-
We start with a color, which is determined the same way we determined what eyepiece to use above..

In this case, the color Blue.
This will determine my Robot’s main color.

The next step is to choose a background, if one is requested.

Our seemingly-random value for the background is 5, so we get Background 5.

Then, I choose a Body. This time, I have Body 10.
The body has to go before all the other body parts, since other things go on top of it, covering up the top of the neck, for instance.

The next up is the face. Again, this has to go earlier than the other pieces, since the eyes, mouth and accessory piece go on top of it.

Next up, a random pair of eyes.

Now, a mouth

Finally, an accessory piece, to accent the robot. In this case, it happens to be a nose. But it could be an antenna, a hat, or anything else.

If I then paste these together, I get a fully assembled Robot:

RoboHash.org

It worked!! I could generate any number of “random” robots, based on whatever text I wanted!

I added some more tweaks, such as being able to pass in a specific size, or request different sets of Robots. I was thrilled. Finally!

At the same time, it seemed like a waste to keep my new little hack to myself.
It had been fun to write, and more, the Robots were cool. I wanted to share them!

I put together a quick page, demonstrating some of the features of the generator.
After posting it online and talking to some people, I realized that most people weren’t going to type in a URL, as it advised. I quickly added a JS loader, which would swap out the main image.
It worked.. Mainly. I’m not a good coder, but it worked well enough for a hurry.

The whole idea was silly. Robots? Really?
I decided to embrace that, and just run with it. The site was just to share my hack, so I didn’t care if it looked ‘unprofessional’.

I threw in a bunch of bits of silliness, such as random quotes from famous robot celebrities..
Each demo robot would have a Random quote (Many asking for help!) if you hovered over them.
The source to the page would have hidden Robots in it.

I had so much fun making it, I asked one of the artists make a robotic “scroll” arrow, encouraging people to keep going down to see the rest of the silliness.

Aftermath

Unordered

After the unveiling, I found I had a problem.
When I wrote the script to pull in all the components and assemble it, I didn’t think about the order it was doing it in.
It was always the same, so I assumed it was alphabetical. It wasn’t.

It was importing the images in the order of their inodes in their directory. Crap!

This worked fine for the one server it ran on, but when people started running the code on their own machines, or when I set up a second machine to handle load, it was generating different Robots!

I had included a unique number at the end of each file, which I had believed would be enough.. But since I was in inode order, rather than filename order, this was just random cruft..

I couldn’t remove it, though, since the hashes were partially based on filename!
I fixed the code so that it would properly sort alphabetically, but this would have broken all the existing hashes. No good at all. A hash is supposed to always stay the same!
Eventually I found that in unix, I could do a “ls -lU” to list them in the native-filesystem order.. I renamed them so that the filesystem order matched the alphabetical order. It was ugly, but it worked.

Gravatar

A few weeks after release, James Devlin wrote about RoboHash, and devised a method to allow a site to use RoboHashes for generic users, but override it with a Gravatar if someone had set one.

It was really clever, but I thought I might be able to do a bit better, if I baked it in.

I ripped open the code, and added support for accepting a RoboHash request, with a Gravatar hash in it, such as

http://robohash.org/620050a4db5104bae758cd75171d64ca?gravatar=hashed

This would make the request to Gravatar– If it found something for that hash, it would use it. If not, it’d Hash it, and display it.

By doing directly on RoboHash.org, I could parse and include all the standard command, such as setting the size. I could also cache it, and serve it out of the same CDN as everything else

Conclusion

I’ve been really happy with the way the project turned out.
A number of different blogs are using it for all their commenters, someone made an iPhone app, and I’ve seen them show up in various online applications.

It’s great to see people using them, and the site is cheap enough to run that it’s not a burden to do so.
Anyone is welcome to use the robots, all I ask is they link back to Robohash.org someplace in their About screen.

Robohash is a cool little project, and I’m glad to have made it, and I’m excited to see people using it

Now, back to coding the Real side-side projects

My Experiences with MongoDB in production over the last year

First off, a disclaimer. I’m writing this article in my off hours, on my personal machine.
I don’t speak for my employer, although my use on the job as well as several projects at home do flavor my impressions.

I’ve been using MongoDB in production for various projects for the last year and a half, and found that there are some things it does incredibly well, and other areas where it tends to fall flat on it’s face.

I’ve seen a lot of people writing up their Mongo Experiences lately, and I’d had this document in the Drafts window for a while, so I thought I’d share my own thoughts with the DB.

Mongo is Fast

First of, Mongo is Fast. Like, seriously, screamingly, How-is-that-possible fast.
When you have a workflow that maps well to what Mongo prefers, Mongo is substantially faster than Couch, Riak, or Postgres.

The key, though, is ensuring your workflow maps to what MongoDB is good at.

Lots of people have written about Mongo’s Write Lock, but I think one point has become somewhat lost in the noise– When MongoDB is writing, the entire DB is locked. No writes can be performed, and no reads can be performed either.
This is normally fine, since writes are very, very fast. In fact, they’ve gotten much faster in Mongo 2.0, since the DB now yields the lock when it sends the command to write to disk, rather than waiting for the disk to return.

The fact remains, however, that when you have a lot of writes, your DB will halt, and reads will block.
Again, to be very, very clear- when MongoDB is writing something, ALL reads to ALL collections on ALL slaves will be blocked.

In the future, 10gen hopes to make this collection-level, which is a start, but other Databases such as Couch have versioned objects, which mean than even individual Objects aren’t locked!

Replica Sets are straightforward

One way to try to solve this is to use Read Slaves.
Replication in Mongo is dead-simple to set up; A competent admin can have a working 3-node replica set in an hour or two.

The intended deployment for MongoDB is that in production, you will always run multiple servers, and thus, always have multiple copies of your data.
This is why until 1.8, any single server might become corrupt if shutdown uncleanly.
This wasn’t a problem– If you’re only running on one server, it was argued, you were doing it wrong!

Essentially Mongo was designed to have Cluster-level safety, rather than server-level safety.

If a machine goes down, no worries, one of the other cluster members will take over for it.

Having multiple read slaves, and using SlaveOK queries means that you can redirect your queries onto these read-only slave replicas, even while the master is blocked by a write.

This generally works well enough, but there are some caveats you should keep in mind:

1) Not all drivers route SlaveOK automatically.
Because Mongo puts a certain degree of logic into the client drivers, you need to be careful to ensure that the driver for your language-of-choice supports the mongo features you want to use.
In some cases you might find that the while core mongo server supports a feature, such as SlaveOK queries, your particular language-driver might not. YMMV.

2) If replication backs up, SlaveOK queries might have bad/old data.
This ought to be obvious, but it’s important to keep in mind. If you store Session data in mongo, make sure you don’t use SlaveOK queries to retrieve it, or logins will mysteriously fail when replication falls behind.

3) Replicas get blocked on writes also!
This was not immediately obvious to me, but it’s important to note.
Writes to the mongoDB block reads. This is due to the aforementioned Global Lock
Less intuitive, is that it reads to the Slaves are also blocked during writes.
This is because during each write, mongo will replicate the write to each slave.
While this write is being written to the slave, as part of normal replication, the slave will be blocked.

There aren’t any good work around here. 10Gen is working to make the Global Lock suck less, but for now, it’s pain on a stick.

Sharding is expensive

If you ask 10Gen about these problems, they’ll patiently explain that they don’t intend for ReplicaSets to be a speed boost, they’re intended to provide data safety.

At the end of day, if you want to get past the blocking writes, you need to shard your data. Note, however, that this is not a panacea- If you have 12 Shards, and you send in a blocking write, 1/12th of your DB will still be blocked!

So here’s my rant about Mongo Sharding. I’m sure if 10Gen reads this, they will say it functions as designed, and I recognize that the way they did it is a valid tradeoff, I just don’t personally agree. Take that for what it is.

Sharding properly requires too many servers.

If you have a database that is getting to be more write-heavy than Mongo can easily keep up with, and you want to divide it up into Shards, you need a lot of machines. Probably 3X as many machines as you think you need.

The reason for this is that each shard needs it’s own replicaset; They don’t share, and you can’t re-use.

That means if you want to divide your DB into 10 partitions, you will need at least 30 servers.

• Replicaset1
• mongo1
• mongo2
• mongo2
• Replicaset2
• mongo1
• mongo2
• mongo2
• Replicaset3
• mongo1
• mongo2
• mongo2

…

• Replicaset10
• mongo1
• mongo2
• mongo2

You’re also supposed to have 3 config servers for each replica set, although these are lightweight enough that you can probably share them.

None if this is unbearable, but it’s something to keep in mind in your design configurations.
If you’re writing a lot, even VERY SMALL writes, you’re going to need a lot of servers.

To be fair, it’s also possible to try to offset this by running multiple Mongo Instances on the same HW.
Beyond adding mental and config overhead (which are bearable), since you’re generally sharding in situations when you’re write limited, this won’t help as much as you might hope.
Essentially, you’re sharing I/O, including replication to all the different servers, across fewer spindles.

What I’d really like to see is a more RAID-6 style deployment, wherein each node added to a mongoDB cluster held 1/nth parity data, as well as it’s own data.
This allows you to distribute requests to each node in the deployment, and still survive if a server fails.

This is more conceptually similar to how Riak handles adding nodes. Each node you add to a Riak server adds both redundancy AND speed.

Development is easy

Finally, Developing against MongoDB is about as straightforward as I could possibly ask for.
Getting it running on my Macbook was trivial, and development is fast, with little learning curve.

In Mongo, I can insert an arbitrary JSON document, such as inserting all email received from the outside world. After this, I can query on any element of the JSON.

This makes it VERY simple to pull all the emails sent from Thunderbird, or any message received in a date range..

I could do these same things with SQL, but I’d generally need to fit everything into a defined schema; For things like Email, or HTML pages, which might have any number of arbitrary headers, a schema-less design is a godsend.

Another thing I really enjoyed is that I can query in familiar styles. For example, to retrieve all messages from a certain sender, I could use a built in JSON query format, which if you squint hard enough, isn’t THAT different from SQL
Contrast this with my impression that Couch/Riak, I believe I need to use MapReduce for everything ((This was my impression in going through basic tutorials and development of mini toy apps; I accept I could be way off))

I like MapReduce for a certain class of problem, but sometimes it’s nice to be able to reach for a simple query, and have it just work. According to 10Gen, these queries are by and away the most popular form of requesting data from MongoDB, and they are going to be adding new forms of queries (such as Group by) in upcoming versions.

Support is available

I have to say that 10Gen has always been very helpful in providing support.
In additional, their new MMS tool is very helpful in seeing where things are having trouble.

Whether through paid (Tickets) or free (IRC/Mailing List) means, they’ve almost always come through with solid answers to almost any problem that comes up.
I have a lot of respect for the company for bringing in senior level people on a regular basis to help answer problems, rather than throwing us to people working off a script.

The only negative I have to say on the support is that too often, the answer is “It’s fixed in the next version”.

This seems to be the default answer to any problem that comes up..

• “MongoS instances are not routing to all mongoD servers”

Fixed in the next version

• “Our MongoD server failed with this error overnight “

Fixed in the next version

• “Our replica set isn’t sure who the master should be”

Fixed in the next version

Don’t get me wrong, it’s great that things are being fixed!
Just know going in that if you’re using MongoDB as your primary DB, you’ll be on a constant upgrade treadmill.

10Gen is still deciding how much to backport, and when, but generally it seems like problems with known workarounds only get fixed in Trunk, and problems without usable workarounds will get a backport to one version back. Don’t expect to use the same version for a year; If you want to avoid major bugs, it’s just not going to happen.

Summary

In Summary, I like MongoDB quite a bit, but like any software, it’s important to know it’s warts.
In a lot of ways, Mongo reminds me of MySQL on MyISAM a decade ago-
It’s Fast, has easy replication, and it’s used by everyone, so you can get help when you need it.
But if you’re not careful, it will blow up in unexpected ways.

If you understand the tradeoffs going in, and you have a relatively DB with relatively infrequent writes, MongoDB can be a godsend. It’s dramatically faster than the alternatives, and we’ve never had problem resulting in dataloss. But like any package you do need to design with it’s limitations in mind.

My afternoon, as told in Adventure Game format.

LOOK

You are currently locked outside of your house. You have left your keys and wallet back in a drawer safely beside your bed. You keep a spare key inside the office, but it is far away.

Nearby, there is a bike, and your roomate’s car.

LOOK SELF

You are an out of shape, 30 year old IT Manager from an urban neighborhood in New England.

INVENTORY

You have:
Your Phone
A Macbook
No Coffee

LOOK CAR

You examine your roomate’s car.
The car is a 2002 Saturn S series, which has seen better days.
You find yourself whistfully wishing your Scion was here, as you tear through your roomate’s things with little regard for personal property.

There appears to be a collection of things in the back seat.

Your keys are still safely in the drawer by your bed. You are unable to drive the car.

SEARCH CENTER COMPARTMENT

You find several CDs, some quarters, and a tire gague.

TAKE TIRE GAGUE

Taken.

TAKE MONEY

Taken. You now have $2.25