[lth@clover sup]$ diff /usr/lib/ruby/gems/1.9.1/gems/lockfile-1.4.3/lib/lockfile.rb{~,}
475c475
<       buf.each do |line|
---
>       buf.split($/).each do |line|

Crazy claim, eh? I figure there’s no better way to get this claim tested that by posting it as a truth!

We’ll here’s the story: Once upon a time, lloyd (that’s me) wanted a parser that didn’t suck that he could embed in a particular product, primarily because he was sick of patching other peoples code: solving problems like:

1. crashes on certain malformed inputs
2. the niggling tendency to scan from the begging of the input text every time a number was encountered
3. the requirement that you had to hold the full json text in memory at one time (usally in addition to an in-memory representation).

To be fair, the project that I was previously using got a lot better, but it still lacked a couple key features that were extremely important to me: stream parsing, and representation independence. At the time (and actually still), we were using JSON as the data representation over our IPC channel (in that same particular product), and this meant that a lot of data was flowing. We had one of two options:

1. throw away json and move to a binary format
2. make a json parser that caused the IPC serialization overhead to fall off our our profiles…

Cause’ I love transparency over the wire (binary still ain’t that scrutable), and because JSON is the representation language of the web (hush now XML people) I decided to try to build this parser.

Here’s the thing, I’m rather stupid and short sighted, and didn’t realize that it’s actually extremely well suited to high level language bindings for other languages, say, ruby or python. Luckily, Brian Lopez is not stupid, and he has brought yajl to ruby.

Initially Brian is reporting that it’s about 4 times as fast as YAML parsing via syck, about 20% faster than the other native JSON parser for ruby](http://flori.github.com/json/) out there, and in general uses up to ½ the memory (now this claim especially is difficult given the exponential nature of the heap mechanism in ruby, so perhaps we need to retry this with some of them gc patches I wrote, or with 1.9).

I’m posting to re-iterate how awesome Brian and I are? No, not really. I’m posting to challenge skeptics to validate these findings, just cause’ I was extremely pleased and surprised to hear these numbers from brian, and a little bit of that rigor that comes from public scrutiny couldn’t hurt…

till the next, lloyd