Iridesco Watercooler
We are Iridesco, and this is our blog. We are the guys behind Harvest and Co-op. Iridesco was born in 2003, and we now operate in downtown New York City. People who speak here: Danny Wen, Shawn Liu, Dee Zsombor, and Barry Hess.Below you will find the things that have caught our attention in design, business, art and culture.
Caching is not enough: spite of having Co-op to serve nearly all requests by gluing together memchached content after going live things were just slow compared to what you would expect from quad Xeon servers. Whenever a cache item became stale we have to go out via an API and refresh the missing pieces from the cluster running Harvest. Going trough logs it revealed that processing the API output was significantly more expensive than just generating the XML on Harvest side. 88+ seconds for just parsing an XML file that took a few milliseconds to produce … just not reasonable. It turns out that the XML parser used by the Hash.from_xml method or ActiveResource in general has a horrible performance. The larger the input worse ActiveResource behaves. Enter Libxml to the rescue, there are countless benchmarks claiming 10x speed improvement. If anything I think they are an understatement as Libxml mostly has a linear performance that cannot be stated about XmlSimple shipped by default with rails. The graph above shows how CPU usage looked before with libxml and after. As you see speed is just one factor. With libxml I can actually increase the ruby processes to handle much larger traffic, whereas a with XmlSimple can tie up an entire cpu core. Libxml is more scalable. If your XML processing needs limit to consuming REST resources via ActiveResource the fix is simple dead simple. Just install the fine FasterFromXml plugin by Brian Durand. |
