On August 26, 2009 in Palo Alto the SDForum SAM SIG hosted LinkedIn engineers Bhupesh Bansal and Jay Kreps to present “Project Voldemort: Scalable Fault Tolerant Distributed Storage at LinkedIn”.
LinkedIn takes web scale computing to extremes. They store and manage high read/write loads of massive data sets. Their applications need high scalability and performance but not necessarily features seen in relational databases. Project Voldemort is a distributed, highly scalable Key Value Storage System based on the Amazon Dynamo project using Hadoop and Pig. They talked about its architecture, design choices and future serving many data intensive applications. They want to build a community to solve the huge challenges they face. (I recommend using LinkedIn to find them.)
I think the use of the name Voldemort is sign of a new generation taking over the nomenclature from retiring nerds who used names from Star Trek and Star Wars. I just hope they are not hoping for a magic solution. :-)
At one point they discussed a process that takes several days to run. That needs fixing pronto. I can recommend a good database expert that might be able tune that to a few seconds.
For a more detailed analysis I suggest Richard Taylor’s post at:http://bandb.blogspot.com/2009/09/project-voldemort.html
Copyright 2009 DJ Cline All rights reserved.