Twitter Open Sources Snowflake, An ID Number Generator

Snowflake IconTired of generating new ideas within a distributed database system for billions of messages? We’re getting tired too! Thankfully, Twittter has open sourced a new product this evening called “Snowflake”, which enables developers to generate new ID numbers, by combining timestamp, worker number, and sequence number, something produced through Apache Zookeeper.

If all this sounds relatively complex, that’s because it is. Twitter is dealing with billions of messages that need to be indexed in a short amount of time. As such, they needed an easier solution beyond integers, the method they previously used. If your startup isn’t dealing with such scalability issues, don’t be jealous! However once you do get to such levels, Facebook and Twitter have both open sourced enough products that you should now have plenty of help.

Given that Twitter is producing so many messages at such a quick rate, they’ve decided to use Cassandra (a project first open-sourced by Facebook) as well as sharded MySQL databases (using gizzard, an automated sharding system first open-sourced by Twitter) to scale quickly. In order to have unique identifiers across each of these messages, the company needed a solution which worked within a distributed system.

The result was Snowflake. If you want the full details you can find them here, and download the code from GitHub here.

If you want to learn more about building the infrastructure for large scale web applications, come to our upcoming Social Developer Summit later this month in San Francisco.

-Social Developer Summit Banner-

Snowflake image found via Discovery Channel.

Related Stories
Mediabistro Course

Podcasting

PodcastingLearn to develop, create, and launch your own podcast! Starting October 23, Steve Belaner, the host of the weekly podcast The Gamut, will teach you how to determine the goals of your podcast, perfect your concept, contact and book guests, market your podcast,  and get your show up and running in just a few weeks. Register now!