Twitter Open Sources Snowflake, An ID Number Generator

Snowflake IconTired of generating new ideas within a distributed database system for billions of messages? We’re getting tired too! Thankfully, Twittter has open sourced a new product this evening called “Snowflake”, which enables developers to generate new ID numbers, by combining timestamp, worker number, and sequence number, something produced through Apache Zookeeper.

If all this sounds relatively complex, that’s because it is. Twitter is dealing with billions of messages that need to be indexed in a short amount of time. As such, they needed an easier solution beyond integers, the method they previously used. If your startup isn’t dealing with such scalability issues, don’t be jealous! However once you do get to such levels, Facebook and Twitter have both open sourced enough products that you should now have plenty of help.

Given that Twitter is producing so many messages at such a quick rate, they’ve decided to use Cassandra (a project first open-sourced by Facebook) as well as sharded MySQL databases (using gizzard, an automated sharding system first open-sourced by Twitter) to scale quickly. In order to have unique identifiers across each of these messages, the company needed a solution which worked within a distributed system.

The result was Snowflake. If you want the full details you can find them here, and download the code from GitHub here.

If you want to learn more about building the infrastructure for large scale web applications, come to our upcoming Social Developer Summit later this month in San Francisco.

-Social Developer Summit Banner-

Snowflake image found via Discovery Channel.

Related Stories
Mediabistro Course

Pinterest Marketing

Pinterest MarketingPin your way to a wider audience! Starting December 1, work with the VP of Marketing at SheKnows to learn how to leverage Pinterest for your brand. In this course, you'll learn how to create a Pinterest Playbook to develop and measure your marketing strategy, grow a follower base, run an effective contest, and stay updated on the best practices for this platform. Register now!