Twitter has detailed its home-grown, real-time analytics system TSAR (Time Series AggregatoR) in a blog post. The system is focused on automating and aggregating data collection, as well as integrating the various components Twitter uses, such as Hadoop and Storm.

In their data collection and processing pipeline, Twitter are using a range of different solutions (Hadoop, MySQL, NoSQL), which all process the data in different ways, and use different languages to do it. TSAR takes the legwork out of getting these systems to talk to each other.

The blog post by Anirudh Todi, TSAR’s key design principles are:

• “Hybrid computation. Process every event twice — in real time, and then again (at a later time) in a batch job. The double processing is orchestrated using Summingbird. This hybrid model confers all the advantages of batch (stability, reproducibility) and streaming (recency) computation.

• Separation of event production from event aggregation. The first processing stage extracts events from source data; in this example, TSAR parses Tweet impression events out of log files deposited by web and mobile clients. The second processing stage buckets and aggregates events. While the “event production” stage differs from application to application, TSAR standardizes and manages the “aggregation” stage.

• Unified data schema. The data schema for a TSAR service is specified in a datastore-independent way. TSAR maps the schema onto diverse datastores and transforms the data as necessary when the schema evolves.

• Integrated service toolkit. TSAR integrates with other essential services that provide data processing, data warehousing, query capability, observability, and alerting, automatically configuring and orchestrating its components.”

TSAR was built on top of the Summingbird system, a high-level extraction library which paired the batch processing capabilities of Hadoop with the real-time powers of Storm. TSAR builds upon these, making it easier for the different technologies to communicate back and forth.

With over 500 million tweets created each day, it’s understandable that Twitter developed their own robust technology to orchestrate their multi-faceted system.

Read more here.
(Image credit: Blog post)

Follow @DataconomyMedia

Interested in more content like this? Sign up to our newsletter, and you wont miss a thing!