Dataconomy
  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
Subscribe
No Result
View All Result
  • AI
  • Tech
  • Cybersecurity
  • Finance
  • DeFi & Blockchain
  • Startups
  • Gaming
Dataconomy
  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
Subscribe
No Result
View All Result
Dataconomy
No Result
View All Result

Meet Apache Samza – LinkedIn’s Stream Processing Framework

byEileen McNulty
January 9, 2015
in News
Home News

With the advent of Big Data and the rapidly growing scale of web-applications, monolithic relational databases were replaced by scalable, partitioned, NoSQL databases and HDFS; individual queries to relational databases were replaced by the likes of Hive and Pig.

This growing scale and partitioned consumption model brought about by these systems, also put forth the need for smooth processing of “streams of events” at scale.

That’s when LinkedIn came up with Samza.

Apache Software Foundation’s incubation project since September 2013, Apache Samza is the distributed stream processing framework that incorporates Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.

Navina Ramesh of LinkedIn explains, “we recognized that some of our use cases couldn’t be implemented in Hadoop due to the large turn-around time that batch processing needed.”

“Unlike Hadoop, which is optimized for throughput, Kafka is optimized for low-latency messaging.  We built a processing system on top of Kafka, allowing us to react to the messages — to join, filter, and count the messages. The new processing system, Apache Samza, solved our batch processing latency problem and has allowed us to process data in near real-time,” she explains in The New Stack article.

It essentially takes care of stream processing needs within LinkedIn, “to provide a lightweight framework for continuous data processing.” Originally open sourced, the framework enables building applications to process feeds of messages.

Read more here.

Follow @DataconomyMedia

(Image credit: Jean and Fred, via Flickr)

 

Stay Ahead of the Curve!

Don't miss out on the latest insights, trends, and analysis in the world of data, technology, and startups. Subscribe to our newsletter and get exclusive content delivered straight to your inbox.

Tags: Apache KafkaApache SamzahdfsHivelinkedinPigStream ProcessingWeekly Newsletter

Related Posts

MediaTek unveils Dimensity 9500 AI chip

MediaTek unveils Dimensity 9500 AI chip

September 23, 2025
Selected AI fraud prevention solutions – September 2025

Selected AI fraud prevention solutions – September 2025

September 22, 2025
A practical guide to connecting Microsoft Dynamics 365 CRM data using ODBC for advanced reporting and BI

A practical guide to connecting Microsoft Dynamics 365 CRM data using ODBC for advanced reporting and BI

September 22, 2025
Coral v1 released with Model Context Protocol runtime

Coral v1 released with Model Context Protocol runtime

September 22, 2025
MIT’s PDDL-INSTRUCT improves Llama-3-8B plan validity

MIT’s PDDL-INSTRUCT improves Llama-3-8B plan validity

September 22, 2025
xAI releases Grok 4 Fast model for all users

xAI releases Grok 4 Fast model for all users

September 22, 2025
Please login to join discussion

LATEST NEWS

MediaTek unveils Dimensity 9500 AI chip

Selected AI fraud prevention solutions – September 2025

A practical guide to connecting Microsoft Dynamics 365 CRM data using ODBC for advanced reporting and BI

Coral v1 released with Model Context Protocol runtime

MIT’s PDDL-INSTRUCT improves Llama-3-8B plan validity

xAI releases Grok 4 Fast model for all users

Dataconomy

COPYRIGHT © DATACONOMY MEDIA GMBH, ALL RIGHTS RESERVED.

  • About
  • Imprint
  • Contact
  • Legal & Privacy

Follow Us

  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
No Result
View All Result
Subscribe

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy Policy.