Dataconomy
  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
Subscribe
No Result
View All Result
Dataconomy
  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
Subscribe
No Result
View All Result
Dataconomy
No Result
View All Result

Meet Apache Samza – LinkedIn’s Stream Processing Framework

by Eileen McNulty
January 9, 2015
in News
Home News
Share on FacebookShare on TwitterShare on LinkedInShare on WhatsAppShare on e-mail

With the advent of Big Data and the rapidly growing scale of web-applications, monolithic relational databases were replaced by scalable, partitioned, NoSQL databases and HDFS; individual queries to relational databases were replaced by the likes of Hive and Pig.

This growing scale and partitioned consumption model brought about by these systems, also put forth the need for smooth processing of “streams of events” at scale.

That’s when LinkedIn came up with Samza.

Apache Software Foundation’s incubation project since September 2013, Apache Samza is the distributed stream processing framework that incorporates Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.

Navina Ramesh of LinkedIn explains, “we recognized that some of our use cases couldn’t be implemented in Hadoop due to the large turn-around time that batch processing needed.”


Join the Partisia Blockchain Hackathon, design the future, gain new skills, and win!


“Unlike Hadoop, which is optimized for throughput, Kafka is optimized for low-latency messaging.  We built a processing system on top of Kafka, allowing us to react to the messages — to join, filter, and count the messages. The new processing system, Apache Samza, solved our batch processing latency problem and has allowed us to process data in near real-time,” she explains in The New Stack article.

It essentially takes care of stream processing needs within LinkedIn, “to provide a lightweight framework for continuous data processing.” Originally open sourced, the framework enables building applications to process feeds of messages.

Read more here.

Follow @DataconomyMedia

(Image credit: Jean and Fred, via Flickr)

 

Tags: Apache KafkaApache SamzahdfsHivelinkedinPigStream ProcessingWeekly Newsletter

Related Posts

How did ChatGPT passed an MBA exam

How did ChatGPT passed an MBA exam?

January 27, 2023
Google code red: ChatGPT and You.com like AI-powered tools threatening the search engine. Moreover, latest Apple Search rumors increased the danger.

Google code red: ChatGPT, You.com and rumors of Apple Search challenge the dominance of search giant

January 26, 2023
T-Mobile data breach 2023 explained: Learn how did the leak happen and explore T-Mobile data breach history. It is not the first time of the company

T-Mobile data breach 2023: The telecom giant got hacked eight times in the last six years

January 20, 2023
Microsoft layoffs 2023: Amazon job cuts that affect 11,000 employees explained. Big tech layoffs continue... Learn why and what will happen next.

Microsoft layoffs will affect more than 11,000 employees

January 18, 2023
Medibank Data Breach Class Action: Compensation can reach up to $20,000 per person

Medibank Data Breach Class Action: Compensation can reach up to $20,000 per person

January 16, 2023
What is DoNotPay AI Lawyer? The world's first robot lawyer ready to give $1 million to represent you. How does it work? Keep reading.

DoNotPay AI lawyer is ready to give $1 million for any case in US

January 12, 2023

Comments 3

  1. Mani Megala says:
    6 years ago

    Great article.. After reading this article i learnt new and useful information about apache samza from this article which helpful to develop my hadoop skills more.. thank you for sharing

    hadoop training | big data training

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

LATEST ARTICLES

AI Text Classifier: OpenAI’s ChatGPT detector indicates AI-generated text

A journey worth taking: Shifting from BPM to DPA

BuzzFeed ChatGPT integration: Buzzfeed stock surges after the OpenAI deal

Adversarial machine learning 101: A new cybersecurity frontier

Fostering a culture of innovation through digital maturity

Nvidia Eye Contact AI can be the savior of your online meetings

Dataconomy

COPYRIGHT © DATACONOMY MEDIA GMBH, ALL RIGHTS RESERVED.

  • About
  • Imprint
  • Contact
  • Legal & Privacy
  • Partnership
  • Writers wanted

Follow Us

  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
No Result
View All Result
Subscribe

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy Policy.