Dataconomy
  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
Subscribe
No Result
View All Result
Dataconomy
  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
Subscribe
No Result
View All Result
Dataconomy
No Result
View All Result

Kubernetes Meets Big Data

by Irshad Raihan
November 8, 2018
in Big Data, Contributors, Understanding Big Data
Home Topics Data Science Big Data
Share on FacebookShare on TwitterShare on LinkedInShare on WhatsAppShare on e-mail

Organizations yearn order and simplicity over chaos and confusion, but the data-driven era we live in challenges these desires on a daily basis. Seemingly every day, massive amounts of transactional and streaming data is being introduced into enterprises. This data must be collected, deciphered, shared, and acted upon.

Cloud-native technologies offer unparalleled scale and the promise of greater agility, both of which are critical in today’s data-intensive era. In fact, I’d go as far as to argue that cloud-native technologies have brought us to a critical inflection point–and could have a long-lasting effect on the way we manage enterprise data.

Take Kubernetes, for example. The orchestration framework provides a single source for easy management of both application infrastructure and data, thereby introducing a much-needed element of simplicity into the big data universe. By enabling persistent storage services to be attached to and served by Linux containers, Kubernetes is helping drive data-intensive workloads like SQL/NoSQL databases and messaging toward containers.

Big Data makes its way into the enterprise data center

How did we get here? To understand the answer, it helps to go back to the early days of Hadoop.


Join the Partisia Blockchain Hackathon, design the future, gain new skills, and win!


Soon after its introduction, it was clear that Hadoop alone was no longer enough to effectively manage emerging data sources and real-time analytics needs, as it was primarily built as a batch processing technology. That resulted in the proliferation of analytics frameworks–such as Spark–designed to address Hadoop’s shortcomings.

This rapidly sprawling ecosystem addressed some big data needs, but it also helped to create some of the chaos as well. Many data analytics applications were often highly volatile and didn’t play by traditional application rules. As a result, they were kept separate from other enterprise applications in the data center.

Now, things are swinging back the other way. Open source cloud-native technologies like Kubernetes are providing a solid platform for managing both applications and data. Meanwhile, solutions are being developed that allow analytics workloads to be run on IT infrastructures, whether those infrastructures are virtualized or containerized.

Shared data context is the key

In the early days of Hadoop, data locality was the mantra. Data was distributed and brought close to compute. Today, storage is being decoupled from compute. We have gone from distributing data to distributing access to data. The inevitable convergence of data analytics workloads and Kubernetes based on-demand cluster provisioning is upon us.

A shared storage repository is key to managing multi-tenant workload isolation, enabling agility, and preventing data duplication. This allows analytics teams to set up customized clusters to suit their needs and meet SLAs without having to re-create or move large data sets.

In addition, developers and data managers can query across unstructured and structured data sources without expensive and cumbersome data movement. Development times are accelerated and products are brought to market faster. The efficiencies brought about through distributed access to a shared storage repository may also result in lower costs and increased utilization.

Unlocking data. Unlocking innovation.

By using a shared data context for multi-tenant workload isolation, data is essentially unlocked and easily accessible by anyone who needs it. Data engineers can dynamically provision clusters with the right resources, versions, and data. Data platform teams can achieve consistency between multiple analytics cluster silos, and IT infrastructure teams can have those clusters use their overall infrastructures that have traditionally been used for other workloads.

Data and applications are finally becoming one with each other again, creating a cohesive and standard means of managing both on the same infrastructure. Getting to this point has taken a few years, but we are finally living in an era where enterprises can now deploy a single infrastructure to manage big data and a host of other needs. Open source and cloud-native technologies have made this possible and will continue to lead the way.

 

Related Posts

What are data silos and how to get rid of them?

Data silos are the silent killers of business efficiency

December 23, 2022
TikTok data practices for data transfer of EU citizens to China and ads catering to kids are under investigation by the EU

EU probes TikTok’s data practices with multiple investigations

November 23, 2022
The role of intelligent data in driving supply chain efficacy

The role of intelligent data in driving supply chain efficacy

November 23, 2022
Big data and artificial intelligence: What's the future for them?

AI and big data are the driving forces behind Industry 4.0

November 7, 2022
What is the impact of artificial intelligence in insurance with examples? Explore AI in insurance use cases and find out insurance companies using artificial intelligence.

The insurance of insurers

September 22, 2022
Business analytics and business intelligence solutions in retail

Navigate through the rough seas of retail with business intelligence as your compass

September 20, 2022

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

LATEST ARTICLES

AI Text Classifier: OpenAI’s ChatGPT detector indicates AI-generated text

A journey worth taking: Shifting from BPM to DPA

BuzzFeed ChatGPT integration: Buzzfeed stock surges after the OpenAI deal

Adversarial machine learning 101: A new cybersecurity frontier

Fostering a culture of innovation through digital maturity

Nvidia Eye Contact AI can be the savior of your online meetings

Dataconomy

COPYRIGHT © DATACONOMY MEDIA GMBH, ALL RIGHTS RESERVED.

  • About
  • Imprint
  • Contact
  • Legal & Privacy
  • Partnership
  • Writers wanted

Follow Us

  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
No Result
View All Result
Subscribe

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy Policy.