Dataconomy
  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
Subscribe
No Result
View All Result
Dataconomy
  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
Subscribe
No Result
View All Result
Dataconomy
No Result
View All Result

Data Science Needs to Fail More, Faster.

by K Young
September 18, 2014
in Data Science
Home Topics Data Science
Share on FacebookShare on TwitterShare on LinkedInShare on WhatsAppShare on e-mail

Darwin never actually said the following quote, but it’s truthy so I’ll use it:

It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change. —Darwin-ish

I was watching a talk by Josh Wills the other day. He was applying Lean engineering concepts to data science. To illustrate how important rapid learning is, he told a story about the team that built the Gossamer Condor and won the Kremer Prize for human-powered flight. They won because they failed more repeatedly than their competition. I’m sure their competition was brilliant, and they definitely had several years head start, but they lost because it took them maybe a year to iterate from design, to build, to test flight, to crash and destroy, and go back to design again. The Gossamer Condor team’s breakthrough insight was this: if the Condor could be repaired and improved in days, then they’d test 100 designs in the time their competition tested 1.

50 years on, and data science is following this same anti-pattern of the teams that didn’t get the Kremer Prize: come up with brilliant ideas, painstakingly move them out to the real world, watch them fail, and then slowly start the brittle process anew. This would be a problem for any profession—the faster you can iterate, the faster you can learn, and the more problems will be solved—but it is an especially pressing problem in data science because there is a huge shortage of data scientists, so inefficiencies mean that many critical problems are not getting solved.

What can we do about it? Luckily, software engineering, which is a sister to data science, has been working through these problems for the last two decades and has some pretty good patterns to build from. Devops is the area of software engineering concerned with moving software from development to real-world use quickly and safely. Devops lets software engineers try more things, and therefore learn faster. Here are the pieces I believe are necessary for data science:

  1. Automated tests: these don’t have to be exhaustive, but there should be an automatic way to know that changes don’t horribly break your user-facing system
  2. 1-Button Deploy: If releasing changes takes more than one step, it will break more frequently, and more importantly in the context of this article, releases will happen less frequently.
  3. 1-Button Rollback: The counterpart to 1-Button Deploy, if an error is discovered in a user-facing system, reverting to a pre-error state must be swift and reliable.
  4. Instrumented Infrastructure: Data science problems often require distributed architectures, non-obvious dependencies, and complex feedback loops. To successfully try many things quickly, it is necessary to spend a minimal amount of time understanding the infrastructure, tuning it, and correcting errors.

It’ll take some work, but I believe Devops is the next crucial frontier for Data Science—a massively underrated piece of this rapidly changing discipline.


Join the Partisia Blockchain Hackathon, design the future, gain new skills, and win!


Yes, I cofounded a company, Mortar, that amongst other things addresses these problems… because I think they are so important to solve.

Follow @DataconomyMedia


K YoungK Young has been CEO of Mortar Data since 2010. Mortar helps data scientists and data engineers spend 100% of their time on problems that are specific to their business—and not on time-wasters like babysitting infrastructure, managing complex deploys, and rebuilding common algorithms from scratch. Mortar’s platform runs pipelines of open technologies including Hadoop, Pig, Java, Python, and Luigi to provide out-of-the-box solutions that can be fully customized. Prior to founding Mortar Data, K built software that reaches one in ten public school students in the U.S. He holds a Computer Science degree from Rice University.

Tags: data scienceDevopsGossamer CondorK YoungKremer PrizeMortar DataWeekly Newsletter

Related Posts

What is ChatGPT Plus, and how to get it? Learn its features, price, and how to join ChatGPT Plus waitlist. Is it worth it? Keep reading and find out

ChatGPT Plus: How does the paid version work?

February 2, 2023
AI Text Classifier: OpenAI's ChatGPT detector can distinguishes AI-generated text

AI Text Classifier: OpenAI’s ChatGPT detector indicates AI-generated text

February 2, 2023
BuzzFeed ChatGPT integration: Buzzfeed stock surges in enthusiasm over OpenAI

BuzzFeed ChatGPT integration: Buzzfeed stock surges after the OpenAI deal

February 2, 2023
Adversarial machine learning 101: A new frontier in cybersecurity

Adversarial machine learning 101: A new cybersecurity frontier

January 31, 2023
What is the Nvidia Eye Contact AI feature? Learn how to get and use the new Nvidia Broadcast feature. Zoom meetings and streams get easier.

Nvidia Eye Contact AI can be the savior of your online meetings

February 2, 2023
How did ChatGPT passed an MBA exam

How did ChatGPT passed an MBA exam?

February 2, 2023

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

LATEST ARTICLES

Cyberpsychology: The psychological underpinnings of cybersecurity risks

ChatGPT Plus: How does the paid version work?

AI Text Classifier: OpenAI’s ChatGPT detector indicates AI-generated text

A journey worth taking: Shifting from BPM to DPA

BuzzFeed ChatGPT integration: Buzzfeed stock surges after the OpenAI deal

Adversarial machine learning 101: A new cybersecurity frontier

Dataconomy

COPYRIGHT © DATACONOMY MEDIA GMBH, ALL RIGHTS RESERVED.

  • About
  • Imprint
  • Contact
  • Legal & Privacy
  • Partnership
  • Writers wanted

Follow Us

  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
No Result
View All Result
Subscribe

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy Policy.