Dataconomy
  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
Subscribe
No Result
View All Result
  • AI
  • Tech
  • Cybersecurity
  • Finance
  • DeFi & Blockchain
  • Startups
  • Gaming
Dataconomy
  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
Subscribe
No Result
View All Result
Dataconomy
No Result
View All Result

Data Science Needs to Fail More, Faster.

byK Young
September 16, 2014
in Articles
Home Resources Articles

Darwin never actually said the following quote, but it’s truthy so I’ll use it:

It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change. —Darwin-ish

I was watching a talk by Josh Wills the other day. He was applying Lean engineering concepts to data science. To illustrate how important rapid learning is, he told a story about the team that built the Gossamer Condor and won the Kremer Prize for human-powered flight. They won because they failed more repeatedly than their competition. I’m sure their competition was brilliant, and they definitely had several years head start, but they lost because it took them maybe a year to iterate from design, to build, to test flight, to crash and destroy, and go back to design again. The Gossamer Condor team’s breakthrough insight was this: if the Condor could be repaired and improved in days, then they’d test 100 designs in the time their competition tested 1.

50 years on, and data science is following this same anti-pattern of the teams that didn’t get the Kremer Prize: come up with brilliant ideas, painstakingly move them out to the real world, watch them fail, and then slowly start the brittle process anew. This would be a problem for any profession—the faster you can iterate, the faster you can learn, and the more problems will be solved—but it is an especially pressing problem in data science because there is a huge shortage of data scientists, so inefficiencies mean that many critical problems are not getting solved.

What can we do about it? Luckily, software engineering, which is a sister to data science, has been working through these problems for the last two decades and has some pretty good patterns to build from. Devops is the area of software engineering concerned with moving software from development to real-world use quickly and safely. Devops lets software engineers try more things, and therefore learn faster. Here are the pieces I believe are necessary for data science:

  1. Automated tests: these don’t have to be exhaustive, but there should be an automatic way to know that changes don’t horribly break your user-facing system
  2. 1-Button Deploy: If releasing changes takes more than one step, it will break more frequently, and more importantly in the context of this article, releases will happen less frequently.
  3. 1-Button Rollback: The counterpart to 1-Button Deploy, if an error is discovered in a user-facing system, reverting to a pre-error state must be swift and reliable.
  4. Instrumented Infrastructure: Data science problems often require distributed architectures, non-obvious dependencies, and complex feedback loops. To successfully try many things quickly, it is necessary to spend a minimal amount of time understanding the infrastructure, tuning it, and correcting errors.

It’ll take some work, but I believe Devops is the next crucial frontier for Data Science—a massively underrated piece of this rapidly changing discipline.

Yes, I cofounded a company, Mortar, that amongst other things addresses these problems… because I think they are so important to solve.

Follow @DataconomyMedia


K YoungK Young has been CEO of Mortar Data since 2010. Mortar helps data scientists and data engineers spend 100% of their time on problems that are specific to their business—and not on time-wasters like babysitting infrastructure, managing complex deploys, and rebuilding common algorithms from scratch. Mortar’s platform runs pipelines of open technologies including Hadoop, Pig, Java, Python, and Luigi to provide out-of-the-box solutions that can be fully customized. Prior to founding Mortar Data, K built software that reaches one in ten public school students in the U.S. He holds a Computer Science degree from Rice University.

Stay Ahead of the Curve!

Don't miss out on the latest insights, trends, and analysis in the world of data, technology, and startups. Subscribe to our newsletter and get exclusive content delivered straight to your inbox.

Tags: data scienceDevopsK YoungMortar DatasurveillanceWeekly Newsletter

Related Posts

When Regulation Embraces Innovation: Xenco Medical Founder and CEO Jason Haider Discusses the Upcoming 2026 CMS Transforming Episode Accountability Model

When Regulation Embraces Innovation: Xenco Medical Founder and CEO Jason Haider Discusses the Upcoming 2026 CMS Transforming Episode Accountability Model

August 26, 2025
DeFAI and the Future of AI Agents

DeFAI and the Future of AI Agents

July 26, 2025
Unifying the fragmented AI ecosystem: A new paradigm for generative AI workflows

Unifying the fragmented AI ecosystem: A new paradigm for generative AI workflows

July 21, 2025

How to plan for technical debt before it buries you

July 21, 2025
Optimizing performance for a global user base

Optimizing performance for a global user base

July 17, 2025
How the right FPS mouse can make or break your game (or workflow)

How the right FPS mouse can make or break your game (or workflow)

July 14, 2025
Please login to join discussion

LATEST NEWS

UK Home Office seeks full Apple iCloud data access

iPhone 17 may drop physical SIM in EU

Zscaler: Salesloft Drift breach exposed customer data

AI boosts developer productivity, human oversight still needed

Windows 11 25H2 enters testing with no new features

ChatGPT logo fixes drive demand for graphic designers

Dataconomy

COPYRIGHT © DATACONOMY MEDIA GMBH, ALL RIGHTS RESERVED.

  • About
  • Imprint
  • Contact
  • Legal & Privacy

Follow Us

  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
No Result
View All Result
Subscribe

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy Policy.