Dataconomy
  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
Subscribe
No Result
View All Result
Dataconomy
  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
Subscribe
No Result
View All Result
Dataconomy
No Result
View All Result

Data Science Needs to Fail More, Faster.

by K Young
September 18, 2014
in Data Science
Home Topics Data Science
Share on FacebookShare on TwitterShare on LinkedInShare on WhatsAppShare on e-mail

Darwin never actually said the following quote, but it’s truthy so I’ll use it:

It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change. —Darwin-ish

I was watching a talk by Josh Wills the other day. He was applying Lean engineering concepts to data science. To illustrate how important rapid learning is, he told a story about the team that built the Gossamer Condor and won the Kremer Prize for human-powered flight. They won because they failed more repeatedly than their competition. I’m sure their competition was brilliant, and they definitely had several years head start, but they lost because it took them maybe a year to iterate from design, to build, to test flight, to crash and destroy, and go back to design again. The Gossamer Condor team’s breakthrough insight was this: if the Condor could be repaired and improved in days, then they’d test 100 designs in the time their competition tested 1.

50 years on, and data science is following this same anti-pattern of the teams that didn’t get the Kremer Prize: come up with brilliant ideas, painstakingly move them out to the real world, watch them fail, and then slowly start the brittle process anew. This would be a problem for any profession—the faster you can iterate, the faster you can learn, and the more problems will be solved—but it is an especially pressing problem in data science because there is a huge shortage of data scientists, so inefficiencies mean that many critical problems are not getting solved.

What can we do about it? Luckily, software engineering, which is a sister to data science, has been working through these problems for the last two decades and has some pretty good patterns to build from. Devops is the area of software engineering concerned with moving software from development to real-world use quickly and safely. Devops lets software engineers try more things, and therefore learn faster. Here are the pieces I believe are necessary for data science:

  1. Automated tests: these don’t have to be exhaustive, but there should be an automatic way to know that changes don’t horribly break your user-facing system
  2. 1-Button Deploy: If releasing changes takes more than one step, it will break more frequently, and more importantly in the context of this article, releases will happen less frequently.
  3. 1-Button Rollback: The counterpart to 1-Button Deploy, if an error is discovered in a user-facing system, reverting to a pre-error state must be swift and reliable.
  4. Instrumented Infrastructure: Data science problems often require distributed architectures, non-obvious dependencies, and complex feedback loops. To successfully try many things quickly, it is necessary to spend a minimal amount of time understanding the infrastructure, tuning it, and correcting errors.

It’ll take some work, but I believe Devops is the next crucial frontier for Data Science—a massively underrated piece of this rapidly changing discipline.

Yes, I cofounded a company, Mortar, that amongst other things addresses these problems… because I think they are so important to solve.

Follow @DataconomyMedia


K YoungK Young has been CEO of Mortar Data since 2010. Mortar helps data scientists and data engineers spend 100% of their time on problems that are specific to their business—and not on time-wasters like babysitting infrastructure, managing complex deploys, and rebuilding common algorithms from scratch. Mortar’s platform runs pipelines of open technologies including Hadoop, Pig, Java, Python, and Luigi to provide out-of-the-box solutions that can be fully customized. Prior to founding Mortar Data, K built software that reaches one in ten public school students in the U.S. He holds a Computer Science degree from Rice University.

Tags: data scienceDevopsGossamer CondorK YoungKremer PrizeMortar DataWeekly Newsletter

Related Posts

Meet Photoleap, the AI photo editor that can fulfill almost every user’s needs

Meet Photoleap, the AI photo editor that can fulfill almost every user’s needs

June 9, 2023
How is artificial intelligence in surgery and healthcare changing our lives?

How is artificial intelligence in surgery and healthcare changing our lives?

June 8, 2023
These text-to-video AI tools brings Tarantino to your Reels

These text-to-video AI tools brings Tarantino to your Reels

June 8, 2023
The pursuit of creative general intelligence comes to fruition

The pursuit of creative general intelligence comes to fruition

June 8, 2023
The best anime AI generator you are looking for

The best anime AI generator you are looking for

June 8, 2023
WordPress has its own AI writing assistant now

WordPress has its own AI writing assistant now

June 7, 2023

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

LATEST ARTICLES

Canva not working right now; yes, it is really annoying

Here are the best Midjourney Prompts that will blow your mind

Decoding the secrets of code execution

Tired of unrealistic beauty filters on TikTok? Bad news, the frustration grows

Meet Photoleap, the AI photo editor that can fulfill almost every user’s needs

What are the best things about IT staff augmentation services?

Dataconomy

COPYRIGHT © DATACONOMY MEDIA GMBH, ALL RIGHTS RESERVED.

  • About
  • Imprint
  • Contact
  • Legal & Privacy
  • Partnership
  • Writers wanted

Follow Us

  • News
  • AI
  • Big Data
  • Machine Learning
  • Trends
    • Blockchain
    • Cybersecurity
    • FinTech
    • Gaming
    • Internet of Things
    • Startups
    • Whitepapers
  • Industry
    • Energy & Environment
    • Finance
    • Healthcare
    • Industrial Goods & Services
    • Marketing & Sales
    • Retail & Consumer
    • Technology & IT
    • Transportation & Logistics
  • Events
  • About
    • About Us
    • Contact
    • Imprint
    • Legal & Privacy
    • Newsletter
    • Partner With Us
    • Writers wanted
No Result
View All Result
Subscribe

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy Policy.