On February 11-13, 2001, a group of senior programmers, project managers, and technical analysts met at the Snowbird ski resort in Utah. Their mission, sparked by numerous conversations between the various members over chat and at conferences, was to define a single movement that would end up radically shaping how software was developed. The result of their efforts was a 12-point document known as The Agile Manifesto.
Twenty years have passed since that manifesto was first published. It is worth putting some context around what the world of programming looked like in 2001. The dot-com era was in full swing, and everyone was racing to build websites that could scale. Amazon was still a few years out from showing a profit, and Google was replacing Altavista as the go-to search engine on the web.
Agile, as the movement became known, focused on the software development process, challenging what had been the dominant Waterfall paradigm with its long cycles and extensive planning stages. Instead, it promoted the idea that software development is an iterative process, one where the precise goals are usually not known ahead of time. Consequently, an iterative approach with short cycles and development goals that could be accomplished within those cycles would generally provide better feedback, making it easier for the client to adjust their thinking as the product moved towards its final state.
As Agile expanded its influence, creating offshoots such as Kanban, Extreme Programming, Scrum, and so forth, the core ideas gained more traction at the C-Suite, and a cottage industry of consultants, scrum masters, and certified Agile gurus emerged as part of this. Yet even as it did so, Agile faced increasing criticism in the areas where it did fail (sometimes spectacularly). Two of the biggest failings of Agile came primarily in areas that had seen comparatively little coverage in the manifesto – the role of design and the role of data.
One of the characteristics that differentiated Agile was the belief that design was an emergent property. A frequent criticism of Agile has been that all too often, this approach has led to both design paralysis as the definition of the product remains too fluid to cogently program to, and integration challenges as multiple teams involved in parallel paths fail to produce common interfaces. This, in turn, leads to fundamental questions about both the notion of a team of equals and the corresponding dangers inherent in a lack of leadership, both of which tend to be played down in the original manifesto.
The second problematic area for Agile has been the shifting roles of programmers/engineers, who tend to focus on the creation of frameworks and processing algorithms, versus data analysts. In the early 2010s, the Big Data era heralded this shift, as the emphasis went from building scalable solutions to handling high volume/high-velocity data (an engineering problem) to using that data to drive both data analytics and machine learning (AI) solutions. The data analyst may use programming tools, but their role is different – the tools are simply a means towards better understanding the inherent, relevant signals coming from the noise of the data. On the other hand, AI specialists increasingly use the data produced by data engineers to create models, often with thousands, millions, or even billions of parameters, to have machines determine their own behavior. These are tool users – using the power of high-speed systems to do most of the modeling and development of “software” computationally – and their needs and priorities are different.
In that scheme of things, especially in conjunction with more than thirty years of open-source development, the role of programmers has gone from being central to all aspects of computing to being necessary primarily as tool builders when existing tools aren’t sufficient to needs. Agile 2000 is just not as relevant today because the emphasis of Agile – tool builders – is not as relevant as the importance of data and machine learning models have escalated.
These were some of Clifford Berg’s considerations and the community that he gathered around him faced when examining what “Agile” really means today. Data Agility was a big part of it – the recognition that the data lifecycle today is not the software lifecycle of two decades ago. In many respects, it is more pipeline oriented: data acquisition, data cleansing and curation, metadata capture, data harmonization, data analysis, data presentation, and data contextualization.
This last point usually doesn’t exist in data cycle charts. Still, it’s one of the most important – contextualization is taking what is learned from the analysis and using it to train the back processes – to make acquisition easier, make harmonization more precise, and make analysis more meaningful, and presentation more robust. It is the process equivalent of back-propagation in machine learning models, and it is fundamental to becoming a data-driven organization.
Agility plays a role here, but it’s one where leadership comes down to determining what the organization as an organism should learn. The new agility is the recognition that communication is not face-to-face. Instead, it is multimodal – synchronous and asynchronous, augmented by artificial intelligence, which increasingly becomes a team-member itself. It is the awareness that there are times where you need the iterative innovation cycles that Agile offers to refine a package, but that there are also times when you need to establish stakes in the ground, working with dedicated creators to help articulate the true intent of a product.
Put another way, Agile in 2020 must-see Agile 2000 as a component, something useful to shape raw concepts into usable forms, but something that also needs to fit in with the broader data fabric being created. We are entering into a profound new era in which the process of creating anything, whether it be a book, a piece of software, an engine for a car or aircraft, or a wedding cake, ultimately will come down to creating a virtual twin of this first, then “instantiating” it through increasingly complex printers. Some of those printers are matter manipulators, some of them are models for decision making; some of them will take pieces of media and composite them into whole new forms. This ultimately will feedback into the digital twin – the virtual learns from the physical and vice versa.
So where does this leave a newer Agile, an Agile 2 if you will? It is an agility that recognizes DevOps and continuous integration, Zoom meetings spanning ten time zones, and git repositories representing the working state of living projects and collaboration in all its many manifest forms. Agile sees nothing as permanent but instead sees “products” as snapshot artifacts of processes over time.
It is arguable whether Agile 2 can be boiled down to a twelve-point manifesto, and perhaps that’s just as well. Manifestos tend to be rigid and unbending around the edges. Agile 2 is adaptive, fluid, and multifaceted, and like the reflection of a crystal on a plane, it will tend to look different depending upon which angle it’s viewed from. That’s okay. Clifford Berg and the Agile 2 movement have taken some significant strides in trying to do this. What will emerge from this is likely a new school of thought, a new best practice, about what a data-driven organization ultimately looks and acts like.
