Agile’s Great Schism

Agile hasn’t died, but the community is fractured. What’s the way forward?

With the seemingly never-ending news of layoffs in tech, and some of those targeted towards Agile groups, people ask, “Has Agile failed?” Indeed, we have been treated to many articles urging us to retire the term Agile, informing us that the revolution failed or, conversely, that Agile has actually won. Recently, Agile coauthor Jim Highsmith announced an effort to reimagine Agile. It’s clear that the Agile ecosystem is in turmoil, but why? I submit that we’ve lost our common story and beliefs, and we need to find new ways to align across the agile world lest we miss out on helping with the next great revolution – Artificial Intelligence. 

The Agile story is nothing short of a bottom-up revolution in business. I was part of the first wave of Agile and was faced with the challenge of applying it to some of the largest companies in the world. I’ve seen the evolution of business and product development, especially in the software space. It is difficult to communicate the impact of the Agile revolution during the last two decades to those who didn’t live through it. My first coding days in the ’90s were faced with Old Testament-level legalese. Requirements would map, screen by screen, field by field, in terse language. Testers sat in different buildings and had no say in estimates. Business analysts would translate all the requirements since developers were shielded from customers as if they were radioactive. Project managers would float estimates, inflating budgets. Changing code late in the process would cause havoc, and if something went wrong, all hell broke loose. The spec would be pulled up, and arguments would ensue. “That’s not in the spec.” Or “You can’t change that after we sign off on this spec.”  Thus the line in the Agile Manifesto “…working software over comprehensive documentation.” Creating software before Agile was torturous. 

We’ve come so far since then. An entire ecosystem of Agile tools and solutions was born from  XP, Scrum, Kanban, ScrumBan, Crystal, LeSS, and SAFe. Consultancies grew up, positioning themselves across the spectrum. We all repeated George Box’s mantra, “All models are wrong, but some are useful,” and cruised toward the future. That is our story, and with it, we changed the world. 

So what happened? I hypothesize that our community grew too large for this short manifesto. Our community comprises thousands of companies and tens of thousands of Agile practitioners. It’s difficult for a community to retain cohesion when it gets too large. Coaches are undoubtedly familiar with the Dunbar number, named after the British anthropologist. Dunbar suggested that the number of social relationships a human can maintain is limited by the amount of time and cognitive resources available for social interaction, putting that number at around 150 people. As groups get larger and subdivide, they lose cohesion; yet, other communities, such as doctors and lawyers, find ways to coordinate actions across their vast ranks. How do they do it? 

I recently came upon Yuval Noah Harari’s bestselling 2015 book Sapiens. Although I questioned several of his positions, I found his take on the importance of a common myth or shared story useful. Harari states that a common story can create a bond that helps humans scale beyond the Dunbar number. Harari is big on the words “myth” and “legal fiction” when talking about companies, agreements, and even the law. To me, the better phrase is “common story.” One only has to violate a nation’s or a company’s laws to see that the consequences of that “fiction” are very real. 

What is Agile’s common story or set of beliefs? To scale our institutions, modern society has created structures and stories that help bind their actions to a standard, regardless of context. A doctor from Texas will likely employ the same techniques and knowledge and ascribe to the same general codes as a doctor from New York. There will be disagreements and varied approaches, but if a doctor strays too far, he or she will be disciplined by the community. A doctor could lose their license. A lawyer could be disbarred. 

If we accept that a key to creating a functioning culture is a common story and the enforcement of its principles, what happens when those principles are violated? There are currently no enforcement mechanisms in Agile, Software Engineering, or product development. The closest we have are certifications, but even then, there are disagreements. Which cert is the best? Do they really prove anything? We talk about an Agile mindset, but how do you enforce it? Or do you? 

When I started in Agile it was easier to define who we were by contrasting our beliefs, as laid out in the Manifesto, with the prevailing control-based processes of the day. Our conferences were filled with excited people practicing their interpretation of these codes, modernizing business from the ground up. Our story and mission: “Waterfall was bad, control-based processes are too much, and we are looking to uncover better ways of doing software.”  

As the Agile community tried to tackle the big companies, the techniques diverged more and more, and some practitioners were and are accused of heresy against the Agile mindset. The Great Schism has occurred, and we’ve all retreated to our sanctuaries.

The problem is that transformations are challenging. They take time, executive buy-in, and, most of all, lots of cash. Now Agilists have a new challenger for these precious funds: Artificial Intelligence. AI has crashed the gates, and our fiddling time is over. Countless dollars are now being routed to the AI revolution, with over 70,000 companies at last count. Like the Internet revolution, everything will be sprinkled with AI holy water, regardless of whether it’s necessary or not. Our refrigerators, toasters, and washing machines no longer simply connect to the Internet (how quaint!) but will use their AI brains to tell you to restock the milk, to ensure that your toast is the perfect shade of brown, and to alert you to the red sock in your load of whites. 

“AI everywhere” is the story shared in the C-suites of America and beyond, driven by both the opportunities of this great tech and the fear of being left behind, similar to what drove the Agile revolution years ago. Yet, Agilists are still wondering how we will leverage an even more complex technology stack with outdated IT systems and overly complex processes. 

“If you failed at agile, you’ll fail at AI”

Jim Highsmith, Agile Manifesto Co-author

Defunding Agile transformations (among nearly ever other previously initiative) might score funding for AI projects, but those projects are every bit as likely to succumb to market misalignment, poor execution, and team morale issues as traditional software projects were in the last two decades. 

Agilists would do well to get out of their camps and come together to form a different story and to create new structures and beliefs to unite us. I welcome the efforts of Highsmith and others in re-imagining agility. The technology landscape remains an increasingly complex domain, with business leaders searching for ways to survive and thrive. Agilists are in a perfect position to assist and shine in an AI-dominated marketplace if we can put down our arms, end the internecine War of Models, and re-discover the common story we share. With that compelling story, we can get back to doing what we do best – helping people uncover better ways of humanely creating awesome products and services.