Scrum

In 2008 I was scheduled to give a talk to the Bay Area APLN meetup group. The meeting was being held at the offices of The Gap in downtown San Francisco. The organizers arranged to meet up for a reception in a bar prior to the meeting. As the guest of honor some of the attendees naturally wanted to talk with me. Waiting patiently and quietly for her turn was Janice Linden-Reed. Janice’s story has become a familiar one. She was a project manager at a company where they had adopted Scrum a couple of years earlier along with the Rally tool for work tracking. They’d received training and coaching from Rally but for the last 6 to 9 months Scrum was proving very painful for them and morale was suffering. Meanwhile, the Rally consultants only had one song to sing, “you aren’t doing it right! If only you followed all the practices of Scrum then everything would be fine.” That company was Posit Science, a company pioneering the commercialization of brain plasticity science, and its staff were a combination of PhD neuroscientists and game developers. They were smart, highly motivated people, who believed they were bringing some good to society with their technology and yet Scrum was making them miserable. They simply refused to buy the argument that the problem was with them.

Over this recent holiday period, Grant Ammons lit up social media with his blog post, “Ditching Scrum for Kanban – the best decision we’ve made as a team.”

What makes Grant’s story interesting is that his is just the latest in a long line of similar stories – a thread that can be traced over at least 8 years since I first met Janice Linden-Reed and first heard about Posit Science and their challenges with Scrum.

Since 2009, the Lean Kanban conference in the United States, has been sponsored by Ultimate Software. Ultimate sponsor our conferences as a thank you and in recognition of what Kanban has done to maintain their corporate culture and enable them to thrive since they made a decision to ditch Scrum. In 2008, Steve Reid and Rafael Santos from Ultimate attended Agile 2008 in Toronto looking for a solution to the misery that Scrum was bringing to their workforce. Their story was similar to Janice’s and similar to Grant’s, Scrum had helped them for a while, it had worked for a while, but after a period of 9 to 18 months, it was clear that it was having an irreconcilable negative impact on staff morale. Ultimate prided themselves on their status in the best 100 places to work list. In the most recent survey they are ranked 21st best large employer to work for in the United States. In 2008, their status on this list was in jeopardy and Scrum was a key source of the dissatisfaction that was bubbling in the workforce.

Every story is a little different but there are clear themes. Perhaps half of our clients and half of the attendees at Lean Kanban training around the world are people looking for answers to challenged, painful or failed Scrum implementations. The first thing to realize is that these are real stories and the facts are what they are: people felt the way they did and they acted according to their situation. Wishful thinking or post hoc analysis that they could have done things differently doesn’t change the facts. So what are the common themes in these and so many other stories of challenged Scrum adoption?…

1. Scrum worked and helped initially

With Posit Science and Pipeline Deals they were startups with smallish engineering teams and they simply needed some initial rigor, some process to follow and Scrum gave them that. With Ultimate, who were already larger, and several hundred engineers, Scrum helped them to focus on short term goals and to deliver more frequently. All of these were good things. So the initial reaction is nearly always positive. While I am only using 3 examples here there are many more such stories where people report, “Scrum helped us initially.” Here is another one by Charles Suscheck. You don’t have to look hard to find more like this on the Web.

2. Morale starts to drop after 9 to 18 months of using Scrum

All of these stories report that following Scrum started to damage morale after a period of time. There are usually several stated reasons for this: missed commitments / trouble planning and estimating; estimation and planning are taking too long and seem random, error prone, or just plain nonsensical; urgent work arrives more quickly than our sprint cadence and there is pressure to reduce the sprint length but we don’t know how to do that given the current overheads and the existing problems planning and estimating; there is pressure to break the sprint boundaries and add urgent work; product owners show favoritism to one stakeholder over others; product owners have trouble balancing demands from multiple stakeholders; stories are being split over multiple sprints and actual customer service is very poor with long lead times and a lack of predictability and transparency; product owners are suffering anxiety because they are making up numbers to facilitate prioritization and planning activities.

Scrum as a process is engineered to use peer pressure to produce results. This is true at the daily Scrum meeting where individuals make commitments on what they will work on and complete today and have to report the following day on whether they made their daily Scrum commitment, while entire teams makes sprint commitments for the collective work they will complete in a single time period such as two weeks. The problem with this peer pressure, commitment-based approach is that it uses very deterministic mechanisms for estimation and planning, in what is actually a non-deterministic domain. Meanwhile, the social engineering in Scrum also uses peer pressure to drive conformance. It is well understood in sociology that pressure to be part of a group can be strong and can often override individual judgment or logic. So when things aren’t working, individuals are pressured into believing the fault is with them, when in fact the fault is the use of deterministic methods in a domain that has a non-deterministic natural philosophy. When the system is to blame, Scrum teaches its practitioners to blame the people and not the system. This is at the core of the stress and dissatisfaction practitioners feel with Scrum.

I am now seeing similar reports from the field with Scaled Agile Framework which has largely adopted scaled up versions of many Scrum practices. It uses similar deterministic approaches to planning, estimation and prioritization over longer time periods. This practice is known as quarterly release train planning. Some of the same complaints – anxiety that the numbers were guessed in order to follow the process, and that attempts to resolve dependencies in advance are error prone and just guess work, while everyone is being held accountable for their commitments. Once, again, the same problem is being repeated – a deterministic approach is being used to plan 3 month windows of work when the underlying domain is non-deterministic. The practitioners are being set up for failure and made to carry the burden of this failure as their own incompetence.

3. When things go wrong the consultants always blame the people

In every one of these stories, when the people and organizations complain about the stress that Scrum is bringing to them individually and organizationally, they are universally told that the fault lies with them. Scrum works! Scrum is without flaws! The problem lies with them, if only they were doing it right everything would be wonderful.

As Ken Schwaber has said, “Scrum works! Scrum is designed to work in a context. Your job is to change your context so that Scrum will work for you.” And I truly believe that this is a genuinely correct statement. It, however, disguises the truly difficult bit, “your job is to change your context.” If we look at Grant Ammons story, or Janice Linden-Reed’s story (we use it in our training classes), or the Ultimate story (captured for a future book), or others on our web site such as Nemetschek Scia what we see is a pattern that these were all firms where the business they were in simply made it impossible for them to control their environment and to create a context in which Scrum would work. It wasn’t that they were doing it wrong, rather, in order for Scrum to work, they would have to have changed their business and their business model. Ultimately, canonical Scrum was poor choice for them. If it helped them initially, it should never have been intended as a final solution, instead it should be a first step on a journey of process change and improvement.

Earlier this week, Larry Maccherone shared this story and cumulative flow diagram via his Facebook page. If you read the comment thread you will see that Mike Beedle, one of the founders of Scrum posts his thoughts. Rather than look at the data and analyze it, he quickly jumps to the conclusion that neither Scrum nor Kanban could help these poor souls because they are evidently very poor at technical practices and need to shape up. Compare Mike’s response to mine, as someone seasoned in analyzing cumulative flow data and with a deep understanding of flow efficiency and lead time. This post and the comment thread deeply underscores the differences between Scrum and Kanban communities. With Scrum the practitioners are almost always belittled and blamed, with Kanban we seek to understand the underlying natural philosophy of the domain, understand how the system is operating within it, and seek to find the systemic problems. With Kanban the system is always the number one suspect, not the people working within that system.

4. Scrum failure stories often involve leaders in the community and professional certified trainers and coaches

There are many stories of failed Kanban in the market: sometimes the failure comes from inappropriate alignment with culture such as resistance to transparency that comes with adopting Kanban boards; sometimes the failure comes from a failure to install and this would generally be avoidable if the protagonists had taken Kanban System Design training; sometimes a proto-Kanban implementation would have been easier initially than a full Kanban and a trained coach would have been able to help the organization scale the depth of their Kanban to the current level of organizational maturity and leadership tolerance. In general, where we see failed Kanban in the market, the failures come in organizations where they had no training, and did not invest in qualified trainers or consultants with credentials from Lean Kanban University.

However, with Scrum the failure stories are often very different. Often large numbers of the work force have been given certified training from Scrum Alliance certified trainers. Often leading consulting firms are involved and all of their consultants had certifications and professional credentials from the Scrum Alliance or other similar bodies. Despite the available expertise, Scrum remains challenged or fails and the story follows the pattern described in 1 through 3 above.

At Posit Science all their Scrum training and coaching was conducted by Rally, the largest Agile consulting firm in the World. At Ultimate the Scrum training had been conducted by Jeff Sutherland and later by Mike Cohn. Despite having two of the leading 3 Scrum experts in the World, Ultimate were unable to make it work.

The conclusion I’ve come to during the last 15 years of Agile is that Scrum has been widely oversold. There is no doubt that it can and does work but the circumstances under which it can be effective are rarely considered. There has in this respect been a considerable amount of professional misconduct in the overselling and the failure to make any reasonable consideration of context when choosing to implement it. The standard practice of blaming the people in the organizations having Scrum forced upon them inappropriately is a practice that is both morally and ethically bankrupt.

The importance of context and appropriate application of methods

Kanban is not an Agile software development method or process. Kanban was deliberately designed as a mechanism that acts on existing underlying processes. There needs to be an existing context into which we apply Kanban. So fundamentally Kanban eliminates a whole class of potential mis-selling of Agile methods. Kanban is the “start with what you do now” method and evolve from there. Every Kanban implementation will have its own unique workflow and policies. As a consequence, Kanban is adaptable to your context and culture.

However, this is not to say that we haven’t learned a lot about applying Kanban in specific situations. We now have the Kanban Appropriateness Appraisal Framework and the Depth of Kanban Assessment Framework. Both of these coaching tools are part of the standard curriculum for Kanban Coaching Professional training. The appropriateness appraisal looks at both the appropriateness of use of pull systems and the cultural fit for an evolutionary approach to change and incremental improvement. Not every organization has a suitable culture or the appropriate level of organizational maturity or leadership tolerance for Kanban to work. We now know that specific styles of Kanban implementation, implementations at different depths, correlate with different levels of organizational maturity, and as a consequence of this we now train Kanban coaches to be aware of this and to initially apply Kanban practices at an appropriate level of depth. The Appropriateness Appraisal and the Depth of Kanban Assessment Framework go hand-in-hand to insure a much greater level of success with Kanban implementations. It is the use of tools such as these and the knowledge and experience that went into creating them that mean you truly get value when employing a qualified Kanban Coaching Professional (KCP).

And meanwhile, it is never the people who take the blame for challenges with Kanban implementations with the exception of instances where the protagonists quite clearly never received any Kanban training or coaching, and did not make any attempt to actually understand what it is or how it works before they attempted to use it. At some level there is no accounting for stupidity. When people stick a board on the wall and declare they are doing Kanban, if it fails to produce the gains they were hoping for then the fault is quite clearly with them. Meanwhile, our community has an extensive capability for learning, including formal events such as Kanban Leadership Retreats which have led to the development of specific consulting tools such as the Depth of Kanban Assessment Framework and the Kanban Appropriateness Appraisal.

If you are feeling the stress and anxiety or a challenged Scrum or Scaled Agile implementation and you are sick of being told that the fault is with you, then there is an alternative path to agility – consider Kanban in 2016 and see how far it can take you!