Agile often isn't

Great ideas rarely survive marketing. Once even the most brilliant vision is distilled and subdivided into a format designed to be sold, it typically loses more than its luster. It is the reason people are disinterested in politics today. And it may become the reason people are less interested in organizing a business.

Ideals, as opposed to ideas, may be immune. Sometimes the reason is because certain ideals may be so unfathomable, so borderless, as to escape all packaging. Freedom, as so poetically and appropriately defined by psychologist Rollo May, “is not rebellion.” Such a simple phrase leaves one free to consider the possibilities, and in so doing, the ideal defines itself.

Some great ideas — including the political ones and the organizational ones — are born from a sense of rebellion. Its founders publish a manifesto designed to stir the soul. The idea of Agile is based on a theory that the nature of the work people produce (in the original case, software) can illuminate the best and also least structured processes with which people collaborate to do that work.

The message of Agile was first articulated in a manifesto, some of whose core principles, like Rollo May’s idea of freedom, attempt to be brief and poetic by setting forth what Agile is not.

Through the marketing of Agile — a process which leads one to think the word itself may require a little “TM” — an identity was given to that which Agile is not, since every great story needs a bad guy. The villain in the Agile story is called Waterfall, which implies the blatant irony of an Ian Fleming novel.

Waterfall™ could coincidentally be categorized as everything the U.S. Government has just declared to be good and ideal, which might in and of itself make it all bad and wrong. It has been explained to me as leveraging the structure of a business’ organizational chart to enable bureaucracy to break down the development process into discrete, schedulable stages that eventually become unmanageable.

Imagine if an alien archaeologist dug up an automotive assembly line without any pre-existing knowledge of Earth history or sociology or Henry Ford. Workflow, for most industries, divides units of work among individuals skilled in that work, who in turn report to managers responsible for its quality and timeliness. That much seems sensible enough. But the staging process, as would be obvious even to an alien archeologist (some of whom may be covertly running assembly lines at this very moment) often appears to feed and nurture itself, creating new and sometimes redundant stages in a cascading fashion that leads to the choice of metaphor.

The downside of Waterfall, allege Agile’s evangelists, is that it removes development teams from direct contact with customers. By partitioning them into committees, the resulting silos prevent team members from collaborating, and from bouncing good ideas off each other’s heads. So the adoption of Agile is not so much about businesses changing their workflow, but rather about individuals in the business changing their behavior in order to prevent the business from falling into the Waterfall trap.Individuals and small groups, often working in pairs, are encouraged to choose their own goals and challenge each other to fulfill them. And since feeling is governed by psychology, the success of the effort is often determined not quantitatively but psychologically; thus many businesses proclaim Agile a success when they feel better about their jobs. Henry Ford, make way for Sigmund Freud.

So when the first serious efforts to quantify the success of Agile were first attempted by university researchers three years ago, they found themselves thinking not as software developers or as business analysts but as sociologists. Using sociological methods, they discovered that many teams that feel they are successful actually are not, by standards which even business analysts would appreciate. In fact, perhaps to ensure their good feelings, teams may actually be secluding themselves from their own customers, cutting themselves off from their main source of feedback, and reducing collaboration with one another. Then to defend these feelings, some teams may invoke a certain cascading, descending, wet process that both Freud and Ford would call a “defense mechanism.”

The Scarlet Letter

Dr. Rashina Hoda is the Research Chair for the International Conference on Agile and Lean Software Methods, whose annual meeting was held last February in Bengaluru, India. Presently, Dr. Hoda teaches software engineering at the University of Auckland, New Zealand. She’s a Certified Scrum Master, which doesn’t mean she plays rugby, but rather that she completed a full course in one of the more explicit practices of Agile development. Specifically, Scrum encourages development teams to self-organize, by choosing tasks in a kind of jump-ball collection process (thus the scrum), assigning Scrum Masters for each task based not so much upon rank or salary but rather who’s up for it, and setting minimal deadlines of 30 days or less. (HP is partnered with an organization that licenses the certification of Scrum Masters.)

Officially, Dr. Hoda is not a sociologist. Nevertheless in 2010, while teaching at Victoria University of Wellington, Dr. Hoda and her colleagues began one of the first comprehensive investigations into the behavior patterns of Agile development teams. Instead of envisioning a brilliant argument touting the benefits of Agile and Scrum and giving herself 30 days to prove it right, she conducted a study using classic Grounded Theory (GT) methodology, created by sociologists for studying human behavior.

The result was therefore a true sociological study, albeit done by computer science professors. Rather than argue a point, they would conduct interviews and questionnaires, analyze the raw data they received in response, and following the principles of GT, produce theory based on only that data.

No one could have anticipated the results. Entitled “Agile Undercover,” the first report from Hoda and her colleagues demonstrated conclusively that Agile development teams were failing to communicate with their customers — not just occasionally, but mainly. And in order to ameliorate the impact of these failures, teams and their companies were making active, intentional efforts to keep customers in the dark about their development practices, including their schedules of deliverables.

“Customers harbor skepticism about Agile methods and pose resistance to involvement,” the Hoda team wrote. “They don’t readily understand Agile practices such as ‘fail fast’ and its intended benefits — that minimum time and effort may be wasted on a project that’s bound to fail — and become prejudiced.”

Although teams may be developing software under what they consider to be short, iterative, Agile cycles, they actually don’t deploy in tandem with those cycles. Instead, the research found, they may wait for windows of opportunity, chances to make a big push when the customer is less likely to mount a resistance.

“What starts to become annoying for the team,” Dr. Hoda tells me in an interview, “is that they’ve done so much hard work in making sure it’s an actual, working piece of software, but they know it’s not going to be ‘on the shelf’ for another six months. And it’s a bit of a demotivation for them, if you know what I mean.”

In larger organizations that Hoda’s team studied, ITC departments (the European term for IT, which includes the “Communications” aspect) are often uninvited to the Agile party. So although development teams may work within 30-day cycles, ITC only deploys updates every six months, at best. In the case of big companies, ironically, this gives development teams a kind of fallback position, a rational excuse for reverting to old habits. With smaller companies, however, developers find themselves scrambling, and customers are left without a clear message.

As Dr. Hoda’s team found, customers disgruntled by an apparent lack of progress tend to spell out their demands more clearly, in the form of tighter, more definitive, less negotiable contracts.

Of the four key goals of the original Agile Manifesto, so far, three have been shot down: Interactions with customers are unpleasant; software is no longer deployed in iterative cycles; and team members, depressed by a lack of communication and individual initiative, are taking steps to feel better about their jobs, since feeling is the key metric here.

The pressure for development teams to appear to be performing as customers expect  is actually higher now than ever before.

Yet the next domino to fall, as explained in Dr. Hoda’s sequel, “Agility in Context,” is absolutely stupefying.

“Teams are very keen on pleasing their customers, and it’s hard for them to bring up issues with customer collaboration,” Hoda tells me. So to keep the customer at bay and out of their hair, development teams hire or appoint a customer proxy. An ambassador, if you will. Or, to be more truthful, a sales associate.

“A customer proxy is a team member who will try to keep up-to-date with the requirements from the customer,” says Hoda, “but not all of the team members will interact with the customer. This one person will.” It’s the job of this proxy to develop a message, a vision, a lofty image of the master plan. And it may not even be the real image, but whatever it is, it should keep the customer mesmerized for now, like a string dangled in front of a cat.

In an ideal situation, Hoda explains, development teams are up-front with their customers from the beginning, explaining the virtues of continual collaboration and iterative development. But more often, customers are not convinced to change their own expectations for deliverables. Some of the reasons why are beyond their control, such as the inflexible work cycles of their own IT/ITC departments and, in some countries including India, laws requiring definitive and all-inclusive contracts to be spelled out and signed in advance by both parties. So the typical course of events is what she describes as “less than ideal.”

Then there is the worst-case scenario, which Hoda calls “Extreme Undercover.” In this scenario, communication with the customer is rigidly restricted, including in many cases — according to the Hoda team’s test subjects — mandates never to mention aloud the dreaded “A-word,” lest it bring shame down upon them.

Regaining Control of Self-control

So if Agile (pardon my language) can indeed lead to successful projects and happy customers, how can teams afflicted by the “Undercover” phenomenon relocate the road to freedom? Theoretically, they should take examples from successful, self-managing, “truly” Agile project teams.

In late 2009, a search for such examples was led by Dr. Likoebe Maruping of the University of Louisville. While teaching at the Sam Walton College of Business at the University of Arkansas, Dr. Maruping and his colleagues researched how Agile development teams adjust their metrics of success in the face of rapidly changing customer requirements (for example, migrations to the cloud, and the adoption of new security models).

Borrowing concepts from sociology, Maruping based his study around control theory. The control in this case comes from the signals or measures or methods used by management not only to steer workers toward desired outcomes, but to communicate to them what those outcomes should be. (Not to be confused with the Control Theory of Motivation, William Glasser’s work which suggests that individuals may conjure the inner need to succeed. Maruping’s field of study is actually closer to B. F. Skinner’s theory of operant conditioning, that individuals’ behaviors can be shaped through experiential programming.)

First, the Maruping team attempted to identify the control modes that development teams used to manage themselves in so-called “self-managing” Agile teams — the signals that leaders send that progress or outcomes are positive. As Dr. Maruping explains to me in an interview, “If you’re going to actually advocate for a specific set of processes that will enable you to achieve your objectives, to what extent are you actually implementing those controls? Rather than only measuring performance targets, to what degree are you also assessing the degree to which you’re following this set of practices?”

This led to their first problem.

“To our knowledge, there are no existing measures for assessing the use of Agile practices in software development teams in the field,” the Maruping team wrote. “Therefore, we developed a new scale to operationalize this construct.” It wasn’t a tremendously complex scale, but it’ll do for now: It bases relative software quality on a minimal number of inherent bugs. At the very least, perhaps the University of Arkansas researchers could discover whether Agile teams were producing bug-free code.

“Many of these teams tend to have self-governing, self-managing approaches to development. A good amount of the way these controls are actually implemented tends to come from the project leaders themselves,” Dr. Maruping tells me.

So the news here isn’t all bad: Project leaders, including those who step up to the plate at the time projects are assigned, do use some methodologies for determining success. They’re just not all the same processes, perhaps because Agile evangelists don’t specify what they should be, and perhaps because, in keeping with the Manifesto, Agile refrains from mandating any one set of tools or methods anyway. But it causes a problem for anyone who wants to derive a set of best practices by example.

Once Maruping’s team developed their own interim set of controls, enabling them to reasonably compare Agile development efforts with each other, they discovered something else that contradicts reason: When Agile does work and software does improve, it’s because managers end up acting more like Skinner than Glasser after all.

“What we actually found was that managers who had had some experience with Agile tend to be the ones placing greater emphasis on implementing behavioral controls,” reports Maruping. “This speaks to a real tension that both managers and their teams are dealing with: There’s a real desire to provide some degree of autonomy, particularly when you’re adopting a methodology that embraces the possibility that requirements are going to shift and morph and change over time. Some degree of autonomy is useful to basically cut loose some of the bureaucracy, to respond to changes as they occur. But at the same time, you want to get away from this free-for-all, anything-goes type of approach as well. So from that perspective, there is the need for some form of managerial intervention, some sort of guidance that channels the focus and the energy of these teams.”

Maruping’s results were graphically plain: Teams that employed certain Agile methodologies, but also found their own signals for communicating progress, produced code with fewer bugs. When developers got more signals, better signals, and perhaps most importantly, the same sets of signals, they did better work. By comparison, teams that actively relied upon “self-control,” with minimal guidance and manager interference, performed much more poorly. And as their adherence to Agile’s tighter iterative cycles increased, their work got worse.

Backwash

Perhaps the final solution could be as simple as realizing teams need definitive guidance, and that reliance upon pure self-control leads to confusion, frustration, and inevitably Dr. Hoda’s “Extreme Undercover” scenario. Maybe Agile could adjust its methods to become more, shall we say, agile — to adopt the particular context, to use her term, that applies specifically to the business at hand.

As Dr. Hoda warns, expect some resistance in the early going.  “What tends to become challenging when you get down to a certain kind of Agile methodology, when you start to do those concrete methods, a lot of people — typically evangelists and consultants — will try to sell it as an all-or-nothing package. Eventually what happens is, people do all sorts of tweaking.”

Hoda’s team learned that under certain conditions, Agile practices work beautifully: specifically, small development teams working with customers who have agreed in advance to the assignment of on-site representatives. She calls this situation the “sweet spot.” But finding the sweet spot, concludes “Agility in Context,” is a roll of the dice.

When the dice come up snake-eyes, “Agility” suggests that development teams may face a choice between adapting the context to suit the new practices (for instance, moving more of the development team onto the customer’s home site, which is not always feasible) or adapting the practices to suit the context. And it’s here that teams face active opposition and possibly even public flouting by Agile evangelists claiming teams are breaking their system.

Dr. Hoda and Dr. Maruping have learned that methodologies inspired by the application of Agile in the enterprise do work. They do improve software quality, increase collaborative efforts, and can (perhaps more rarely, but it’s possible) spark customer involvement. But it takes the adoption and subsequently the failure of what purports to be “concrete Agile methods” for businesses to become so inspired. As a result, the methods work are more solutions to Agile rather than “Agile solutions.”

As one of the world’s foremost supporters of the principles of Agile, Rashina Hoda suggests that Agile spend a cleansing moment beneath a cool waterfall, and consider the benefits of applying its own medicine. As her team writes in “Agility,” “Being agile is all about having the courage to change.”