Martin Cron

A friend of mine recently posted a somewhat anti-“Agile” rant on Facebook a while ago and I had a momentary urge to throw some arguments back at him. After all, I cannot be silent while one slanders the glorious movement!

Actually, I can.  Any glory of the movement has been extracted and defiled long ago.  Most “Agile” implementations I’ve seen have either been bottom-up revolts from under-empowered developers or top-down mandates from over-empowered managers. Neither approach actually works.

It’s almost as if mixing a prescription of meetings with obscure rugby jargon isn’t enough to fix all of an organization’s problems. You can, if you need to, substitute “XP” or “Lean” or “Kanban” into that last sentence, In the case of Kanban, you get to use Japanese manufacturing jargon instead of rugby jargon because business school guys steeped in a long tradition of Toyota worship really like that.

Obligatory response to “If it’s not working, you’re not doing it right.” Of course we’re not doing it right. Nobody does it right.

I’m not about to bust out the Gantt charts and sharpen my pencil for 200 page technical specification writing, though. Because as much as I’m no longer really pro-agile, I remain militantly anti-waterfall.

The secret ingredient? It’s HATE. Usually it’s love, but…

Why? Let’s look at what happens in a rigid waterfall project:

Firstly, all of the decisions are made at the beginning of the project, a time when (hopefully) everyone knows the least.  Also, as the project is probably going to be relatively long, the business has a perverse incentive to cram as much as possible into the specs instead of thinking small and designing only the things that they know they actually need.

Next, the developers will start to push back and say “woah, that’s too much!” in a way that tends to get framed as a negotiation instead of a collaboration. The business guys are almost better negotiators than the developers, so the developers walk away feeling steamrolled. Now the developers start to resent the business as they are committed to a date and deliverable that they couldn’t give meaningful consent to. The macho culture of overwork (long hours, late nights, weekends) only makes that worse.

As this is a big project that you have one shot at, the stakes are really high to get things right the first time. Developers and architects over-develop and over-architect, respectively.  They build themselves hooks and layers and infrastructure and generalizations that they might someday end up needing, just in case.  The software, an abstract and ethereal beast by nature, becomes even more disconnected from any customer-facing business reality. Developers start having arguments with religious zeal about things like style and formatting. The developers start to hate each other.

After months of sound and fury which seems to signify nothing, the business starts to worry. Why haven’t we seen anything? Let’s have more frequent status meetings.  Developers feel distrusted and start to hate the business for breathing down their necks.

After a chunk of working at an unsustainable pace, the developers say “Hey! Now we have something to look at! Are you happy now?” Followed almost immediately by “What? You want that to behave differently? But this is what you signed off on!” and “Of course that part isn’t done yet, I didn’t tell you that part was done, did I?” and “Stop complaining about the fonts, we can fix the fonts later. Sheesh”

Eyes are rolled. The business starts to hate the developers for being so difficult and condescending.

Once the thing more-or-less works, it’s bundled up and sent over the wall to QA. As QA now has less time than they managed to negotiate but no room to adjust. That’s what you get for being at the tail end of a fixed-date schedule.  When they find “real” bugs, developers resent them for making them look bad. When they find things that aren’t “real” bugs, developers berate them for being wrong.

The testers start to feel that they are the only people who care about the users, and that the crappy developers don’t even care enough to fix the goddamn bugs. They fight among themselves about what bugs are “priority one” and what bugs are “priority two” and bug #2628 is really a duplicate of bug #2599. The intellectual output of some of the best minds in our generation has been spent arguing over differing priority levels of bugs that will probably never get fixed anyway.

Eventually, the “critical” defect count gets low enough that the business decides to release the darn thing, even though the QA folks think they are wrong to do so, there’s nothing they can do to stop it.

Maybe there’s a launch party. We’re done! Bust out the French Champagne! Everyone feels a little more uneasy than celebratory, however.

Customers start to get their hands on it.  It’s radically (and pointlessly) different from the last version, so existing customers hate it because it changes their expectations. It also has a lot of bugs in it, so new customers get to hate it as well.  Tech support hates everyone for releasing this half-baked beast into the wild and putting them in a difficult position. Everyone scrambles to put together “hot fixes”, the releases of which never warrant French Champagne.

Upon hearing all of the complaints, the business obliquely blames QA for not finding all of the bugs (as if the existence of bugs that they didn’t create is somehow their fault) with the semi-rhetorical “did anyone even test this?” QA hates this, naturally, and now they get to hate the developers for making them look bad.

Those who have the strength to keep going get to jump into another cycle! As all of the problems were obviously caused by lack of up-front design and documentation, so we’ll do even more of that. Also, as the code has degraded into a dangerous pile of spaghetti, broken glass, and human blood, we’ll just completely rewrite it with an entirely new architecture and “paradigm” to have fresh arguments over.

Part of the tragedy is that throughout this whole process, everyone is doing their best. The business people aren’t “bad” business people, they just lack super-human foresight. The developers aren’t “bad” developers, they just have normal emotional reactions to work they are invested in. The QA people aren’t “bad” QA people, they just lack super-human omniscience.  Though I have worked with some QA people who were to close to omniscient it was scary, that’s a different story.

The whole arrangement is an engine for hatred and despair, an engine fueled by tribalism and resentment. The only thing holding these isolated hate-filled groups together is their even greater hatred for everyone else.

It’s also an engine for guilt and shame. People feel that they are never following the “proper” process “properly”, and they sneer at those who don’t even try to follow the process as either “cowboy coders” or “gullible fools who drank the agile kool-aid”.

So, go ahead. Mock “agile” all you want, but let’s remember the historical context, shall we?

Concept #1. Knowledge Work

Software developers are largely “knowledge workers”, which (broadly) means that the job they do day-to-day is something that their manager can’t do, at least not as readily.

Note: this isn’t the typical tribalist anti-business developer rant. I’m not talking about “pointy haired bosses”. I’m talking about the fact that much of the time, developers are managed by people who either aren’t currently or haven’t ever been, software developers themselves.  I think that’s fine, but it leads to something that I’m calling the “developer blind spot”

Concept #2. Signal Attenuation

Even the best transmission medium has some signal attenuation.

As an idea goes through multiple people, it can change subtly, so an idea such as “for my service types, I like to encapsulate construction instead of using ‘new foo()’ to get an instance, so my consuming code isn’t tightly coupled to a specific concrete implementation” gets mangled to become “never ever use ‘new foo()‘” which a developer will hear and say “WTF? That doesn’t even make any sense!“. Because it doesn’t make any sense, not once the signal has degraded that far.

Concept #3. Fashion and Skepticism

Many ideas come and go, and it’s time consuming to try to differentiate solid ideas from fads, snake oil, and deep-sounding fluff that some executive got from a breathlessly enthusiastic book he read on an airplane.

In a perfect world, executives would limit their airplane reading to trashy fiction.  Feel free to contact me for specific suggestions as I read a lot of really trashy fiction that I don’t exactly feel comfortable blogging about.

Conclusion

Show. Don’t tell.

If you’re managing or leading a software team and want your team to do things differently, it’s possible that your (brilliant) advice will hit your team right in their blind spot and be ignored.  If you really want to plant an idea into a developer’s head, you need to obliquely inject it, Inception-style. Demonstrate new ideas in a proper real-world context, without appeals to top-down authority, blog posts written by random dudes (especially mine), or light airplane reading. Connect real-world developers with their peers and let them have meaningful conversations without latency and signal attenuation.  Give people enough creative space and room to experiment that they feel invested in these ideas. Anything new must actually make sense with the dozens of micro-scale cost-benefit decisions that good developers are making all day every day. Otherwise, they’re just checking mysterious magic checkboxes on some mysterious magic checklist because some manager says so for mysterious magic reasons.

I recently got to experience the exquisite joy and excruciating pain of helping to build a new software team for a large, well-established company. We had the mandate to do things differently and to help other teams modernize. While a lot of things went very well, I think the most helpful, or at least entertaining, things I can write about are what didn’t go well and why. So, at taking the risk of being a prescriptive hypocrite, here’s a list of what I think you should not do.

1. Do not muddle the message.

It’s about Agile! It’s about Lean Startup! It’s about DevOps! It’s about a green-field project! It’s about The Cloud! It’s about Open Source! It’s about automation! It’s about whatever personal pet perspective you care about.  Yes, when your mandate is to innovate, you’re going to work with different tools and techniques, but If you’re known as “that cloud team” or “that agile team” you’ve lost control of your own identity. You should be about innovation, experimentation, and letting go of the way things have always been done. Everything else is just implementation details.

2. Do not change the experiment halfway through.

A few months into the project, a new executive informed us of the new reality. “You guys have been operating all on your own. That ends now.” Looking back on it now, that was the moment when all value in the experiment was lost. We started working with the existing processes (which nearly everyone agreed were terrible).

Between the radical process change and the lack of clear messaging, any person could take any result and use that to validate their own existing opinions.  If the project were a huge success, it was because of technology X. If it were a failure, that was obviously because of process Y.  

I don’t expect a lot of scientific rigor, but seriously….

3. Do not keep things a secret.

I fear that I’m being a little unkind to the anonymous executive in the point above. He was reacting to the fact that we were completely opaque.  Apparently (I didn’t know this at the time) other groups in other offices were either not informed that we existed or specifically told to not interact with us.  A more sustainable approach would have been to have full transparency, along with broad autonomy, from day one. Other groups should know what’s going on, and even to give advice and suggestions, but not to swoop in and micro-manage.

4. Do not piss all over other people.

One of the first things that our team did was to look at an existing project underway from an existing team and to throw it away entirely. From a purely technological perspective, that may have been a good decision, but it was, if I may speak plainly, a total dick move. That, coupled with our opacity, created the perception that we were a bunch of elitists. I always wanted to be elite, but not elitist.  We never wanted to be a group of better technologists. Just technologists with a different mandate and a different toolkit.

The boastful and tone deaf claims of “look at this team getting done in six months what that other team spent several years on” didn’t help matters. If anyone on that other team reads this. You have my sincere apologies.

5. Don’t conflate being an “innovator” with being an “agent of change”.

If your organization is large enough that an innovation lab makes sense, you’ll also need change agents who can evangelize and support changes. These are different things. Your change agents can point at what the innovation lab is doing, which will help work around the developer blind spot, but being a change agent requires more patience and political savvy than being an innovator.

Change agents are there to dismantle the dysfunctional engine, innovators are there to create fuel for new engines.

One of the good things about writing is that it forces you to organize your thinking. While organizing my thinking around the role of dedicated “software architects” I actually changed my opinion. Slightly.

My original extremely persuasive argument was that the main goal of a software architect should be to minimize the day-to-day pain of the developers on the team and should therefore have hands-on involvement with the code so that the pain is felt and understood.

While reading that over, I realized it was a bit too myopic and dev-centric (sorry, I’m only human) so I broadened the goal to minimize the day-to-day pain of the cross-functional team that is responsible for developing and testing and deploying and supporting the product. After all, Architectural decisions that make things easier for those developing the product but harder for those supporting the product are most likely bad decisions. You know, the whole “optimize the whole” concept that goes all the way back to the Toyota Production System.

So then, I guess a good software architect would be someone who still feels the day-to-day pain of their architecture decisions as an actual hands-on-contributor to the cross functional team, regardless of discipline. Ideally it should be someone who has gone through at least one cross-disciplinary crop-rotation.

As long as you don’t abuse them (mostly by implicitly imposing constraints from the source concept), metaphors can be useful shorthand to talk about and explore ideas. I think that if you’re thoughtful, you can use metaphors from just about every source. I go back to biological metaphors a lot: thinking about ecosystems, monocultures, germ theory, immune response, etc. One of my very favorites is the agricultural practice of crop rotation.

If you didn’t grow up on a farm (as I sort of did… long story) the concept may be foreign. The basic idea is to use the same piece of land to grow different plants over sequential seasons. You know, rotate the crops. There are a few reasons to do this. One is that different plants interact with the soil differently. Some pull nitrogen out, some put nitrogen in. It also helps with pest control, as bugs who like to eat crop X might not like to eat crop Y. Wikipedia has a good writeup.

I started thinking about this metaphor many years ago after I was on a project where the dedicated tester left and I had to step in and be entirely test-focused for several months. It was a good experience for me. For one, I learned that even though I had a QA background (software testing was my first real job) I wasn’t half as good at being a test-focused developer (aka SDET) as the guy I was trying to replace.  I had to learn some different tools and techniques and it felt great.

The surprising part was, after this project was over and I was doing lead development on a different project, I was a better developer as a result of it. Like the nitrogen left behind in the soil after growing legumes, validation strategies were left behind in my brain after letting myself focus on the test discipline.

I like to imagine a healthy-sized cross-functional team where the different team members deliberately rotate through which particular discipline/kind of effort that they “own” or pay attention to. The rotation could be along the frontend/middle/backend axis, or the dev/test/support axis, or whatever distinct kinds of efforts the team requires.  Even though there is a meaningful cost to letting your people get up to speed in multiple related disciplines (just as there is a meaningful cost in planting different crops from year to year) the benefits in terms of big-picture vision, creativity, preventing burnout, and team empathy could be huge.

A few months ago, Mrs. Cron insisted that I go to the Emergency Room. She insisted that writhing around in pain on the bathroom floor isn’t a great way to spend an evening. She eventually brought me around to her way of thinking. Persuasive woman, Mrs. Cron.

I noticed that after giving me some industrial-strength painkillers, most of the effort was spent trying to look inside my generally opaque body without cutting me open (thanks!). There were blood tests, urine tests, ultrasounds, x-rays, MRIs, CT scans. Probably other tests as well, I don’t remember too clearly thanks to the aforementioned industrial-strength painkillers.

It turns out that I had gall stones, and they needed to remove my gallbladder. When it was time to cut me open, the surgeon already knew exactly what he would find inside and how to take care of it. The incisions were small and I recovered quickly. I’ll never be an underwear model, but that was never a reasonable goal of mine anyway.

The parallel to professional software development is pretty obvious. Many of the tools that we use are about looking inside a complex opaque thing without having to cut it open. We use debuggers, profilers, log analyzers, static analyzers, code coverage tools, and the like. Just like how the ER staff had a diverse set of tools for looking into me, it’s good for coders to have a diverse toolkit for looking into an application.

I just started using the free tier of New Relic for keeping track of how a very complex (and yes, very opaque) legacy application is behaving and it is my new favorite thing in the universe.