Saturday, June 25, 2016

Paying the Piper, Techie-Style

Growth based on debt is unsustainable, artificial.
-- Jose Manuel Barroso

Debt is beautiful only after it is repaid.

-- Russian Proverb

Technical debt.  It sounds like a scary new investment mechanism that too-big-to-fail banks created after the whole CDO/credit swap/Big Short thing ended... poorly.  But it's much scarier than that.  It's the time bomb lurking in every software product, waiting for the right time to jump out and take down your newest beautifully crafted feature.  It's the source of zero-day security holes, back door hacks, and plain old everyday performance issues.  It's a fact of life for every software company and the bane of every application architect's existence.  In my office, it's also the prime catalyst for torrents of muttered (or not) curse words.  Sometimes, it even comes with a name, usually the name of the person who thought he'd found a brilliant shortcut... at the time.

So what is it?  It's old code, shortcuts that felt necessary, compromises in quality and flexibility based on ignorance or tight deadlines.  It's the code that you wish you hadn't written, or that you'd like to smack someone else for writing.  In short, it's software development's loan against future functionality.  Like Popeye's friend Wimpy, it will gladly pay you next year for two features hacked together today.  And when the bill comes due, it sucks.

So how do you keep technical debt from taking over your product?  First, you need to recognize whether technical debt is a problem that you have to worry about.  So ask yourself these questions?

  1. Do you have a software product?
  2. Do you plan to keep adding features to that product?
  3. Do you still plan to have that product two years from now?

If you answered yes to these three questions, then yes, you need to worry about technical debt.  If you answered no to #2 or #3, congratulations!  You are the proud owner of a legacy product and can look forward to many years of saying, "No, we aren't investing in this anymore so I can't add that enhancement.  Please see our list of new products and let me know if you're interested in migrating."  If you answered no to #1, thanks for reading, Mom.  You can stop now and just tell me how great you  thought the article was.

For those of us with active, growing software products, technical debt is a constant worry, or at least it should be.  Because even if you have genius architects, brilliant developers, and aggressively detailed QA testers, you still can't escape one fact: software decays.  Even perfect code, designed to account for every known variable at the time, will feel slow, awkward, and difficult to work with three years from now.  Technology moves quickly, and user expectations move with it.  So join the rest of us, Mr. Perfect, and start thinking about tech debt.

There are different kinds of tech debt, and you need to know which kind you're dealing with before you can manage it.  Let's look at these different debt instruments, in order of urgency.

1. Visiting The Software Loan Shark

Sometimes you just don't have time to do things the right way.  Whether you're working against a tight deadline (a commitment made by "Management," of course, since you would never over-commit, right?) or scrambling to fix a Sev 1 in production, you look at your code and you think, I can do this now or I can do it right.  You look around to see if anyone's watching, then you hard-code that    client-specific logic, or you wire that UI element directly to the database, because it's Friday and you know there's a cold beer waiting for you somewhere.  You walk the changes over to the architect for the code review, because this one needs some "explaining."  You tell him, "I know this isn't the right way to do it, but the right way would take another week.  And see?  I put '// *hack*' right there in the comments.  We can come back and fix it on Monday."

You, my friend, just made a visit to the software loan shark, and the vig is high.  If I had a dollar for every time a developer investigated a production issue and said, "This was a [name of our fastest coder] special," I wouldn't be writing this blog, because I would have retired.  And, of course, by that point we had built whole features around that shortcut, so it took weeks (sometimes months) to rebuild that "time-saver" so that it worked for anything other than the very specific case that Speedy Gonzales was thinking of at the time.

So, before you go in search of that beer, create a new user story to remediate your shortcut, and 'fess up in front of everyone.  It's OK to find the quick fix once in a while, especially if it makes your biggest customer or your boss's boss's boss happy.  Just don't make a habit of it, and pay it off quickly.  You don't want to get hooked on this stuff if you want to keep your job.

2. Emergent Debt

The second kind of tech debt is what I call "Emergent Debt."  This comes from a combination of myopic planning and a general lack of omniscience.  Even when you take the time to design your solution carefully, your design is only as good as your planning horizon.  If your product roadmap only extends six months into the future -- due to genuine market uncertainty or "because we're Agile, man..." -- then you have two choices:
  1. Design for what you know you'll need
  2. Guess
Either way, you're working with limited information, and you're bound to miss something.  Even with an extensive roadmap, any decisions beyond this year are, at best, educated guesses about what your customers will find valuable.  You're going to miss something.

Welcome to the land of emergent debt.  That design that was a frickin' work of ART! two years ago is now a stinking pile of shortsighted delusions.  What were you thinking?!?  Oh, right, you're not omniscient.  So stop mourning your tarnished monument to technology and start fixing it.  Ask yourself:
  • What's salvageable?  Is it still usable?
  • How much will it slow us down to keep building around it?
  • Should we rebuild or build a replacement beside the old code?
  • What's the cost of the new solution relative to continuing to work around the old one?
Once you understand the benefits and the costs, you're ready to choose a new course.  Don't start until you truly understand that balance, because some code, even if it's old, can still do its old job just fine for years while its younger siblings steadily grow up to take over its functions.  Don't jump into a rewrite just because you're offended by your younger self's myopia.  Write the user stories, put them in the backlog, and prioritize them against the new features that were already there.

3. Code Decay

Even high-functioning code gets old, and sometimes you have to replace it, not because it isn't extensible, but because the rest of the application has passed it by.  Code written for different hardware configurations, leveraging ancient libraries, can become a bottleneck or even a source of crashes when faster components start racing through their actions before waiting for the old-timer to hobble through its logic chain. Like a 1964 Maserati in a world of Teslas, your code is still beautiful, but it's getting expensive to keep around.  Time to upgrade.

The good news is that this is a chance to learn from the past.  Study the old code, see what it does well (if slowly) and what pieces you had to write because they didn't exist in the developer kit yet.  Look for opportunities to leverage and extend a clean design and then rewrite it with half the lines of code.  Look for performance upgrades so that the new version can last for another several years and cause bottlenecks in other places before it needs to be updated again.

You often see code decay early, so you have time to plan to replace the old code.  Put the user stories in your backlog and keep an eye on the current code in production.  Since these rewrites can often take some time, make sure that you start the work before it becomes an emergency (see: Loan Shark).

4. Moronic Predecessor Syndrome

This category may or may not represent actual technical debt, but in my experience it creates the most noise in every software organization.  I call it "Moronic Predecessor Syndrome" (MoPS for short).  Here's how this works: the smart developer that you just hired a week ago walks up to you in the hallway and says, "Whoever built this component I'm working on was a complete moron.  We'll have to rewrite the entire thing."

You reply, "Hmm, he seemed pretty smart for the 4 years he worked here.  Why do you say that?"

He replies with a series of beeps and clicking noises that in reality are English speech, but you've stopped listening because you've had this conversation before, every time you hired or promoted a new developer.

You see, developers are smart people who thrive on complexity and problem-solving.  I've kept brilliant teams working together in stultifyingly boring industries simply because the problems were interesting (the secret is to keep feeding them problems at the proper rate so they don't notice that no one understands what they do).  Sometimes, though, this problem-solving tendency runs out of control, especially when a developer is bored or feels a need to prove himself, as when he enters a new role.  This is where MoPS rears its ugly head, and you have to decide if you're facing a real problem or an aesthetic difference of opinion.

Once the beeps and clicking noises wind down, try asking a few questions:

  • Why do you think the code is all wrong?  Is it functionally deficient or is it built "the wrong way?"
  • What's the impact of the current design?  Is it slowing us down or making it difficult to add new functionality?
  • Does the rest of the team who works with this code agree with you or are you the only one seeing this? (Try to avoid using the word "visionary" here: it will sound sarcastic).
  • How long do we have before the problems you foresee come to pass?
Note that I didn't ask, "How long will it take to fix this?" because the answer is inevitably, "6-12 months."  Seriously, every time.  I have never seen a breakout of MoPS with less than a 6 month price tag, so before you even go down that road, make sure that you're dealing with a functional issue and not a difference in technical tastes.  If the issue is aesthetic, bring some other people into the conversation to talk about how they make this grossly disfigured code work for them.  Offer to look at alternatives in the future, but don't waste time rewriting functional code just because someone thinks it's ugly.

If you determine that you do have a problem, then follow the same steps as for Emergent Debt: assess the risk, understand the cost and benefits, plan the work.

And did I mention that you should create a user story?

Paying Off Your Debts

Development teams get weird when it comes to paying off their technical debt.  There seems to be this shame attached to fixing old code, as though the company should have known better and written code that would stand the test of time.  Yet tech debt accrues in a myriad of ways, and as long as you aren't making weekly trips the software loan shark, you don't have anything to be ashamed of.  This is life in the digital world, so stop trying to sneak your debt payments into your new feature estimates!  (And yes, I know who you are).

Eliminating technical debt is just like any other piece of work: you're investing to improve your product, and you expect certain benefits from doing the work.  Just because they're often invisible to users doesn't mean they don't exist, and everyone wants a product that stays usable over time, right?  Of course, you can't blackmail the budget keepers with "do this or your precious site will crash," so you need to make your case.  New features have business benefits (new revenue, increased customer satisfaction, competitive advantages) to offset their costs, so why aren't you doing this for your technical debt?

Here's how the conversation usually goes when a developer wants to fix old code:

Developer: "We need to fix this."
Product Owner: "Why?"
Developer: "Because old code sucks and the new code will be better."
Product Owner: "Shut up and go build my new feature."

What if, instead, we created a business case for out debt payments?  All you have to do is quantify "better."  If spending 4 weeks building a new service means that the next 5 features can be built in half the time, that's obviously a good investment.  If rebuilding the calc engine means that our product stays performant for another 3 years, that's probably also a good investment, assuming that it doesn't take a year to build.  The question behind every business case is simple: how can you make the benefits quantifiably outweigh the costs?  And since costs are easily measured (people * time = $$$) the more quantifiable the benefit, the better.  If you tell me that you can guarantee that a $10,000 investment will return $100,000 in the next year, I'd be an idiot not to take it.  If you tell me that you want $10,000 so you can make something "better," then you're asking for a lot of trust.

And if you can't quantify the benefit?  Then you probably have a bad case of MoPS.  Take two beers and a good look in the mirror and call me on Monday.

Monday, June 13, 2016

In Defense of the Plan, Part 2

I wrote recently about the need for long-term planning, even (perhaps especially) in an Agile development environment.  I left you with a choice: plan or fail.

The fun part about writing on the internet is that you don't really have to solve problems: you just have to point them out so that other people can agree with you that yes, there is a problem there, and someone should do something about it.  I'm a problem-solver by nature, though, so I can't just leave it at that.  If I'm going to tell people they need to do something, then I'm constitutionally required to help them do it.  So we've already covered why you need a plan for your product development.  Let's talk about how.

Let's start with a few objections, because I never met a straw man I didn't like (to poke holes in):

  1. I can't tell you when we'll have a whole feature set done because we're using Scrum/Kanban/Lean Agile/some other Japanese-sounding development process.
  2. I can't plan because I don't know how long it will take to build something six months from now.
  3. I can't create a release plan because I don't want executives to see it and think that my team has committed to specific deliveries way out in the future, where all the uncertainty lies.
Let's take these in order:

1. Your development methodology doesn't matter.

I hate to break it to you, but no matter how proud you are of your particular style of software code widget management (i.e., your methodology), it has no impact on your ability to predict what your team can accomplish over the long run.  You see, that delivery method is the micro view, and the little irregularities that can torch a sprint or a Kanban board in the short term all even out over time.  Release planning, on the other hand, is an exercise in macro planning that's almost Seussian in its simplicity:

I have this box.
I have these rocks.
How many rocks can I fit in this box?
It's somewhere between a little and lots.

(For those who prefer allegory over metaphor, rocks = major features, the box = your release)

Individual vacations, surprisingly difficult features, requirements changes, these are all daily problems that have no place in the long-term plan, and using them to keep you from looking at the  big picture is either laziness or a misunderstanding of the viewpoint.  Think in big chunks, average velocities, and conservative predictions and the picture will quickly clear up.

Ask yourself:
  1. How much time do we have to build this release?
  2. At a rough order of magnitude, how long will it take to build each major feature that we want to include?
  3. How many of those major features can we get done in the time we have?
Ta-da!  Instant release plan.

2. When you estimate has little impact on what you estimate.

There's no doubt that development teams learn things as they build that make their estimates more precise and their designs more accurate, but claiming that you can't estimate a feature now because you plan to build it later is a fallacy.  How do I know this?  Reorder your product backlog and move those features to the top of the list.  Go ahead, I'll wait....

Can you estimate them now?  That's what I thought.

I understand that there's complexity here.  What if one feature depends upon another?  What if building that framework first makes it easier to build all the features that will use it?  What if your customers decide that they want another feature that isn't on the list?  These are all things that could be true and might change your project.  But how many of them actually impact your estimates?

Sometimes development teams become infatuated by complexity. Like that pretty girl you can't stop thinking about, they see it everywhere they look. But sometimes it's just a skinny dude with long hair. Rather than making a long list of the things that could happen to blow up your plan, how about making a small list of reasonable assumptions that will help build your plan?  Assume that you'll be smart and build the framework first, then estimate your effort based upon its existence. Build a plan based on what you know now rather than waiting until you know everything (hint: you never will).

3. Learn the difference between planning and presenting

Above all, your release plan is for you.  It's a guide and a goal for your team, helping you understand how long you have to build things and when it's time to move on.  It helps you shape the picture of what this thing is that you're trying to build and, most importantly, what done looks like.  If this were a road trip, your release plan would be the map that tells you whether you're driving to New York or Hawaii and helps you choose the destination with the highest likelihood for success.  Along the way, it also helps you set intermediate goals, plan your next steps, and ensure that you don't wander too far off the path chasing shiny objects.

Your stakeholders, customers, and whoever is funding your project (read: executives or investors) also want a hint as to what they'll get for their time and money, and this is where presenting comes in.  The smart team lead knows how to set achievable targets that account for the risk inherent in software delivery, then hit every one.  Some people call this "under-promising and over-delivering."  I prefer to call it "risk mitigation and expectation management."  Of course, you can't mitigate risks if you haven't identified them, and you can't manage expectations if you don't know what you're capable of delivering.  In other words, you need a plan (and if you didn't see that coming, you really haven't been paying attention).  With the plan you've built for yourself and your team, you can easily account for the estimates that feel accurate but make you the most nervous or for those parts of the product that present the highest levels of uncertainty in execution.  Build contingency into your plan and commit to exceed expectations whenever possible.  I like to set stretch features in every iteration or release, giving my team room to deliver everything that we've promised, even if we hit some bumps, then give our users a little more when things go as well as we'd hoped.

In other words, if you're confident that you can get to New York in the time allotted, tell your executives how fabulous Cleveland is in the fall, then try to surprise them with a trip to the Big Apple.

Why Plan?

This still feels like a lot of work, so why are we doing it again?  Because:
  • It sets the expectations for your team and help them understand what they'll have to show for all the work they're about to do.
  • Like the framework of a building, it provides a rough guide for where every feature goes and how long it should take to build it, helping you identify the most efficient path to completion.  When features get too big, it warns you before they take over the project.
  • It helps you identify risks and opportunities before you stumble across them, that you can mitigate them or take the best advantage of them.
  • It lays a foundation for communicating to everyone who cares about your product -- and the more people who care, the better, no matter how annoying they may be -- and helps you manage expectations and chart your progress.  This is a significantly better approach than saying, "We'll tell you when we're done," an approach that has never in the history of time failed to aggravate the people with the money and always leads to more bad attention, not less.
So forget what your Agile guru said: the sprint is the means, not the end.  Sit down with your team and contemplate your glorious future together.  It starts with a simple question:

"Does anyone know where we're going?"

Wednesday, June 08, 2016

In Defense of the Plan, Part 1

Have you ever sat down with a pile of Legos and tried to build something with whichever pieces came to hand?  I have, and it turned out looking something like this:

I didn't really know where I was going with it, and then I couldn't find any of the round pieces to use as wheels, and pretty soon my car became a house, then it was just sort of this rock/tree stump thing with stuff growing out of it.  When I was a kid, though, I used to buy the Lego kits.  You know, the ones that could build a spaceship or a race car or, if your parents were rich, a medieval castle.  When I followed all of the steps, I ended up with this:

It took a lot of work, and the instructions weren't always helpful, but I had a plan.  I knew what I would have to show for my efforts, and with the application of some patience, concentration, and some judiciously applied glue to fix the pieces that I broke trying to force them into place, I got my spaceship (with bonus moon rovers!).  I had a great sense of accomplishment when I finished that project, which lasted until my little sister played with it and broke it in half.  Then my dad stepped on a moon rover and it was time for it all to go back in the big box with all the other pieces.

So what's with all the Lego talk?  As I look around at the state of the software industry, the metaphor seems apt.  As we celebrate the 15th anniversary of the Agile Manifesto this year, people are reassessing the value and the viability of Agile development practices.  Many, including Dave Thomas, declared that at least the term "Agile" should be declared dead.  Matthew Kern provided an exhaustive summary of the larger conversation along with some insightful analysis on how the original vision has been co-opted, but the Agile brand will live on for as long as consultancies can bill hours teaching it, practicing it, and building extensive methodologies around it.

I don't know if Agile is dead yet, but I certainly see some signs of poor health.  I've written before about the dangers of overly simplifying the messy process of software development and I've poked fun at the sunny-minded Agile zealots who think that a Scrum Master certification and a framed copy of the Agile Manifesto will solve all of life's problems.  Now I have another bone to pick with the Sacred Manifesto and its adherents.

Let's clear something up first: I was agile before we had Agile.  I developed my own iterative delivery methodology when I was a frustrated project manager trying to figure out why building software was so damn much more difficult than building a house, road, or nuclear submarine.  I was teaching teams how to break functions down and deliver them in pieces when the consultants were still trying to figure out whether "software development" was a practice they wanted to add to their centers of excellence.  In fact, I'm not even sure they had "centers of excellence" back then.  They might have just called them "centers."  The excellence (and associated higher billing rates) came later.  I see the value in acknowledging that software development is essentially a creative process in which continuous discovery plays an essential part.  I get it, and I know that, even if "Agile" dies, agility, at the individual, team, and corporate level, is still critical to successfully delivering functional products.

I also live in the real world where, if you want people to pay you for something, you have to actually give it to them.  And that's where I see Agile Development as a practical methodology failing us today.

It all started with Line 4:
[We have come to value] Responding to Change over Following a Plan
On the face of it, this makes perfect sense.  If your plan no longer matches reality, you need either a new plan or a new reality.  Until science comes through with that multiverse-hopping technology they've been promising us for years -- and I am absolutely canceling my Scientific American subscription this year if they don't deliver -- you're only left with one practical option: change the plan.  Clinging to a plan when it no longer makes sense is foolish.  The essence of agility is the ability to respond to changing demands and market conditions for the good of the product.

Once again, though, we've gone too far.  "Don't let your plan overrule your common sense" has become "planning is futile."  Not only have we lost sight of the big picture, it seems like some teams are determined to deny that anything like a big picture could ever exist.  Their battle cry of "the sprint is everything!" places them in a state of Agile existentialism, with no future, no past, only today.  This defeatist approach to the fourth tenet of the Manifesto subverts its original purpose while using it as a club to pound the long-term memory out of an organization's brain.

As often happens when ideas take on momentum, we've swung from one extreme to the other.  Where the smart Agile team used to say, "I know where I want to go, but I can't tell you exactly how long it will take to get there (because, uncertainty)," now the extremist Agile team says, "I don't know where we're going or how long it will take, but when we get there I'll tell you what we did (because nothing is certain and planning is for evil management types)."  This twisted logic leaves teams in a Kafkaesque nightmare of failing to deliver because they failed to plan and then choosing not to plan because delivery is so uncertain.

What we need is a little release planning.

What would you do if you took your car to a mechanic and he said, "Well, I can see what's wrong, but I won't know if I can fix it until I take the whole engine apart and look at all the pieces to see if there's anything else I haven't seen yet.  I'll start taking it apart now and I'll let you know the next step in a couple of days."  Would you leave your car with him or find someone who knew what he was talking about?  Yet this is what overly Agile teams tell their customers on a regular basis: "We'll start coding and we'll let you know what you're getting when we're done building it."

Contrary to popular opinion, "release planning" isn't a sneaky way for project managers to make their teams commit to delivering features they don't yet understand.  Nor is it an even sneakier way of reverting to waterfall project management.  Release planning is how you figure out what you're going to build and roughly how long it's going to take, so that you can decide whether the destination will be worth the trip before you leave the house.  It's what you're supposed to do when you build your backlog and define your minimum viable product and before you begin sprint planning.  It tells you what the finished product will look like and when you'll have it, based upon what you know right now.  In true Agile form, we aren't committing anyone to anything, because we realize that change happens, but in service to agility, we also want to make sure that we're on the right track before we write a line of code.

If you don't bother to look at the big picture, bad things happen.  Just a few things I've seen over the years:
  • A team was two months into a release before they realized that the way they had designed the features only worked for very specific use cases. They had to scrap what they'd built and start over from scratch.
  • Another team committed to an initial release of a minimum viable product within four months, but didn't estimate beyond the next 2-week sprint.  Three months into the project, they realized that there was no way they could deliver a viable product in only one more month and had to choose between releasing a stunted, useless product or asking for more time.  After six months, they finally released a product that still had fewer features than they originally planned.
  • A team made commitments to a client to deliver a set of features within six months.  While the list of features felt too long to successfully deliver, they didn't think that they had a choice.  To save time, they skipped the initial planning phase and jumped right into coding.  After several false starts that cost them several weeks of time, they finally got going.  When it was time to deliver the release, they rushed through testing in order to hit the deadline and spent the next two months stabilizing the product, effectively turning their six month release into eight months.
You could look at each one of these examples and say, "See? Software development is uncertain.  You never know what could happen, so long-term planning is a waste of time."  Yet for every one of these sad stories I could give you two examples where a little bit of forward thinking and rough estimation kept a project from disaster.  Release planning isn't about forcing false certainty on an uncertain world, but rather about embracing uncertainty.  It's about stating what you know and what you don't know, and solving for both.

Another advantage of putting a little energy into forward thinking: you can compartmentalize your features, deciding how much time you can afford to spend on each one as part of the entire release.  This allows you to decide where to draw the line at "good enough" rather than continuing to polish a feature before moving on to the next one.  Without this framing, it's easy for a complex feature to shove the others aside, leaving you with a difficult conversation as the end of the release approaches.

There's something to be said for flexibility and for the ability to meet every unforeseen circumstance with the full attention of a development team.  But if you've given up on planning altogether, then you've gone from agility to chaos, from disciplined improvisation to making it up as you go and hoping for the best.  Keep it up. and even though you might want a snazzy spaceship, you're going to end up with an ugly rock.