Paying the Piper, Techie-Style

Growth based on debt is unsustainable, artificial.
-- Jose Manuel Barroso

Debt is beautiful only after it is repaid.

-- Russian Proverb

Technical debt.  It sounds like a scary new investment mechanism that too-big-to-fail banks created after the whole CDO/credit swap/Big Short thing ended... poorly.  But it's much scarier than that.  It's the time bomb lurking in every software product, waiting for the right time to jump out and take down your newest beautifully crafted feature.  It's the source of zero-day security holes, back door hacks, and plain old everyday performance issues.  It's a fact of life for every software company and the bane of every application architect's existence.  In my office, it's also the prime catalyst for torrents of muttered (or not) curse words.  Sometimes, it even comes with a name, usually the name of the person who thought he'd found a brilliant shortcut... at the time.

So what is it?  It's old code, shortcuts that felt necessary, compromises in quality and flexibility based on ignorance or tight deadlines.  It's the code that you wish you hadn't written, or that you'd like to smack someone else for writing.  In short, it's software development's loan against future functionality.  Like Popeye's friend Wimpy, it will gladly pay you next year for two features hacked together today.  And when the bill comes due, it sucks.

So how do you keep technical debt from taking over your product?  First, you need to recognize whether technical debt is a problem that you have to worry about.  So ask yourself these questions?

  1. Do you have a software product?
  2. Do you plan to keep adding features to that product?
  3. Do you still plan to have that product two years from now?

If you answered yes to these three questions, then yes, you need to worry about technical debt.  If you answered no to #2 or #3, congratulations!  You are the proud owner of a legacy product and can look forward to many years of saying, "No, we aren't investing in this anymore so I can't add that enhancement.  Please see our list of new products and let me know if you're interested in migrating."  If you answered no to #1, thanks for reading, Mom.  You can stop now and just tell me how great you  thought the article was.

For those of us with active, growing software products, technical debt is a constant worry, or at least it should be.  Because even if you have genius architects, brilliant developers, and aggressively detailed QA testers, you still can't escape one fact: software decays.  Even perfect code, designed to account for every known variable at the time, will feel slow, awkward, and difficult to work with three years from now.  Technology moves quickly, and user expectations move with it.  So join the rest of us, Mr. Perfect, and start thinking about tech debt.

There are different kinds of tech debt, and you need to know which kind you're dealing with before you can manage it.  Let's look at these different debt instruments, in order of urgency.

1. Visiting The Software Loan Shark

Sometimes you just don't have time to do things the right way.  Whether you're working against a tight deadline (a commitment made by "Management," of course, since you would never over-commit, right?) or scrambling to fix a Sev 1 in production, you look at your code and you think, I can do this now or I can do it right.  You look around to see if anyone's watching, then you hard-code that    client-specific logic, or you wire that UI element directly to the database, because it's Friday and you know there's a cold beer waiting for you somewhere.  You walk the changes over to the architect for the code review, because this one needs some "explaining."  You tell him, "I know this isn't the right way to do it, but the right way would take another week.  And see?  I put '// *hack*' right there in the comments.  We can come back and fix it on Monday."

You, my friend, just made a visit to the software loan shark, and the vig is high.  If I had a dollar for every time a developer investigated a production issue and said, "This was a [name of our fastest coder] special," I wouldn't be writing this blog, because I would have retired.  And, of course, by that point we had built whole features around that shortcut, so it took weeks (sometimes months) to rebuild that "time-saver" so that it worked for anything other than the very specific case that Speedy Gonzales was thinking of at the time.

So, before you go in search of that beer, create a new user story to remediate your shortcut, and 'fess up in front of everyone.  It's OK to find the quick fix once in a while, especially if it makes your biggest customer or your boss's boss's boss happy.  Just don't make a habit of it, and pay it off quickly.  You don't want to get hooked on this stuff if you want to keep your job.

2. Emergent Debt

The second kind of tech debt is what I call "Emergent Debt."  This comes from a combination of myopic planning and a general lack of omniscience.  Even when you take the time to design your solution carefully, your design is only as good as your planning horizon.  If your product roadmap only extends six months into the future -- due to genuine market uncertainty or "because we're Agile, man..." -- then you have two choices:
  1. Design for what you know you'll need
  2. Guess
Either way, you're working with limited information, and you're bound to miss something.  Even with an extensive roadmap, any decisions beyond this year are, at best, educated guesses about what your customers will find valuable.  You're going to miss something.

Welcome to the land of emergent debt.  That design that was a frickin' work of ART! two years ago is now a stinking pile of shortsighted delusions.  What were you thinking?!?  Oh, right, you're not omniscient.  So stop mourning your tarnished monument to technology and start fixing it.  Ask yourself:
  • What's salvageable?  Is it still usable?
  • How much will it slow us down to keep building around it?
  • Should we rebuild or build a replacement beside the old code?
  • What's the cost of the new solution relative to continuing to work around the old one?
Once you understand the benefits and the costs, you're ready to choose a new course.  Don't start until you truly understand that balance, because some code, even if it's old, can still do its old job just fine for years while its younger siblings steadily grow up to take over its functions.  Don't jump into a rewrite just because you're offended by your younger self's myopia.  Write the user stories, put them in the backlog, and prioritize them against the new features that were already there.

3. Code Decay

Even high-functioning code gets old, and sometimes you have to replace it, not because it isn't extensible, but because the rest of the application has passed it by.  Code written for different hardware configurations, leveraging ancient libraries, can become a bottleneck or even a source of crashes when faster components start racing through their actions before waiting for the old-timer to hobble through its logic chain. Like a 1964 Maserati in a world of Teslas, your code is still beautiful, but it's getting expensive to keep around.  Time to upgrade.

The good news is that this is a chance to learn from the past.  Study the old code, see what it does well (if slowly) and what pieces you had to write because they didn't exist in the developer kit yet.  Look for opportunities to leverage and extend a clean design and then rewrite it with half the lines of code.  Look for performance upgrades so that the new version can last for another several years and cause bottlenecks in other places before it needs to be updated again.

You often see code decay early, so you have time to plan to replace the old code.  Put the user stories in your backlog and keep an eye on the current code in production.  Since these rewrites can often take some time, make sure that you start the work before it becomes an emergency (see: Loan Shark).

4. Moronic Predecessor Syndrome

This category may or may not represent actual technical debt, but in my experience it creates the most noise in every software organization.  I call it "Moronic Predecessor Syndrome" (MoPS for short).  Here's how this works: the smart developer that you just hired a week ago walks up to you in the hallway and says, "Whoever built this component I'm working on was a complete moron.  We'll have to rewrite the entire thing."

You reply, "Hmm, he seemed pretty smart for the 4 years he worked here.  Why do you say that?"

He replies with a series of beeps and clicking noises that in reality are English speech, but you've stopped listening because you've had this conversation before, every time you hired or promoted a new developer.

You see, developers are smart people who thrive on complexity and problem-solving.  I've kept brilliant teams working together in stultifyingly boring industries simply because the problems were interesting (the secret is to keep feeding them problems at the proper rate so they don't notice that no one understands what they do).  Sometimes, though, this problem-solving tendency runs out of control, especially when a developer is bored or feels a need to prove himself, as when he enters a new role.  This is where MoPS rears its ugly head, and you have to decide if you're facing a real problem or an aesthetic difference of opinion.

Once the beeps and clicking noises wind down, try asking a few questions:

  • Why do you think the code is all wrong?  Is it functionally deficient or is it built "the wrong way?"
  • What's the impact of the current design?  Is it slowing us down or making it difficult to add new functionality?
  • Does the rest of the team who works with this code agree with you or are you the only one seeing this? (Try to avoid using the word "visionary" here: it will sound sarcastic).
  • How long do we have before the problems you foresee come to pass?
Note that I didn't ask, "How long will it take to fix this?" because the answer is inevitably, "6-12 months."  Seriously, every time.  I have never seen a breakout of MoPS with less than a 6 month price tag, so before you even go down that road, make sure that you're dealing with a functional issue and not a difference in technical tastes.  If the issue is aesthetic, bring some other people into the conversation to talk about how they make this grossly disfigured code work for them.  Offer to look at alternatives in the future, but don't waste time rewriting functional code just because someone thinks it's ugly.

If you determine that you do have a problem, then follow the same steps as for Emergent Debt: assess the risk, understand the cost and benefits, plan the work.

And did I mention that you should create a user story?

Paying Off Your Debts

Development teams get weird when it comes to paying off their technical debt.  There seems to be this shame attached to fixing old code, as though the company should have known better and written code that would stand the test of time.  Yet tech debt accrues in a myriad of ways, and as long as you aren't making weekly trips the software loan shark, you don't have anything to be ashamed of.  This is life in the digital world, so stop trying to sneak your debt payments into your new feature estimates!  (And yes, I know who you are).

Eliminating technical debt is just like any other piece of work: you're investing to improve your product, and you expect certain benefits from doing the work.  Just because they're often invisible to users doesn't mean they don't exist, and everyone wants a product that stays usable over time, right?  Of course, you can't blackmail the budget keepers with "do this or your precious site will crash," so you need to make your case.  New features have business benefits (new revenue, increased customer satisfaction, competitive advantages) to offset their costs, so why aren't you doing this for your technical debt?

Here's how the conversation usually goes when a developer wants to fix old code:

Developer: "We need to fix this."
Product Owner: "Why?"
Developer: "Because old code sucks and the new code will be better."
Product Owner: "Shut up and go build my new feature."

What if, instead, we created a business case for out debt payments?  All you have to do is quantify "better."  If spending 4 weeks building a new service means that the next 5 features can be built in half the time, that's obviously a good investment.  If rebuilding the calc engine means that our product stays performant for another 3 years, that's probably also a good investment, assuming that it doesn't take a year to build.  The question behind every business case is simple: how can you make the benefits quantifiably outweigh the costs?  And since costs are easily measured (people * time = $$$) the more quantifiable the benefit, the better.  If you tell me that you can guarantee that a $10,000 investment will return $100,000 in the next year, I'd be an idiot not to take it.  If you tell me that you want $10,000 so you can make something "better," then you're asking for a lot of trust.

And if you can't quantify the benefit?  Then you probably have a bad case of MoPS.  Take two beers and a good look in the mirror and call me on Monday.
Post a Comment

Top Posts

The Giving Season

Wanted: Someone to Make My Life Easier

Do You Really Want to Be CTO?