My only bone of contention is that taking on debt is usually a conscious choice. Technical debt to my mind is the price of making a pragmatic decision, it’s the trade-off that lingers.
Making poor architectural decisions or writing poor quality code isn’t tech debt. They are just institutionalised consequences. The way the world now works.
Legacy code isn’t tech debt either: it was right for the time.
What you have instead is knowledge debt as engineers turn over, and the once intuitive and elegant parts of the codebase are utterly alien to the new hires who learned their craft in a totally different way. This isn’t tech debt because engineers can’t intuitively anticipate the business’s future hiring decisions, particularly when those are made independently of the team.
If you want to establish tech debt you have to define the schedule to pay it off and there should be some record of that work somewhere, with accountability attached. It is not merely the accumulation of things you don’t like the look of or find hard to work with.
Similarly, I think MVP is a conscious choice to incur product debt because it is far too risky and irresponsible to go all-in with your perfect vision right off the bat. You want your initial buy in and a chance to bail out or pivot before all is lost.
Edit: as an example that comes to mind, a bank running on COBOL doesn’t have tech debt unless their COBOL engineers are aware of their missed opportunities. If they think COBOL is no longer fit for purpose, it’s not debt, it’s time to reinvest.
You may go in debt because you looked at the housing market and decided that now is likely a good time to buy a house with low interest rates. You take on a large load of debt, but it is done wisely based on currently information.
You may go in debt as a business operating on credit to fund an expansion that should pay off quite nicely in a few years time. Maybe it will fail, but it isn't a bad risk based on current knowledge.
You may go in debt for an education. Depending upon the degree and extent of debt this could be good or bad.
You may go in debt by spending beyond your means on a credit card. This is generally unwise.
You may go in debt because of a medical emergency that is entirely beyond your control (especially in some countries, but even some countries with universal healthcare it is possible to need treatment that isn't covered).
All of these have something similar when dealing with technical debt. Sometimes it is a wise investment. Sometimes it is a risk that may or may not pay off. Sometimes it is beyond one's control. Sometimes it is poor planning that will be regretted later.
I think of all the code written as technical debt. As soon as you write it, it becomes a burden - just like regular financial debt. However instead of thinking of code like a credit card loan that will take a few years to pay off, think of it like a mortgage that you have for 30 years. At the 30 year mark, you will generally have retired the code, hopefully. If you're not so lucky, you should expect to see balloon payments coming shortly. I know this isn't a perfect metaphor because honestly "technical debt" applies to all code, not just slightly bad code. That's very different from a loan.
The accounting world considers code an asset and allows you to amortize its creation.
Making progress in the world is fighting against entropy. Just because it is hard to fight the disorganizing efforts of the universe doesn’t mean that your output is a liability.
> liability: a thing for which someone is responsible, especially a debt or financial obligation.
According to the balance sheet equation,
> Assets = Liabilities + Owners Equity
Certainly all code is a liability - it must be at least understood and maintained from time to time (80% of lifetime cost is in the maintenance).
And while an accountant might also mark it an asset - not all code is an asset - and we recognize that fact when we delete for badness or disuse - or rewrite it to make extending and maintaining it easier.
As intimated, I'm not sure that the fuzzy language and world of economics is the best way to view such things, however ...
I'd accept that debts are a subset of liabilities, but not that 'liabilities are debts'.
A debt implies an externality, while a liability implies some generic exposure -- it's that which I focus on when considering (all) code as a risk to an organisation.
I suspect, however, that we agree on the fundamentals. There's some cost and risk to developing and trying to maintain any code base, and accepting that can usefully inform development and operations work better than thinking of code as a (near-)risk-free asset.
Code is a liability it can be hacked if not written securely. It can be out of date if the language changes or Windows changes and the company upgrades, which means parts of it need to be rewritten.
I used to program COBOL on IBM 370 DOS/VSE systems they later went to Visual BASIC on Window 3.1 PCs. The conversion took a long time and most code had to be rewritten because COBOL is not BASIC. Management didn't care about that, they just wanted a working project. We all did our best going overtime without pay and working weekends to meet deadlines.
COBOL was a liability when they wanted to convert to Visual BASIC. There was liability to convert the code, convert from DB2 to SQL Server on the database end.
That's not debt. That's an asset. It's inventory. It has a maintenance cost in the sense that a truck has a cost of upkeep to keep it on the road.
Technical debt is configuration and plugins. You are able to deliver something now with the benefit of changing behaviour later but at the cost of having an enormously increased test surface and invasion into your type system with possibly hostile parameters.
> Without the code the company is nothing, so it's definitely not 'debt'.
I reconcile the concept that code is a debt with your statement in the following way:
The feature produced by a piece of code can be an asset, but the code itself is a debt.
Thus, for a new feature to be valuable, you want the value added of the feature (or asset) to out-weight the cost of the code (or debt).
As a general rule, the complexity that comes along with a piece of code grows as the code-base and feature set grows, thus, for a young codebase, the value of a feature more easily out-weights the cost of the code.
Anything in any other business terms that needed to be used, created or maintained to create value for customers would not be debt.
'Technical debt' is just not debt, it's just an interesting way to think about it, and a neat use of lingo.
There are any number of other business practices for which one could try to apply the same analogy, but it wouldn't work because very quickly the term wouldn't seem right.
For example 'training employees' for a situation, knowing they might need to be trained differently in the future, is not going to be regarded as debt.
Same thing for processes, investment in equipment, parts, materials etc..
It's a metaphor not actual money that can create a negative value.
Although, you could say that a lot of profitable companies have negative value codebases where the cost to rebuild them is less than the cost to maintain.
This probably explains why some companies get replaced by small nimble startups.
> Making poor architectural decisions or writing poor quality code isn’t tech debt
I think it is. Technical debt is all about trading current productivity for future productivity. Poor architectural decisions can be caused by several factors. Inexperienced devs, not taking enough time to understand problem, not spending enough time brainstorming the problem. These are all decisions that trade current productivity for future productivity, which is technical debt.
Also most technical debt is not accrued consciously. Very few people are doing cost benefit analysis on technical debt. It's usually made intuitively by project and product managers who don't directly interact with the costs of technical debt but directly experience the benefits.
> Inexperienced devs, not taking enough time to understand problem, not spending enough time brainstorming the problem
Requirements changing is also a big one. 4 years into one project that I had created and maintained a "one to many" relationship changed to a "many to many" in some of the tables central to the system. Something that I had clarified wouldn't when originally designing the system.
I don’t believe we can equate jonesing for a fresh hit to writing well intentioned but ultimately difficult code or the product of inexperience.
Besides which, it trivialises the seriousness and the tragedy of addiction. If you think your tech debt is similar to that you are well equipped to quit your job and work elsewhere. An addict does not have such an easy choice, they can’t leave their addiction to someone else and move on.
You choose to take the payday loan or hit up a shark, addiction or no.
The point where I concur is when you gamble the short term against the future despite all evidence that you will benefit from spending time to save time.
all analogies and metaphors are in some ways inexact - indeed the whole concept of technical debt as a thing correlating to real debt is fraught with edge cases that don't exactly line up. Nonetheless if the metaphor has some utility then individual expressions of it such as mine will often have some utility - hence your concurring on a single point while finding other implications problematic.
The victim in that metaphor is not an individual coder who can go work elsewhere, it is the org who takes on technical debt and sometimes can't function at all without the code in question.
> I don’t believe we can equate jonesing for a fresh hit to writing well intentioned but ultimately difficult code or the product of inexperience.
That's not what "technical dept" is. Choosing a quick solution instead of a proper one that would take more effort is. I've made that conscious more than once myself: to hit a deadline, or because I didn't know if the MVP would even survive long enough to worry about code maintenance.
Poor quality code is debatable. What seems like a great architecture and clean design to one, may be regarded as the worst thing on earth by another. Code quality measurement is often very subjective and it's difficult to come up with objective criteria. For example, someone coming from embedded world will be appalled by some OOP designs, ridiculous usage of so called "patterns", etc. And the same thing will be regarded as "clean" and high quality by someone else.
Totally agree. In my opinion, with this type of MVP strategy, work done should always be in the service of something else. Initially, an MVP is created to test out ideas and also to learn what structurally it takes to build your product.
After the MVP, you may wish to add more features or support more users, it is at that point that you can assess what is debt and what is not. It's not a matter of "cruftiness" but a matter of what's holding you back from adding additional features or supporting a larger number of users.
If we frame all work as doing something that will deliver some sort of value (be it customer value or unlock the potential to add more features to add customer value), then it becomes a more constructive conversation about work that needs to be done, debt or otherwise.
I'm not a fan of just looking at whether code is crufty or not to determine if it's "technical debt". If you look at code long enough, you'll find something you don't like and want to rewrite, but it has to be in the service of some other goal or else you'll just become an architecture astronaut building wonderful things that don't provide value.
Legacy code isn’t tech debt either: it was right for the time...
Edit: as an example that comes to mind, a bank running on COBOL doesn’t have tech debt unless their COBOL engineers are aware of their missed opportunities. If they think COBOL is no longer fit for purpose, it’s not debt, it’s time to reinvest.
Should there be an analogy for depreciation? Some of the "best practices" in my current C++ shop are in direct contradiction to the Law of Demeter and related philosophical stances in Object Oriented programming (particularly ones espoused by Martin Fowler) despite the fact that my current shop advocates for OO. Neither the old position or the my current shop's position is so bad that it's not viable, though some hotheads in the field act as though this could be the case.
A lot of the churn seems to be arbitrary and driven more by fashion than solid first principles. (Which would include cost-benefit analysis, for which there is often insufficient data, and for which no one wants to pay the cost to gather the data.)
Yeah, I see too often devs shrug off their "technical debt" when what they're really describing is simply sloppy work. These economic metaphors always break down when you challenge the idea that devs/dev managers are always acting in the best interest of the product. I've seen teams cheer their "velocity" because they're closing lots of issues, when most of the things they're focused on shouldn't have even been an issue in the first place.
For the cases like legacy code and turnover, perhaps "tech depreciation" is the more appropriate economic term. You didn't really borrow against the future to have more today, rather something naturally decreases in value over time.
Right. Technical debt should not be understood on its own but as part of a larger "technical balance sheet" metaphor: technical depreciation, technical amortization, and so on.
Where I work, we actually do use technical debt to cover unknown defects as well. So if a person writes bad code, or if we have some legacy code that has suffered from rot, we'd call that technical debt.
I think there's a lot of value to be found in separating the known debt from the unknown debt though, and giving it a name. I might have to start doing it.
"Legacy code isn’t tech debt either: it was right for the time." really? Isn't that what technical debt is, solution that no longer fits the current problem?
If the code still fit the problem then you wouldn't need to pay a cobol programmer. If its hard to change the code because its too expensive to change, then you have tech debt.
But it's only debt if business decisions mean you must change it. I've seen far too many programmers crawl through legacy code that need not be replaced (it does its job) merely because it's using an outdated technology. But if there's no business justification for that, I'd argue it's not debt at all. There is absolutely nothing wrong with some ugly COBOL code doing its job with no current need for modification. That's not debt. Debt is something you have to pay back.
Great comment, if you allow I would summarize by rephrasing one sentence making it shorter:
What you have instead is knowledge debt as engineers turn over, and the once intuitive and elegant parts of the codebase are utterly alien to the new hires who learned their craft in a totally different way. This isn’t tech debt because engineers can’t intuitively anticipate the business’s future.
I count knowledge debt as a superset. You can’t determine the state of a codebase without first knowing the decisions that were made when it was being written.
This, and I will accept being wrong, is why I think we see code we didn’t write and immediately see how we would do it better. What we’re doing is sending ourselves back in time and replacing their shoes with our own. They didn’t know what you know and you have no idea what they knew.
A potentially effective approach is to introduce an immutable decision log. We have these people in many other fields - they record the decisions made and they become part of the overall reasoning.
I reckon anyone in a position of power should have an independent decision log. Tech architects, CTOs...
> Legacy code isn’t tech debt either: it was right for the time.
Doesn’t this assume that code as written was of the highest quality? Under tight deadlines or misaligned priorities code quality suffers. Even if it works or “is right” doesn’t mean it wasn’t done to a standard that maximizes productivity.
> Legacy code isn’t tech debt either: it was right for the time.
The time it was written often isn't when the technical debt was incurred, when maintenance to keep it right, technologically, for the changing time was deferred, that's when a decision was made to incur a higher price later to save now: that's when the (technical) debt was incurred.
> as an example that comes to mind, a bank running on COBOL doesn’t have tech debt unless their COBOL engineers are aware of their missed opportunities
The existence of technical debt is unrelated to awareness of the deferred costs. “technical debt” isn't a statement about the existing technical staff’s perception of the system.
>What you have instead is knowledge debt as engineers turn over ... This isn’t tech debt
You know, if there only were a way to pass knowledge from one person to another aside from directly talking to that person. Perhaps if we had means of preserving knowledge, and even making it a part of the job description... But alas.
Joking aside, not writing technical documentation and product design docs is very much technical debt.
A lot of information that is lost with turnover is information that could have been written down and passed on.
Documentation is not jut code comments. It is:
-A solid bug tracking system, with clean and reproducible bug descriptions
-A version control history, where every change has a why in addition to a brief how, and which links with the bug tracker
-Well-written tests, which not only verify how a system works, but showcase how to use the APIs.
-Documentation for tests explaining why this input was chosen, what kind of output is expected, and why it was written that way
-High level comments in code (especially libraries) explaining how the API should be used
-High level documentation explaining the structure of the code that is checked in, and lives with the code. The documentation can be in Markdown or a plaintext README, but should be there
-A "code map" telling you what the code in various directories does, and where to look for things.
-And, most importantly, a lot of design docs - for every feature and algorithm. "We are solving this problem. Here are the alternatives considered. Here is what we are going to do and why. Here is how we'll implement this. We used ideas from this, this, and this paper".
Yes, all of that is needed. The price you pay is engineering time. But if you take on the debt, your new engineers will have Hours Of Fun[1] trying to figure out, say, whether the imaginary nanometers[2] model parameter increasing as a result of their change is a good or a bad thing.
What I am saying is documentation debt is technical debt. Documentation is just as important as the code. And having it does help[3].
You might think that your code doesn't need all of that. And that's fine; but that means you plan on never paying that debt. Maybe you plan for your startup to fail/sell out in a few years. Maybe you plan to leave that job. But anything long term means you have to write things down.
And if you haven't been doing so - now is as good time as ever to start.
[2]True story. There was a good reason for why the distance parameter had complex-valued nanometers as units; it was an elegant way to do improve a model while staying backwards-compatible. It was written down nowhere, and have fun googling "complex nanometers". They only exist in that code, AFAIK.
There is information which is readily transmitted through non-experiential means: tacit knowledge. Books, lectures, conversations.
There is information that must be experienced, drilled, and committed to habit and muscle memory: tacit knowledge. Drill, practice, experiment, coaching. Often 1:1 or at least 1:few.
It's highly arguable that individual knowledge does not become cultural knowledge until a means of transmitting it from one generation to the next becomes both established on institutionalised. It need not be a large institution, but it must have persistence.
Within any knowledge domain, the explicit knowledge transmits more readily than the tacit. With time and complexity, I suspect a situation much analogous to Amdahl's law is reached, in which generational knowledge transfer comes to be dominated by the tacit component -- it is harder, slower, and more expensive to transmit, and tends to irreducibility.
Technical cultures which become or incorporate a tradition of shared stories and lore, as Unix and Linux have done, seem to achieve this better than others.
And that neglects issues of semantic and cultural drift which affect written texts.
Though I'd come to this conclusion some years ago, I'm finding reading Mortimer J. Adler's How to Read a Book that this is a significant theme of that work.
> I heard a recent conference talk where it was referenced as “Revenue Generating Code”, rather than “Legacy Code”.
The two categories overlap, but not all code generating revenue fits in the traditional scope of “Legacy Code” and not all “Legacy Code” is generating revenue.
> I found it be a pretty accurate summation that gives some perspective from the other side.
It’s not a perfect overlap. I agree with you there. My personal experience has led me to believe that “Revenue Generating Code” is the most common variety of technical debt.
Technical debt that does not generate revenue in any way is often far easier to replace/refactor.
I look at legacy code like this:
- Could we remove this from the product?
- Could we replace this part of the product with a totally new system?
- Are we OK with potentially introducing subtle bugs in this code as we work through refactoring it.
If the answer to the above questions are “No”, it’s likely revenue generating code (or part of some compliance process).
In this case the “other side” is non-technical stakeholders who don’t inherently understand the importance of resolving technical debt.
Technical debt is a great metaphor exactly because it suggest the suboptimal code is due to a deliberate strategic decision.
If you just say that some code is bad you will get unproductive reactions like people getting defensive, trying to assign blame etc. Calling it technical debt allow you to call attention to an issue while avoiding blaming anybody.
It was right for the time, and required maintenance and updating; I consider skipping said maintenance to be a debt of the time it would take to do that maintenance, with the interest being the time spent working around it as you work on other things. In a lot of cases, the interest rate is pretty low, and is worth it.
The goal of an MVP is not to write crappy code to get something quickly out of the door but to not build every feature into all details to get all of the parts together and see how they work first. It's about the code you don't write, not about the quality of what you do write.
This is great and well articulated! Would you agree legacy code can often turn into tech debt by virtue of being neglected / unmaintained? The debt here could perhaps be measured by operational overhead
I cut my teeth as a developer on the Windows operating system. Because we were a platform, we had learned that when we shipped an API it became more or less permanent. So technical debt was a huge part of our planning and thought process.
When I joined Netflix I was assigned to add DASH support to the Silverlight web player. I spent weeks working on the new architecture, refactoring the streaming code, etc. One day my manager stopped by my desk and said that "I needed to wrap it up. It was taking too long." I was still weeks away from being done. I explained to that I was cleaning up a ton of technical debt and that's why it was taking so long.
He explained to me that they had different values on his team.
1. Most code will be rewritten every 2 to 3 years.
2. For code that doesn't get rewritten often, preserving battle tested code trumps paying down technical debt
Just remember that technical debt doesn't apply to all software.
>> I explained to that I was cleaning up a ton of technical debt and that's why it was taking so long.
> The only point here that jumps at me is that the manager came a few weeks to late to explain the development values.
Well...(and I'm not having a go at rsweeney21). They were asked to implement a feature, not to spend ages cleaning up technical debt, and being "was still weeks away from being done" would be an alarm bell for me as a senior dev/manager. If the feature couldn't ever possibly be implemented due to technical debt then GP should have raised this with their manager far earlier rather than just going ahead and fixing this silently. If the feature can be developed and released, despite technical dept, then go ahead and point out to your manager (as soon as possible) this is going to be horrible for the dev(s) working on the next iteration, but the chances are your manager will likely insist you get on with the feature. Communication is at fault here.
We all work in projects that have technical debt and we'd all love to go back and fix these things. But you also have that obvious tension which is the fact that you also need to release features which have been identified by the business as another positive cash-income thing because you're competing against other services - and sometimes these features are very time critical.
Sounds like GP poster seems reasonably experienced (working at MS, Netflix) and should probably know this (unless there are other mitigating factors not included in your comment)
Yeah, I feel like this one was on me. I was new to the job and wanted to impress my manager. I could feel the refactoring getting out of hand. I should have had a conversation with my manager to ensure I was going in the right direction before he felt the need to tell me I was taking too long.
Don't be too hard on yourself; it takes two to tango. You are your manager's work, as much as your code is yours. Unless you lied about progress it's a miscommunication with no right or wrong.
This is central to Fowler's point, which is that technical debt only needs to be paid off if it gets "activated" a lot - this is where the metaphor breaks down, because typical debt accrues over time.
If you never touch code again, but that code was rushed and is ugly, there's no good reason to pay off the debt.
Similarly, if the code's getting thrown out (and if you design code to get thrown out), there's less reason to prioritize cleanup.
That's often overlooked, yeah - if it ain't broke, don't fix it. This goes against a lot of people that read code for the first time, myself included.
A recent example: React introduced hooks, which effectively means that there's a new de facto standard way to write components and you shouldn't be writing components as JS classes anymore.
The kneejerk reaction is then "We need to rewrite all our class-based components to use hooks!", which can take quite a long time (and it's tedious, boring busywork).
Luckily the React documentation on hooks itself [0] basically tells you to not bother unless you're actually going to work in that component anyway, and even then you should make sure everyone else in your team understands them; classes are fine, it'll keep working for a long time, resist your kneejerk reaction.
Yeah, I try to align tech debt cleanup with 'touch points', since that's generally an indicator that the code needs work done on it anyway.
It's sometimes controversial to say "We should not prioritize tech debt" but I feel pretty strongly that it's true. We aren't paid to write beautiful code, we're paid to write code that gets the job done. If code is getting the job done, even if it's fragile, our job is done. Only if that code is impeding new work should we prioritize it.
No, this is still a payment against technical debt. From the top of Martin Fowler's article:
"The extra effort that it takes to add new features is the interest paid on the debt."
If adding a new feature starts with, "just throw out all the code and start from scratch", that's a much larger cost over the long term than building code that's easier to maintain and refresh.
With very few exceptions code that is "easy to maintain and refresh" over any appreciable time horizon (say,longer than 2-3 years) requires the kind of engineering effort that the modern world of "CS trivia is good interview material" generally derides. I'm not even talking about "real" engineering processes as might be used for space systems or whatever; I mean just basic requirements analysis and technical documentation.
The cost of rewrite at that point is typically less than cost of straightjacketing the existing codebase to support all new requirements, otherwise it is less likely to happen. So, defaulting it is.
(Mind you I'm not fully sold myself that 'technical debt' is a useful analogy at all)
You can't fully make that distinction until it happens.
Sometimes you can say "I know this will need to be rewritten later" and be pretty sure you're seeing technical debt be accrued.
But other times you have no idea. It might be a tool that was intended to be used for a few weeks, but turned out to be incredibly useful and lasted years. Had you known, you might have written it better if the first place.
Technical debt isn't like financial debt in that there's nobody with an accounts sheet telling you that you owe them anything from the start. You don't necessarily make a deal with anyone. So you can't judge its existence in the same way.
Or the target's moving fast enough that it's inefficient to build a gold-plated solution when a quick hack will get you out of trouble for now.
Research code is much the same. Your code doesn't have to be great (or even OK, to be honest) as long as it runs well enough to provide the results you need for your paper.
A technology no longer being supported (e.g. Flash, Silverlight, IE) is not technical debt though, it's external influences.
Likewise, rewriting a front-end in the framework of the week doesn't have to be because of technical debt; Angular 1.x was fine and it's still fine, however you'll see a lot of upgrade or rewrite projects happening right now not because of tech debt, but because people simply don't want to work in Angular 1.x anymore, or an employer can't find competent people that want to work with it anymore.
Likewise, COBOL applications aren't bad because of technical debt per sè, it's just become very hard to find people that can maintain it.
Nice straw man. Nobody said "all code will be rewritten every 2-3 years". He said "most code" will be rewritten every 2-3 years. Also, like I said, these were values on his team. We were a front-end application and every 2-3 years Netflix would rebuild their player UI which usually involved a complete rewrite.
I'm just saying that if you have it in your head most code is going to be rewritten every 2 years, it's understandable if you don't bother making things easy to work on, and just hack it together.
An argument that "eh, most code is rewritten anyway, who cares" is what I'm reacting to, and have seen countless times myself.
But I don't want to say there is never a time someone will say, "this code is going away, don't spend too much time fixing it now" and it's really really good advise.
Its funny that you mention specifically this related to UI code. I was thinking about it after your first comment, and basically every product that I've worked on that fit this style of development has been UI related. And indeed, most UI projects have had almost complete rewrites every 2-3 years for one reason or another.
The ones that didn't were effectively stalled projects. This makes me think that technical debt really can be thought about in a completely different way for frontend code in many cases.
A few years make sense for UI for consumer products. The evolution of displays has been tremendous in the last decade and a half. Desktop screens have doubled in size and resolution, while mobile and tablets just appeared overnight and when from nothing to where they are today.
Products have changed, but also UI design tends to be driven (or at least influenced) by marketing. And marketing trends change regularly driving a need for markup and styling to be revisited.
In theory, this is just marketing, but in practice if often turns into much more.
Well, that sounds about right from my experience, for some type of code and organizations.
The original developers have left after a few years, newcomers don't understand the system and may not even try to. It will ultimately get rewritten in newer technologies and frameworks, that happen to be nice for their resume, whether it was obsolete or not.
Indeed. 2-3 years is short, that way of thinking could discourage doing things right in the first place because ‘it’ll be rewritten it in 2-3 years so it doesn’t matter’.
Sometimes it's because developers just want to change tech to enhance their cv or keep themselves entertained. That's scary.
Sometimes it's rewritten because it's so unstable because debt wasn't paid off but total cost of writing it twice is much more than paying off as you go along. That's scary too.
Within consumer tech, it's more like "Plan to throw it all away; you will anyway."
I used to measure code lifetime in half-lives at Google: the half-life of most of the code there was about a year, meaning that after a year, 50% of your code will have been deleted. It's pretty accurate: by the time I left after 5 years, about 97% of the code I'd written had been deleted. Ironically, I'm told that my one contribution (after 10 years) that still exists is an attribute-renaming CL that I wrote over break at a team summit; basically the whole team agreed it was a good idea, we would never have a chance to fix the problem later, so I just went and did it before the framework got too entrenched to change. Meanwhile stuff I slaved over for months, sometimes even stuff that was directly sponsored by a VP (who is no longer there) or got commendations from the CEO (who is also no longer there) was gone within 1-2 years.
Often times if something needs to be rewritten, it means that the feature became popular enough that it needs to scale up. It is not a bad problem to have.
Even well written code might not scale or be flexible to changing requirements. The feature could even be removed. The effort it took paying off tech debt prematurely would have been a waste.
Expecting code to be replaced every 2 or 3 years is nervous-making for a couple of reasons.
First, new code is always less reliable than old code, so -- all other things being equal -- devs should lean toward keeping the old code rather than replacing it.
Second, code should be architected so that any changes required to adapt to changing conditions should be limited in scope. The bulk of code for most projects should only very rarely need changing to adapt to changing conditions.
You're presumably used to operating in an enterprise environment with O(1000) customers and O(10) developers, not a consumer environment with O(100M) customers and O(1000) developers.
Reliability issues at large consumer tech companies will usually be caught by the QA/canary/SRE process. If they're not, you'll hear about them soon enough with 100M users banging on the product, and then just do a rollback and fix it on a more leisurely pace. Consumer expectations in non-tech industries have gotten so low that you can burn down whole towns, poison people, and leak millions of records of personal data with few consequences to the company; not being able to access your favorite video for 15 minutes is comparatively minor.
Also, the type of changes in market conditions consumer companies face are usually not those you can architect around. They include things like "We are no longer shipping DVDs to customers; we are streaming content online", "We no longer write web software, we're a mobile-first company", and "our business model is no longer selling personal data, it's payments". There is no architectural fix for "your product is canceled".
> You're presumably used to operating in an enterprise environment with O(1000) customers and O(10) developers, not a consumer environment with O(100M) customers and O(1000) developers.
I have a great deal of experience with both environments.
> Is it really? It's another way of saying that conditions in the tech industry change every 2-3 years, which isn't all that controversial a statement.
I'd say it's an outright incorrect statement, fads come and go but the revolutions (like the move to web apps) are rare and take and take decades to complete. If everything around you changes every 2-3 years then you're hanging around fashionistas not engineers.
The example at hand was netflix... The bulk of the platform seems pretty stable to me. If they are rewriting their best code every 2-3 years that'd be a red-flag to me.
I worked on Google Search for 5 years. You would probably say Google Search is pretty stable - at least, when was the last time you noticed it go down? The half-life of code I worked on was about 1 year. Over a 2-3 year period, 75-90% of it would be gone.
We did things like incrementally rewrite a million-line binary from C++ into Java while it was running, or completely change the indexing system from batch processing to continuous updating, or grow the number of documents indexed from about 80B to 1T. There's a huge iceberg of development that you don't see, much of which has to do with scalability (you generally have to rewrite every time a key metric grows by a factor of 10) and much of which is experimental, trying out new features to see what resonates with the userbase.
GWS (Google Web Server). Powers websearch, responsible for rendering everything you see when you run a search. It used to have a Wikipedia page but basically everything on the Wikipedia page was wrong so the admins eventually deleted it (edit: apparently it's back. Basically everything on the wiki page is still wrong, but at least it doesn't claim GWS is an Apache derivative now). It's the oldest continually-pushed binary at Google (originally written 1999 by Craig Silverstein; at least when I left it still had code in it from Craig, Marissa Mayer, and Sergey Brin). It's changed programming languages twice (from C to C++ around 2005, and from C++ to Java etc. around 2010), and when I left was a frankenmonster of C++, Java, and two proprietary DSLs. My understanding is that it still exists, though most of my friends at Google have since moved on to other teams.
Thank you. Can you talk about the reasons that necessitated the rewrite from c++ to java? I didn’t realize Brin wrote any code that wasn’t python either.
I personally wasn't a big fan, I liked C++ (and Python - there was an embedded Python interpreter in 09-10 that was a casualty of the rewrite, and IMHO Python was a lot more productive than Java for experimentation).
There were some pragmatic reasons though. It's very difficult to multithread C++ correctly, while Java at least has a proper memory model and thread support (note that this was before C++11; at the time C++ had no standardized memory model at all). Debugging core dumps in production sucked. Most of the newer parts of the company (GMail, Docs, Google+, etc.) were written in Java, and the rewrite let us share code with them. Compile times sucked, and Java let us pluginize the architecture, load code at runtime, and build & push each component team's codebase independently (as well as shut them off independently if they started crashing).
I wonder how much of that is due to using the current fad tech... my longest running major piece of code (untouched 15 years+), I implemented in PL/pgSQL. Rock solid, well understood technology, isolated from the current hip web framework that is probably calling into the views. Somehow I think the technology choice is a big part of what ultimately made the program a great investment for the business.
> Just remember that technical debt doesn't apply to all software.
I'm not following. You still need code that could be rewritten and deleted every few years. Any code that couldn't just be deleted as needed wouldn't fit that plan.
So whatever forced code to outlive its welcome is technical debt in that scenario. Examples could be code that used outdated middleware, maintained arcane ETLs to keep data accurate, consumed said data in unconventional ways, code that wasn't repackaged and deployed regularly, etc.
How many bosses have you had that suggested paying off technical debt? For me it was always a never ending "next month, after we have rushed out this new feature".
Some, but usually the rule is to not let them know - if they don't understand the problem, they shouldn't be burdened with deciding when and how the problem should be solved either.
[I recall posting this here in the past, but worth re-posting.]
The "handy rule" from the book Team Geek -- the newer edition is called Debugging Teams[1]. It's from chapter "Offensive versus Defensive work":
[...] After this bad experience, Ben began to categorize all work as either “offensive” or “defensive.” Offensive work is typically effort toward new user-visible features—shiny things that are easy to show outsiders and get them excited about, or things that noticeably advance the sexiness of a product (e.g., improved UI, speed, or interoperability). Defensive work is effort aimed at the long-term health of a product (e.g., code refactoring, feature rewrites, schema changes, data migra- tion, or improved emergency monitoring). Defensive activities make the product more maintainable, stable, and reliable. And yet, despite the fact that they’re absolutely critical, you get no political credit for doing them. If you spend all your time on them, people perceive your product as holding still. And to make wordplay on an old maxim: “Perception is nine-tenths of the law.”
We now have a handy rule we live by: a team should never spend more than one-third to one-half of its time and energy on defensive work, no matter how much technical debt there is. Any more time spent is a recipe for political suicide.
> If you spend all your time on them, people perceive your product as holding still.
This is precisely what bothers me about the vast majority of software out there. It's hard to find software that doesn't suffer from what I call, for lack of a better name, "forced evolution": the devs keep fiddling with the UI and adding new features of dubious usefulness. Meanwhile, the quality slowly, but surely deteriorates: the software gets bloated, runs slower, becomes harder to use, etc.
This was how I said it, buried off in the documentation of an open source project:
I'm not a fan of the way most software is developed, where features accrete on features in endless succession like barnacles attaching to the hull of a ship until there's more barnacles than ship. I think most developers don't spend enough time on quality, on refactoring, or on cleanliness, and I think they don't weigh the costs of every new feature (and every piece of old compatibility cruft) as heavily as they should. I'm also not a fan of the kind of personal rigidity where one never questions one's own past decisions and thoughts. I regularly make stupid design mistakes, and I don't want to have to live with them forever. Software projects should at least try to value simplicity and purposeful design, and try to avoid the Second System Effect.
> If you spend all your time on them, people perceive your product as holding still
I've always found this hilarious... we spend thousands on hiring developers and one of the primary goals is to find a well rounded person who is okay doing maintenance tasks and ensuring proper code quality and knows how to write effective tests. A short time later, I come on to a project just for a status update and find the developers cutting tests and maintenance tasks at the orders of the PO who then complains that tickets are getting larger and larger.
Devs should not ever say the word "test" to a PM/PO. If the question is, "why isn't the feature ready?" the answer is "because the work took longer than I expected." If you say "because I've writing tests" you are sabotaging the team. Tests aren't any of the PO's business, they are just part of the software that supports the feature.
This is an ideal reality. If the PO is at your desk every day to ask why the thing isn’t done, most developers are eventually going to drop everything non-essential to the functioning just to get them out of their hair.
Sometimes I think that offensive work is always prioritized ovee defensive work. And then execs and stakeholders think when the code is a mess and a simple change takes a day and a half it's the developers fault. And it's difficult to say the least to convince stakeholders to defer a feature for a little bit of refactoring.
This why I think it might be useful to allow the stakeholders and pms to decide the % of time spent on defensive work instead of what defensive work to do. And then you have a specific metric on technical debt(we budgeted 30% for defensive work and we've only spent 10% that's a 1000hrs of technical debt)
Something to specifically point to when a stakeholder asks why is this taking so long or why is the application buggy. And ideally this caused a company wide increase in the allotted time for defensive work.
Bonus:It will also give developers who like working on clean projects a number to ask when being hired.
That would solve a problem I saw at a previous job: new features would be described as technical debt, so they could take up tech debt time at the expense of _real_ debt.
My team spends 80% of our time paying down tech debt...by building a new system that will let us throw away the old system later this year, I mean next year, I mean in 2 years, I mean when the last zombie client with no staffing migrates off of it.
I think the OPs paradigm holds. If the Code Refactoring has a huge performance impact then it's Offensive work, if it has no impact, then it's defensive.
Not exactly, because sometimes there's refactoring which has 0 customer benefit. Sometimes it doesn't improve the UX/latency/etc. Sometimes it just makes the codebase easier to maintain, the point of what the OP is saying is that this work is hard to justify. If you spend an entire year making a codebase "easier to maintain" but don't deliver anything tangible that is all defensive work and not offensive work and there's no reclassification which can solve that.
Even worse, introducing shiny features in half-baked bug ridden way gets credit, the person who gets to fix all the bugs and possibly reimplement the whole thing correctly gets no credit.
I have a similar idea. I call it the "horizon". There is a certain amount of time where you can go off and do stuff that doesn't result in stake holder visible change. How much time you've got depends a lot on the organisation, but I find that 2 weeks is a good rule of thumb. If you sail over that "horizon" your stakeholders lose sight of you and start to get worried. Ever day you spend over the horizon increases the risk of the stakeholders taking drastic action (from pressing the big, red "stop the presses" button and changing direction all the way to cancelling the project). The key is always to deliver value within that horizon. As long as you keep doing that regularly and predictably, usually nobody will care what else you do.
This is why I think it's good for technical debt to be partially accounted for by development management types - write down how much offensive and defensive work is required to sanely complete a project or just stay ahead of the curve and let the total of those two parts be reported as the cost of the feature - it's a benefit to the long term health of your company.
Gotta love it, this is the same dumb ideology Agile spews. Develop develop develop, forget about fixing issues - They'll magically go away... It's clear that these people do not work in Enterprise environments
I've been thinking about the difference between video game software and most CRUD software. With most CRUD software you are expecting future changes, so you have to be more conscious about tech debt. But with video games (at least older ones), once you ship you're more or less done.
I wonder if that gives room to take on extreme debt at the last minute before shipping a game. An example might be when Steve Ellis added multiplayer to Goldeneye on N64 a couple months before launch. Since they were going to ship a finished product, he was probably able to build it much faster without having to worry about code quality.
One thing that's interesting about this insight is that many games are moving away from this model, with regular updates and continued development. This doesn't invalidate your point, of course, but I also wonder how big of a mental shift this must be, because historically, when a game was done, it was done.
That's a great point. One of my favorite games, Enter the Gungeon, had plans for a major release of DLC, but they canned it and replaced it with a much smaller scope of free updates because they had so much technical debt that they couldn't attempt the big DLC release confidently.
It's mostly restrictive on the side of game design. Once your goal is to support a large quantity of software features you automatically fall into the mold of a style of gameplay that will minimize impact on feature development.
In the classical model of ship-once gamedev, certain big-ticket software features tend to be thrown out when the designers are in control, because they're so costly to get right and don't surface as much immediate impact:
* Large-scale persistence(just store a minimal representation and instance that)
* Online features(too much synchronization logic - it hinders almost every other feature!)
* Complex AI and procedural simulation(too much chaotic behavior, limited ability to control outcomes)
The transition we've made has, in essence, been to construct microtransaction service products that do support these more complex features: the game is a platform for the sale of assets, and it is massively online with total persistence and sometimes even robust AI. And when you start doing that, your ability to revise the design diminishes substantially. For the most part, you can't take away assets and features that are already there without a substantial player backlash, which means that games of today tend towards accumulating their way into an incoherent maximalist experience with no particular focus or intended playstyle, all of it following templates intended to encourage monetization:
Make sure the players can customize their character cosmetically, and give them a huge array of skills and powers, but nerf all the unique aspects of these powers so that the game will play roughly the same regardless of what you choose. Present them with opportunities to draw in their friends, and add hooks and incentives to come back repeatedly and to check the in-game store for deals. Provide play opportunities for all segments and demographics of players(following the lines of the Bartle player types).
When this model works, it does extremely well in the mainstream(Fortnite basically ticks all the boxes and is outstanding in its ability to progress as a live service) - but it has the downside of being a compromise for everyone. It makes the game too Byzantine in its complexity, too restrictive in its customization, too carefree in its changes to competitive dynamics, too willing to let annoying minor bugs persist - to say nothing of the cost and complexity needed to pull it off.
So I think there's definitely room for more of a spectrum of products, from the mega-blockbuster down to tiny boutique indie games. But the marketplace is still catching up with the idea of providing full service, and a lot of games are still released with the focus being on developing a franchise or on addressing a specific genre. In many cases the capital requirements needed to do services crush that idea out of the gate.
Well, online games that are basically sold as partially or fully a "service" (e.g. Fortnite), and game engines that are primarily marketed to developers of games are, but I'm not sure I agree that applies to games overall.
It's certainly only a part of games, but this also happens outside of games like Fortnite.
For example, one of my favorite games is Celeste, a fully offline, single player platformer that's about 18 months old. They've had a lot of releases[1], and at least one of them sounded like, to me, like a decision that was made purely because it was easier to implement technically. (I won't get into it here because it's pretty off-topic, but if anyone is interested, I can elaborate. It's "The Prologue Dash cutscene is skippable") There's also one more major content update coming that appears to have new mechanics, and it's already late. That said the game is amazing and I don't fault them for any of that at all...
IMO technical debt does not discern between different types of software. If you make a mistake (or are not 100% correct) you will need to fix it unless you are willing to provide your customers less than perfect software.
Maybe an appropriate way to look at this is that the developer working on the feature could only need the company to take $100 in debt, while another developer needs the company to take on $100,000 in debt for the same work. Typically this difference, unfortunately, does not reflect in the individual developers' salaries.
No, the customers only care about working software, not maintainable, perfect software.
Technical Debt is about taking shortcuts to have something working now, in exchange for more work maintaining and evolving it in the future. If the software doesn't work, it's not debt, it's a bug.
If you are not changing the software, it doesn't matter that a list is hardcoded instead of parameterizable, that the modules are tightly coupled and can't be reused, or that the build must be done following a particular set of incantations in a specific machine. If you are, these are all examples of debt that will come back charging interest.
Properly planned and managed, technical debt is a tool much like actual debt (a loan) that can accelerate your growth and your time to market. Thing is, most teams don't know that they are taking on a debt and don't realize how much interest they are paying each day.
I think the path toward optimizing software for (a) maintainability (b) completeness and (c) bug-free are not necessarily mutually exclusive. I think in general software is fighting all of these battles simultaneously. And so by definition software that is incomplete (ie. without all the features) has the possibility (probability?) of being buggy. Otherwise if it's complete why add features?
Given what you knew at the time you created a definition, and per that definition you could have written 100% maintainable 100% bug-free 100% complete code. It's in uncovering your lack of/ incomplete understanding of your problem that you may need to go back in to that code. Even though at the time it was considered 100% bug-free, complete, and maintainable.
100% maintainable software is NEVER a goal. A hard coded list that will never change is better than a configuration file that needs to be installed someplace and loaded as runtime. If you are right that the list will never change you have made the software more maintainable as you never have to deal all the code to load the configuration from whatever path the user has installed it to. If you are wrong though you will have a lot of work tracking all the releases and what is hard coded in each until you retrofit a loader.
This is the type of choice you need to make upfront often when designing software. If you get it right your system is more maintainable. If you get it wrong your system is less maintainable. Sometimes you don't know in advance which is right.
Technical debt doesn't make the software worse, it makes the codebase worse which degrades the ability to maintain and add features in the future (which is paying interest on the debt, so to speak).
So, in the context of oldschool console video games that ship and then end development permanently, it's absolutely a reasonable decision to add some technical debt if it means fitting one more nice-to-have feature in before launch. Because you don't pay interest or pay down that debt ever. The code is shipped and then it retires, presumably.
The days of fully offline games are long gone though. It doesn't mean that game studios take on less ambitious projects though. Most games are superglued together to get over the finish line and shipped with a massive day-1 patch.
Probably wasn't even true for GoldenEye. They used the same engine for Perfect Dark; sure the assets and game scripts would be changed but the multiplayer code had to be ported and cleaned up I'm sure.
The key defining characteristic of technical debt though is our inability to accurately estimate its true cost. If we could frame the question like you present it the battle would be won
Under schedule pressure that's certainly true, but subsequent products will often start by taking a skeleton of a previous one. In this way tech debt could live on for many additional games unless it's addressed.
Is this actually true? The engine lives for multiple games, and the tooling is often reused as-is, but do games start with existing projects, or do we add new technical debt through the cleansing baptism of copy-paste?
In 1982: all games were written from scratch. You might copy the good parts from a previous game, but there generally wasn't much: poor abstractions meant the game logic and UI was mixed together (in their defense, on the limited CPUs of the day you couldn't have made proper code separation work, and of course the limited CPUs also meant you couldn't create such complex systems where it mattered.)
As games entered the mid 90s games started to get releases, and game engines came out. However each release was treated like an all new project even if some code was reused. Games would customize the engine in ways that wouldn't apply to the next game. Bugs in the engine would be worked around by ensuring that part of the game could not be reached (thus resulting in games that you could never get the 100% complete stat as the engine knew you didn't go through some locked door without a key). Some of the core engine was reusable, but for the most part the engine and game were tested as one.
Sports games tended to be the first to figure this out: every year was a new release of essentially the same features, but different player stats. As such it is becomes obvious quickly the game engine maintenance is a cost, there were few new features (until the next console offered better graphics). Most other games were latter because part of the new game was engine changes to support it, so it was hard to find a stable core to the engine.
Gradually games have seen more and more value from separating the engine from the game. As this has happened engine changes are no longer for one game, but for the possible use by all future games. Part of this is game designers have figured out the right abstractions, things that previously would have looked unrelated and thus change different areas of the engine are now seen as variations on a theme if you squint just right - and we know how to squint
Of course there have always been, and probably always will be one-off games written like 1982. When your game is small this is okay.
Different game companies have reached this at different times. Infocom had mostly figured out their reusable engine in 1982.
> But with video games (at least older ones), once you ship you're more or less done.
Technical debt doesn't care about your release status and can kick in long before you reach the finish line, which I suspect happened with Perfect Dark and many subsequent rare games. How many game delays and cancellations have technical debt as the root cause?
In enterprise projects I've seen enough over engineered crap added at the very start of the project that hampered development for the entire lifecycle.
Yes, I think a more general way to put that would be to say the lifecycle of the software affects how important these things are. It’s not debt if it’s not going to slow you down in the future.
It’s common to cram in some debt before a release in the applications world too, with the understanding that meeting the deadline is more important than finessing every detail in the implementation and that it will be cleaned up shortly thereafter. (Yes it’s possible for that part to fall by the wayside, what else is new.)
I like the metaphor of technical debt, but...this essay does not address the most prevalent obstacle with paying down technical debt, which is that the people making the decisions don't care. They are in short-term mode, just trying to get this year's objectives met and IPO or sell or get a higher up job in a different company or division. They really don't care about the long-term health of this software product. Technical debt, as a metaphor, may make perfect sense to them, but they aren't the one who will pay the interest.
Now, if you're working on open source and will be in this codebase for a decade, or this is your company or your software, it makes the tradeoffs much clearer, and that works because your incentives are right. So this metaphor works better for helping the typical developer make choices well, than for helping the typical manager/director/CEO make choices well.
this is usually a communication problem from engineering. you have to be able to describe the impact of the risk and benefits and make a business case for it. no one cares if you want to upgrade a framework. but if you are going to improve customer trust by using a newer version of something and result in less bugs / downtime it sounds a lot better
Sure. But the risk of disruption is usually actually higher, in the short term, by paying off the technical debt. In the long run, of course, it will likely result in fewer/less risk of technical problems impacting customers. But the risk of there being a disruption right now, is nearly always higher if we pay down technical debt, than if we put it off. So again, you have to have something other than a short-term outlook on the part of the decision maker.
Which does happen, of course, but not always, and not almost always either.
One of the hardest challenges of paying off technical debt is that there's no visible improvement to the stakeholder. From a business perspective, they often think it's working now, why not leave it alone?
Also, the metaphor starts to fall apart if you consider good debt vs bad debt in terms of financial investments. No examples of good technical debt come to mind, other than the benefit of shipping quickly.
>the metaphor starts to fall apart if you consider good debt vs bad debt in terms of financial investments. No examples of good technical debt come to mind, other than the benefit of shipping quickly.
I think that's where the metaphor works best. Good technical debt lets you ship sooner than you otherwise would have, similar to how a mortgage lets you afford a home sooner than you otherwise would have. Paying interest isn't ideal, but it may work out a lot better than renting. Likewise, code cruft isn't ideal, but shipping 6 months sooner might pay off a lot more than having a perfect code base can.
A pretty good talk from Railsconf a few years
back. The debt metaphor can be thought of as two axis: planned vs unplanned, and low vs high interest.
If you explain to the stakeholder that by not paying down technical debt, they're only paying off the interest, but they have to keep paying it forever, perhaps that will sink in. The opportunity cost of those interest payments is profit-generating work. By failing to address technical debt they are constraining the business.
Good technical debt... well, what's the expected lifetime of a piece of software? How long do you think it'll be maintained and growing, before it is either no longer relevant/needed, or replaced by a new product?
Over the right time scale, technical debt can be a huge win. That's "good debt".
Consider the race against time - either time to market (get a product out the door asap, for valid business reasons), or because it's currently in proof of concept stage and will either be replaced or fixed later, once it becomes a real project destined for production. In either case, technical debt is a good deal. Even if it costs you more in the long run, the time saved in the short run can be the difference between success and failure.
Technical debt it the intentional decisions you make to go faster. You can pay it off or not if you want to.
Unfortunately every project I have been on has very little of this. The real problem I see are not intentional decisions, but "If only I had known". The team that choose a framework and 3/4ths the way though the framework company dropped the product (I know a product shipping on WinCE 6 - MS dropped support for that on them 1 week before code freeze). The team that started on some new fad that seemed good only to be the first to find the issues when used in their situation (the fad is overall good, but the limits are not understood).
I've come to the sad conclusion that excessive code quality is a smell. And most attempts to head off technical debt are really just premature optimization, which is its own sort of evil.
I think you need to answer why shipping quickly would benefit the company. Shipping quickly is not an end but a means to an end. The point is the state of the market and the impact you plan to have on it, with an added constraint of some deadline.
Perhaps it’s an AR app a company wants to ship in time for a sporting event or art show, and the business has only enough runway to get to a potential windfall of revenue from it?
That being said I think for tech debt to be “good”, the real challenge must be well understood, as well as the alternatives, it must be documented somewhere and all stakeholders must be aware of it, because I’m going to force them to consider it in every prioritization meeting until it’s done or not needed any more.
Personally, if I have a client that’s constantly questioning my expert analysis, the thing for which they entered into a contract with me, I would question whether that relationship should continue. Seeking info is fine and expected; shutting it down is not.
That is the benefit, shipping quickly. It is exactly the same benefit as financial debt: It lets you start working now.
Technical debt is good if what you can create quickly and dirty actually improves your productivity. That is what tools can do.
In the real world if I incur in debt and my debt pays for a tool like a car-truck that lets me move myself and my things faster, I am improving my productivity. Sometimes tens of times.
Think of the difference between a sewing machine and hand sewing. Hundreds of times slower.
I can pay back my debt faster with good debt.
In software, if I can make a tool that in a quickly and dirty way is able to do things in seconds that take me hours and fatigue me, then it makes sense.
Sometimes you can even use your quick and dirty tool in order to clean the dirtiness of your own tool way faster. That is what bootstrapping is all about.
That is what compiler writers have always done. They started in assembler, then created the core of the language, and then they used their own language in order to improve the compiler.
Other writers used languages like Lisp with super slow compilation, most of the time interpreted, in order to create the core of the language in days instead of years.
You don't see the benefit of shipping quickly. For me it is one of the most important things in the world. People's salaries cost more in direct proportion of the time it takes to do something.
I wrote an essay about communicating to stakeholders the value of paying down tech debt. If I wrote it over again I'd amplify the section called "Later".
> One of the hardest challenges of paying off technical debt is that there's no visible improvement to the stakeholder. From a business perspective, they often think it's working now, why not leave it alone?
I think the article somewhat addresses this where it talks about not needing to pay off debt in areas where it doesn't need to be changed, so perhaps it's a valid argument that if it's "working now" to leave it alone. As for the hard challenge of when to pay off (or accrue) technical debt, it really comes down to individual judgements from an experienced technical leader who has a high level and long term frame of reference. The focus should ultimately be on optimizing delivering overall value to the company or mission. One thing to avoid is to have non technical managers making decisions about technical debt, since it will likely be an uninformed decision.
There's an argument that it's only technical debt if you're paying interest on it. i.e., it is costing you something: time, pain, preventing improvements, etc.
Consider a financial instrument that is interest only on a perpetual timeline, and you are not required to pay down the principle. If the cost of paying the interest forever is still less than paying down the principle, why wouldn't you just keep paying interest? Like you said, it takes an experienced, competent leader to make a good judgement about what the true costs are either way.
I think an interesting way to address this impact, is to start adding more time to each task and address it as technical debt, which is both informative and actually true.
For example after a few months of no refactoring, you start adding 10% to all estimations based on technical debt. It grows more the more refactoring is kept at bay.
Similar to how we add a certain percentage to each task for communication, bug tracking, qa, scope changes or anything else.
You can keep an arbitrary count, and perhaps even have a "technical debt" reduction amount for certain backlog tasks.
I haven't ever tried this though. But it's an interesting way to communicate technical debt (and is very close to the metaphor).
> Other than the benefit of shipping quickly.
How about the benefit of not having to pay the full cost ever if the early release disproved the business case successfully? That is a big financial saving for the stakeholder.
It's the same as buying insurance, it reduces risk and exposure in exchange for money. Ask management if they want insurance or risk and cash. You pay money to reduce risk.
It's a decision they are capable of making when put into non technical terms. They assume the risk of catastrophic failure if they dont buy insurance. (Obviously major refactoring has its own risks.)
You need to tie it to a business need. I have a single project which is being reviewed for both performance. If you don't need a new feature, why change the code? If you do need a new feature: offer a trade-off. You can do it quickly, buy it will be slow, or you can take additional time to improve the performance. Which choice you makes depends on your performance benchmarks/buckets.
My previous job had an excessive amount of technical debt in it's core product. So much so that simple code changes that should have taken a few hours were now taking days, or in some extreme cases weeks. For example there were financial calculations that should have been confined to one class and treated like a black box, but over time bits of the calculation had spread out all over the application from the front end, the server layer and in to the DB. A simple change had a ripple effect on other parts of the application which made bug hunting and testing a slow and laborious process.
After much grumbling from the devs the CEO prepared a presentation where he told us technical debt was our friend. We had a choice to pay down the debt, or add new features. New feature would get us more customers, and more income, which would mean we could hire more developers to eventually pay down the debt. I felt it was a very wishfull thinking type of argument. We were having increasing numbers of outages due to the lack of a proper testing process in our CI pipeline and our DB was massively overloaded. Adding new feature was literally adding to our technical debt and increasing our risk of outages, something that wouldn't help get new customers.
My issue with the above argument is that the problem with debt isn't the amount or type, but the person holding the debt. A 30-something in a safe job (e.g. doctor) earning €100k a year can safely hold €300k of debt, so long as they are responsible and always strive to reduce the amount owed.
A 20 year old working in a fast food joint on minimum wage, but holding €50k of debt is in serious danger. They aren't in a stable job and they aren't being responsible.
Some companies are like the doctor. They have debt, but are responsible and actively work to reduce it. Some are like the 20 year old who thinks that they can deal with it later, oblivious to the rising interest and danger they are putting themselves in.
I dislike the term "technical debt" as it makes it appear that it is something that must be paid. It takes a stance that there is a correct way to do something and that if you owe a debt it's because you've strayed from the correct way and that you need to restructure your work to make it conform more to this correct way.
I prefer to see engineering as a series of trade-offs. These trade-offs come about often as not having enough information about future scenarios and yes, sometimes they come about because you are in a rush to do something. Either way, whether something should be redone is dependent upon a lot of factors and calling something "technical debt" is a sort of way to politicize work as though it is "the right thing to do" and msut be done. This feels too simplistic to me.
Often, shortcuts we take will live for a long time in our code and often it doesn't matter that it's "wrong". If we call something technical debt, we should only call it that at the time that we determine work definitely needs to be done, not as we're going along and thinking "this is not the ideal design" as this will just lead to YAGNI designs.
Having debt doesn't mean you're doing it "wrong"--debt is a useful tool, both in the financial and technical sense.
Debt only becomes a problem when you take out too much of it. If you owe 90% of your monthly income in interest payments, it'll take a long time to a buy a new car. If you have nothing but hacks/shortcuts/etc in your code, it'll take a long time to add new features.
Debt is only debt when you have to service it. Shortcuts that aren't affecting your ability to develop new features are not debt.
> I dislike the term "technical debt" as it makes it appear that it is something that must be paid.
You don't have to pay other debts either, you just have to be prepare to deal with consequences of that (bad credit scores, bankruptcy). Real debt too you often have choices in what you pay off today, versus what can wait for next month or next year.
There's also nothing inherently wrong with taking on debt. Most people will have all sorts of debt in their lifetime, and some economists believe that debt is much more interesting economically than wealth ever is. Debt is not a moral proposition and calling something "technical debt" isn't saying that it is bad/wrong, it's saying that we're writing IOUs we may or may not have to cash at some point. It isn't necessary to avoid debt at all costs because it isn't wrong to have some debt on the books (beyond very conservative religious readings that feel that all debt is sinful, of course).
There is a lot of technical debt you can leave in the system without consequence.
Tech debt mainly becomes an issue when making changes to the code. If no changes are needed in the code and it is working correctly, it is best to leave the debt in there.
I think that gets to other people's points that if it doesn't have consequences it is isn't likely "tech debt". If you don't "owe" it to someone, and often that someone is "future you", then it isn't really tech debt, it was just a trade-off or a poor aesthetic. We have other fun names for ugly aesthetic choices that work and get the job done and aren't likely to be replaced such as "jerry-rigged" and "Rube Goldberg machine" and "duct-taped shambles".
You can default on debt. Debt isn't bad unless it's allowed to grow beyond the means to pay for it. Large companies can have more debt without additional risk.
Debt=/=bad
If you miss a deadline because adding a feature causes to many issues: you may have defaulted.
This is painful for me as a Rails dev because the real technical debt is always a lack of test coverage. Cleaning up cruft is great, except without tests I'm just going to break my application in the process (and probably not know it.) Without first defining the behavior of the application, there's a reduced incentive to refactor anything. In this way, there's effectively two layers to the technical debt.
This becomes a compounding problem, where the technical debt gets so great that it becomes more appealing to just keep pushing out features and dealing with the fallout. Everywhere I've worked, the code has ended up exactly this yucky. And yes I've contributed to it too. I just don't think that the idea of paying for someone to go back and make the code nicer with no tangible changes to the application's external behavior doesn't seem to resonate with my CTOs. Maybe I just need to find better places to work.
I have experienced the exact same thing in Python and it feels bad to not make changes out of fear of breaking something. So much so that I won't do refactoring on a large codebase written in a dynamic language with not great code coverage (i.e. pretty much every one I've worked on professionally).
This is where static typing can help a _lot_ in my experience. Your types (and your function signatures) are contracts you still need to adhere to after you're done refactoring. Keep making changes until the compiler tells you it's all good and then usually it is.
That cruft is due to "requirements". It's easier to modify a system with less cruft, because modifying a system means adding, deleting or changing requirements. This is easier to do when there are fewer of them.
If you think of a new requirement, it's easier to be sure that it doesn't conflict with six requirements than with sixty, or six hundred.
Additionally, requirements are hard to remove/change once their implementations are deployed and depended on.
Basically we are already saddled with debt at the requirement level, before we even consider the implementation quality.
Thus, the first tool in the fight against technical debt is to very carefully manage the acceptance of new requirements. If a requirement doesn't exist, then its code doesn't exist, and that's as clean and maintainable as code can possibly be.
Secondly, try to gather all of the requirements up-front before implementing anything, and then resist futher requirement creep. If most of the requirements in a system were gradually introduced after implementation began, that will tend to degrade the quality.
I kind of like the idea of calling it technical debt- because I can declare technical bankruptcy (move companies). But really it doesn't work, partly because technical debt very often is actually the result of doing a slap-dash job in the past. If this code was unimportant enough to just throw together in the past you really need to ask yourself whether it's important enough today to fix. This feeds in to one of the fundamental personality types - someone who may be more interested in making the code beautiful and streamlined and neat than actually improving the bottom line.
There are cases where you need to consider technical debt, but most often it should be considered as something you create rather than something you inherit- because the creation of the technical debt is where you're making the judgement:
* Is delivering now important
* Is this code going to need to adapt and develop in the future
* Is the code this is building upon going to stick around or will this code likely be superseded.
* (Selfish developer: Am I going to be around to deal with this)
I think when trying to categorise technical debt the Technical Debt Quadrant is a better tool [0]. It's already linked to in the parent article.
A lot of people are shoving all technical debt in the Deliberate / Prudent category. However I've seen a lot of code bases written by inexperienced developers and it would have be done completely differently if they had any experience (both of the inadvertent categories). This means actually updating the code base can be a complete pain. Sometimes you know if you're editing the code you risk creating huge regression issues and suddenly something that should have taken a short time can spiral out control if you want to add more functionality.
My framework to think through technical debt: I don't think of it as debt (negatively connoted) but business investment.
When you take a debt it's usually to finance something. Focusing the wording on debt highlights the negative side of the trade off. It's like we're suddenly all turned fiscal austerity hawks, where any bit of debt is evil.
I feel like developers are each placed somewhere on a continuum from purist to entrepreneurs. Purists can spend weeks to tweak an SQL query, while entrepreneurs would be happy with a no-code Zapier MVP that actually lands customers
It's not to say that one is always better than the other. A balanced team probably needs both. In our company, I'm more on the entrepreneur side (which I guess is good for launching a startup) but the recent addition of a "purist" to the team has improved our codebase significantly.
People who understand financial debt don't think of it as evil. It's a tool they use.
Technical debt is the same way, and I often talk about it that way at work. But it's a phrase that means something that "business investment" doesn't. When I tell my boss that we're incurring technical debt (and what exactly that debt is), he is already in the "business investment" mindset on the matter. I'm the one trying to reign that in and focus on making the future easier by avoiding some of that debt now. But I fully understand both sides of the matter.
That fits with my belief that the notion of having technical debt is a luxury that only a measure of success brings. Until you're in a place where you have a excess of resources (time and money) all you have is code.
Obviously with experience you can make decisions that can lower the volume of technical debt once you reach that breakpoint, but in my view the right play is always to get to that moment as efficiently as possible and that never involves re-architecting or rewriting until then.
The reality of most code, especially as it gets closer and closer to the end user, is that over time there will be requirements added and removed. It is really hard to perfectly design such a system from the start.
Thus some level of technical debt is inevitable and it isn't really debt. For instance, an overriding goal of zero technical debt is not always the right decision. You may be expecting additional requirements and have not come up with a better conceptual model.
This focus on the technical makes you lose sight of underlying issues which are often organizational rather than technical. Is someone neglecting the code intentionally to get a promotion for a rewrite? Is there a ship now or feature mentality? Are your engineers empowered to change the code?
Other industries and systems have the same issues. What do they call this?
It's not future requirement changes. Its that the conceptual model that you constructed for your original requirements doesn't accommodate a new requirement. Now if you construct a new conceptual model, you will have to modify the original code. It sometimes isn't feasible to do this to each new requirement especially if you suspect that the next new requirement will require yet another change to your conceptual model.
This can happen because the product roadmap changes (new PM, business changes, etc...)
Try writing UI for several different designers over the course of several years for several different iterations of features when there is no common design language.
Technical debt is about quality of code, not about changes in your conceptual model.
Lets say I had a warehouse that stored food, all stored neatly in its own section, each with the least frequently moved at the top where I need a forklift to access, and the most frequently used at the bottom where they are easy to access.
Someone orders too many boxes of rice, and there's no room in the rice section, but there's room on the lower shelves of the sauces so it gets stored there, and some are left on the floor, partially blocking forklift access.
The immediate problem of rice storage is solved, but every time I need sauces I have to grab the forklift, and it may take longer to get there and back because the route is blocked. This will persist until I refactor the warehouse so that there is more space to store the rice appropriately.
Now the manager decides we should be able to store refrigerated goods, and I have no refrigerators yet. The work to refactor the shelves to make room for the refrigerators is a conceptual change in how I use the warehouse, and not a direct consequence of how I've previously been storing goods. It's not tech debt.
Quality of your code can be affected because your conceptual model is muddy. Makes it more complex, harder to understand, harder to change. Makes you implement a requirement in two different ways.
I think it is worth noting that different organizations/teams may require differing amounts of work related to technical debt. This is often driven by business culture, priorities, team experience, long term business goals and many more.
From experience, a lot of people/companies have difficulties marketing what/how/why/when technical debt should be tackled for both internal and external stakeholders. This leads to a less than optimal structure where "new" or "shiny" work is preferable and rewarded, as opposed to a more balanced approach.
I disagree that productivity cannot be measured. Measure productivity by the time it takes a developer to achieve a proof of accomplishing the business requirement through test automation. If it takes a developer 1 hour versus two business days of 8 hours there is a 16x factor of productivity.
Tech debt are all the factors that erode productivity. This includes many technical factors from inefficient systems design to unnecessary testing. It also includes slowness from weak developers and developer training.
I know it puts me in an extreme minority, but I am a big fan of extreme simplicity, which is not what it might suggest. Extreme simplicity removes everything in a system present for the easiness of the developer so that all that is left is only that which is necessary to accomplish a business requirement as directly as possible. If there is less to read (including dependencies) there is less to maintain and less to test. Simplicity is not easy.
The confusion is that a system must achieve a certain level of easiness or it will be rejected by some developers outright or require training at great expense. My basic premise for making any challenging decision is to never compromise on integrity, which includes being honest about the challenge at hand. You can embrace that challenge directly and solve for it or you can hide from it behind layers of easiness.
This assumes goal complexity is measurable and you can run randomized experiments to measure what you call "productivity". Neither of these assumptions holds in the real world.
E.g. changing some business logic can be 10 minutes of work to implement a hack (which will pass tests, mind you), or 2 weeks of strenuous yak shaving for a "principled" solution. Which would you rather pick? And more importantly, are you prepared to select for developers who favor 10 minute hacks to principled solutions?
You measure productivity by the time it takes to complete a task. That is both objective and measurable.
How do you define a principled solution versus a hack? All that matters is whether the result achieves the desired business goals without regression, and that is what tests are for. That being said I would gladly pick the 10 minute solution that passes all the tests.
So much of development is bullshit posturing for developers to justify their existence with unnecessary tasks to make things easier for themselves.
You measure productivity that way on the assembly line or other menial tasks where it's obvious how to do things and the work isn't very complicated. Trying to do this with creative professions is a disastrous mistake.
>> So much of development is bullshit posturing for developers to justify their existence
That applies to any profession. Developers aren't unique in this regard.
> You measure productivity that way on the assembly line or other menial tasks where it's obvious how to do things
Isn't it obvious how to do things in programming? Many developers, in my corporate experience, occupy themselves with how to do things in code. That is exceedingly unfortunate. How to write code should be an obvious disqualifier before obtaining employment. Nobody would hire a lawyer, for instance, who didn't know how to a legal argument.
Writing code should be as obvious as writing an essay, but I suspect many developers that are hiring probably have trouble writing essays as well.
Anyways, if you had the choice between a 10 minute pizza and a 3 day pizza with ingredients more evenly spaced apart which would you choose?
>> Isn't it obvious how to do things in programming?
Not in my field, no. But I might be an oddity. I get hired to do the kinds of things others have tried and failed to do. So while they may be "productive" by whatever arbitrary metric (solving small problems real quick), they fail at gnarly, "deep" shit which can't be done in a day, or, sometimes, in a week, but which does need to be done.
That is why the software gods invented test automation.
> paying interest on it is
That is why simplicity is important, because less is more. All code is ultimately debt as it demands some amount of maintenance. When there is less to maintain there is ultimately less debt.
> And you will hardly hire a lawyer to work in unknown to him legal domain in foreign country in foreigh language.
Has that ever happened to you as a software developer? The one time I was told to learn a new language I was allowed the time and space to learn it.
The problem with the concept of technical debt is the implicit suggestion that you know what is debt and what is not up front. An awful lot of debt is stuff that people thought was needed but turned out to be totally off base and now weighs everything down. And large piece of that - not by any means all or even most, but a large piece - is people pre-emptively optimising the design to satisfy precepts put about by people like Fowler himself.
To reduce/remove tech debt from my code, the only patterns i found helpful is: Separation of concern: The chunks of files/codes that work or removed together.
One typical example in real world is React Hook.
By moving boring stuff into a folder called "hooks", we centralize concerns into its own file, so that we could manage them effectively, or we don't need to think about it anymore.
I find the phrase 'Technical Debt' to be unhelpful. If I am in financial debt I will definitely need to pay it off unless I go through bankruptcy. Technical 'Debt' seems to operate by bad analogy: I may opt to completely wipe out the subsystem that has accrued the debt and replace it with something else, or it could turn out that the subsystem solves some problem that wasn't business-critical anyhow.
I've had to restrain developers from "turd polishing" systems that were poorly thought out to begin with. Code gardening in a subsystem that was simply the algorithmically wrong way of doing it (that we might have had to have shipped quickly in order to stay afloat) isn't a first class activity. The problem is with the metaphor of technical 'debt' is that implies a balance sheet, but many (most?) of these bills will never come due.
Technical debt really applies when time-saving decisions in the past cost more time in the future. As such, you can't always be 100% sure whether you're incurring it at the time or not. You can only be sure later, once you have to pay it.
But there are definitely times that you feel certain it's going to end up being technical debt. When you implement a system that doesn't scale, but you know it has to eventually scale massively, that's going to be technical debt.
You could write it right now to handle that load, but it'd take a lot of time and money. Or you could write it the quick and dirty way, and get money coming in immediately and put off the scaling until it's actually necessary. You know it's coming. But you decided to incur that debt in order to make money sooner.
IME, technical debt is usually about time or money, and getting something done cheaply now and worrying about the full cost later when it's more applicable.
It's a tool, not a problem, so long as you're managing it and not just blindly choosing to do things the quick and dirty way all the time.
That's the difficult part of being new on a project. If you aren't building the replacement is progress being made to fix the issue.
With regards to gardening: if you're spending all your time weeding but you haven't planted any seeds what are you growing?
Something takes 20 hours to implement, but you do a hack job in 2 hours. Next time someone touch that code, they probably need to spend the 20 hours to re-implement it. Technical debt can can be made less costly by using the test-first approach, you start by writing a test that confirms what you are going to implement works. When doing test-first, it's often fine to just do the bare minimal work. If something needs to be added/fixed, write another test, then do the bare minimal change, repeat. Eventually the code will be so complex a small change will take hours, and that's when you have to pay the technical debt, but if you have all the tests, you can make a clean re-write, and the tests confirms that the new clean code works just like the old code.
Businesses should probably start tracking technical debt as a liability on their books. Maybe depreciate code as an asset like a business does with a truck or piece of equipment. Management would probably understand the concept better if it was represented on the financials.
If you know a piece of crufty code is going to be irrelevant in the near term after some major architectural change or new feature, it isn't really worth it to pay down the principal with minor improvements or shims.
I think it's marginally useful to think of low vs high interest debt in terms of technical debt. I can make assumptions that certain areas of software will not change in the future, but that in itself becomes technical debt. Now future features have to be designed around the idea that certain features are off limits. Sounds extremely risky to me to try to hedge your bets on this kind of foresight. On the other hand, I think it's extremely useful to examine different libraries / modules as having different levels of technical debt, but the metaphor stays intact. You just look at your software not as a single loan, but many micro? loans.
Technical Debt becomes really toxic when mission critical customer facing functionality is based on it. Just this morning we received a ticket where cleaning up technical debt caused a major issue for the customer, when we looked even further we realized that this customer relied heavily on the state brought on by technical debt and that resolving this issue will require an even more careful refactoring to ensure that the debt is cleared but the overall functionality is unchanged.
As a workaround for the time sensitive issue we literally copied the old code and overloaded our class so that the customer runs the old code while we develop a robust fix.
> Given this, usually the best route is to do what we usually do with financial debts, pay the principal off gradually. On the first feature I'll spend an extra couple of days to remove some of the cruft.
This should be managed carefully to prevent a lot of independent gradual improvements that don't follow a clear architectural pattern and end up conflicting with each other.
Furthermore, all code changes to existing modules require validation of the affected functionalities, which are often hard to spot. Hence, I'd say that a wide scale refactoring effort needs a clear testing plan before it's put into practice.
I think technical debt extends beyond poor quality code that prevents additions or modifications. Having worked with a large amount of scientific software it extends into the efficiency and robustness of software.
Frequently, efficient methods or parallelization are not used and the least creative method is implemented to solve a scientific question. Functions are not written in a modular way but merely for a “quick and dirty” analysis.
The end result is that you spend more time getting things to run or waiting for them to run than if you had gone back and reworked the code to begin with. It’s a serious drag on productivity.
A startup without technical debt, is mismanaged. The financial runway is the first consideration for survival. Spending a second on anything that doesn't get the startup to the next phase is a second squandered.
This article frames technical debt in terms of sub-par code, which is part of tech debt but only a part. Legacy systems that are nearing end of life, patches and updates that haven't been applied, all of that represents technical debt as well. And when it comes to some of it, it can be actual financial debt too. Plenty of legacy systems sit around a long time past their sell-by date because of the financial cost of replacement, which may be orders of magnitude more than the cost of maintenance.
Funny thing about technical debt with programming is that sometimes, and in my experience I did not had to pay it back... Software went to trash just because business changed direction.
A lot of people seems to complain about the wording. I understand the analogy doesn't cover 100% of the issue but unless somebody finds a better one this is the best we have. Good enough so that stakeholders do sometimes allow us to take care of it. Yes probably not all the time, and not enough, that's why we need to get better at communicating about it. In a corporate environment, communications skill are as important as technical skills, and we should seek to improve both.
As a side note, I noticed Fowler linked the CannotMeasureProductivity page using camel case, and that took me right back!
I remembered using PhpWiki and being able to auto-link pages just by using camel case. I don’t believe MediaWiki has that capability. And it’s such a standard I had forgotten it even existed.
Something very soothing about the idea that links could be handled so gracefully.
This is not just about the code, but also about the feature this code implements.
A feature implemented through "indebted" code tends to be buggier and less performant. There are exceptions, of course, but by-and-large this has been my experience.
So the "interest" is not repaid by just the programmers, but also by the users, over and over again as they use the feature.
How many companies are profiting from a poorly written codebase that just works well enough to be a parketable product and isn't being maintained and there is no plan to maintain it? I can think of many utilities not being actively developed that are still part of or a product in their own right. Surely that isn't debt in any sense?
The concept of 'technical debt' seemed to emerge around the time that the concept of 'scoping' went out of of fashion. The best engineers that I have ever worked with were truly masters of scoping projects.
Technical debt is a bad name for this phenomenon. The word debt implies that we know it exists at the time of taking on the debt and that it can be paid back.
In reality a lot of technical debt can never be paid back without restarting from scratch, and a lot of it is accumulated unknowingly only to be discovered way later in the project.
I would say technical debt is only that which you already know about but have made an intentional decision to take out a “loan” to ship quickly.
I’m having trouble thinking of an example of financial debt that can be taken on unknowingly. Maybe identity theft? But that's less like an engineering tradeoff... more like NSA backdoor type stuff.
Debt heavy choices made knowingly are more rare, as programmers consciously avoid it or choose debt that won't accumulate beyond certain bounds.
Whatever it is, I think most projects deal with "technical debt" accumulated unknowingly. There's no theory around how to create a "good design" and thus it's often shooting in the dark or using preexisting patterns. Most software projects have design flaws that only become evident later. In my career, I have dealt with this type of problem more-so then debt accumulated knowingly. Additionally my current company is calling this type of debt, technical debt. I have rarely rarely seen technical debt taken on knowingly and documented.
> Debt heavy choices made knowingly are more rare, as programmers consciously avoid it or choose debt that won't accumulate beyond certain bounds
I don't know what kind of software you work on, but I disagree. It's incredibly common, from what I've worked on, that we have an existing subsystem X, and project management wants it to interact with this new subsystem Y but it was never designed in a way to do so, so either we can hack it in real quick, build a moderately pleasing integration with only a little technical debt, or completely rebuild X from the ground up in a way that makes sense and leaves no technical debt. From my experience, bad managers will want the quick hack while engineers and good managers will take the middle option. Nobody opts for the complete rebuild.
The design of subsystem Y is likely technical debt. The reason is, because of agile, software developers know that they must code defensively to account for feature changes or additions in the future. That means you take on technical debt every time you make a subsystem that isn't modular, meaning that if system Y is not infrastructure it should be able to be pulled out and used somewhere else. To not account for this feature is to not account for future change.
A good way to test if your subsystem is modular is to ask yourself, can you pull your subsystem out of the current system and use it in a completely different app in a completely different context? If the answer is no, then likely the module is either IO or too tightly coupled with other systems. Too much Mocking in your tests is another sign of a design flaw. It's a signal that your code is dependent on other systems rather than a module that can be pulled out and reinserted somewhere else.
Another design flaw is, is system Y itself made up of modules or interconnected dependencies? If subsystem Y needs to reconfigured to do something slightly different can I pull out 10% of the system and replace it with new modules? Or does pulling out 10% of the system involve pulling out another 80% of the system as a dependency? You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.
Most programmers choose dependencies because modules are harder. Also the dependency style of programming is heavily promoted. Additionally, most programmers are Unaware of what it means to make a modular component, they organize code following nomenclature rather than a systematic theory, and they think they are doing modular design but they are not.
Likely your subsystem Y was designed in such a way that it is tightly coupled to another subsystem and can't be used outside of a certain context. Possibly it is tightly coupled with IO or some specific database schema. Likely system Y itself is made up of modules that are interdependent. Also likely it was programmed by someone who did all of this unknowingly.
So it could be a matter of perspective. What you see as debt taken on deliberately I see as debt from the past "discovered" and interest accrued because the debt can't be paid back without a rebuild.
I Suspect my husband of cheating his attitudes were enough to raise suspicions and I need to be more convinced enough with evidence,I came online to read some hackers review andi contacted one of them which he demanded for some certain amount of money and I paid, after payment he came out with nothing before a friend recommended Hack Wizard to me i contacted him immediately, he monitor my husband illicit activities; his phone Calls, Text Messages,Whatsapp,Snapchat,Facebook,Hangouts, Instagram, Twitter and Email I was able to spy them with little deposit. His job is fast within 2hrs and 100% deliverywith no trace,He have many service to render, Clear Criminal Record,Repair your Credit Score, Credit top-up, Increase your Grade, Monitor your Children etc Contact him at CYBERWIZARD1995 @ GMAIL COM Whatsapp: +1 662 727 5740
One good analogy is how the chefs in the kitchen clean up there table right after they are done with a dish, If you do not clean up, There will be a delay in the next dish and the time to make a new dish increases exponentially and the a dish is never delivered within a reliable time interval.
I always thought "technical debt" was just a buzzword consultants use to make the client pay more money upfront. To me it translates to simply "work we haven't done yet".
My only bone of contention is that taking on debt is usually a conscious choice. Technical debt to my mind is the price of making a pragmatic decision, it’s the trade-off that lingers.
Making poor architectural decisions or writing poor quality code isn’t tech debt. They are just institutionalised consequences. The way the world now works.
Legacy code isn’t tech debt either: it was right for the time.
What you have instead is knowledge debt as engineers turn over, and the once intuitive and elegant parts of the codebase are utterly alien to the new hires who learned their craft in a totally different way. This isn’t tech debt because engineers can’t intuitively anticipate the business’s future hiring decisions, particularly when those are made independently of the team.
If you want to establish tech debt you have to define the schedule to pay it off and there should be some record of that work somewhere, with accountability attached. It is not merely the accumulation of things you don’t like the look of or find hard to work with.
Similarly, I think MVP is a conscious choice to incur product debt because it is far too risky and irresponsible to go all-in with your perfect vision right off the bat. You want your initial buy in and a chance to bail out or pivot before all is lost.
Edit: as an example that comes to mind, a bank running on COBOL doesn’t have tech debt unless their COBOL engineers are aware of their missed opportunities. If they think COBOL is no longer fit for purpose, it’s not debt, it’s time to reinvest.
>Making poor architectural decisions or writing poor quality code isn’t tech debt.
I compare technical debt to debt.
You may go in debt because you looked at the housing market and decided that now is likely a good time to buy a house with low interest rates. You take on a large load of debt, but it is done wisely based on currently information.
You may go in debt as a business operating on credit to fund an expansion that should pay off quite nicely in a few years time. Maybe it will fail, but it isn't a bad risk based on current knowledge.
You may go in debt for an education. Depending upon the degree and extent of debt this could be good or bad.
You may go in debt by spending beyond your means on a credit card. This is generally unwise.
You may go in debt because of a medical emergency that is entirely beyond your control (especially in some countries, but even some countries with universal healthcare it is possible to need treatment that isn't covered).
All of these have something similar when dealing with technical debt. Sometimes it is a wise investment. Sometimes it is a risk that may or may not pay off. Sometimes it is beyond one's control. Sometimes it is poor planning that will be regretted later.
I think of all the code written as technical debt. As soon as you write it, it becomes a burden - just like regular financial debt. However instead of thinking of code like a credit card loan that will take a few years to pay off, think of it like a mortgage that you have for 30 years. At the 30 year mark, you will generally have retired the code, hopefully. If you're not so lucky, you should expect to see balloon payments coming shortly. I know this isn't a perfect metaphor because honestly "technical debt" applies to all code, not just slightly bad code. That's very different from a loan.
I suggest that all code is a liability, rather than a debt per se.
This assumes all things can be usefully viewed through the prism and language of economics (though my gut feel is few things can or should be).
The accounting world considers code an asset and allows you to amortize its creation.
Making progress in the world is fighting against entropy. Just because it is hard to fight the disorganizing efforts of the universe doesn’t mean that your output is a liability.
Liabilities are debts.
> liability: a thing for which someone is responsible, especially a debt or financial obligation.
According to the balance sheet equation,
> Assets = Liabilities + Owners Equity
Certainly all code is a liability - it must be at least understood and maintained from time to time (80% of lifetime cost is in the maintenance).
And while an accountant might also mark it an asset - not all code is an asset - and we recognize that fact when we delete for badness or disuse - or rewrite it to make extending and maintaining it easier.
As intimated, I'm not sure that the fuzzy language and world of economics is the best way to view such things, however ...
I'd accept that debts are a subset of liabilities, but not that 'liabilities are debts'.
A debt implies an externality, while a liability implies some generic exposure -- it's that which I focus on when considering (all) code as a risk to an organisation.
I suspect, however, that we agree on the fundamentals. There's some cost and risk to developing and trying to maintain any code base, and accepting that can usefully inform development and operations work better than thinking of code as a (near-)risk-free asset.
Code is a liability it can be hacked if not written securely. It can be out of date if the language changes or Windows changes and the company upgrades, which means parts of it need to be rewritten.
I used to program COBOL on IBM 370 DOS/VSE systems they later went to Visual BASIC on Window 3.1 PCs. The conversion took a long time and most code had to be rewritten because COBOL is not BASIC. Management didn't care about that, they just wanted a working project. We all did our best going overtime without pay and working weekends to meet deadlines.
COBOL was a liability when they wanted to convert to Visual BASIC. There was liability to convert the code, convert from DB2 to SQL Server on the database end.
Overtime without pay is financial debt.
That's not debt. That's an asset. It's inventory. It has a maintenance cost in the sense that a truck has a cost of upkeep to keep it on the road.
Technical debt is configuration and plugins. You are able to deliver something now with the benefit of changing behaviour later but at the cost of having an enormously increased test surface and invasion into your type system with possibly hostile parameters.
I would say debt == investment.
You're betting that the maintenance foregone in the present is paid back with interest in the future.
This "pay back" could be the success of the company which allows you to pay for more developers and tools.
Is it worth it? Time will tell.
Without the code the company is nothing, so it's definitely not 'debt'.
If that were the case, the company would have a negative value.
Code is asset that requires maintenance.
But yes, it's definitely worth reminding ourselves of the ongoing cost of maintaining code.
> Without the code the company is nothing, so it's definitely not 'debt'.
I reconcile the concept that code is a debt with your statement in the following way:
The feature produced by a piece of code can be an asset, but the code itself is a debt.
Thus, for a new feature to be valuable, you want the value added of the feature (or asset) to out-weight the cost of the code (or debt).
As a general rule, the complexity that comes along with a piece of code grows as the code-base and feature set grows, thus, for a young codebase, the value of a feature more easily out-weights the cost of the code.
Anything in any other business terms that needed to be used, created or maintained to create value for customers would not be debt.
'Technical debt' is just not debt, it's just an interesting way to think about it, and a neat use of lingo.
There are any number of other business practices for which one could try to apply the same analogy, but it wouldn't work because very quickly the term wouldn't seem right.
For example 'training employees' for a situation, knowing they might need to be trained differently in the future, is not going to be regarded as debt.
Same thing for processes, investment in equipment, parts, materials etc..
It's a metaphor not actual money that can create a negative value.
Although, you could say that a lot of profitable companies have negative value codebases where the cost to rebuild them is less than the cost to maintain.
This probably explains why some companies get replaced by small nimble startups.
> Making poor architectural decisions or writing poor quality code isn’t tech debt
I think it is. Technical debt is all about trading current productivity for future productivity. Poor architectural decisions can be caused by several factors. Inexperienced devs, not taking enough time to understand problem, not spending enough time brainstorming the problem. These are all decisions that trade current productivity for future productivity, which is technical debt.
Also most technical debt is not accrued consciously. Very few people are doing cost benefit analysis on technical debt. It's usually made intuitively by project and product managers who don't directly interact with the costs of technical debt but directly experience the benefits.
> Technical debt is all about trading current productivity for future productivity.
This is a very succinct and accurate description, and one I haven't come across. Thank you for this phrasing :)
> Inexperienced devs, not taking enough time to understand problem, not spending enough time brainstorming the problem
Requirements changing is also a big one. 4 years into one project that I had created and maintained a "one to many" relationship changed to a "many to many" in some of the tables central to the system. Something that I had clarified wouldn't when originally designing the system.
>writing poor quality code isn’t tech debt.
sure it is, in the same way that taking one of those payday loans to cover a coke habit is still real debt, it's just real bad debt.
I don’t believe we can equate jonesing for a fresh hit to writing well intentioned but ultimately difficult code or the product of inexperience.
Besides which, it trivialises the seriousness and the tragedy of addiction. If you think your tech debt is similar to that you are well equipped to quit your job and work elsewhere. An addict does not have such an easy choice, they can’t leave their addiction to someone else and move on.
You choose to take the payday loan or hit up a shark, addiction or no.
The point where I concur is when you gamble the short term against the future despite all evidence that you will benefit from spending time to save time.
all analogies and metaphors are in some ways inexact - indeed the whole concept of technical debt as a thing correlating to real debt is fraught with edge cases that don't exactly line up. Nonetheless if the metaphor has some utility then individual expressions of it such as mine will often have some utility - hence your concurring on a single point while finding other implications problematic.
The victim in that metaphor is not an individual coder who can go work elsewhere, it is the org who takes on technical debt and sometimes can't function at all without the code in question.
>I don’t believe we can equate jonesing for a fresh hit to writing well intentioned but ultimately difficult code or the product of inexperience.
Agreed. The addict might recover. Organizations with poor culture are doomed.
> I don’t believe we can equate jonesing for a fresh hit to writing well intentioned but ultimately difficult code or the product of inexperience.
That's not what "technical dept" is. Choosing a quick solution instead of a proper one that would take more effort is. I've made that conscious more than once myself: to hit a deadline, or because I didn't know if the MVP would even survive long enough to worry about code maintenance.
Poor quality code is debatable. What seems like a great architecture and clean design to one, may be regarded as the worst thing on earth by another. Code quality measurement is often very subjective and it's difficult to come up with objective criteria. For example, someone coming from embedded world will be appalled by some OOP designs, ridiculous usage of so called "patterns", etc. And the same thing will be regarded as "clean" and high quality by someone else.
Totally agree. In my opinion, with this type of MVP strategy, work done should always be in the service of something else. Initially, an MVP is created to test out ideas and also to learn what structurally it takes to build your product.
After the MVP, you may wish to add more features or support more users, it is at that point that you can assess what is debt and what is not. It's not a matter of "cruftiness" but a matter of what's holding you back from adding additional features or supporting a larger number of users.
If we frame all work as doing something that will deliver some sort of value (be it customer value or unlock the potential to add more features to add customer value), then it becomes a more constructive conversation about work that needs to be done, debt or otherwise.
I'm not a fan of just looking at whether code is crufty or not to determine if it's "technical debt". If you look at code long enough, you'll find something you don't like and want to rewrite, but it has to be in the service of some other goal or else you'll just become an architecture astronaut building wonderful things that don't provide value.
Legacy code isn’t tech debt either: it was right for the time...
Edit: as an example that comes to mind, a bank running on COBOL doesn’t have tech debt unless their COBOL engineers are aware of their missed opportunities. If they think COBOL is no longer fit for purpose, it’s not debt, it’s time to reinvest.
Should there be an analogy for depreciation? Some of the "best practices" in my current C++ shop are in direct contradiction to the Law of Demeter and related philosophical stances in Object Oriented programming (particularly ones espoused by Martin Fowler) despite the fact that my current shop advocates for OO. Neither the old position or the my current shop's position is so bad that it's not viable, though some hotheads in the field act as though this could be the case.
A lot of the churn seems to be arbitrary and driven more by fashion than solid first principles. (Which would include cost-benefit analysis, for which there is often insufficient data, and for which no one wants to pay the cost to gather the data.)
Yeah, I see too often devs shrug off their "technical debt" when what they're really describing is simply sloppy work. These economic metaphors always break down when you challenge the idea that devs/dev managers are always acting in the best interest of the product. I've seen teams cheer their "velocity" because they're closing lots of issues, when most of the things they're focused on shouldn't have even been an issue in the first place.
For the cases like legacy code and turnover, perhaps "tech depreciation" is the more appropriate economic term. You didn't really borrow against the future to have more today, rather something naturally decreases in value over time.
Right. Technical debt should not be understood on its own but as part of a larger "technical balance sheet" metaphor: technical depreciation, technical amortization, and so on.
Where I work, we actually do use technical debt to cover unknown defects as well. So if a person writes bad code, or if we have some legacy code that has suffered from rot, we'd call that technical debt.
I think there's a lot of value to be found in separating the known debt from the unknown debt though, and giving it a name. I might have to start doing it.
So thanks for making me think about it.
"Legacy code isn’t tech debt either: it was right for the time." really? Isn't that what technical debt is, solution that no longer fits the current problem?
No, solution still fits the problem space but requires more than investment to improve efficiency/quality/etc of future management of the technology.
The cobol example is great - it is code still solving the problem but is no longer efficient or productive to find and pay cobol programmers.
If the code still fit the problem then you wouldn't need to pay a cobol programmer. If its hard to change the code because its too expensive to change, then you have tech debt.
But it's only debt if business decisions mean you must change it. I've seen far too many programmers crawl through legacy code that need not be replaced (it does its job) merely because it's using an outdated technology. But if there's no business justification for that, I'd argue it's not debt at all. There is absolutely nothing wrong with some ugly COBOL code doing its job with no current need for modification. That's not debt. Debt is something you have to pay back.
Great comment, if you allow I would summarize by rephrasing one sentence making it shorter:
What you have instead is knowledge debt as engineers turn over, and the once intuitive and elegant parts of the codebase are utterly alien to the new hires who learned their craft in a totally different way. This isn’t tech debt because engineers can’t intuitively anticipate the business’s future.
That is it! Thank you for sharing your thoughts.
I don't see the distinction between knowledge and technical debt unless you count knowledge debt as a subset of technical debt.
You make a choice to accumulate knowledge debt by not writing docs, not cross training team mates, etc.
I count knowledge debt as a superset. You can’t determine the state of a codebase without first knowing the decisions that were made when it was being written.
This, and I will accept being wrong, is why I think we see code we didn’t write and immediately see how we would do it better. What we’re doing is sending ourselves back in time and replacing their shoes with our own. They didn’t know what you know and you have no idea what they knew.
A potentially effective approach is to introduce an immutable decision log. We have these people in many other fields - they record the decisions made and they become part of the overall reasoning.
I reckon anyone in a position of power should have an independent decision log. Tech architects, CTOs...
> Legacy code isn’t tech debt either: it was right for the time.
Doesn’t this assume that code as written was of the highest quality? Under tight deadlines or misaligned priorities code quality suffers. Even if it works or “is right” doesn’t mean it wasn’t done to a standard that maximizes productivity.
I don’t believe so, I’m just saying it wasn’t seen as a debt at the time. It might well have been the best they could do given the constraints.
If they consciously chose to sacrifice on it, that’s the debt.
> Legacy code isn’t tech debt either: it was right for the time.
The time it was written often isn't when the technical debt was incurred, when maintenance to keep it right, technologically, for the changing time was deferred, that's when a decision was made to incur a higher price later to save now: that's when the (technical) debt was incurred.
> as an example that comes to mind, a bank running on COBOL doesn’t have tech debt unless their COBOL engineers are aware of their missed opportunities
The existence of technical debt is unrelated to awareness of the deferred costs. “technical debt” isn't a statement about the existing technical staff’s perception of the system.
>What you have instead is knowledge debt as engineers turn over ... This isn’t tech debt
You know, if there only were a way to pass knowledge from one person to another aside from directly talking to that person. Perhaps if we had means of preserving knowledge, and even making it a part of the job description... But alas.
Joking aside, not writing technical documentation and product design docs is very much technical debt.
A lot of information that is lost with turnover is information that could have been written down and passed on.
Documentation is not jut code comments. It is:
-A solid bug tracking system, with clean and reproducible bug descriptions
-A version control history, where every change has a why in addition to a brief how, and which links with the bug tracker
-Well-written tests, which not only verify how a system works, but showcase how to use the APIs.
-Documentation for tests explaining why this input was chosen, what kind of output is expected, and why it was written that way
-High level comments in code (especially libraries) explaining how the API should be used
-High level documentation explaining the structure of the code that is checked in, and lives with the code. The documentation can be in Markdown or a plaintext README, but should be there
-A "code map" telling you what the code in various directories does, and where to look for things.
-And, most importantly, a lot of design docs - for every feature and algorithm. "We are solving this problem. Here are the alternatives considered. Here is what we are going to do and why. Here is how we'll implement this. We used ideas from this, this, and this paper".
Yes, all of that is needed. The price you pay is engineering time. But if you take on the debt, your new engineers will have Hours Of Fun[1] trying to figure out, say, whether the imaginary nanometers[2] model parameter increasing as a result of their change is a good or a bad thing.
What I am saying is documentation debt is technical debt. Documentation is just as important as the code. And having it does help[3].
You might think that your code doesn't need all of that. And that's fine; but that means you plan on never paying that debt. Maybe you plan for your startup to fail/sell out in a few years. Maybe you plan to leave that job. But anything long term means you have to write things down.
And if you haven't been doing so - now is as good time as ever to start.
[1]http://wiki.c2.com/?HoursOfFun
[2]True story. There was a good reason for why the distance parameter had complex-valued nanometers as units; it was an elegant way to do improve a model while staying backwards-compatible. It was written down nowhere, and have fun googling "complex nanometers". They only exist in that code, AFAIK.
There is information which is readily transmitted through non-experiential means: tacit knowledge. Books, lectures, conversations.
There is information that must be experienced, drilled, and committed to habit and muscle memory: tacit knowledge. Drill, practice, experiment, coaching. Often 1:1 or at least 1:few.
It's highly arguable that individual knowledge does not become cultural knowledge until a means of transmitting it from one generation to the next becomes both established on institutionalised. It need not be a large institution, but it must have persistence.
Within any knowledge domain, the explicit knowledge transmits more readily than the tacit. With time and complexity, I suspect a situation much analogous to Amdahl's law is reached, in which generational knowledge transfer comes to be dominated by the tacit component -- it is harder, slower, and more expensive to transmit, and tends to irreducibility.
Technical cultures which become or incorporate a tradition of shared stories and lore, as Unix and Linux have done, seem to achieve this better than others.
And that neglects issues of semantic and cultural drift which affect written texts.
Though I'd come to this conclusion some years ago, I'm finding reading Mortimer J. Adler's How to Read a Book that this is a significant theme of that work.
https://en.wikipedia.org/wiki/How_to_Read_a_Book
> Legacy code isn’t tech debt either
“Legacy code” is usually a label applied exactly to identify that code which has been determined to represent technical debt.
I heard a recent conference talk where it was referenced as “Revenue Generating Code”, rather than “Legacy Code”.
I found it be a pretty accurate summation that gives some perspective from the other side.
> I heard a recent conference talk where it was referenced as “Revenue Generating Code”, rather than “Legacy Code”.
The two categories overlap, but not all code generating revenue fits in the traditional scope of “Legacy Code” and not all “Legacy Code” is generating revenue.
> I found it be a pretty accurate summation that gives some perspective from the other side.
What “other side” are you referring to?
It’s not a perfect overlap. I agree with you there. My personal experience has led me to believe that “Revenue Generating Code” is the most common variety of technical debt.
Technical debt that does not generate revenue in any way is often far easier to replace/refactor.
I look at legacy code like this:
- Could we remove this from the product? - Could we replace this part of the product with a totally new system? - Are we OK with potentially introducing subtle bugs in this code as we work through refactoring it.
If the answer to the above questions are “No”, it’s likely revenue generating code (or part of some compliance process).
In this case the “other side” is non-technical stakeholders who don’t inherently understand the importance of resolving technical debt.
Agreed. Technical debt is the accrued cost of maintenance -- underpaying maintenance is a typical recipe for poor outcomes.
Technical debt is a great metaphor exactly because it suggest the suboptimal code is due to a deliberate strategic decision.
If you just say that some code is bad you will get unproductive reactions like people getting defensive, trying to assign blame etc. Calling it technical debt allow you to call attention to an issue while avoiding blaming anybody.
If you buy thousands of dollars of unnecessary crap with your credit card, you’re still in “debt”, so the analogy still holds.
It was right for the time, and required maintenance and updating; I consider skipping said maintenance to be a debt of the time it would take to do that maintenance, with the interest being the time spent working around it as you work on other things. In a lot of cases, the interest rate is pretty low, and is worth it.
The goal of an MVP is not to write crappy code to get something quickly out of the door but to not build every feature into all details to get all of the parts together and see how they work first. It's about the code you don't write, not about the quality of what you do write.
This is great and well articulated! Would you agree legacy code can often turn into tech debt by virtue of being neglected / unmaintained? The debt here could perhaps be measured by operational overhead
I cut my teeth as a developer on the Windows operating system. Because we were a platform, we had learned that when we shipped an API it became more or less permanent. So technical debt was a huge part of our planning and thought process.
When I joined Netflix I was assigned to add DASH support to the Silverlight web player. I spent weeks working on the new architecture, refactoring the streaming code, etc. One day my manager stopped by my desk and said that "I needed to wrap it up. It was taking too long." I was still weeks away from being done. I explained to that I was cleaning up a ton of technical debt and that's why it was taking so long.
He explained to me that they had different values on his team.
1. Most code will be rewritten every 2 to 3 years. 2. For code that doesn't get rewritten often, preserving battle tested code trumps paying down technical debt
Just remember that technical debt doesn't apply to all software.
The only point here that jumps at me is that the manager came a few weeks to late to explain the development values.
>> I explained to that I was cleaning up a ton of technical debt and that's why it was taking so long.
> The only point here that jumps at me is that the manager came a few weeks to late to explain the development values.
Well...(and I'm not having a go at rsweeney21). They were asked to implement a feature, not to spend ages cleaning up technical debt, and being "was still weeks away from being done" would be an alarm bell for me as a senior dev/manager. If the feature couldn't ever possibly be implemented due to technical debt then GP should have raised this with their manager far earlier rather than just going ahead and fixing this silently. If the feature can be developed and released, despite technical dept, then go ahead and point out to your manager (as soon as possible) this is going to be horrible for the dev(s) working on the next iteration, but the chances are your manager will likely insist you get on with the feature. Communication is at fault here.
We all work in projects that have technical debt and we'd all love to go back and fix these things. But you also have that obvious tension which is the fact that you also need to release features which have been identified by the business as another positive cash-income thing because you're competing against other services - and sometimes these features are very time critical.
Sounds like GP poster seems reasonably experienced (working at MS, Netflix) and should probably know this (unless there are other mitigating factors not included in your comment)
Yeah, I feel like this one was on me. I was new to the job and wanted to impress my manager. I could feel the refactoring getting out of hand. I should have had a conversation with my manager to ensure I was going in the right direction before he felt the need to tell me I was taking too long.
Don't be too hard on yourself; it takes two to tango. You are your manager's work, as much as your code is yours. Unless you lied about progress it's a miscommunication with no right or wrong.
Hey mate, hope I didn't come off too critical of you personally. And you know in my younger days I'd likely have been the same.
No offense taken. You're all good! :-)
Sounds similar to some of my learning experience too, as a maker and manager
This is central to Fowler's point, which is that technical debt only needs to be paid off if it gets "activated" a lot - this is where the metaphor breaks down, because typical debt accrues over time.
If you never touch code again, but that code was rushed and is ugly, there's no good reason to pay off the debt.
Similarly, if the code's getting thrown out (and if you design code to get thrown out), there's less reason to prioritize cleanup.
That's often overlooked, yeah - if it ain't broke, don't fix it. This goes against a lot of people that read code for the first time, myself included.
A recent example: React introduced hooks, which effectively means that there's a new de facto standard way to write components and you shouldn't be writing components as JS classes anymore.
The kneejerk reaction is then "We need to rewrite all our class-based components to use hooks!", which can take quite a long time (and it's tedious, boring busywork).
Luckily the React documentation on hooks itself [0] basically tells you to not bother unless you're actually going to work in that component anyway, and even then you should make sure everyone else in your team understands them; classes are fine, it'll keep working for a long time, resist your kneejerk reaction.
[0] https://reactjs.org/docs/hooks-faq.html#should-i-use-hooks-c...
Yeah, I try to align tech debt cleanup with 'touch points', since that's generally an indicator that the code needs work done on it anyway.
It's sometimes controversial to say "We should not prioritize tech debt" but I feel pretty strongly that it's true. We aren't paid to write beautiful code, we're paid to write code that gets the job done. If code is getting the job done, even if it's fragile, our job is done. Only if that code is impeding new work should we prioritize it.
> 1. Most code will be rewritten every 2 to 3 years.
a.k.a. defaulting on the technical debt
No, this is still a payment against technical debt. From the top of Martin Fowler's article:
"The extra effort that it takes to add new features is the interest paid on the debt."
If adding a new feature starts with, "just throw out all the code and start from scratch", that's a much larger cost over the long term than building code that's easier to maintain and refresh.
With very few exceptions code that is "easy to maintain and refresh" over any appreciable time horizon (say,longer than 2-3 years) requires the kind of engineering effort that the modern world of "CS trivia is good interview material" generally derides. I'm not even talking about "real" engineering processes as might be used for space systems or whatever; I mean just basic requirements analysis and technical documentation.
The cost of rewrite at that point is typically less than cost of straightjacketing the existing codebase to support all new requirements, otherwise it is less likely to happen. So, defaulting it is.
(Mind you I'm not fully sold myself that 'technical debt' is a useful analogy at all)
Not necessarily. More like an asset that has finished it's designed lifespan and the org has moved to a new asset.
That begs the question, what distinguishes actual technical debt from short-lived assets?
You can't fully make that distinction until it happens.
Sometimes you can say "I know this will need to be rewritten later" and be pretty sure you're seeing technical debt be accrued.
But other times you have no idea. It might be a tool that was intended to be used for a few weeks, but turned out to be incredibly useful and lasted years. Had you known, you might have written it better if the first place.
Technical debt isn't like financial debt in that there's nobody with an accounts sheet telling you that you owe them anything from the start. You don't necessarily make a deal with anyone. So you can't judge its existence in the same way.
Schrodinger cat's technical debt.
Or the target's moving fast enough that it's inefficient to build a gold-plated solution when a quick hack will get you out of trouble for now.
Research code is much the same. Your code doesn't have to be great (or even OK, to be honest) as long as it runs well enough to provide the results you need for your paper.
A technology no longer being supported (e.g. Flash, Silverlight, IE) is not technical debt though, it's external influences.
Likewise, rewriting a front-end in the framework of the week doesn't have to be because of technical debt; Angular 1.x was fine and it's still fine, however you'll see a lot of upgrade or rewrite projects happening right now not because of tech debt, but because people simply don't want to work in Angular 1.x anymore, or an employer can't find competent people that want to work with it anymore.
Likewise, COBOL applications aren't bad because of technical debt per sè, it's just become very hard to find people that can maintain it.
Never really seen that in practice, code is "touched" but rewritten?
It's downright stupid to say all code will be rewritten every 2-3 years, because in reality this never happens as completely as it's stated.
As the article mentions, it's about figuring out what makes sense to fix now, or wait until later.
But yes technical debt applies to all projects. The way we interact with it just varies.
Nice straw man. Nobody said "all code will be rewritten every 2-3 years". He said "most code" will be rewritten every 2-3 years. Also, like I said, these were values on his team. We were a front-end application and every 2-3 years Netflix would rebuild their player UI which usually involved a complete rewrite.
I'm just saying that if you have it in your head most code is going to be rewritten every 2 years, it's understandable if you don't bother making things easy to work on, and just hack it together.
An argument that "eh, most code is rewritten anyway, who cares" is what I'm reacting to, and have seen countless times myself.
But I don't want to say there is never a time someone will say, "this code is going away, don't spend too much time fixing it now" and it's really really good advise.
Its funny that you mention specifically this related to UI code. I was thinking about it after your first comment, and basically every product that I've worked on that fit this style of development has been UI related. And indeed, most UI projects have had almost complete rewrites every 2-3 years for one reason or another.
The ones that didn't were effectively stalled projects. This makes me think that technical debt really can be thought about in a completely different way for frontend code in many cases.
A few years make sense for UI for consumer products. The evolution of displays has been tremendous in the last decade and a half. Desktop screens have doubled in size and resolution, while mobile and tablets just appeared overnight and when from nothing to where they are today.
Products have changed, but also UI design tends to be driven (or at least influenced) by marketing. And marketing trends change regularly driving a need for markup and styling to be revisited.
In theory, this is just marketing, but in practice if often turns into much more.
In the case stated though, they were talking about a silverlight player.
I don’t think they quite got to the point where they had to rewrite it.
Sure they did. They rewrote it in another language and phased out the Silverlight language completely.
Well, that sounds about right from my experience, for some type of code and organizations.
The original developers have left after a few years, newcomers don't understand the system and may not even try to. It will ultimately get rewritten in newer technologies and frameworks, that happen to be nice for their resume, whether it was obsolete or not.
Indeed. 2-3 years is short, that way of thinking could discourage doing things right in the first place because ‘it’ll be rewritten it in 2-3 years so it doesn’t matter’.
Given that you were working on Silverlight, his decision to stop you spending more time on it was remarkably prescient.
It should have been communicated earlier though, so you wouldn't have wasted weeks on soon-to-be dead tech.
> Most code will be rewritten every 2 to 3 years.
That's kinda scary.
Is it really? It's another way of saying that conditions in the tech industry change every 2-3 years, which isn't all that controversial a statement.
Sometimes it's because developers just want to change tech to enhance their cv or keep themselves entertained. That's scary.
Sometimes it's rewritten because it's so unstable because debt wasn't paid off but total cost of writing it twice is much more than paying off as you go along. That's scary too.
I thought this was the old quote Fred Brooks quote, "Plan to throw one away; you will anyway."
Although then you butt up against the "Second System Effect", so you're damned if you do, damned if you don't.
Within consumer tech, it's more like "Plan to throw it all away; you will anyway."
I used to measure code lifetime in half-lives at Google: the half-life of most of the code there was about a year, meaning that after a year, 50% of your code will have been deleted. It's pretty accurate: by the time I left after 5 years, about 97% of the code I'd written had been deleted. Ironically, I'm told that my one contribution (after 10 years) that still exists is an attribute-renaming CL that I wrote over break at a team summit; basically the whole team agreed it was a good idea, we would never have a chance to fix the problem later, so I just went and did it before the framework got too entrenched to change. Meanwhile stuff I slaved over for months, sometimes even stuff that was directly sponsored by a VP (who is no longer there) or got commendations from the CEO (who is also no longer there) was gone within 1-2 years.
In my experience prototypes / proof of concpts always end up in production. Might as well make a half decent first attempt.
Often times if something needs to be rewritten, it means that the feature became popular enough that it needs to scale up. It is not a bad problem to have.
Even well written code might not scale or be flexible to changing requirements. The feature could even be removed. The effort it took paying off tech debt prematurely would have been a waste.
I think that's a different statement, actually.
Expecting code to be replaced every 2 or 3 years is nervous-making for a couple of reasons.
First, new code is always less reliable than old code, so -- all other things being equal -- devs should lean toward keeping the old code rather than replacing it.
Second, code should be architected so that any changes required to adapt to changing conditions should be limited in scope. The bulk of code for most projects should only very rarely need changing to adapt to changing conditions.
You're presumably used to operating in an enterprise environment with O(1000) customers and O(10) developers, not a consumer environment with O(100M) customers and O(1000) developers.
Reliability issues at large consumer tech companies will usually be caught by the QA/canary/SRE process. If they're not, you'll hear about them soon enough with 100M users banging on the product, and then just do a rollback and fix it on a more leisurely pace. Consumer expectations in non-tech industries have gotten so low that you can burn down whole towns, poison people, and leak millions of records of personal data with few consequences to the company; not being able to access your favorite video for 15 minutes is comparatively minor.
Also, the type of changes in market conditions consumer companies face are usually not those you can architect around. They include things like "We are no longer shipping DVDs to customers; we are streaming content online", "We no longer write web software, we're a mobile-first company", and "our business model is no longer selling personal data, it's payments". There is no architectural fix for "your product is canceled".
> You're presumably used to operating in an enterprise environment with O(1000) customers and O(10) developers, not a consumer environment with O(100M) customers and O(1000) developers.
I have a great deal of experience with both environments.
I have no idea what you're trying to say. O(1) and O(1000000) are the same thing, the O() notation ignores the constant factor.
> Is it really? It's another way of saying that conditions in the tech industry change every 2-3 years, which isn't all that controversial a statement.
I'd say it's an outright incorrect statement, fads come and go but the revolutions (like the move to web apps) are rare and take and take decades to complete. If everything around you changes every 2-3 years then you're hanging around fashionistas not engineers.
Uh... Really?
The example at hand was netflix... The bulk of the platform seems pretty stable to me. If they are rewriting their best code every 2-3 years that'd be a red-flag to me.
I worked on Google Search for 5 years. You would probably say Google Search is pretty stable - at least, when was the last time you noticed it go down? The half-life of code I worked on was about 1 year. Over a 2-3 year period, 75-90% of it would be gone.
We did things like incrementally rewrite a million-line binary from C++ into Java while it was running, or completely change the indexing system from batch processing to continuous updating, or grow the number of documents indexed from about 80B to 1T. There's a huge iceberg of development that you don't see, much of which has to do with scalability (you generally have to rewrite every time a key metric grows by a factor of 10) and much of which is experimental, trying out new features to see what resonates with the userbase.
Very curious about what that Java program is.
GWS (Google Web Server). Powers websearch, responsible for rendering everything you see when you run a search. It used to have a Wikipedia page but basically everything on the Wikipedia page was wrong so the admins eventually deleted it (edit: apparently it's back. Basically everything on the wiki page is still wrong, but at least it doesn't claim GWS is an Apache derivative now). It's the oldest continually-pushed binary at Google (originally written 1999 by Craig Silverstein; at least when I left it still had code in it from Craig, Marissa Mayer, and Sergey Brin). It's changed programming languages twice (from C to C++ around 2005, and from C++ to Java etc. around 2010), and when I left was a frankenmonster of C++, Java, and two proprietary DSLs. My understanding is that it still exists, though most of my friends at Google have since moved on to other teams.
Thank you. Can you talk about the reasons that necessitated the rewrite from c++ to java? I didn’t realize Brin wrote any code that wasn’t python either.
I personally wasn't a big fan, I liked C++ (and Python - there was an embedded Python interpreter in 09-10 that was a casualty of the rewrite, and IMHO Python was a lot more productive than Java for experimentation).
There were some pragmatic reasons though. It's very difficult to multithread C++ correctly, while Java at least has a proper memory model and thread support (note that this was before C++11; at the time C++ had no standardized memory model at all). Debugging core dumps in production sucked. Most of the newer parts of the company (GMail, Docs, Google+, etc.) were written in Java, and the rewrite let us share code with them. Compile times sucked, and Java let us pluginize the architecture, load code at runtime, and build & push each component team's codebase independently (as well as shut them off independently if they started crashing).
I wonder how much of that is due to using the current fad tech... my longest running major piece of code (untouched 15 years+), I implemented in PL/pgSQL. Rock solid, well understood technology, isolated from the current hip web framework that is probably calling into the views. Somehow I think the technology choice is a big part of what ultimately made the program a great investment for the business.
> Just remember that technical debt doesn't apply to all software.
I'm not following. You still need code that could be rewritten and deleted every few years. Any code that couldn't just be deleted as needed wouldn't fit that plan.
So whatever forced code to outlive its welcome is technical debt in that scenario. Examples could be code that used outdated middleware, maintained arcane ETLs to keep data accurate, consumed said data in unconventional ways, code that wasn't repackaged and deployed regularly, etc.
> I spent weeks working on the new architecture, refactoring the streaming code
Seems weird to do that without talking to your boss about it.
How many bosses have you had that suggested paying off technical debt? For me it was always a never ending "next month, after we have rushed out this new feature".
Some, but usually the rule is to not let them know - if they don't understand the problem, they shouldn't be burdened with deciding when and how the problem should be solved either.
[I recall posting this here in the past, but worth re-posting.]
The "handy rule" from the book Team Geek -- the newer edition is called Debugging Teams[1]. It's from chapter "Offensive versus Defensive work":
[...] After this bad experience, Ben began to categorize all work as either “offensive” or “defensive.” Offensive work is typically effort toward new user-visible features—shiny things that are easy to show outsiders and get them excited about, or things that noticeably advance the sexiness of a product (e.g., improved UI, speed, or interoperability). Defensive work is effort aimed at the long-term health of a product (e.g., code refactoring, feature rewrites, schema changes, data migra- tion, or improved emergency monitoring). Defensive activities make the product more maintainable, stable, and reliable. And yet, despite the fact that they’re absolutely critical, you get no political credit for doing them. If you spend all your time on them, people perceive your product as holding still. And to make wordplay on an old maxim: “Perception is nine-tenths of the law.”
We now have a handy rule we live by: a team should never spend more than one-third to one-half of its time and energy on defensive work, no matter how much technical debt there is. Any more time spent is a recipe for political suicide.
[1] http://shop.oreilly.com/product/0636920042372.do
> If you spend all your time on them, people perceive your product as holding still.
This is precisely what bothers me about the vast majority of software out there. It's hard to find software that doesn't suffer from what I call, for lack of a better name, "forced evolution": the devs keep fiddling with the UI and adding new features of dubious usefulness. Meanwhile, the quality slowly, but surely deteriorates: the software gets bloated, runs slower, becomes harder to use, etc.
+1000
This was how I said it, buried off in the documentation of an open source project:
I'm not a fan of the way most software is developed, where features accrete on features in endless succession like barnacles attaching to the hull of a ship until there's more barnacles than ship. I think most developers don't spend enough time on quality, on refactoring, or on cleanliness, and I think they don't weigh the costs of every new feature (and every piece of old compatibility cruft) as heavily as they should. I'm also not a fan of the kind of personal rigidity where one never questions one's own past decisions and thoughts. I regularly make stupid design mistakes, and I don't want to have to live with them forever. Software projects should at least try to value simplicity and purposeful design, and try to avoid the Second System Effect.
> If you spend all your time on them, people perceive your product as holding still
I've always found this hilarious... we spend thousands on hiring developers and one of the primary goals is to find a well rounded person who is okay doing maintenance tasks and ensuring proper code quality and knows how to write effective tests. A short time later, I come on to a project just for a status update and find the developers cutting tests and maintenance tasks at the orders of the PO who then complains that tickets are getting larger and larger.
Devs should not ever say the word "test" to a PM/PO. If the question is, "why isn't the feature ready?" the answer is "because the work took longer than I expected." If you say "because I've writing tests" you are sabotaging the team. Tests aren't any of the PO's business, they are just part of the software that supports the feature.
This is an ideal reality. If the PO is at your desk every day to ask why the thing isn’t done, most developers are eventually going to drop everything non-essential to the functioning just to get them out of their hair.
Sometimes I think that offensive work is always prioritized ovee defensive work. And then execs and stakeholders think when the code is a mess and a simple change takes a day and a half it's the developers fault. And it's difficult to say the least to convince stakeholders to defer a feature for a little bit of refactoring.
This why I think it might be useful to allow the stakeholders and pms to decide the % of time spent on defensive work instead of what defensive work to do. And then you have a specific metric on technical debt(we budgeted 30% for defensive work and we've only spent 10% that's a 1000hrs of technical debt) Something to specifically point to when a stakeholder asks why is this taking so long or why is the application buggy. And ideally this caused a company wide increase in the allotted time for defensive work.
Bonus:It will also give developers who like working on clean projects a number to ask when being hired.
> instead of what defensive work to do
That would solve a problem I saw at a previous job: new features would be described as technical debt, so they could take up tech debt time at the expense of _real_ debt.
My team spends 80% of our time paying down tech debt...by building a new system that will let us throw away the old system later this year, I mean next year, I mean in 2 years, I mean when the last zombie client with no staffing migrates off of it.
That's beautifully absurd. Bravo, management
I think that's a leaky abstraction. Code refactoring can have a huge performance impact that is 100% noticed by the end user.
I think the OPs paradigm holds. If the Code Refactoring has a huge performance impact then it's Offensive work, if it has no impact, then it's defensive.
IMO you just pointed out the key to selling the work.
Reclassification.
Not exactly, because sometimes there's refactoring which has 0 customer benefit. Sometimes it doesn't improve the UX/latency/etc. Sometimes it just makes the codebase easier to maintain, the point of what the OP is saying is that this work is hard to justify. If you spend an entire year making a codebase "easier to maintain" but don't deliver anything tangible that is all defensive work and not offensive work and there's no reclassification which can solve that.
It’s dirty accounting tricks for a greater good. I kinda resent the fact that we have to resort to this.
A huge performance impact is a shiny thing that is easy to show outsiders and get them excited about. There is no dirty accounting here.
The dirty accounting is that doing work to prevent bugs gets no credit, but letting bugs happen and then fixing them gets credit.
Even worse, introducing shiny features in half-baked bug ridden way gets credit, the person who gets to fix all the bugs and possibly reimplement the whole thing correctly gets no credit.
If you're optimizing code while you're refactoring then you're not really refactoring. Refactoring shouldn't change what's happening.
If you want the exact same behavior, then you cannot refactor. Of course refactoring can and will affect performance.
Refactoring should maintain identical functionality not counting performance unless it's a requirement.
Refactoring means retaining the exact behavior, by definition.
Of course, many refactorings will make it easier to apply performance optimizations later.
That makes no sense. If you must retain exact behavior, you must compile to the same assembly instructions.
Generally refactoring worries only about the functionality that you test in unit tests and integration tests, not counting performance.
Seems like "finding another job" deserves a spot in offensive work:
-- it's likely revenue-positive
-- it's visible
-- it involves understanding customer needs
I have a similar idea. I call it the "horizon". There is a certain amount of time where you can go off and do stuff that doesn't result in stake holder visible change. How much time you've got depends a lot on the organisation, but I find that 2 weeks is a good rule of thumb. If you sail over that "horizon" your stakeholders lose sight of you and start to get worried. Ever day you spend over the horizon increases the risk of the stakeholders taking drastic action (from pressing the big, red "stop the presses" button and changing direction all the way to cancelling the project). The key is always to deliver value within that horizon. As long as you keep doing that regularly and predictably, usually nobody will care what else you do.
This is why I think it's good for technical debt to be partially accounted for by development management types - write down how much offensive and defensive work is required to sanely complete a project or just stay ahead of the curve and let the total of those two parts be reported as the cost of the feature - it's a benefit to the long term health of your company.
Gotta love it, this is the same dumb ideology Agile spews. Develop develop develop, forget about fixing issues - They'll magically go away... It's clear that these people do not work in Enterprise environments
I've been thinking about the difference between video game software and most CRUD software. With most CRUD software you are expecting future changes, so you have to be more conscious about tech debt. But with video games (at least older ones), once you ship you're more or less done.
I wonder if that gives room to take on extreme debt at the last minute before shipping a game. An example might be when Steve Ellis added multiplayer to Goldeneye on N64 a couple months before launch. Since they were going to ship a finished product, he was probably able to build it much faster without having to worry about code quality.
https://www.engadget.com/2012/08/14/goldeneye-007s-multiplay...
One thing that's interesting about this insight is that many games are moving away from this model, with regular updates and continued development. This doesn't invalidate your point, of course, but I also wonder how big of a mental shift this must be, because historically, when a game was done, it was done.
That's a great point. One of my favorite games, Enter the Gungeon, had plans for a major release of DLC, but they canned it and replaced it with a much smaller scope of free updates because they had so much technical debt that they couldn't attempt the big DLC release confidently.
https://www.reddit.com/r/EnterTheGungeon/comments/9ykpgc/onc...
Ah, thanks for that! I love that game too, and forgot about this, it’s a great example.
Ah nice! :)
I wonder if that contributes to games being more conservative with the features they offer.
(I'm guessing the biggest factor is the massive investment required to make a game nowadays.)
It's mostly restrictive on the side of game design. Once your goal is to support a large quantity of software features you automatically fall into the mold of a style of gameplay that will minimize impact on feature development.
In the classical model of ship-once gamedev, certain big-ticket software features tend to be thrown out when the designers are in control, because they're so costly to get right and don't surface as much immediate impact:
* Large-scale persistence(just store a minimal representation and instance that)
* Online features(too much synchronization logic - it hinders almost every other feature!)
* Complex AI and procedural simulation(too much chaotic behavior, limited ability to control outcomes)
The transition we've made has, in essence, been to construct microtransaction service products that do support these more complex features: the game is a platform for the sale of assets, and it is massively online with total persistence and sometimes even robust AI. And when you start doing that, your ability to revise the design diminishes substantially. For the most part, you can't take away assets and features that are already there without a substantial player backlash, which means that games of today tend towards accumulating their way into an incoherent maximalist experience with no particular focus or intended playstyle, all of it following templates intended to encourage monetization:
Make sure the players can customize their character cosmetically, and give them a huge array of skills and powers, but nerf all the unique aspects of these powers so that the game will play roughly the same regardless of what you choose. Present them with opportunities to draw in their friends, and add hooks and incentives to come back repeatedly and to check the in-game store for deals. Provide play opportunities for all segments and demographics of players(following the lines of the Bartle player types).
When this model works, it does extremely well in the mainstream(Fortnite basically ticks all the boxes and is outstanding in its ability to progress as a live service) - but it has the downside of being a compromise for everyone. It makes the game too Byzantine in its complexity, too restrictive in its customization, too carefree in its changes to competitive dynamics, too willing to let annoying minor bugs persist - to say nothing of the cost and complexity needed to pull it off.
So I think there's definitely room for more of a spectrum of products, from the mega-blockbuster down to tiny boutique indie games. But the marketplace is still catching up with the idea of providing full service, and a lot of games are still released with the focus being on developing a franchise or on addressing a specific genre. In many cases the capital requirements needed to do services crush that idea out of the gate.
Well, online games that are basically sold as partially or fully a "service" (e.g. Fortnite), and game engines that are primarily marketed to developers of games are, but I'm not sure I agree that applies to games overall.
It's certainly only a part of games, but this also happens outside of games like Fortnite.
For example, one of my favorite games is Celeste, a fully offline, single player platformer that's about 18 months old. They've had a lot of releases[1], and at least one of them sounded like, to me, like a decision that was made purely because it was easier to implement technically. (I won't get into it here because it's pretty off-topic, but if anyone is interested, I can elaborate. It's "The Prologue Dash cutscene is skippable") There's also one more major content update coming that appears to have new mechanics, and it's already late. That said the game is amazing and I don't fault them for any of that at all...
1: http://www.celestegame.com/changelog.html
IMO technical debt does not discern between different types of software. If you make a mistake (or are not 100% correct) you will need to fix it unless you are willing to provide your customers less than perfect software.
Maybe an appropriate way to look at this is that the developer working on the feature could only need the company to take $100 in debt, while another developer needs the company to take on $100,000 in debt for the same work. Typically this difference, unfortunately, does not reflect in the individual developers' salaries.
No, the customers only care about working software, not maintainable, perfect software.
Technical Debt is about taking shortcuts to have something working now, in exchange for more work maintaining and evolving it in the future. If the software doesn't work, it's not debt, it's a bug.
If you are not changing the software, it doesn't matter that a list is hardcoded instead of parameterizable, that the modules are tightly coupled and can't be reused, or that the build must be done following a particular set of incantations in a specific machine. If you are, these are all examples of debt that will come back charging interest.
Properly planned and managed, technical debt is a tool much like actual debt (a loan) that can accelerate your growth and your time to market. Thing is, most teams don't know that they are taking on a debt and don't realize how much interest they are paying each day.
I think the path toward optimizing software for (a) maintainability (b) completeness and (c) bug-free are not necessarily mutually exclusive. I think in general software is fighting all of these battles simultaneously. And so by definition software that is incomplete (ie. without all the features) has the possibility (probability?) of being buggy. Otherwise if it's complete why add features?
Given what you knew at the time you created a definition, and per that definition you could have written 100% maintainable 100% bug-free 100% complete code. It's in uncovering your lack of/ incomplete understanding of your problem that you may need to go back in to that code. Even though at the time it was considered 100% bug-free, complete, and maintainable.
100% maintainable software is NEVER a goal. A hard coded list that will never change is better than a configuration file that needs to be installed someplace and loaded as runtime. If you are right that the list will never change you have made the software more maintainable as you never have to deal all the code to load the configuration from whatever path the user has installed it to. If you are wrong though you will have a lot of work tracking all the releases and what is hard coded in each until you retrofit a loader.
This is the type of choice you need to make upfront often when designing software. If you get it right your system is more maintainable. If you get it wrong your system is less maintainable. Sometimes you don't know in advance which is right.
Technical debt doesn't make the software worse, it makes the codebase worse which degrades the ability to maintain and add features in the future (which is paying interest on the debt, so to speak).
So, in the context of oldschool console video games that ship and then end development permanently, it's absolutely a reasonable decision to add some technical debt if it means fitting one more nice-to-have feature in before launch. Because you don't pay interest or pay down that debt ever. The code is shipped and then it retires, presumably.
The days of fully offline games are long gone though. It doesn't mean that game studios take on less ambitious projects though. Most games are superglued together to get over the finish line and shipped with a massive day-1 patch.
Agreed, the parent mentioned GoldenEye for N64 but it definitely isn't true for modern games
Probably wasn't even true for GoldenEye. They used the same engine for Perfect Dark; sure the assets and game scripts would be changed but the multiplayer code had to be ported and cleaned up I'm sure.
The key defining characteristic of technical debt though is our inability to accurately estimate its true cost. If we could frame the question like you present it the battle would be won
Under schedule pressure that's certainly true, but subsequent products will often start by taking a skeleton of a previous one. In this way tech debt could live on for many additional games unless it's addressed.
Is this actually true? The engine lives for multiple games, and the tooling is often reused as-is, but do games start with existing projects, or do we add new technical debt through the cleansing baptism of copy-paste?
What year are you asking?
In 1982: all games were written from scratch. You might copy the good parts from a previous game, but there generally wasn't much: poor abstractions meant the game logic and UI was mixed together (in their defense, on the limited CPUs of the day you couldn't have made proper code separation work, and of course the limited CPUs also meant you couldn't create such complex systems where it mattered.)
As games entered the mid 90s games started to get releases, and game engines came out. However each release was treated like an all new project even if some code was reused. Games would customize the engine in ways that wouldn't apply to the next game. Bugs in the engine would be worked around by ensuring that part of the game could not be reached (thus resulting in games that you could never get the 100% complete stat as the engine knew you didn't go through some locked door without a key). Some of the core engine was reusable, but for the most part the engine and game were tested as one.
Sports games tended to be the first to figure this out: every year was a new release of essentially the same features, but different player stats. As such it is becomes obvious quickly the game engine maintenance is a cost, there were few new features (until the next console offered better graphics). Most other games were latter because part of the new game was engine changes to support it, so it was hard to find a stable core to the engine.
Gradually games have seen more and more value from separating the engine from the game. As this has happened engine changes are no longer for one game, but for the possible use by all future games. Part of this is game designers have figured out the right abstractions, things that previously would have looked unrelated and thus change different areas of the engine are now seen as variations on a theme if you squint just right - and we know how to squint
Of course there have always been, and probably always will be one-off games written like 1982. When your game is small this is okay.
Different game companies have reached this at different times. Infocom had mostly figured out their reusable engine in 1982.
> But with video games (at least older ones), once you ship you're more or less done.
Technical debt doesn't care about your release status and can kick in long before you reach the finish line, which I suspect happened with Perfect Dark and many subsequent rare games. How many game delays and cancellations have technical debt as the root cause?
In enterprise projects I've seen enough over engineered crap added at the very start of the project that hampered development for the entire lifecycle.
Yes, I think a more general way to put that would be to say the lifecycle of the software affects how important these things are. It’s not debt if it’s not going to slow you down in the future.
It’s common to cram in some debt before a release in the applications world too, with the understanding that meeting the deadline is more important than finessing every detail in the implementation and that it will be cleaned up shortly thereafter. (Yes it’s possible for that part to fall by the wayside, what else is new.)
I like the metaphor of technical debt, but...this essay does not address the most prevalent obstacle with paying down technical debt, which is that the people making the decisions don't care. They are in short-term mode, just trying to get this year's objectives met and IPO or sell or get a higher up job in a different company or division. They really don't care about the long-term health of this software product. Technical debt, as a metaphor, may make perfect sense to them, but they aren't the one who will pay the interest.
Now, if you're working on open source and will be in this codebase for a decade, or this is your company or your software, it makes the tradeoffs much clearer, and that works because your incentives are right. So this metaphor works better for helping the typical developer make choices well, than for helping the typical manager/director/CEO make choices well.
this is usually a communication problem from engineering. you have to be able to describe the impact of the risk and benefits and make a business case for it. no one cares if you want to upgrade a framework. but if you are going to improve customer trust by using a newer version of something and result in less bugs / downtime it sounds a lot better
Sure. But the risk of disruption is usually actually higher, in the short term, by paying off the technical debt. In the long run, of course, it will likely result in fewer/less risk of technical problems impacting customers. But the risk of there being a disruption right now, is nearly always higher if we pay down technical debt, than if we put it off. So again, you have to have something other than a short-term outlook on the part of the decision maker.
Which does happen, of course, but not always, and not almost always either.
One of the hardest challenges of paying off technical debt is that there's no visible improvement to the stakeholder. From a business perspective, they often think it's working now, why not leave it alone?
Also, the metaphor starts to fall apart if you consider good debt vs bad debt in terms of financial investments. No examples of good technical debt come to mind, other than the benefit of shipping quickly.
>the metaphor starts to fall apart if you consider good debt vs bad debt in terms of financial investments. No examples of good technical debt come to mind, other than the benefit of shipping quickly.
I think that's where the metaphor works best. Good technical debt lets you ship sooner than you otherwise would have, similar to how a mortgage lets you afford a home sooner than you otherwise would have. Paying interest isn't ideal, but it may work out a lot better than renting. Likewise, code cruft isn't ideal, but shipping 6 months sooner might pay off a lot more than having a perfect code base can.
https://www.youtube.com/watch?v=XakfJ2spb3w
A pretty good talk from Railsconf a few years back. The debt metaphor can be thought of as two axis: planned vs unplanned, and low vs high interest.
If you explain to the stakeholder that by not paying down technical debt, they're only paying off the interest, but they have to keep paying it forever, perhaps that will sink in. The opportunity cost of those interest payments is profit-generating work. By failing to address technical debt they are constraining the business.
I've come to adopt the metaphor of Technical Tax instead for this reason.
Technical Debt is something for which you have a repayment plan and a timeline on which it will be repayed and you are making the repayments.
Technical Tax is everything else. And your tax rate will keep climbing higher and higher unless you make a change at the policy level.
Good technical debt... well, what's the expected lifetime of a piece of software? How long do you think it'll be maintained and growing, before it is either no longer relevant/needed, or replaced by a new product?
Over the right time scale, technical debt can be a huge win. That's "good debt".
Consider the race against time - either time to market (get a product out the door asap, for valid business reasons), or because it's currently in proof of concept stage and will either be replaced or fixed later, once it becomes a real project destined for production. In either case, technical debt is a good deal. Even if it costs you more in the long run, the time saved in the short run can be the difference between success and failure.
> currently in proof of concept stage > replaced or fixed later > becomes a real project destined for production
These are the justifications that lead to Technical Debt.
Unless you write a self destruct system that prevents the software from starting after a certain date... there is no such thing as 'good debt'.
Projects come from POCs, they get moved into Production and are never replaced, just patched.
Technical debt it the intentional decisions you make to go faster. You can pay it off or not if you want to.
Unfortunately every project I have been on has very little of this. The real problem I see are not intentional decisions, but "If only I had known". The team that choose a framework and 3/4ths the way though the framework company dropped the product (I know a product shipping on WinCE 6 - MS dropped support for that on them 1 week before code freeze). The team that started on some new fad that seemed good only to be the first to find the issues when used in their situation (the fad is overall good, but the limits are not understood).
I've come to the sad conclusion that excessive code quality is a smell. And most attempts to head off technical debt are really just premature optimization, which is its own sort of evil.
Don't burn the village in order to save it.
> other than the benefit of shipping quickly
I think you need to answer why shipping quickly would benefit the company. Shipping quickly is not an end but a means to an end. The point is the state of the market and the impact you plan to have on it, with an added constraint of some deadline.
Perhaps it’s an AR app a company wants to ship in time for a sporting event or art show, and the business has only enough runway to get to a potential windfall of revenue from it?
That being said I think for tech debt to be “good”, the real challenge must be well understood, as well as the alternatives, it must be documented somewhere and all stakeholders must be aware of it, because I’m going to force them to consider it in every prioritization meeting until it’s done or not needed any more.
Personally, if I have a client that’s constantly questioning my expert analysis, the thing for which they entered into a contract with me, I would question whether that relationship should continue. Seeking info is fine and expected; shutting it down is not.
That is the benefit, shipping quickly. It is exactly the same benefit as financial debt: It lets you start working now.
Technical debt is good if what you can create quickly and dirty actually improves your productivity. That is what tools can do.
In the real world if I incur in debt and my debt pays for a tool like a car-truck that lets me move myself and my things faster, I am improving my productivity. Sometimes tens of times.
Think of the difference between a sewing machine and hand sewing. Hundreds of times slower.
I can pay back my debt faster with good debt.
In software, if I can make a tool that in a quickly and dirty way is able to do things in seconds that take me hours and fatigue me, then it makes sense.
Sometimes you can even use your quick and dirty tool in order to clean the dirtiness of your own tool way faster. That is what bootstrapping is all about.
That is what compiler writers have always done. They started in assembler, then created the core of the language, and then they used their own language in order to improve the compiler.
Other writers used languages like Lisp with super slow compilation, most of the time interpreted, in order to create the core of the language in days instead of years.
You don't see the benefit of shipping quickly. For me it is one of the most important things in the world. People's salaries cost more in direct proportion of the time it takes to do something.
I wrote an essay about communicating to stakeholders the value of paying down tech debt. If I wrote it over again I'd amplify the section called "Later".
https://medium.com/@brlewis/fighting-technical-debt-in-an-ag...
> One of the hardest challenges of paying off technical debt is that there's no visible improvement to the stakeholder. From a business perspective, they often think it's working now, why not leave it alone?
I think the article somewhat addresses this where it talks about not needing to pay off debt in areas where it doesn't need to be changed, so perhaps it's a valid argument that if it's "working now" to leave it alone. As for the hard challenge of when to pay off (or accrue) technical debt, it really comes down to individual judgements from an experienced technical leader who has a high level and long term frame of reference. The focus should ultimately be on optimizing delivering overall value to the company or mission. One thing to avoid is to have non technical managers making decisions about technical debt, since it will likely be an uninformed decision.
There's an argument that it's only technical debt if you're paying interest on it. i.e., it is costing you something: time, pain, preventing improvements, etc.
Consider a financial instrument that is interest only on a perpetual timeline, and you are not required to pay down the principle. If the cost of paying the interest forever is still less than paying down the principle, why wouldn't you just keep paying interest? Like you said, it takes an experienced, competent leader to make a good judgement about what the true costs are either way.
I think an interesting way to address this impact, is to start adding more time to each task and address it as technical debt, which is both informative and actually true.
For example after a few months of no refactoring, you start adding 10% to all estimations based on technical debt. It grows more the more refactoring is kept at bay. Similar to how we add a certain percentage to each task for communication, bug tracking, qa, scope changes or anything else. You can keep an arbitrary count, and perhaps even have a "technical debt" reduction amount for certain backlog tasks.
I haven't ever tried this though. But it's an interesting way to communicate technical debt (and is very close to the metaphor).
> Other than the benefit of shipping quickly. How about the benefit of not having to pay the full cost ever if the early release disproved the business case successfully? That is a big financial saving for the stakeholder.
>no visible improvement to the stakeholder.
It's the same as buying insurance, it reduces risk and exposure in exchange for money. Ask management if they want insurance or risk and cash. You pay money to reduce risk. It's a decision they are capable of making when put into non technical terms. They assume the risk of catastrophic failure if they dont buy insurance. (Obviously major refactoring has its own risks.)
You need to tie it to a business need. I have a single project which is being reviewed for both performance. If you don't need a new feature, why change the code? If you do need a new feature: offer a trade-off. You can do it quickly, buy it will be slow, or you can take additional time to improve the performance. Which choice you makes depends on your performance benchmarks/buckets.
My previous job had an excessive amount of technical debt in it's core product. So much so that simple code changes that should have taken a few hours were now taking days, or in some extreme cases weeks. For example there were financial calculations that should have been confined to one class and treated like a black box, but over time bits of the calculation had spread out all over the application from the front end, the server layer and in to the DB. A simple change had a ripple effect on other parts of the application which made bug hunting and testing a slow and laborious process.
After much grumbling from the devs the CEO prepared a presentation where he told us technical debt was our friend. We had a choice to pay down the debt, or add new features. New feature would get us more customers, and more income, which would mean we could hire more developers to eventually pay down the debt. I felt it was a very wishfull thinking type of argument. We were having increasing numbers of outages due to the lack of a proper testing process in our CI pipeline and our DB was massively overloaded. Adding new feature was literally adding to our technical debt and increasing our risk of outages, something that wouldn't help get new customers.
My issue with the above argument is that the problem with debt isn't the amount or type, but the person holding the debt. A 30-something in a safe job (e.g. doctor) earning €100k a year can safely hold €300k of debt, so long as they are responsible and always strive to reduce the amount owed.
A 20 year old working in a fast food joint on minimum wage, but holding €50k of debt is in serious danger. They aren't in a stable job and they aren't being responsible.
Some companies are like the doctor. They have debt, but are responsible and actively work to reduce it. Some are like the 20 year old who thinks that they can deal with it later, oblivious to the rising interest and danger they are putting themselves in.
I dislike the term "technical debt" as it makes it appear that it is something that must be paid. It takes a stance that there is a correct way to do something and that if you owe a debt it's because you've strayed from the correct way and that you need to restructure your work to make it conform more to this correct way.
I prefer to see engineering as a series of trade-offs. These trade-offs come about often as not having enough information about future scenarios and yes, sometimes they come about because you are in a rush to do something. Either way, whether something should be redone is dependent upon a lot of factors and calling something "technical debt" is a sort of way to politicize work as though it is "the right thing to do" and msut be done. This feels too simplistic to me.
Often, shortcuts we take will live for a long time in our code and often it doesn't matter that it's "wrong". If we call something technical debt, we should only call it that at the time that we determine work definitely needs to be done, not as we're going along and thinking "this is not the ideal design" as this will just lead to YAGNI designs.
Having debt doesn't mean you're doing it "wrong"--debt is a useful tool, both in the financial and technical sense.
Debt only becomes a problem when you take out too much of it. If you owe 90% of your monthly income in interest payments, it'll take a long time to a buy a new car. If you have nothing but hacks/shortcuts/etc in your code, it'll take a long time to add new features.
Debt is only debt when you have to service it. Shortcuts that aren't affecting your ability to develop new features are not debt.
> I dislike the term "technical debt" as it makes it appear that it is something that must be paid.
You don't have to pay other debts either, you just have to be prepare to deal with consequences of that (bad credit scores, bankruptcy). Real debt too you often have choices in what you pay off today, versus what can wait for next month or next year.
There's also nothing inherently wrong with taking on debt. Most people will have all sorts of debt in their lifetime, and some economists believe that debt is much more interesting economically than wealth ever is. Debt is not a moral proposition and calling something "technical debt" isn't saying that it is bad/wrong, it's saying that we're writing IOUs we may or may not have to cash at some point. It isn't necessary to avoid debt at all costs because it isn't wrong to have some debt on the books (beyond very conservative religious readings that feel that all debt is sinful, of course).
There is a lot of technical debt you can leave in the system without consequence.
Tech debt mainly becomes an issue when making changes to the code. If no changes are needed in the code and it is working correctly, it is best to leave the debt in there.
I think that gets to other people's points that if it doesn't have consequences it is isn't likely "tech debt". If you don't "owe" it to someone, and often that someone is "future you", then it isn't really tech debt, it was just a trade-off or a poor aesthetic. We have other fun names for ugly aesthetic choices that work and get the job done and aren't likely to be replaced such as "jerry-rigged" and "Rube Goldberg machine" and "duct-taped shambles".
You can default on debt. Debt isn't bad unless it's allowed to grow beyond the means to pay for it. Large companies can have more debt without additional risk. Debt=/=bad
If you miss a deadline because adding a feature causes to many issues: you may have defaulted.
This is painful for me as a Rails dev because the real technical debt is always a lack of test coverage. Cleaning up cruft is great, except without tests I'm just going to break my application in the process (and probably not know it.) Without first defining the behavior of the application, there's a reduced incentive to refactor anything. In this way, there's effectively two layers to the technical debt.
This becomes a compounding problem, where the technical debt gets so great that it becomes more appealing to just keep pushing out features and dealing with the fallout. Everywhere I've worked, the code has ended up exactly this yucky. And yes I've contributed to it too. I just don't think that the idea of paying for someone to go back and make the code nicer with no tangible changes to the application's external behavior doesn't seem to resonate with my CTOs. Maybe I just need to find better places to work.
I have experienced the exact same thing in Python and it feels bad to not make changes out of fear of breaking something. So much so that I won't do refactoring on a large codebase written in a dynamic language with not great code coverage (i.e. pretty much every one I've worked on professionally).
This is where static typing can help a _lot_ in my experience. Your types (and your function signatures) are contracts you still need to adhere to after you're done refactoring. Keep making changes until the compiler tells you it's all good and then usually it is.
That cruft is due to "requirements". It's easier to modify a system with less cruft, because modifying a system means adding, deleting or changing requirements. This is easier to do when there are fewer of them.
If you think of a new requirement, it's easier to be sure that it doesn't conflict with six requirements than with sixty, or six hundred.
Additionally, requirements are hard to remove/change once their implementations are deployed and depended on.
Basically we are already saddled with debt at the requirement level, before we even consider the implementation quality.
Thus, the first tool in the fight against technical debt is to very carefully manage the acceptance of new requirements. If a requirement doesn't exist, then its code doesn't exist, and that's as clean and maintainable as code can possibly be.
Secondly, try to gather all of the requirements up-front before implementing anything, and then resist futher requirement creep. If most of the requirements in a system were gradually introduced after implementation began, that will tend to degrade the quality.
I kind of like the idea of calling it technical debt- because I can declare technical bankruptcy (move companies). But really it doesn't work, partly because technical debt very often is actually the result of doing a slap-dash job in the past. If this code was unimportant enough to just throw together in the past you really need to ask yourself whether it's important enough today to fix. This feeds in to one of the fundamental personality types - someone who may be more interested in making the code beautiful and streamlined and neat than actually improving the bottom line.
There are cases where you need to consider technical debt, but most often it should be considered as something you create rather than something you inherit- because the creation of the technical debt is where you're making the judgement:
* Is delivering now important
* Is this code going to need to adapt and develop in the future
* Is the code this is building upon going to stick around or will this code likely be superseded.
* (Selfish developer: Am I going to be around to deal with this)
I think when trying to categorise technical debt the Technical Debt Quadrant is a better tool [0]. It's already linked to in the parent article.
A lot of people are shoving all technical debt in the Deliberate / Prudent category. However I've seen a lot of code bases written by inexperienced developers and it would have be done completely differently if they had any experience (both of the inadvertent categories). This means actually updating the code base can be a complete pain. Sometimes you know if you're editing the code you risk creating huge regression issues and suddenly something that should have taken a short time can spiral out control if you want to add more functionality.
[0] https://martinfowler.com/bliki/TechnicalDebtQuadrant.html
My framework to think through technical debt: I don't think of it as debt (negatively connoted) but business investment.
When you take a debt it's usually to finance something. Focusing the wording on debt highlights the negative side of the trade off. It's like we're suddenly all turned fiscal austerity hawks, where any bit of debt is evil.
I feel like developers are each placed somewhere on a continuum from purist to entrepreneurs. Purists can spend weeks to tweak an SQL query, while entrepreneurs would be happy with a no-code Zapier MVP that actually lands customers
It's not to say that one is always better than the other. A balanced team probably needs both. In our company, I'm more on the entrepreneur side (which I guess is good for launching a startup) but the recent addition of a "purist" to the team has improved our codebase significantly.
People who understand financial debt don't think of it as evil. It's a tool they use.
Technical debt is the same way, and I often talk about it that way at work. But it's a phrase that means something that "business investment" doesn't. When I tell my boss that we're incurring technical debt (and what exactly that debt is), he is already in the "business investment" mindset on the matter. I'm the one trying to reign that in and focus on making the future easier by avoiding some of that debt now. But I fully understand both sides of the matter.
That fits with my belief that the notion of having technical debt is a luxury that only a measure of success brings. Until you're in a place where you have a excess of resources (time and money) all you have is code.
Obviously with experience you can make decisions that can lower the volume of technical debt once you reach that breakpoint, but in my view the right play is always to get to that moment as efficiently as possible and that never involves re-architecting or rewriting until then.
'technical debt' is a misnomer.
The reality of most code, especially as it gets closer and closer to the end user, is that over time there will be requirements added and removed. It is really hard to perfectly design such a system from the start.
Thus some level of technical debt is inevitable and it isn't really debt. For instance, an overriding goal of zero technical debt is not always the right decision. You may be expecting additional requirements and have not come up with a better conceptual model.
This focus on the technical makes you lose sight of underlying issues which are often organizational rather than technical. Is someone neglecting the code intentionally to get a promotion for a rewrite? Is there a ship now or feature mentality? Are your engineers empowered to change the code?
Other industries and systems have the same issues. What do they call this?
You can't redefine technical debt to include future requirement changes, and then say that definition is wrong and it isn't really debt.
It's not future requirement changes. Its that the conceptual model that you constructed for your original requirements doesn't accommodate a new requirement. Now if you construct a new conceptual model, you will have to modify the original code. It sometimes isn't feasible to do this to each new requirement especially if you suspect that the next new requirement will require yet another change to your conceptual model.
This can happen because the product roadmap changes (new PM, business changes, etc...)
Try writing UI for several different designers over the course of several years for several different iterations of features when there is no common design language.
Technical debt is about quality of code, not about changes in your conceptual model.
Lets say I had a warehouse that stored food, all stored neatly in its own section, each with the least frequently moved at the top where I need a forklift to access, and the most frequently used at the bottom where they are easy to access.
Someone orders too many boxes of rice, and there's no room in the rice section, but there's room on the lower shelves of the sauces so it gets stored there, and some are left on the floor, partially blocking forklift access.
The immediate problem of rice storage is solved, but every time I need sauces I have to grab the forklift, and it may take longer to get there and back because the route is blocked. This will persist until I refactor the warehouse so that there is more space to store the rice appropriately.
Now the manager decides we should be able to store refrigerated goods, and I have no refrigerators yet. The work to refactor the shelves to make room for the refrigerators is a conceptual change in how I use the warehouse, and not a direct consequence of how I've previously been storing goods. It's not tech debt.
Quality of your code can be affected because your conceptual model is muddy. Makes it more complex, harder to understand, harder to change. Makes you implement a requirement in two different ways.
It looks as if people have different definitions of technical debt.
I think it is worth noting that different organizations/teams may require differing amounts of work related to technical debt. This is often driven by business culture, priorities, team experience, long term business goals and many more.
From experience, a lot of people/companies have difficulties marketing what/how/why/when technical debt should be tackled for both internal and external stakeholders. This leads to a less than optimal structure where "new" or "shiny" work is preferable and rewarded, as opposed to a more balanced approach.
I disagree that productivity cannot be measured. Measure productivity by the time it takes a developer to achieve a proof of accomplishing the business requirement through test automation. If it takes a developer 1 hour versus two business days of 8 hours there is a 16x factor of productivity.
Tech debt are all the factors that erode productivity. This includes many technical factors from inefficient systems design to unnecessary testing. It also includes slowness from weak developers and developer training.
I know it puts me in an extreme minority, but I am a big fan of extreme simplicity, which is not what it might suggest. Extreme simplicity removes everything in a system present for the easiness of the developer so that all that is left is only that which is necessary to accomplish a business requirement as directly as possible. If there is less to read (including dependencies) there is less to maintain and less to test. Simplicity is not easy.
The confusion is that a system must achieve a certain level of easiness or it will be rejected by some developers outright or require training at great expense. My basic premise for making any challenging decision is to never compromise on integrity, which includes being honest about the challenge at hand. You can embrace that challenge directly and solve for it or you can hide from it behind layers of easiness.
This assumes goal complexity is measurable and you can run randomized experiments to measure what you call "productivity". Neither of these assumptions holds in the real world.
E.g. changing some business logic can be 10 minutes of work to implement a hack (which will pass tests, mind you), or 2 weeks of strenuous yak shaving for a "principled" solution. Which would you rather pick? And more importantly, are you prepared to select for developers who favor 10 minute hacks to principled solutions?
You measure productivity by the time it takes to complete a task. That is both objective and measurable.
How do you define a principled solution versus a hack? All that matters is whether the result achieves the desired business goals without regression, and that is what tests are for. That being said I would gladly pick the 10 minute solution that passes all the tests.
So much of development is bullshit posturing for developers to justify their existence with unnecessary tasks to make things easier for themselves.
You measure productivity that way on the assembly line or other menial tasks where it's obvious how to do things and the work isn't very complicated. Trying to do this with creative professions is a disastrous mistake.
>> So much of development is bullshit posturing for developers to justify their existence
That applies to any profession. Developers aren't unique in this regard.
> You measure productivity that way on the assembly line or other menial tasks where it's obvious how to do things
Isn't it obvious how to do things in programming? Many developers, in my corporate experience, occupy themselves with how to do things in code. That is exceedingly unfortunate. How to write code should be an obvious disqualifier before obtaining employment. Nobody would hire a lawyer, for instance, who didn't know how to a legal argument.
Writing code should be as obvious as writing an essay, but I suspect many developers that are hiring probably have trouble writing essays as well.
Anyways, if you had the choice between a 10 minute pizza and a 3 day pizza with ingredients more evenly spaced apart which would you choose?
>> Isn't it obvious how to do things in programming?
Not in my field, no. But I might be an oddity. I get hired to do the kinds of things others have tried and failed to do. So while they may be "productive" by whatever arbitrary metric (solving small problems real quick), they fail at gnarly, "deep" shit which can't be done in a day, or, sometimes, in a week, but which does need to be done.
Writing code is not a problem, changing it without breaking is. Taking tech debt is not a problem, paying interest on it is.
And you will hardly hire a lawyer to work in unknown to him legal domain in foreign country in foreigh language.
> changing it without breaking is
That is why the software gods invented test automation.
> paying interest on it is
That is why simplicity is important, because less is more. All code is ultimately debt as it demands some amount of maintenance. When there is less to maintain there is ultimately less debt.
> And you will hardly hire a lawyer to work in unknown to him legal domain in foreign country in foreigh language.
Has that ever happened to you as a software developer? The one time I was told to learn a new language I was allowed the time and space to learn it.
The problem with the concept of technical debt is the implicit suggestion that you know what is debt and what is not up front. An awful lot of debt is stuff that people thought was needed but turned out to be totally off base and now weighs everything down. And large piece of that - not by any means all or even most, but a large piece - is people pre-emptively optimising the design to satisfy precepts put about by people like Fowler himself.
To reduce/remove tech debt from my code, the only patterns i found helpful is: Separation of concern: The chunks of files/codes that work or removed together.
One typical example in real world is React Hook.
By moving boring stuff into a folder called "hooks", we centralize concerns into its own file, so that we could manage them effectively, or we don't need to think about it anymore.
I find the phrase 'Technical Debt' to be unhelpful. If I am in financial debt I will definitely need to pay it off unless I go through bankruptcy. Technical 'Debt' seems to operate by bad analogy: I may opt to completely wipe out the subsystem that has accrued the debt and replace it with something else, or it could turn out that the subsystem solves some problem that wasn't business-critical anyhow.
I've had to restrain developers from "turd polishing" systems that were poorly thought out to begin with. Code gardening in a subsystem that was simply the algorithmically wrong way of doing it (that we might have had to have shipped quickly in order to stay afloat) isn't a first class activity. The problem is with the metaphor of technical 'debt' is that implies a balance sheet, but many (most?) of these bills will never come due.
Technical debt really applies when time-saving decisions in the past cost more time in the future. As such, you can't always be 100% sure whether you're incurring it at the time or not. You can only be sure later, once you have to pay it.
But there are definitely times that you feel certain it's going to end up being technical debt. When you implement a system that doesn't scale, but you know it has to eventually scale massively, that's going to be technical debt.
You could write it right now to handle that load, but it'd take a lot of time and money. Or you could write it the quick and dirty way, and get money coming in immediately and put off the scaling until it's actually necessary. You know it's coming. But you decided to incur that debt in order to make money sooner.
IME, technical debt is usually about time or money, and getting something done cheaply now and worrying about the full cost later when it's more applicable.
It's a tool, not a problem, so long as you're managing it and not just blindly choosing to do things the quick and dirty way all the time.
That's the difficult part of being new on a project. If you aren't building the replacement is progress being made to fix the issue. With regards to gardening: if you're spending all your time weeding but you haven't planted any seeds what are you growing?
Hopefully the manager has enough sense to help a new person see the priorities. As a manager I've definitely 'called time' on weeding. Good analogy.
Something takes 20 hours to implement, but you do a hack job in 2 hours. Next time someone touch that code, they probably need to spend the 20 hours to re-implement it. Technical debt can can be made less costly by using the test-first approach, you start by writing a test that confirms what you are going to implement works. When doing test-first, it's often fine to just do the bare minimal work. If something needs to be added/fixed, write another test, then do the bare minimal change, repeat. Eventually the code will be so complex a small change will take hours, and that's when you have to pay the technical debt, but if you have all the tests, you can make a clean re-write, and the tests confirms that the new clean code works just like the old code.
Businesses should probably start tracking technical debt as a liability on their books. Maybe depreciate code as an asset like a business does with a truck or piece of equipment. Management would probably understand the concept better if it was represented on the financials.
A related concept: defaulting on technical debt.
If you know a piece of crufty code is going to be irrelevant in the near term after some major architectural change or new feature, it isn't really worth it to pay down the principal with minor improvements or shims.
I think it's marginally useful to think of low vs high interest debt in terms of technical debt. I can make assumptions that certain areas of software will not change in the future, but that in itself becomes technical debt. Now future features have to be designed around the idea that certain features are off limits. Sounds extremely risky to me to try to hedge your bets on this kind of foresight. On the other hand, I think it's extremely useful to examine different libraries / modules as having different levels of technical debt, but the metaphor stays intact. You just look at your software not as a single loan, but many micro? loans.
Technical Debt becomes really toxic when mission critical customer facing functionality is based on it. Just this morning we received a ticket where cleaning up technical debt caused a major issue for the customer, when we looked even further we realized that this customer relied heavily on the state brought on by technical debt and that resolving this issue will require an even more careful refactoring to ensure that the debt is cleared but the overall functionality is unchanged.
As a workaround for the time sensitive issue we literally copied the old code and overloaded our class so that the customer runs the old code while we develop a robust fix.
> Given this, usually the best route is to do what we usually do with financial debts, pay the principal off gradually. On the first feature I'll spend an extra couple of days to remove some of the cruft.
This should be managed carefully to prevent a lot of independent gradual improvements that don't follow a clear architectural pattern and end up conflicting with each other.
Furthermore, all code changes to existing modules require validation of the affected functionalities, which are often hard to spot. Hence, I'd say that a wide scale refactoring effort needs a clear testing plan before it's put into practice.
I think technical debt extends beyond poor quality code that prevents additions or modifications. Having worked with a large amount of scientific software it extends into the efficiency and robustness of software.
Frequently, efficient methods or parallelization are not used and the least creative method is implemented to solve a scientific question. Functions are not written in a modular way but merely for a “quick and dirty” analysis.
The end result is that you spend more time getting things to run or waiting for them to run than if you had gone back and reworked the code to begin with. It’s a serious drag on productivity.
A startup without technical debt, is mismanaged. The financial runway is the first consideration for survival. Spending a second on anything that doesn't get the startup to the next phase is a second squandered.
Any exception to that?
Any market-leading product which is known for having the worst codebase out of all the competitors?
This article frames technical debt in terms of sub-par code, which is part of tech debt but only a part. Legacy systems that are nearing end of life, patches and updates that haven't been applied, all of that represents technical debt as well. And when it comes to some of it, it can be actual financial debt too. Plenty of legacy systems sit around a long time past their sell-by date because of the financial cost of replacement, which may be orders of magnitude more than the cost of maintenance.
Funny thing about technical debt with programming is that sometimes, and in my experience I did not had to pay it back... Software went to trash just because business changed direction.
A lot of people seems to complain about the wording. I understand the analogy doesn't cover 100% of the issue but unless somebody finds a better one this is the best we have. Good enough so that stakeholders do sometimes allow us to take care of it. Yes probably not all the time, and not enough, that's why we need to get better at communicating about it. In a corporate environment, communications skill are as important as technical skills, and we should seek to improve both.
Technical debt is like real world debt.
You can either take the debt and try to increase your business revenue more than the interest of the debt.
The same is true for the technical debt,or decision not to modernize / write shitty solutions to fix something quickly.
In real world any business man would ask before taking the debt: is there a chance paying it back.
But no one asks about paying technical debt back,or if ignoring it will yield a high enough return ..
For anyone interested, a new book on the topic just came out
"Managing Technical Debt: Reducing Friction in Software Development (Sei Series in Software Engineering)"
https://www.amazon.com/Managing-Technical-Debt-Development-E...
As a side note, I noticed Fowler linked the CannotMeasureProductivity page using camel case, and that took me right back!
I remembered using PhpWiki and being able to auto-link pages just by using camel case. I don’t believe MediaWiki has that capability. And it’s such a standard I had forgotten it even existed.
Something very soothing about the idea that links could be handled so gracefully.
This is not just about the code, but also about the feature this code implements.
A feature implemented through "indebted" code tends to be buggier and less performant. There are exceptions, of course, but by-and-large this has been my experience.
So the "interest" is not repaid by just the programmers, but also by the users, over and over again as they use the feature.
How many companies are profiting from a poorly written codebase that just works well enough to be a parketable product and isn't being maintained and there is no plan to maintain it? I can think of many utilities not being actively developed that are still part of or a product in their own right. Surely that isn't debt in any sense?
It's not "technical debt", it's "escalating risk". Technical debt gives the wrong impression to business people.
https://twitter.com/jessitron/status/1123310331957145601
I really like the comparison of TD with the Tetris game : "You can’t win. You can only control how quickly you lose"
Already posted on HN here : https://news.ycombinator.com/item?id=19353352
The concept of 'technical debt' seemed to emerge around the time that the concept of 'scoping' went out of of fashion. The best engineers that I have ever worked with were truly masters of scoping projects.
> Stated like that, this sounds like a simple matter of working the numbers
I don't think it is that simple, because the attempt to remove cruft might introduce new one and can make it even harder to reason about the code.
He goes on to say it isn't that simple because we're bad at estimating and measuring productivity.
It's only debt if you have to pay it back.
Technical debt is a bad name for this phenomenon. The word debt implies that we know it exists at the time of taking on the debt and that it can be paid back.
In reality a lot of technical debt can never be paid back without restarting from scratch, and a lot of it is accumulated unknowingly only to be discovered way later in the project.
I would say technical debt is only that which you already know about but have made an intentional decision to take out a “loan” to ship quickly.
I’m having trouble thinking of an example of financial debt that can be taken on unknowingly. Maybe identity theft? But that's less like an engineering tradeoff... more like NSA backdoor type stuff.
Debt heavy choices made knowingly are more rare, as programmers consciously avoid it or choose debt that won't accumulate beyond certain bounds.
Whatever it is, I think most projects deal with "technical debt" accumulated unknowingly. There's no theory around how to create a "good design" and thus it's often shooting in the dark or using preexisting patterns. Most software projects have design flaws that only become evident later. In my career, I have dealt with this type of problem more-so then debt accumulated knowingly. Additionally my current company is calling this type of debt, technical debt. I have rarely rarely seen technical debt taken on knowingly and documented.
> Debt heavy choices made knowingly are more rare, as programmers consciously avoid it or choose debt that won't accumulate beyond certain bounds
I don't know what kind of software you work on, but I disagree. It's incredibly common, from what I've worked on, that we have an existing subsystem X, and project management wants it to interact with this new subsystem Y but it was never designed in a way to do so, so either we can hack it in real quick, build a moderately pleasing integration with only a little technical debt, or completely rebuild X from the ground up in a way that makes sense and leaves no technical debt. From my experience, bad managers will want the quick hack while engineers and good managers will take the middle option. Nobody opts for the complete rebuild.
The design of subsystem Y is likely technical debt. The reason is, because of agile, software developers know that they must code defensively to account for feature changes or additions in the future. That means you take on technical debt every time you make a subsystem that isn't modular, meaning that if system Y is not infrastructure it should be able to be pulled out and used somewhere else. To not account for this feature is to not account for future change.
A good way to test if your subsystem is modular is to ask yourself, can you pull your subsystem out of the current system and use it in a completely different app in a completely different context? If the answer is no, then likely the module is either IO or too tightly coupled with other systems. Too much Mocking in your tests is another sign of a design flaw. It's a signal that your code is dependent on other systems rather than a module that can be pulled out and reinserted somewhere else.
Another design flaw is, is system Y itself made up of modules or interconnected dependencies? If subsystem Y needs to reconfigured to do something slightly different can I pull out 10% of the system and replace it with new modules? Or does pulling out 10% of the system involve pulling out another 80% of the system as a dependency? You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.
Most programmers choose dependencies because modules are harder. Also the dependency style of programming is heavily promoted. Additionally, most programmers are Unaware of what it means to make a modular component, they organize code following nomenclature rather than a systematic theory, and they think they are doing modular design but they are not.
Likely your subsystem Y was designed in such a way that it is tightly coupled to another subsystem and can't be used outside of a certain context. Possibly it is tightly coupled with IO or some specific database schema. Likely system Y itself is made up of modules that are interdependent. Also likely it was programmed by someone who did all of this unknowingly.
So it could be a matter of perspective. What you see as debt taken on deliberately I see as debt from the past "discovered" and interest accrued because the debt can't be paid back without a rebuild.
At that point, it's not technical debt, it's sunk cost.
I Suspect my husband of cheating his attitudes were enough to raise suspicions and I need to be more convinced enough with evidence,I came online to read some hackers review andi contacted one of them which he demanded for some certain amount of money and I paid, after payment he came out with nothing before a friend recommended Hack Wizard to me i contacted him immediately, he monitor my husband illicit activities; his phone Calls, Text Messages,Whatsapp,Snapchat,Facebook,Hangouts, Instagram, Twitter and Email I was able to spy them with little deposit. His job is fast within 2hrs and 100% deliverywith no trace,He have many service to render, Clear Criminal Record,Repair your Credit Score, Credit top-up, Increase your Grade, Monitor your Children etc Contact him at CYBERWIZARD1995 @ GMAIL COM Whatsapp: +1 662 727 5740
One good analogy is how the chefs in the kitchen clean up there table right after they are done with a dish, If you do not clean up, There will be a delay in the next dish and the time to make a new dish increases exponentially and the a dish is never delivered within a reliable time interval.
"Thinking of this as paying interest versus paying of principal can help decide which cruft to tackle"
I think there is a typo here -- this:
"paying of principal"
should probably be this:
"paying off principal"
or similar phrase like "paying towards principal" or "making a payment towards principal", etc.
I always thought "technical debt" was just a buzzword consultants use to make the client pay more money upfront. To me it translates to simply "work we haven't done yet".