Unit Testing Costs Too Much And Slows A Project Down
The company I contract through has a tradition of having monthly lunch where all of us contractors get together, eat, share war stories and on occasion present new techniques or ideas. Today's presentation was on fundamentals of unit testing. The presenter had put the presentation together as an introduction and he was intending on presenting to his client and wanted feedback. Lots of good conversation was had around getting back to the basics with unit testing and how to teach the technique to new developers.
One of my fellow contractors mentioned that management at his client had made the decree that all new code should be unit tested and have 100% code coverage. His response was that unit testing slowed the development process down and adding cost to the project. But he would do it if the customer was going to pay for the extra effort.
I interjected and said that I disagreed with his basic assertion. I said that unit testing over time ups the quality of code and reduces cost over all, by finding bugs before they make their way into production. The room almost universally erupted in disagreement. One member said he had never worked on a project where unit testing prevented bugs from getting to production. One asked if it was a fair statement that unit testing slows down the initial startup of a project. I said maybe.
The way I see it there are three discussions happening here. First, it is a reasonable expectation that management can dictate all code will be unit tested with 100% coverage to a team completely unfamiliar with the technique and expect them to be successful. The next involves the practice of unit testing and the overhead it adds to the production of a codebase. Finally is the assertion that the cost of unit testing adds to the cost of the overall project long term. I'll attempt to make an argument on all these topics.
Before continuing, let me say I don't write this as a scathing rebuke of my comrades. We are all in different situations and under different constraints. This is simply the way I think about these topics and my experiences. The sass is all in good fun.
Top Down Mandates by Management
When management is pushing development to adopt unit testing to me they are saying they want the number of bugs reaching the production environment and affecting the users to be dramatically decreased. They may have read an article somewhere that the testing practices have shown to produce fewer bugs in production.
But is it reasonable to mandate unit testing and 100% code coverage to a team that is not doing it already? Or worse has no experience with the technique? Probably not, you are dooming that team to failure and a lot of wasted effort. I personally think this is where the unit testing slows you down and adds cost mindset starts.
Let's step back and consider this from other life experiences learning something new.
When I was a boy, I wanted to learn to ride a bike. All my friends were riding bikes and it looked like fun. Up to that point in my life, if I wanted to get from point A to point B I walked. I had many years of experience walking. But the first time I got on a bike to get from point A to B I ended up with dents in my skull, scraped knees and elbows and a hospital bill. It took me five times as long to get to point B, if I got there at all.
At that point, I could very easily have walked away from bike riding and declared my my bike riding friends elitist jerks who have never really had to get from point A to point B in the real world. Riding bikes is dangerous! It leads to hospital bills! And to top it off there is no benefit; it actually takes longer!
Instead I stuck with it. I learned to keep my balance. I started wearing a helmet as my default bike riding gear. Eventually, it became second nature to ride a bike. I could walk any time I wanted to and I could ride a bike and do it really fast.
Management making the kind of mandate described above is the equivalent of telling that boy he was going to get on a race bike, ride in a BMX championship race and expecting him to win. That poor kid is still trying to figure out how to get his feet on the pedals and not fall over.
It is an unreasonable expectation. Introducing unit testing to an inexperienced team takes time and practice. Forcing it into an existing codebase is going to frustrate newbies. Managing to a metric like code coverage is going to encourage gaming the test framework to eek out numbers that don't matter. I look at coverage as a valuable heuristic that loses value as the number goes up. At 10% coverage, you have a problem; at 80% there might be value in spending your time elsewhere (sorry Uncle Bob).
Developers Unit Testing is Slow
Does testing slow the development process down? On the surface of that question it seems like the answer is obvious, yes. You are writing more code. You are forced to change the structure of your code to make it testable. You have to put in the mental effort to make abstractions that collaborate together and to think about the edge cases of all the possible ways this bit of code could be used that will break. That certainly is slower than sitting down and writing procedural code, committing to to source control and shipping it. Well, duh!?
I would argue that that is exactly the kind of work you are being paid to do; thinking. The act of writing code is the easy part. Any cut and paste developer can do it. The result of just writing code without deep though and consideration is where we get technical debt from. It is how we introduce subtle bugs that have long term damaging effects to not only the maintainability of an application but the data that that application relies on.
A well maintained and abstracted codebase is easy to change and add new features too. Unit testing as a technique encourages you to make that code in the first place.
Attempting to introduce unit testing to an established codebase is painful and hard. Yeah, it takes time. It might feel like you are spinning your wheels and you are making things worse. It is going to be a slog. But the effort yields fruit over time. There is a whole cult of personality surrounding Michael Feathers, who wrote the definitive guide on the topic. Yeah, I am a total fanboy.
But when starting a new codebase with testing in mind from the beginning, things are a bit easier; especially to an experienced tester. Someone with experience and a well covered codebase can add new features quicker, isolate bugs faster, fix issues easier and have some level of certainty that a bug will never show back up in production again. They have the freedom to experiment with the codebase and try new tools & techniques without fear that they are breaking things. It opens up a whole world of possibilities.
How many codebases have you worked on that everyone was scared to death to touch that one area? It is riddled with bugs and performance issues, but nobody will touch it. The team just carries around the dead weight around their neck like a noose waiting and hoping it doesn't hang them.
Unit Testing is Costly
Does testing add to the cost of a project disproportionately to not unit testing? Let's take an example from today's presentation. The presenter showed a bug in which a comparison had a simple bug. I don't recall the exact code, but the gist of it was this:
if(thing.value > .05m) {...}
The correct code was this:
if(thing.value >= .05m) {...}
His business rule clearly stated that it should be applied if the value of the thing was greater than or equal to .05. This line of logic takes say less than a minute to write. Adding unit tests to exercise the logic that considers edge cases might add an additional 100 lines of code to the codebase and take that developer say 10 minutes to produce. Given the developers bill rate, that is a pretty huge inflation of cost to validate that single line of code. Now extrapolate that out to the lifetime of the project and hot damn what a waste! Or is it?
The act of the developer catching this issue on his local machine and fixing it forever has value. Say the developer didn't catch the bug until he deployed it to his development environment. Do we calculate the cost of the wasted deployment? Or the effort to commit the fix to the codebase and redeploy? How much is that costing?
What if the developer doesn't catch the bug in development and it escapes into test. Your tester is now spending wasted effort testing the feature. That has cost as well. Sure, your tester is probably doing that testing anyway. But when they find that bug, how much work is added to log the bug? To take screenshots? To write up reproduction steps? Meetings? Bug reports? To retest and validate the fix? We have to add that cost on top of the developer's effort to fix the bug.
What if our tester misses the bug and it leaks into production? What impact does that bug now have on our cost in fixing it? Are we now involving release management and others in the cost?
What if our bug makes it in to production and goes undiscovered for six months? What cost do we add to correct the data that has been corrupted? The customers we lose to miscalculated bills? The lawsuits that are filed? God forbid the King 5 news reports that are aired?
Focusing on the developer cost of unit testing is myopic at best. It is at worst professional negligence. If you want to read more about that, Uncle Bob has a book for you as well.
Unit testing is not going to catch all bugs from escaping into production. That is impossible, bugs happen. Hell servers die, hard drives fail, networks drop. Chaos Monkey will always be on our backs. But the cost of preventing as many bugs as possible from getting there, is worth it in my estimation.
"Olympia Washington_2012 03 24_0581" By [Harvey Barrison](Harvey Barrison) is licensed under CC BY 2.0