This article originally appeared in the October issue of No Fluff Just Stuff magazine. The copyright recently reverted to me, so here is the reprint for the general public. As an unneeded aside, I’m really happy about the way this article turned out. I hope you enjoy it too.
In our industry no one recommends increasing waste, there are no waste evangelists, and no pro-waste lobby. So why is there so much of it? As we scramble frantically through the day to please our customers we are often not even aware of the pointless trail of half done work and unwanted features we leave in our wake. This article helps you understand what waste is, where it comes from, and what you can do about it by mapping the seven wastes of Lean Manufacturing into the software development field. If you want both better software and happier customers then quit wasting time and read this article.
There’s something very natural about seeing waste in the world around us. We think about waste almost every day in our personal lives. For example, just in the last few minutes I’ve thought to myself: I should recycle these stacks of magazines (wasting space), I should watch less Tool Academy (wasting time), and I should really start running again (wasting away). Many of us are equally good at spotting waste at work: pointless meetings, fancy meals for the sales staff, and expensive development tools (SharePoint anyone?). And every year, around January 1st, we recommit ourselves to avoiding the same wastes as the previous year, willfully ignoring our past failure to lose weight, quit smoking, or watch less reality television. Likewise with work, every person in an organization from CEO to janitor can rail about wasted time in meetings yet somehow the situation never improves. Clearly, finding solutions to waste requires more than a statement of intent.
Eliminating waste and improving productivity should be easy and uncontroversial. There are really only two rules to follow: 1) stop doing things that aren’t worth the effort or 2) do the same things with less effort. These are the basic building blocks of improvement, whether they be related to your personal or professional life. If you’re wasting your time watching trash TV then stop! Skip those quarterly meetings where half the audience is playing Bejeweled on their phones. In fact, why not dial in to every meeting you can and write your code to the background noise of a glacially paced decision process?
If these two rules of waste are so simple, then why is making a lasting change so hard? Why do we have the same resolutions every New Year, and why aren’t we hurtling forward at work to ever higher levels of productivity? The reality is that we are good and seeing waste but not good at seeing its causes, and we often eliminate a wasteful practice without finding a suitable replacement. Consider the classic resolution of losing weight. We say to ourselves, “I’m going to eat less junk food this year” without asking why we have this tendency in the first place. Caffeine and lack of sleep are both known to contribute to overeating, but we don’t often focus on these causes. Excessive meetings form an even better example. Why do some workplaces have so many meetings? This is often a sign of a broken decision making process. Here’s a fun way to fill up your daily calendar: when you can’t make a decision then call a meeting and get everyone to talk about it. But be sure to leave the agenda and result open ended, so the conversation can be free form and lead to no particular resolution. Your days will quickly fill up leaving you busy yet unproductive, and you can use your quick lunch breaks to bemoan your culture of meetings. No, on the contrary, fixing a culture of meetings requires you to both find the cause and make a replacement. First identify the real problem, difficulties making decisions, and then experiment with new ways to solve the problem. Perhaps you need to discuss and embrace decision forms other than “unanimous”, or learn to hold shorter, more specific meetings, or just create a team decision board and handle issues in the team room instead of the conference room.
The antithesis of removing waste is what I call “Write Only Process Improvement”. As a group tries to adopt a better process, they are only allowed to add new practices without removing unneeded ones. This is the bane of agile pilot groups around the world. Your team is certainly free to try a Scrum style iteration with story cards, a burndown chart, and iterative planning. But you still need to adhere to the enterprise standards of full requirements documents, weekly status reports, and quarterly deployments. Feel free to add whatever you want to the process, just don’t dare remove anything. This arrangement requires duplicate work, quickly consumes the time of the ScrumMaster, and leads to the conclusion that agile methods are slower than the alternative. Of course a new practice is slower if you’re not allowed to abandon the old way of doing things. The Agile Manifesto has a wonderful parody site called the “Manifesto for Half-Arsed Agile Software Development“. While the original manifesto claims to value “Working software over comprehensive documentation”, the parody adds the line “as long as that software is comprehensively documented.” This rings perfectly true for so many Agile pilot projects: please try your new process, but be sure to follow the old one as well. New practices are adopted despite “Write Only Process Improvement”, never because of it.
To reduce cost and be more effective we must eliminate waste. By now you know we do this in two ways: find the real cause of software waste and find suitable replacements to eliminate the cause. To help us understand waste we can turn to the Lean community for inspiration. Lean is a broad and growing movement, and was first brought to the software world by Mary and Tom Poppendieck in the book Lean Software Development. Lean provides, among other things, a set of categories for recognizing waste. Learning how these categories apply to software is a good first step for improvements. The categories of waste are:
1. Partially Done Work
2. Extra Features
5. Task Switching
Let’s discuss each one in turn, and see what insights they can reveal.
Partially Done Work
So you own a bicycle factory (trust me on this one), and you make 1000 mountain bike frames with the plan to later make the handlebars and ship the whole units. Guess what happens when your customers decide they want road bikes instead? Yup, you’re stuck with 1000 useless frames. It would have been better to produce one entire bike at a time. Then, you would never be left with excess inventory. Or maybe, instead of a bicycle tycoon, you are a humble developer needing to write a new fangled Service Oriented Architecture. Someone (usually a consultant) says “You need to design Contract First” so you spend two weeks creating ten different WSDL service descriptors by hand. Guess what happens when the architecture group decides to switch to REST and plain old XML services? Yup, you just worked for two weeks and have nothing to show for it. In-process inventory is a risk and a cost, whether it is half built bicycles or partially complete web services. Changing requirements and technologies will leave you with wasted effort sooner or later. When this happens to me I think, “I wish I’d spent two weeks at home instead, it would have had the same value to my company.” Partially done work manifests in many ways in software development.
Your new project is starting and you’ve got a great team of three developers. After the first planning meeting you divide up the story cards… Sean takes feature x, Felix takes feature y, and you take feature Z. Unfortunately, at the end of the week your team gets re-prioritized and which of the features is complete? Possibly none of them, your three person team has delivered no business value at all. This is similar to what happened in the bike factory, where one person made 1000 frames. Instead, we had three people work on three frames and none of them got quite done. A better approach is to have three people work on one feature until it is complete. This gives you the least amount of time that any feature is partially done.
You bust your hump all week and finally finish your feature on Friday afternoon, just in time to check in and head home – except that you get version control merge errors. Maybe your merge errors are simple and you can still make it home to watch the Friday night “Must See TV” with your family. Or maybe you end up working until when Saturday Night Live comes on. Bummer. Unsynchronized code is a risk, just waiting to slash down your productivity. Check in early, check in often, avoid costly delays. And what about version control branches and distributed version control systems? Any code not checked into the branch that gets deployed is partially done work and will eventually cost you in terms of productivity. There are some unique environments where DVCS makes sense, but in general you should be careful. Anything on a branch is waste; whether you call it a fork or a branch is immaterial.
Customers almost always want new features. You only have to ride along on a few sales calls before realizing this. Sometimes you get lucky, and a feature the customer wants is in the next release. Unfortunately, you may have to make the customer wait six months until the next release date. This is a problem. Your customers get value from using the software you write and your company gets value from having customers. Any software you write is inventory, and it doesn’t do customers any good to have the software written if they can’t use it. Avoid the risk and cost by deploying frequently.
Back at your bike factory you started to take eliminating Partially Done Work seriously, and now only complete, fully built bicycles roll off your assembly line, one at a time. But now you’ve got another problem. It seems customers don’t actually need bright yellow shock absorbent banana seats (who could have predicted?). But that’s what you built and you need to charge a little extra for each bike because of it, even though your customers neither want nor need it.
This happens all the time with software, and an oft quoted Standish Group study found that almost 65% of software features are rarely or never used. The implication is that you can practically double your development speed by simply not writing in the features no one wants. Easier said than done. I, for instance, would love a yellow banana seat even without a shock absorber. Still, there are a couple of options to avoid extra, value-wasting features.
Create a currency based cost/benefit model for all new features. Estimate what revenues and income the new feature will likely produce, and what development costs it requires to create. Be sure to pick the same unit of measure for these estimates. Can you sort the following data: 5 weeks, $10,000, 40 developer hours? Of course not, all these measurements use a different unit. But if you convert them all into a currency such as dollars, then choosing priorities becomes a lot easier. Clear and simple economic models, while never perfect, enable the whole team to make better decisions. Now be extra cautious in what you develop. In the words of Mary Poppendieck, “If there isn’t a clear and present economic need for the feature, it should not be developed.” When in doubt, leave it out.
Get Close to the Customer
You’ve just heard that your biggest customer needs your software to send email (all systems eventually need this feature). How sure are you that this a real requirement? Did an end user tell you this? Or did an end user tell the sales person who told the product owner who discussed it with the business analyst and now there is a “Must Send Email” item in your product backlog? A string of message passing like this results in garbled priorities and misunderstood requirements even when intentions are good. Add to this the perverse incentives in many enterprises, such as delivery bonuses, and you can quickly end up developing the wrong product for the wrong market. The closer you are to the end user, and their value stream, the better.
Rework in manufacturing is easy to identify. Sending a bike frame back through painting to turn it from blue to red is rework. You shouldn’t have painted it blue in the first place and now you’ve wasted time and paint. How does this map to software development? Discovering something we once knew but have now forgotten is one definition. In big companies it’s quite common to have two different groups independently verify a technical failure. You hear anecdotal evidence of it after work wherever developers gather to drink.
Dev #1 – Ugh, our new web framework sucks.
Dev #2 – Yes, we tried the same framework last year and it does suck.
How great would it be if this conversation could have happened six months ago before the second attempt at using the web framework. Sharing and broadcasting failures is just as important as broadcasting successes. Good decisions come from knowing what you should not do as much as what you should do.
How do you find out about failed approaches today instead of tomorrow after you’ve invested energy into the failure? Google, of course. Blog posts and Stack Overflow-like forums are an excellent chronicle of failure. You search for what you are trying to do, you find an article by someone trying the same, and then you scroll down to read the comments lambasting the original author for their horrible idea. You can even take a more direct approach. When contemplating using a new technology, my first search always contains the word “sucks”. EJB Sucks, Groovy Sucks, Flex Sucks. Evangelists are great at posting glowing, moderately informed reviews on new technologies… but if you want to know the sharp edges and failure points in a framework then seek out the opinion of those bitter and frustrated enough to write a scathing review. Plus, reading about epic failure is always a bit of fun. As long as it’s not yours.
The first few days at a new job or project are usually pretty easy: you’re a trainee and not expected to contribute much. You’re often given an Intranet link and a few printed documents and left alone to somehow magically “ramp up”. If you make minimum wage as a fast food fry cook then a little down time can be tolerated, after all minimum wage is around 10 times less than what programmers make. But why in the world do we tolerate ramp up time in IT? The problem only gets worse as companies invest in closed source and proprietary software. You’re at a severe disadvantage if you can’t hire new employees that already have experience in the web framework, persistence layer, and even programming language of your software system.
Fortunately, there is an easy answer to long ramp up times and knowledge handoffs, but many programmers just don’t want to hear it. If you want a new team member to check in working code on their first day on the job then start pair programming. If you want to ensure just-in-time delivery of system knowledge to your new teammate then sit the entire team in the same room and let the newbie ask questions when they are needed. For some reason, many enterprise developers resist team rooms, flexible workspaces, and pair programming. Few people like writing documentation, but when given the choice they’ll cite the need for better architecture documents and “living documentation” rather than make a simple change to the work environment. Having been on both sides recently, first watching new team members come into an agile environment and then later myself rolling into a pair programming team, I now understand that a team room and pair programming is the easiest and quickest way to eliminate ramp up time. You don’t even need to pair forever, just the first few weeks. But you may keep doing it anyway.
Switching tasks takes time. You need to mentally and sometimes physically reset. If you have three separate week-long tasks, then in an ideal world you will take longer than three weeks to complete the tasks. Why? Even if switching tasks takes only 30 seconds each, completing three week-long tasks requires two context switches, resulting in a minimum of three weeks and one minute to complete the work. And 30 seconds for task switching is ludicrously short. Facebook doesn’t check itself you know.
So we just need to stop switching tasks. Except you can’t. There is always maintenance work arriving at an uneven rate. You can’t eliminate that disruption but you can minimize it. One way is with a separate maintenance team, but I can’t imagine living through that drudgery for more than a few weeks before looking for a new job. Plus, living with your code in production brings valuable lessons. A temporary maintenance team where you serve for only a few weeks has worked for me in the past, mainly because there is a light at the end of the tunnel and you know when your tour of duty is up. A better solution is having maintenance days. For example, every Friday the team must work solely on defects and maintenance issues. Having everyone work maintenance is fair, and it also minimizes interruption. When a small team is needed to solve a problem then the right people are available and feature work isn’t being interrupted. Just be careful. Senior team members tend to have more vacation and sometimes take Fridays off near the end of the year. If this is the case, pick a different day. Also, some defects take longer than one day to fix. You may find defect Friday stretching into defect Monday as well, splintering the team.
Developers do a lot of waiting. Maybe the requirements aren’t ready, or the design isn’t approved, or QA hasn’t certified the build. Wait, wait, wait. The only time we enjoy waiting is when the power goes out. Then we recognize our helplessness and turn the dimly lit office into some sort of indoor business-casual football match. Just like summer camp, my favorite memories from work involve the power being cut off. Good times. The bad times, on the other hand, all involve waiting for approval. The requirements are almost done, and you could start work on some of it, but you’re not allowed because they haven’t officially been approved. Or your design is trapped in some sort of review loop, where not enough commitment has been made to start work.
You can’t just throw out approval. Things need to be approved. Agreements must be made. But you can change the frequency and granularity of approvals. If your large requirements document is not approved, then why have a large document in the first place? Why not break it up into smaller pieces, in which case portions of it can be approved and you can move on. If your design documents need approval, then why not break these up into manageable portions as well. In Waterfall shops, or other bureaucratic environments, the process police don’t normally care if you produce more, smaller documents. Just avoid having no documentation. Short iterations aren’t just for code. If you’re stuck with delays, break down the problem into smaller pieces and deliver less more often.
It’s not a waste of time to work on defects. They do need fixing and you can’t eliminate waste by simply declaring that you’ll no longer be doing defect work (although this might be a good way to shorten your product’s lifespan if you get really tired of the defect work). Quality Assurance departments recognized the real waste with defects years ago and have been asking to be involved early in the life cycle for many years now. What QA “gets” is that inspecting products for defects after the product is built is wasteful, it’s much better to inspect before it’s built to prevent the defects. QA departments world-wide frequently lobby to be brought in earlier in the process. If only we could be in the requirements meetings, they say, then we could ensure the product quality, which is absurd. High quality cannot be achieved through better use of meetings or documentation.
Quality comes from every single person on the team caring about a better product. If quality is the domain of a single person or department, then it’s likely quality does not describe your product. Moving QA forward in the process needs to mean partnering on acceptance tests that constitute the design of the product. Having a developer and tester pairing on the same workstation is a good sign, and automated acceptance test frameworks facilitate this. Fitnesse, easyb, whatever. Having a developer and tester work in different tools, where neither has access to the other, leads to strong test ownership and little collaboration. It is easy to give the advice that “whenever a defect is found, a test should be created so that it cannot happen again.” To do this, you need to pick a development tool where your testers can create a failing test for the developers, and then throw out that other one you were using before.
The point of all this is not to categorize waste. Learning about the categories helps you recognize waste, which is good, but the real goal is reducing cost and becoming more effective. The underlying causes of waste are not always obvious, and real improvements are seldom straight forward. The first step is to recognize the waste, and the second step must be replacing the old way of doing things with something new. Perhaps you are building unnecessary software and need to change the way you approve feature requests. Or maybe you’re drowning under the burden of failed installations and need to move product deployment into the development life-cycle. Whatever it is, each company and situation is unique. No advice or software practice is universally applicable. It’s up to you to figure out what needs to be done on your team and then start trying something new and different. Good luck.
About the Author
Hamlet D’Arcy, Canoo Engineering AG
Hamlet D’Arcy has been writing software for over a decade, and has spent considerable time coding in C++, Java, and Groovy. He’s passionate about learning new languages and different ways to think about problems. Hamlet is a founder of the Basel-based Hackergarten open source coding group, and regularly participates and speaks at local and international user groups and conferences. Hamlet is a committer on the Groovy and CodeNarc projects, and is a contributor on a few other open source projects (including Griffon, GPars, the IDEA Groovy Plugin). He blogs regularly at http://hamletdarcy.blogspot.com and can be found on Twitter as HamletDRC (http://twitter.com/hamletdrc).
 Mary and Tom Poppendieck, Lean Software Development 2003
 Jim Johnson, Standish Group. XP 2002 Conference Presentation
 Mary and Tom Poppendieck, Implementing Lean Software Development 2006