• Home
  • Events
  • About
  • Waste!

    March 8th, 2011

    This article originally appeared in the October issue of No Fluff Just Stuff magazine. The copyright recently reverted to me, so here is the reprint for the general public. As an unneeded aside, I’m really happy about the way this article turned out. I hope you enjoy it too.

    In our industry no one recommends increasing waste, there are no waste evangelists, and no pro-waste lobby. So why is there so much of it? As we scramble frantically through the day to please our customers we are often not even aware of the pointless trail of half done work and unwanted features we leave in our wake. This article helps you understand what waste is, where it comes from, and what you can do about it by mapping the seven wastes of Lean Manufacturing into the software development field. If you want both better software and happier customers then quit wasting time and read this article.

    There’s something very natural about seeing waste in the world around us. We think about waste almost every day in our personal lives. For example, just in the last few minutes I’ve thought to myself: I should recycle these stacks of magazines (wasting space), I should watch less Tool Academy (wasting time), and I should really start running again (wasting away). Many of us are equally good at spotting waste at work: pointless meetings, fancy meals for the sales staff, and expensive development tools (SharePoint anyone?). And every year, around January 1st, we recommit ourselves to avoiding the same wastes as the previous year, willfully ignoring our past failure to lose weight, quit smoking, or watch less reality television. Likewise with work, every person in an organization from CEO to janitor can rail about wasted time in meetings yet somehow the situation never improves. Clearly, finding solutions to waste requires more than a statement of intent.

    Eliminating waste and improving productivity should be easy and uncontroversial. There are really only two rules to follow: 1) stop doing things that aren’t worth the effort or 2) do the same things with less effort. These are the basic building blocks of improvement, whether they be related to your personal or professional life. If you’re wasting your time watching trash TV then stop! Skip those quarterly meetings where half the audience is playing Bejeweled on their phones. In fact, why not dial in to every meeting you can and write your code to the background noise of a glacially paced decision process?

    If these two rules of waste are so simple, then why is making a lasting change so hard? Why do we have the same resolutions every New Year, and why aren’t we hurtling forward at work to ever higher levels of productivity? The reality is that we are good and seeing waste but not good at seeing its causes, and we often eliminate a wasteful practice without finding a suitable replacement. Consider the classic resolution of losing weight. We say to ourselves, “I’m going to eat less junk food this year” without asking why we have this tendency in the first place. Caffeine and lack of sleep are both known to contribute to overeating, but we don’t often focus on these causes. Excessive meetings form an even better example. Why do some workplaces have so many meetings? This is often a sign of a broken decision making process. Here’s a fun way to fill up your daily calendar: when you can’t make a decision then call a meeting and get everyone to talk about it. But be sure to leave the agenda and result open ended, so the conversation can be free form and lead to no particular resolution. Your days will quickly fill up leaving you busy yet unproductive, and you can use your quick lunch breaks to bemoan your culture of meetings. No, on the contrary, fixing a culture of meetings requires you to both find the cause and make a replacement. First identify the real problem, difficulties making decisions, and then experiment with new ways to solve the problem. Perhaps you need to discuss and embrace decision forms other than “unanimous”, or learn to hold shorter, more specific meetings, or just create a team decision board and handle issues in the team room instead of the conference room.

    The antithesis of removing waste is what I call “Write Only Process Improvement”. As a group tries to adopt a better process, they are only allowed to add new practices without removing unneeded ones. This is the bane of agile pilot groups around the world. Your team is certainly free to try a Scrum style iteration with story cards, a burndown chart, and iterative planning. But you still need to adhere to the enterprise standards of full requirements documents, weekly status reports, and quarterly deployments. Feel free to add whatever you want to the process, just don’t dare remove anything. This arrangement requires duplicate work, quickly consumes the time of the ScrumMaster, and leads to the conclusion that agile methods are slower than the alternative. Of course a new practice is slower if you’re not allowed to abandon the old way of doing things. The Agile Manifesto has a wonderful parody site called the “Manifesto for Half-Arsed Agile Software Development“. While the original manifesto claims to value “Working software over comprehensive documentation”, the parody adds the line “as long as that software is comprehensively documented.” This rings perfectly true for so many Agile pilot projects: please try your new process, but be sure to follow the old one as well. New practices are adopted despite “Write Only Process Improvement”, never because of it.

    To reduce cost and be more effective we must eliminate waste. By now you know we do this in two ways: find the real cause of software waste and find suitable replacements to eliminate the cause. To help us understand waste we can turn to the Lean community for inspiration. Lean is a broad and growing movement, and was first brought to the software world by Mary and Tom Poppendieck in the book Lean Software Development[1]. Lean provides, among other things, a set of categories for recognizing waste. Learning how these categories apply to software is a good first step for improvements. The categories of waste are:

    1. Partially Done Work
    2. Extra Features
    3. Relearning
    4. Handoffs
    5. Task Switching
    6. Delays
    7. Defects

    Let’s discuss each one in turn, and see what insights they can reveal.

    Partially Done Work

    So you own a bicycle factory (trust me on this one), and you make 1000 mountain bike frames with the plan to later make the handlebars and ship the whole units. Guess what happens when your customers decide they want road bikes instead? Yup, you’re stuck with 1000 useless frames. It would have been better to produce one entire bike at a time. Then, you would never be left with excess inventory. Or maybe, instead of a bicycle tycoon, you are a humble developer needing to write a new fangled Service Oriented Architecture. Someone (usually a consultant) says “You need to design Contract First” so you spend two weeks creating ten different WSDL service descriptors by hand. Guess what happens when the architecture group decides to switch to REST and plain old XML services? Yup, you just worked for two weeks and have nothing to show for it. In-process inventory is a risk and a cost, whether it is half built bicycles or partially complete web services. Changing requirements and technologies will leave you with wasted effort sooner or later. When this happens to me I think, “I wish I’d spent two weeks at home instead, it would have had the same value to my company.” Partially done work manifests in many ways in software development.

    Dividing Work

    Your new project is starting and you’ve got a great team of three developers. After the first planning meeting you divide up the story cards… Sean takes feature x, Felix takes feature y, and you take feature Z. Unfortunately, at the end of the week your team gets re-prioritized and which of the features is complete? Possibly none of them, your three person team has delivered no business value at all. This is similar to what happened in the bike factory, where one person made 1000 frames. Instead, we had three people work on three frames and none of them got quite done. A better approach is to have three people work on one feature until it is complete. This gives you the least amount of time that any feature is partially done.

    Unsynchronized Code

    You bust your hump all week and finally finish your feature on Friday afternoon, just in time to check in and head home – except that you get version control merge errors. Maybe your merge errors are simple and you can still make it home to watch the Friday night “Must See TV” with your family. Or maybe you end up working until when Saturday Night Live comes on. Bummer. Unsynchronized code is a risk, just waiting to slash down your productivity. Check in early, check in often, avoid costly delays. And what about version control branches and distributed version control systems? Any code not checked into the branch that gets deployed is partially done work and will eventually cost you in terms of productivity. There are some unique environments where DVCS makes sense, but in general you should be careful. Anything on a branch is waste; whether you call it a fork or a branch is immaterial.

    Undeployed Code

    Customers almost always want new features. You only have to ride along on a few sales calls before realizing this. Sometimes you get lucky, and a feature the customer wants is in the next release. Unfortunately, you may have to make the customer wait six months until the next release date. This is a problem. Your customers get value from using the software you write and your company gets value from having customers. Any software you write is inventory, and it doesn’t do customers any good to have the software written if they can’t use it. Avoid the risk and cost by deploying frequently.

    Extra Features

    Back at your bike factory you started to take eliminating Partially Done Work seriously, and now only complete, fully built bicycles roll off your assembly line, one at a time. But now you’ve got another problem. It seems customers don’t actually need bright yellow shock absorbent banana seats (who could have predicted?). But that’s what you built and you need to charge a little extra for each bike because of it, even though your customers neither want nor need it.

    This happens all the time with software, and an oft quoted Standish Group[2] study found that almost 65% of software features are rarely or never used. The implication is that you can practically double your development speed by simply not writing in the features no one wants. Easier said than done. I, for instance, would love a yellow banana seat even without a shock absorber. Still, there are a couple of options to avoid extra, value-wasting features.

    Economic Models

    Create a currency based cost/benefit model for all new features. Estimate what revenues and income the new feature will likely produce, and what development costs it requires to create. Be sure to pick the same unit of measure for these estimates. Can you sort the following data: 5 weeks, $10,000, 40 developer hours? Of course not, all these measurements use a different unit. But if you convert them all into a currency such as dollars, then choosing priorities becomes a lot easier. Clear and simple economic models, while never perfect, enable the whole team to make better decisions. Now be extra cautious in what you develop. In the words of Mary Poppendieck, “If there isn’t a clear and present economic need for the feature, it should not be developed.”[3] When in doubt, leave it out.

    Get Close to the Customer

    You’ve just heard that your biggest customer needs your software to send email (all systems eventually need this feature). How sure are you that this a real requirement? Did an end user tell you this? Or did an end user tell the sales person who told the product owner who discussed it with the business analyst and now there is a “Must Send Email” item in your product backlog? A string of message passing like this results in garbled priorities and misunderstood requirements even when intentions are good. Add to this the perverse incentives in many enterprises, such as delivery bonuses, and you can quickly end up developing the wrong product for the wrong market. The closer you are to the end user, and their value stream, the better.

    Relearning

    Rework in manufacturing is easy to identify. Sending a bike frame back through painting to turn it from blue to red is rework. You shouldn’t have painted it blue in the first place and now you’ve wasted time and paint. How does this map to software development? Discovering something we once knew but have now forgotten is one definition. In big companies it’s quite common to have two different groups independently verify a technical failure. You hear anecdotal evidence of it after work wherever developers gather to drink.

    Dev #1 – Ugh, our new web framework sucks.

    Dev #2 – Yes, we tried the same framework last year and it does suck.

    How great would it be if this conversation could have happened six months ago before the second attempt at using the web framework. Sharing and broadcasting failures is just as important as broadcasting successes. Good decisions come from knowing what you should not do as much as what you should do.

    How do you find out about failed approaches today instead of tomorrow after you’ve invested energy into the failure? Google, of course. Blog posts and Stack Overflow-like forums are an excellent chronicle of failure. You search for what you are trying to do, you find an article by someone trying the same, and then you scroll down to read the comments lambasting the original author for their horrible idea. You can even take a more direct approach. When contemplating using a new technology, my first search always contains the word “sucks”. EJB Sucks, Groovy Sucks, Flex Sucks. Evangelists are great at posting glowing, moderately informed reviews on new technologies… but if you want to know the sharp edges and failure points in a framework then seek out the opinion of those bitter and frustrated enough to write a scathing review. Plus, reading about epic failure is always a bit of fun. As long as it’s not yours.

    Handoffs

    The first few days at a new job or project are usually pretty easy: you’re a trainee and not expected to contribute much. You’re often given an Intranet link and a few printed documents and left alone to somehow magically “ramp up”. If you make minimum wage as a fast food fry cook then a little down time can be tolerated, after all minimum wage is around 10 times less than what programmers make. But why in the world do we tolerate ramp up time in IT? The problem only gets worse as companies invest in closed source and proprietary software. You’re at a severe disadvantage if you can’t hire new employees that already have experience in the web framework, persistence layer, and even programming language of your software system.

    Fortunately, there is an easy answer to long ramp up times and knowledge handoffs, but many programmers just don’t want to hear it. If you want a new team member to check in working code on their first day on the job then start pair programming. If you want to ensure just-in-time delivery of system knowledge to your new teammate then sit the entire team in the same room and let the newbie ask questions when they are needed. For some reason, many enterprise developers resist team rooms, flexible workspaces, and pair programming. Few people like writing documentation, but when given the choice they’ll cite the need for better architecture documents and “living documentation” rather than make a simple change to the work environment. Having been on both sides recently, first watching new team members come into an agile environment and then later myself rolling into a pair programming team, I now understand that a team room and pair programming is the easiest and quickest way to eliminate ramp up time. You don’t even need to pair forever, just the first few weeks. But you may keep doing it anyway.

    Task Switching

    Switching tasks takes time. You need to mentally and sometimes physically reset. If you have three separate week-long tasks, then in an ideal world you will take longer than three weeks to complete the tasks. Why? Even if switching tasks takes only 30 seconds each, completing three week-long tasks requires two context switches, resulting in a minimum of three weeks and one minute to complete the work. And 30 seconds for task switching is ludicrously short. Facebook doesn’t check itself you know.

    So we just need to stop switching tasks. Except you can’t. There is always maintenance work arriving at an uneven rate. You can’t eliminate that disruption but you can minimize it. One way is with a separate maintenance team, but I can’t imagine living through that drudgery for more than a few weeks before looking for a new job. Plus, living with your code in production brings valuable lessons. A temporary maintenance team where you serve for only a few weeks has worked for me in the past, mainly because there is a light at the end of the tunnel and you know when your tour of duty is up. A better solution is having maintenance days. For example, every Friday the team must work solely on defects and maintenance issues. Having everyone work maintenance is fair, and it also minimizes interruption. When a small team is needed to solve a problem then the right people are available and feature work isn’t being interrupted. Just be careful. Senior team members tend to have more vacation and sometimes take Fridays off near the end of the year. If this is the case, pick a different day. Also, some defects take longer than one day to fix. You may find defect Friday stretching into defect Monday as well, splintering the team.

    Delays

    Developers do a lot of waiting. Maybe the requirements aren’t ready, or the design isn’t approved, or QA hasn’t certified the build. Wait, wait, wait. The only time we enjoy waiting is when the power goes out. Then we recognize our helplessness and turn the dimly lit office into some sort of indoor business-casual football match. Just like summer camp, my favorite memories from work involve the power being cut off. Good times. The bad times, on the other hand, all involve waiting for approval. The requirements are almost done, and you could start work on some of it, but you’re not allowed because they haven’t officially been approved. Or your design is trapped in some sort of review loop, where not enough commitment has been made to start work.

    You can’t just throw out approval. Things need to be approved. Agreements must be made. But you can change the frequency and granularity of approvals. If your large requirements document is not approved, then why have a large document in the first place? Why not break it up into smaller pieces, in which case portions of it can be approved and you can move on. If your design documents need approval, then why not break these up into manageable portions as well. In Waterfall shops, or other bureaucratic environments, the process police don’t normally care if you produce more, smaller documents. Just avoid having no documentation. Short iterations aren’t just for code. If you’re stuck with delays, break down the problem into smaller pieces and deliver less more often.

    Defects

    It’s not a waste of time to work on defects. They do need fixing and you can’t eliminate waste by simply declaring that you’ll no longer be doing defect work (although this might be a good way to shorten your product’s lifespan if you get really tired of the defect work). Quality Assurance departments recognized the real waste with defects years ago and have been asking to be involved early in the life cycle for many years now. What QA “gets” is that inspecting products for defects after the product is built is wasteful, it’s much better to inspect before it’s built to prevent the defects. QA departments world-wide frequently lobby to be brought in earlier in the process. If only we could be in the requirements meetings, they say, then we could ensure the product quality, which is absurd. High quality cannot be achieved through better use of meetings or documentation.

    Quality comes from every single person on the team caring about a better product. If quality is the domain of a single person or department, then it’s likely quality does not describe your product. Moving QA forward in the process needs to mean partnering on acceptance tests that constitute the design of the product. Having a developer and tester pairing on the same workstation is a good sign, and automated acceptance test frameworks facilitate this. Fitnesse, easyb, whatever. Having a developer and tester work in different tools, where neither has access to the other, leads to strong test ownership and little collaboration. It is easy to give the advice that “whenever a defect is found, a test should be created so that it cannot happen again.” To do this, you need to pick a development tool where your testers can create a failing test for the developers, and then throw out that other one you were using before.

    In Conclusion…

    The point of all this is not to categorize waste. Learning about the categories helps you recognize waste, which is good, but the real goal is reducing cost and becoming more effective. The underlying causes of waste are not always obvious, and real improvements are seldom straight forward. The first step is to recognize the waste, and the second step must be replacing the old way of doing things with something new. Perhaps you are building unnecessary software and need to change the way you approve feature requests. Or maybe you’re drowning under the burden of failed installations and need to move product deployment into the development life-cycle. Whatever it is, each company and situation is unique. No advice or software practice is universally applicable. It’s up to you to figure out what needs to be done on your team and then start trying something new and different. Good luck.

    About the Author

    Hamlet D’Arcy, Canoo Engineering AG

    Hamlet D’Arcy has been writing software for over a decade, and has spent considerable time coding in C++, Java, and Groovy. He’s passionate about learning new languages and different ways to think about problems. Hamlet is a founder of the Basel-based Hackergarten open source coding group, and regularly participates and speaks at local and international user groups and conferences. Hamlet is a committer on the Groovy and CodeNarc projects, and is a contributor on a few other open source projects (including Griffon, GPars, the IDEA Groovy Plugin). He blogs regularly at http://hamletdarcy.blogspot.com and can be found on Twitter as HamletDRC (http://twitter.com/hamletdrc).

    References

    [1] Mary and Tom Poppendieck, Lean Software Development 2003
    [2] Jim Johnson, Standish Group. XP 2002 Conference Presentation
    [3] Mary and Tom Poppendieck, Implementing Lean Software Development 2006


    Lean Groovy 7 – Optimize the Whole

    March 2nd, 2011

    This article originally appeared in the October 2010 edition of GroovyMag, the Groovy and Grails magazine. A video of related material is available on InfoQ titled Lean Groovy. Previous articles in this series are on the Canoo website: Part 1: Eliminate Waste, Part 2: Build Quality In, Part 3: Create Knowledge, Part 4: Defer Commitment, Part 5: Deliver Fast, and Part 6: Respect People. Lastly, if you like this, you may want to check out some of my older blog posts from my personal site under the ‘craft’ category. Enjoy!

    The Groovy Programming Language advertises itself as an “agile and dynamic Language for the JVM”, but what does this mean exactly? This series of articles explains Lean Software Development, and shows how your choice of programming language can make your entire process remain nimble and adaptive. Each month’s article will cover one of the seven Lean Software Development principles and explain how Groovy and the associated ecosystem help eliminate waste, defer commitment, and build quality into your product.

    About this series

    Groovy is an agile programming language. In order to explain what this means, these articles are structured around the seven principles of Lean Software Development. The previous installments included a short introduction to Lean, explained how an expressive language can eliminate waste in unit testing, showed how the easyb Behavior Driven Development framework and Groovy Builders can build quality into your life-cycle, demonstrated how the Spock Testing Framework can create project knowledge, discussed how tools like Real Options and dynamic languages for deferred commitment, and most recently looked at how Groovy’s Grapes module system, alongside modules like Groovy Web Services, streamline interactions between development and other groups. Today, in the final article, we’ll look at optimizing the whole by applying Groovy to domains that cross department boundaries: how Gradle improves the build process and easyb serves as a collaboration mechanism between different organization roles. The series ends with a discussion on the best practices of mixing Java and Groovy.

    Some of these code examples are basic while others are more advanced. The articles explore why the features of Groovy are important rather than the mechanics of any one Groovy feature. Hopefully you’ll leave with some new ideas about how to use Groovy, how to convince your team that Groovy is worthwhile, or most importantly – how to increase your productivity.

    Lean revisited

    Part 1 of this series contained a more in-depth explanation of Lean, and I won’t repeat it here. But as a quick refresher here are the 7 lean principles along with a summary for those principles covered previously:

    1. Eliminate Waste – Reduce work in progress and half done work
    2. Build Quality In – Instead of settling for finding defects, optimize your process to create fewer defects
    3. Create Knowledge – Create fast feedback cycle and encode project information in executable form
    4. Defer Commitment
    5. Deliver Fast – Increase velocity by working in small batches and removing bottlenecks
    6. Respect People – Create an environment that lets every single person contribute
    7. Optimize the Whole

    Principle 7 – Optimize the Whole

    The successful software project is dependent on more than just the proper functioning of each part of the life-cycle: coding, testing, analysis, and deployment. Success hinges on how well all of these pieces fit together. Optimizing the Whole in Lean refers to making sure you always measure up. Improving your business is too often an exercise in sub-optimization: you may spend all your effort on improving the unit testing experience without realizing that it’s your interactions with operations that are derailing the project, or you may perfect your team’s delivery process while the rest of the company continues behaving wastefully. In either case, it pays to step back and examine the situation as a whole. If you’re optimizing your unit testing with Groovy, then step back and ask how you can improve the entire project and not just testing. If you’re working with your team on a better software life-cycle, then step back and ask how you could improve the life-cycle of all the teams.

    Writing code is a lot more than just making the unit tests pass. More importantly you must know which tests to write and what the customer needs. And this means having developers work closely with customers or business analysts. Oftentimes the customer or analysts document the behavior of the system in one format (user stories), developers re-document it in another (functional tests), and QA re-encodes it in another (test plans). Once the code is complete, can the requirements be generated from the tests? Can the story cards or test plan be used as input to the tests? Probably not, since most testing tools and writing tools are so different. Luckily, easyb is not like most testing tools.

    Easyb stories have 3 states: pending, failing, and passing. Failing and passing should be obvious. Pending means the story is written but has not been worked on. Listing 1 shows how to write a pending story.

    Listing 1: A Pending easyb Story

    Easyb stories were covered in part 2 of this series (what, you don’t have all the GroovyMag back issues?). In case you missed it, this is a form of a behavior driven test case. Normally you would write setup arrangements, system actions, and assertions within these blocks. And in most test tools a unit test without any assertions is passing. The tests are green… ship it. Not so with easyb. The output of running this easyb story can be seen in listing 2.

    Listing 2: Pending Story Output

    Leaving out the code for stories (the closure block) makes those stories “Pending”. After documenting an entire iteration worth of work in stories, it is easy to have your build generate an HTML or text based report showing which stores are passing, which are failing, and which are pending. Story cards are great, but too often their information is duplicated in several places. Why not save time and just use easyb as a collaboration tool? The syntax is natural and simple, and most people should be able to cope with writing one. Even so, it would be a small amount of work to generate pending stories from wiki pages or other documents. For your team, optimizing the whole might mean using easyb stories as a collaboration medium.

    The output for listing 2 is the text output from easyb. Naturally, this pending story report can be html formatted as well. I prefer text reports because they tend to integrate with continuous integration servers a lot easier. If you want the html report, just change the “-txtstory” in listing 3 to “-html”.

    Listing 3: Running easyb Text Reports

    Besides collaboration, another frequent optimization problem is to have developers create the perfectly productive code environment only to burn several days fighting with Ant and Maven to get the build and deployment working correctly. Ant offers endless flexibility if you’re willing to write it all yourself and Maven gives you smooth automation if you’re capable of adhering to their strict project conventions and plug-in environment. Either way, I hope you like XML. In the best case scenario, there is a junior developer around to whom you can delegate the unsavory task of build automation. Life gets easier if you can trick someone into thinking the “Build Master” role looks good on a resume.

    Luckily there is a third way: Gradle. Gradle is an enterprise build tool designed around the philosophy of making the simple things simple and the complex things possible, which is a space somewhere between the endless flexibility of Ant and the strict rules of Maven. Gradle uses convention based builds by default, allowing you the ease of Maven for new projects, but also exposes all of these conventions as configurable properties. For old builds that do not conform to a project layout standard, you only need to change your build script not your entire project in order to get the build running. And there is no XML. In fact, Gradle uses Groovy as a Domain Specific Language for it’s build scripts. The complete script for building a mixed Java and Groovy project with several dependencies retrieved from Maven repositories is quite short:

    Listing 4: Mixed Java and Groovy Build File

    This should impress you if you’ve spent any time maintaining builds. This build produces all of the standard targets you would expect(clean, compile, test, etc), adds several Maven repositories, and declares two dependencies. The big advantage of Gradle isn’t the Groovy build scripts, though. It is that Gradle was designed and built for the enterprise. Multiple projects and multiple artifacts are no problem for Gradle, and it offers a plug-in system that is more than just spawning off new tasks and hoping they complete. And one of the nicest features features is the Gradle Wrapper, which lets you build a project without having Gradle installed… perfect for continuous integration servers and open source projects.

    Better developer testing is one aspect Groovy brings to a project. But once there is a little Groovy skill in your organization, then options start to open elsewhere. You can start to consider using easyb to eliminate duplication and waste, you can adopt a new build system without fearing the hurdle of learning a new language, and you can even consider new frameworks like Grails when you need to quickly create a prototype application! Having Groovy skill in-house keeps options open. And having more options means you can make better decisions, for your team and your company.

    Sidebar: Gradle is not a Groovy project

    Gradle is an Enterprise Java Build system that happens to use Groovy as a DSL scripting language. More than just a wrapper around Ant or Maven, Gradle targets the complex, multi-project, and multi-artifact builds that Maven has so much trouble supporting without sacrificing any of the build by convention simplicity for those that can use it.

    Not only is the build script a real scripting language, but the build itself is a set of domain objects that you can manipulate. Want to dynamically create tasks in Groovy? Sure. Want to query the build graph before executing any targets? Go for it. And as for multi-project build support, those used to waiting for Maven to build all the projects in the master build will appreciate the ability of Gradle to build sub-projects independently from their parent.

    Gradle has an active user community and a fair (and growing) amount of online documentation available. It is a build tool worth a look, even for those projects that have nothing to do with Groovy.

    Best Practices

    This series has focused on using Groovy along-side Java in the same project to improve overall productivity. Hopefully you have some new ideas around how to use these technologies together. Before finishing this last article, I want to share some of my own personal experiences attempting to make Java and Groovy coexist in production. It’s not all wine and roses, and here are a few important things I learned along the way.

    Yes, Groovy allows you to define an interface in Groovy, implement it in Java, and subclass it once again in Groovy. No, this is not how you should be writing your system. Groovy excels at integrating with Java, making scenarios like this possible. But at some point you need to make an architectural decision about which languages are first class languages and which are extension languages. Choose cautiously the option of allowing bi-directional relationships between Groovy and Java; that is, allowing both Groovy classes to reference Java classes and vice versa. Relying on the Groovy compiler in this way may make it harder in the future to either add a new extension language or remove Groovy from your project altogether (not that it will happen!). If you want to limit risk, the best thing to do is allow uni-directional dependencies from Groovy to Java until you’re more comfortable with the language.

    Test Driven Development is one path to creating a usable API in your object model. It make sense that one of the best ways to design a practical and easy to use component is to start by writing a consumer of that component. Only the most masochist or stubborn programmers will live with a crummy API they have to use when it is within their power to change it. So what happens when your production code is written in Java and your test driven tests are written in Groovy? If you are careless you will end up with an API optimized for Groovy and not Java. If you’re releasing your library to the public or other development teams then this may be an issue. The best advice is to be careful. Groovy does not require you to catch checked exceptions, while Java does. Make sure your production code has a thoughtful design around checked versus unchecked exceptions. Just because your IDE makes it easy to generate reams of try catch blocks doesn’t mean you should be lazy about API design. As another example, Groovy makes working with Maps and Lists easy and convenient. But watch out if your public API becomes too ‘listy’ or too ‘mappy’. Having a method return an empty list for nothing and a one element list for something is a great match for the functional style of Groovy, but can be difficult for Java to consume (not that I’m recommending returning null!). And a constructor that takes a map instead of a set of arguments allows Groovy code to pass method parameters as named arguments, but it’s a very painful design for Java clients.

    Lastly, do not throw out types. The message of Groovy and other dynamic languages isn’t that types don’t matter. They do. It’s that the names of types don’t matter. If you want to make your Groovy dynamic and support duck typing then do it where you need it. But don’t do it everywhere. Types serve as documentation. Documentation that never goes out of date. If your method takes a String then declare it as such. If it takes a List then write it in there. When was the last time you needed to mock a String or a List? Probably never. Reserve dynamic typing for those times when you don’t want to commit to a named abstraction, don’t just use it all the time.

    Summary

    Over the last few years Groovy has become an important part of my software development practice. I believe Groovy has a big role to play in the Enterprise: those organizations that move a little slower and deliberate a little longer than the rest of us. Those places where risks are calculated conservatively and decisions are not made lightly. The irony is that the companies least willing to adopt a new language are those that need it the most. You don’t have to tell startups to eliminate waste and deliver fast, they do it by necessity. The problem with adopting Groovy is the risk. Architects and decision makers are often unsure of the benefits and worried about the unknown.

    This is why testing with Groovy is such a good fit. Leaving Groovy out of the production code means that there is no deployment footprint and no dependency on Groovy from any of your users or customers. This limits your risk and leaves your options open. And the benefits of Groovy are real and tangible. The biggest hurdle to overcome is respecting people. Is your organization willing to let the people doing the work make the decision about how it gets done?

    Relating Groovy back to the seven Lean Software Principles was more than just a fun exercise. I genuinely believe Groovy is an agile language, and that technology choices like this are an important part of staying nimble. Some readers may be more convinced by eliminating waste and building quality in, while others might find deliver fast and create knowledge more compelling. In the end, each software project is different and it is up to you to decide whether benefits shown here are worth advocating for within your team. As lean says, “There is no one right way.”

    As a parting recommendation I suggest an experiment. Add the latest version of Groovy to your project as a testing dependency and have everyone one the development team write Groovy unit tests for 30 days. Schedule a retrospective at the end of the month and facilitate an honest discussion about whether or not you really want to continue on with Groovy. At worst, your team comes away with new skills and a better idea of the project needs at the expense of having some unit tests they’d like to rewrite into Java. At best, you’ve started down a path where your chosen technologies are helping you achieve an agile life-cycle rather than hindering you. In the words of the Pragmatic Programmer Tip Number 5: “Be a catalyst for change”.

    Learn More

    Several references were used to create this article series, and all are enjoyable, informative reads in their own right:

    [1] Extreme Programming Explained, 2nd Edition – Beck and Anders
    [2] Implementing Lean Software Development – Poppendieck
    [3] The Pragmatic Programmer – Hunt and Thomas
    [4] Toyota Production System – Ohno
    [6] Groovy – http://groovy.codehaus.org/
    [5] Gradle – http://www.gradle.org/
    [6] Easyb – http://www.easyb.org/
    [7] Spock – http://code.google.com/p/spock/
    [8] GroovyWS – http://groovy.codehaus.org/GroovyWS

    Hamlet D’Arcy Canoo Engineering AG

    Hamlet D’Arcy has been writing software for over a decade, and has spent considerable time coding in C++, Java, and Groovy. He’s passionate about learning new languages and different ways to think about problems, and recently he’s been discovering the joys of both F# and Scheme. He’s a committer on several projects including Groovy and JConch, and is a contributor on a few open source projects (including Griffon, CodeNarc, and the IDEA Groovy Plugin). He blogs regularly at http://hamletdarcy.blogspot.com and http://canoo.com/ blog and can be found on Twitter as @HamletDRC.


    Lean Groovy Full Video and an Interview

    December 23rd, 2010

    This week is a bumper week for me. I’m going away for the holidays and I wanted to clear out all the old blog posts I had in the “pending” state. Then both InfoQ and DZone finally go around to publishing a video and an interview respectively.

    First up is an interview I did with fellow JetBrains Academy Member Radoslaw Holewa. In the interview we talk about open source, Groovy, static code analysis, and of course IntelliJ IDEA. Radoslaw and I will both be heading to the 33rd Degree and GeeCon conferences this Spring, so register now and meet up with us!

    Second is a full 90 minute video of my presenting at 2gx. The topic is “Slimmed Down Software – A Lean, Groovy Approach”, based on a series of articles I wrote this year. It’s not a bad performance… but geez 90 minutes is a long time. It’s like trying to perform a one man show. I made the unfortunate decision to include a poop joke about midway through that I kinda regret. Oh well. I doubt I will ever truly grow out of potty humor.

    Enjoy and see you in 2011!


    Slimmed Down Software – A Lean, Groovy Approach Part 6 – Respect People

    December 7th, 2010

    This article originally appeared in the September 2010 edition of GroovyMag, the Groovy and Grails magazine. Part 7 is currently available for download from the magazine’s site, and more will come each month. Previous articles in this series are on the Canoo website: Part 1: Eliminate Waste, Part 2: Build Quality In, Part 3: Create Knowledge, Part 4: Defer Commitment, and Part 5: Deliver Fast. Lastly, if you like this, you may want to check out some of my older blog posts from my personal site under the ‘craft’ category. Enjoy!

    The Groovy Programming Language advertises itself as an “agile and dynamic Language for the JVM”, but what does this mean exactly? This series of articles explains Lean Software Development, and shows how your choice of programming language can make your entire process remain nimble and adaptive. Each month’s article will cover one of the seven Lean Software Development principles and explain how Groovy and the associated ecosystem help eliminate waste, defer commitment, and build quality into your product.

    About this series

    Groovy is an agile programming language. In order to explain what this means, these articles are structured around the seven principles of Lean Software Development. The previous installments included a short introduction to Lean, explained how an expressive language can eliminate waste in unit testing, showed how the easyb Behavior Driven Development framework and Groovy Builders can build quality into your life-cycle, demonstrated how the Spock Testing Framework can create project knowledge, and discussed how tools like Real Options and dynamic languages defer commitment. This month we’ll see how Groovy’s Grapes module system, alongside modules like Groovy Web Services, streamline interactions between development and other groups like operations and QA (an important aspect of respecting people). Finally, we’ll look at optimizing the whole by applying Groovy to domains that cross department boundaries: how Gradle improves the build process and easyb serves as a collaboration mechanism between different organization roles. The series will end with a discussion on the best practices of mixing Java and Groovy.

    Some of the code examples are basic while others are advanced. The articles explore why the features of Groovy are important rather than the mechanics of any one Groovy feature. Hopefully you’ll leave with some new ideas about how to use Groovy, how to convince your team that Groovy is worthwhile, or most importantly, how to increase your productivity.

    Lean revisited

    Part 1 of this series contained a more in-depth explanation of Lean, and I won’t repeat it here. But as a quick refresher here are the 7 lean principles along with a summary for those principles covered previously:

    1. Eliminate Waste – Reduce work in progress and half done work
    2. Build Quality In – Instead of settling for finding defects, optimize your process to create fewer defects
    3. Create Knowledge – Create fast feedback cycle and encode project information in executable form
    4. Defer commitment – Delay decisions until the last responsible moment
    5. Deliver Fast – Increase velocity by working in small batches and removing bottlenecks
    6. Respect People
    7. Optimize the Whole

    Principle 6 – Respect People

    Respecting people means creating an environment where the human element drives innovation and value. Creative problem solving and spontaneity are our most valuable assets, and great companies arrange their development process to let every single employee be able to exercise these traits.

    My experience with a big Capability Maturity Model (CMM) rollout was exactly the antithesis of this: we brought in several outside consultants who proceeded to observe the way we worked, then created a disciplined process we were required to follow. In their hearts, the consultants believed they were respecting people because the process they documented and codified was the one that we ourselves, the workers, had created ad hoc over the last several years. But our ability to improvise and make changes to the process greatly diminished. Trying something new meant filling out paper work and appearing before a process change group, so most people stopped bothering. The consultants never understood why we refused to display any sort of ownership over the new process. Well, it’s hard to think you own something when you aren’t allowed to change it. A development process must be malleable and open to experimentation, not locked down. There is nothing wrong with bringing in some outside help, but be careful who you choose. Consultants leave you with a some new documents or artifacts; coaches leave you with new skills. Seek that latter.

    I’ve seen many development-led initiatives fail to respect people as well. At a previous company, we had a distinct pattern for making big technical decisions: call all the best people into a room to discuss the possibilities then assign a small group to deliver the changes. This ensured that the best developers solved the major issues, but everyone not in the council had neither input nor visibility into changes that affected their work. It was incredibly demoralizing to be a junior developer and know a big, new web framework was being developed behind closed doors, with no obvious way to influence its outcome. If you’re having a meeting with all the best people in your department, then there is only one thing you should be doing: firing everyone else. Seriously, what’s the value proposition of tolerating mediocrity? Why employ someone only to ignore their most valuable asset, their creative problem solving process?

    The open source community, and the Groovy community in particular, is an excellent case study in how to respect people. The entire Groovy product development process is open and transparent. If you want to know what is going on, then subscribe to the Groovy-Developer mailing list and the Groovy JIRA ticketing system on The Codehaus. Both resources are open to the public and almost nothing happens that is not documented in one of these two places. If you want to provide input to the future direction of Groovy then just get involved in the email discussion and start commenting on the issues. Of course, the real beauty of Groovy is how the language grows and accepts changes. More than other languages I have seen, Groovy relies on their actual users for improvements. If several users ask for the same utility method then, more than likely, that method will show up in the next release. Methods like the new tr(), stripMargin(), stripIndent, expand(), and unexpand() were all added for the forthcoming 1.8 release. Instead of relying on research papers and doctoral theses for inspiration on how to grow the language, Groovy looks directly to you, the users. Next time you need to copy and paste utility methods between projects, or somehow else share that StringUtils you have, why not just create a new JIRA ticket and attach the source? The community will thank you and you’ll feel good about yourself.

    The big question now is how you can get your company to start respecting people. Pitching an open source style work-flow isn’t going to get you very far. Most likely, the changes you need to make to start respecting people means changing a deep and long-standing business culture. Nathaniel Schutta summed up this common scenario perfectly: “Changing culture is hard, but for many organizations, it’s the critical first step towards better software.” Regardless of how open to new ideas your company is, the best approach to introducing change is often leading by example. The easiest way to convince a skeptical developer or operations team member of the value of Groovy is use it to make someone’s life easier. Repetitive environment configuration steps? Find a way to script it. Lengthy troubleshooting cycles? Ship operations some helpful .groovy files.

    Sidebar: Agile Stage 0

    Be careful promoting agility within your workplace: your company simply may not be ready. Andy Hunt of the Pragmatic Programmers describes an unwilling culture as “Agile Stage 0”, meaning it is not ready for agile. Symptoms include: behaving territorially, being inflexible, growing uncomfortable with uncertainty, treating developers like a commodity, and believing development is a linear process. If this describes your workplace, then expect a slow pace of adoption and find ways to enjoy the small victories.

    A sustainable pace for programmers requires that we are allowed to work with technologies we love. Great techies crave new technology, and good companies crave great techies. In general, which job description do you think appeals to the best programmers: one listing Java and JUnit or one listing Java and Groovy? Fostering pride in workmanship is a key part of Respecting People, and the cultural change of allowing new ideas and languages like Groovy to flourish in your workplace can improve morale, attract new employees, and retain the ones you have. I’ve seen this first hand on a past project. For companies that aren’t early adopters, that have lots of legacy code, using Groovy outside of production source code, in testing, scripting, and builds, can be a great way to energize and respect the team.

    Part of Respecting People is moving responsibility and decisions to the lowest level possible. The person most capable of making a decision about how to perform a job is the person doing that job. Creating knowledge of Groovy within your team gives everyone more and better choices for completing tasks. Consider the scenario of needing to create an ad-hoc web service client, perhaps for the QA or operations team to consume some of your web services. The GroovyWS module provides an easy way to do this, as seen in Listing 1.

    Listing 1: Deployable SOAP Web Services Script

    This script “Grabs” the GroovyWS module, contacts a web service endpoint, and dynamically creates a web service client capable of calling operations on the service. This is the entire application, nothing is hidden, including a deployment module! You can email this script to QA or Ops without a Jar and without an accompanying cryptic command line invocation. The @Grab Groovy annotation is a way to retrieve dependencies from repositories. Running this Groovy script will download the GroovyWS module and put it in the script’s classpath. The user only needs Groovy installed, one small change to the user’s HOME directory so that the Maven repositories can be found, and they will never again have to receive large Jar files or confusing scripts to run. And, unlike many Java web service solutions, there are no generated Java files that you’ll need to step through later trying to find out what is broken. The Grapes module system is a boon for anyone needing to help out other teams (which is pretty much all of us).

    Sidebar: Configuring Grapes

    Grapes needs to be told once where to find the Maven repositories. You do this in the ~/.groovy/grapeConfig.xml. On Unix systems, ~ is the user’s home directory, or $USERHOME. On Windows, this is something like “c:/Documents and Settings/username”. The GroovyWS website [4] has clear instructions on the XML to copy into this file.

    It is easy advice to tell someone they should learn a scripting language. Having Groovy available to write scripts like these, that leverage the Groovy module system and are easily run by other teams, has clear value when compared to trying to write the Java equivalent. But a problem arises when developers are allowed to use dynamic scripting languages only for scripting. How do you learn a language effectively when you can only use it for one task a month? You don’t. Using a scripting language only for scripting is a mistake. It ensures that only the most self-motivated developers will take the time to write a script, and the majority of developers will have no clue how to maintain it should anything ever break. Effective scripting will only happen after a language has been adopted, it is not a means to language adoption. Groovy testing is a better gateway towards making these other opportunities viable.

    Next steps

    W. Edwards Deming was an American statistician and professor who played a significant role in creating Japan’s reputation for high quality products. Deming emphasized a whole-systems, multi-dimensional view of the manufacturing process called the System of Profound Knowledge. Deming summarized his views in a short list of 14 points, one of which contained this: Eliminate annual performance ratings for salaried workers. I wholeheartedly agree. Annual performance reviews feed rivalry and office politics, producing an environment of fear around the first of the year. And for peer-reviews, I have never seen these amount to anything more than a popularity contest. The number one action to take to ensure a good year-end peer review is to remember to bring in treats and pastries once a month. So much for focusing on business value. Annual reviews do nothing to improve good teams and simply waste time for poor teams.

    Without reviews how will team members know how to improve? How will we know who to promote? How much do we give in raises? One of my favorite agile thinkers, Esther Derby, recently republished several articles on her website addressing just these issues [7], [8], [9]. Ongoing, face-to-face coaching sessions need to replace a year end evaluations. And if someone isn’t doing their job, then why wait for the year end cycle to fire the person? Likewise with those who are over-performing. If you don’t find ways to reward top performers more often than at year end, then you’ll find many of them move on in the middle of the year! Profit sharing is a much fairer compensation plan than raises, but most of us aren’t doing it for the money anyway. Before disagreeing too quickly, watch the YouTube video called “Drive: The surprising truth about what motivates us” [10]. You’ll realize reward is a complex subject; but reward definitely doesn’t come from a year end review process. A good step to building an honest, trust based, and high performing workplace is to throw out those cursed year end reviews.

    Next Month: Optimize the Whole

    Improving your business is too often an exercise in sub-optimization: you may spend all your effort on improving the unit testing experience without realizing it’s your interactions with operations that’s derailing the project, or you may perfect your own team’s delivery process while the rest of the company continues behaving wastefully. In either case, it pays to step back and examine the situation as a whole. If you’re optimizing your unit testing with Groovy, then step back and ask how you can improve the entire project and not just testing. If you’re working with your team on a better software life-cycle, then step back and ask how you could improve the life-cycle of all the teams. Next month we’ll discuss how to “measure up” and optimize the whole, looking specifically at how to use easyb to facilitate cross-department interactions and Gradle for full project automation.

    Learn More

    I’ll be presenting a session on Lean Groovy in Denver at the Uberconf conference in July. I hope to see you there! Otherwise, several references were used to create this article, and all are enjoyable, informative reads in their own right:
    [1] Lean Software Development – Poppendieck
    [2] Implementing Lean Software Development – Poppendieck
    [3] http://www.gembapantarei.com/2008/02/exploring_the_respect_for_people_principle_of_the.html
    [4] http://groovy.codehaus.org/GroovyWS
    [5] http://ntschutta.com/jat/2008/06/21/culture-kills/
    [6] http://blog.toolshed.com/2008/06/stage-0-not-rea.html
    [7] http://www.estherderby.com/2010/07/eliminate-performance-reviews.html
    [8] http://www.estherderby.com/2009/01/the-pay-off-in-merit-pay-not.html
    [9] http://www.estherderby.com/2010/07/performance-without-appraisal-addressing-the-most-common-concerns.html
    [10] http://www.youtube.com/watch?v=u6XAPnuFjJc

    Hamlet D’Arcy Canoo Engineering AG
    Hamlet D’Arcy has been writing software for over a decade, and has spent considerable time coding in C++, Java, and Groovy. He’s passionate about learning new languages and different ways to think about problems, and recently he’s been discovering the joys of both F# and Scheme. He’s a committer on several projects including Groovy, CodeNarc and JConch, and is a contributor on a few open source projects (including Griffon and the IDEA Groovy Plugin). He blogs regularly at http://hamletdarcy.blogspot.com and http://canoo.com/blog.


    Slimmed Down Software- A Lean, Groovy Approach Part 5- Deliver Fast

    October 28th, 2010

    This article originally appeared in the August 2010 edition of GroovyMag, the Groovy and Grails magazine. Parts 6 and 7 are currently available for download from the magazine’s site, and more will come each month. Previous articles in this series are on the Canoo website: Part 1: Eliminate Waste, Part 2: Build Quality In, Part 3: Create Knowledge, and Part 4: Defer Commitment. Lastly, if you like this, you may want to check out some of my older blog posts from my personal site under the ‘craft’ category. Enjoy!

    The Groovy Programming Language advertises itself as an “agile and dynamic Language for the JVM”, but what does this mean exactly? This series of articles explains Lean Software Development, and shows how your choice of programming language can make your entire process remain nimble and adaptive. Each month’s article will cover one of the seven Lean Software Development principles and explain how Groovy and the associated ecosystem help eliminate waste, defer commitment, and build quality into your product.

    About this series

    Groovy is an agile programming language. In order to explain what this means, these articles are structured around the seven principles of Lean Software Development. The previous installments included a short introduction to Lean, explained how an expressive language can eliminate waste in unit testing, showed how the easyb Behavior Driven Development framework and Groovy Builders can build quality into your life-cycle, demonstrated how the Spock Testing Framework can create project knowledge, and discussed how tools like Real Options and dynamic languages defer commitment. In later months we’ll see how Groovy’s Grapes module system, alongside modules like Groovy Web Services, streamline interactions between development and other groups like operations and QA (an important aspect of respecting people). Finally, we’ll look at optimizing the whole by applying Groovy to domains that cross department boundaries: how Gradle improves the build process and easyb serves as a collaboration mechanism between different organization roles. The series will end with a discussion on the best practices of mixing Java and Groovy.

    Some of the code examples are basic while others are advanced. The articles explore why the features of Groovy are important rather than the mechanics of any one Groovy feature. Hopefully you’ll leave with some new ideas about how to use Groovy, how to convince your team that Groovy is worthwhile, or most importantly, how to increase your productivity.

    Lean Revisited

    Part 1 of this series contained a more in-depth explanation of Lean, and I won’t repeat it here. But as a quick refresher, here are the 7 lean principles along with a summary for those principles covered previously:

    1. Eliminate Waste – Reduce work in progress and half done work
    2. Build Quality In – Instead of settling for finding defects, optimize your process to create fewer defects
    3. Create Knowledge – Create fast feedback cycle and encode project information in executable form
    4. Defer commitment – Delay decisions until the last responsible moment
    5. Deliver Fast
    6. Respect People
    7. Optimize the Whole

    Principle 5 – Deliver Fast

    The easiest way to deliver quickly is to deliver less. There is a kind of mathematical simplicity to that statement. Short iterations are a reflection of this principle: pick one thing to do, finish it, release it, then move to the next thing. The assertion agile makes is that you’ll go faster over time by continuing to work in small batches. The rhythm of the iteration is the same each time: take a small piece of work, finish it, integrate and release, then start over.

    It is clear and logical that developing features in small increments means you deliver sooner, and there is value in that. Delivering a partial system on an earlier calendar date means the customer starts accruing a return on investment before project completion, and this can even result in the system paying for itself before the project is even finished. But will delivering smaller increments really increase your output in the long term? The answer from queuing theory research is a resounding “yes”.

    Queuing theory is the study of waiting in lines, or queues, and concerns itself with making your wait as short as possible. It can be applied to software development if we look at our overall development process as a system that takes ideas and visions as input and produces working, tested, and documented software as output. In software production, delivering fast means getting ideas and visions to flow faster through the system, producing working software sooner. Studying how queues work, whether it is supermarket queues, bank teller lines, our your team’s product backlog, has produced several findings that should influence how you schedule and manage work:

    • As variability increases, delivery becomes slower – The more people that show up at a bank, then the longer they will each on average have to wait. The more features crammed into your product or iteration backlog, then the longer each takes to deliver.
    • As batch size increases, delivery becomes slower – Guess what happens when there is one user story in your backlog that is twice as large as all the others? The same thing that happens when someone shows up in a supermarket express lane pushing a cart overstuffed with goods: processing time increases and delivery becomes slower.
    • As utilization increases, delivery becomes slower – If your web servers are operating at 100% capacity then your operations team (possibly you!) should be getting paged so that the problem can be fixed. In a traffic jam, everyone has to wait longer, and the same applies to the development process. You shouldn’t schedule your servers or your team to be utilized 100%. It leaves no time for task switching, maintenance, improvements, or innovation.

    There is always more work to do to improve your system, even for teams with a small batch size, steady arrival rates, steady processing rates, and built in slack time. The best way to increase throughput in a system is to find the biggest bottleneck and fix it. Work often piles up when developers hand off features to QA. If so, then it’s time to take more ownership of quality and start writing your own automated acceptance tests. Easyb is an excellent choice for writing Groovy based acceptance tests. If work is piling up in operations, waiting to be deployed to the test environment or production environment, then it’s time to help with project and deployment automation. Currently, Gradle is the best choice for project automation tools, and its features outweigh the more limited scripting abilities of the competitors like Maven, GMaven, or Gant, and the front end to Gradle is conveniently Groovy based.

    The worst way to increase throughput in a system is to increase productivity in a non-bottleneck area. If your deployment process has no bottlenecks, then migrating from Maven to Gradle will not help you deliver faster. Improving test coverage in a system with no defects and rework isn’t going to help deliver fast. The only locations in your process that should be improved are the bottlenecks, and even then it makes sense only to work on the worst bottleneck, fix it, and then move on to the next. Addressing capacity bottlenecks can reduce cycle time even in organizations that are already efficient.

    I spend most of my day writing code, some of it Groovy and some of it Java. In the Java projects, I’m always struck by how large and comprehensive small changes quickly become, even early on in a feature change or bug fix. In Java, have you ever refactored the public API of a class and then tried to run the tests only to find that a dozen or so of them won’t compile? You need to laboriously integrate your changes into the test files before checking that your changes even work. This always comes as an interruption to my workflow. What I wanted to do was make a small change and test it, but what I got was a huge and unexpected variation in the arrival of goods. It was as if the javac compiler showed up in my supermarket express lane and cut right in front of me with a shopping cart packed full with compile errors. Against my desire, batch size increased, and the arrival of goods (the work I needed to get done) started to vary. Both queuing theory and my personal experience tell me that this slows me down and less work gets done overall.

    What I love about programming Groovy is that you can take a small batch approach to your code. To run a Groovy test, the only source unit that needs to compile and run is the test class you’re trying to execute. And even that is an exaggeration: if you try to run a single test method through your IDE then the other methods can contain quite invalid code!

    The small batch cycle of Groovy is this: make your changes to the production code; rerun the single unit test you care about. Iterate until your changes are complete and tested, then run the entire test suite, integrating your change back into the rest of the project when needed. Take a small piece of work, finish it, integrate and release, then start over. Small batch testing is better than the alternative, just as small batch iterations are better than the alternative.

    Neal Ford once said, “I would rather program Groovy in vi than Java in IDEA.” While I don’t entirely agree with the statement (uh, ever heard of emacs?), the truth behind is that the Flow of Groovy is to have two files open: the production code and the unit test. You don’t need to have any more files than that open at one time, and you don’t exactly need a fancy IDE that allows you to navigate giant software projects with science fiction accuracy. You need two windows, and nowadays they both fit on one decently sized monitor. Of course, with the excellent Groovy support in the IDEs available, there is no reason to switch back to 1980s technology. You can have your small batch and your fancy IDE too.

    The Deliver Fast principle complements the Defer Commitment principle covered last month. The faster you can deliver, the longer you can delay decisions. To deliver fast, let the lessons learned from queuing theory guide your process improvements. Reduce batch size, even out variations, and create a little slack time. Then find the biggest bottleneck in your process and apply a little Groovy magic to make it go away.

    Sidebar: Little’s Law and throughput

    Little’s Law is a simple and intuitive observation with big ramifications. It states: Inventory = Arrival Rate × Flow Time. You are already familiar with this formula from visiting supermarkets. When the line is 5 people long, and the clerk takes 2 minutes to checkout each person, then your wait will be 10 minutes. The Inventory is 5 people and the Flow Time is ½ a person per minutes. Your Arrival Rate is therefore 10 minutes: 5 = n × ½.

    Little’s Law implies that you can deliver faster by reducing work in process, or the amount of items in the system. As your inventory diminishes, items flow through the process faster. When lines are shorter at the supermarket, then more customers can make it through the checkout counters. And you will move user stories to done faster by reducing the amount of stories in-process.

    Be careful, as this is a simplification. Reducing work-in-progress without making any other changes will also reduce throughput. Small batches, slack, and reduced variability should still be your goals. As H.H. Munro said, “A little inaccuracy sometimes saves a ton of explanation.”

    Next steps

    Delays are costly. Time spent waiting is clearly a waste: you are by definition doing nothing. Delays, on the other hand, are a different type of waste more akin to lost opportunity cost. You may follow the advice here, and bring a lean software project to market in 6 months. Or maybe you don’t and it takes you 7 months. A naïve way to calculate the extra cost is one month of paying the development team. But what about the cost of not having your product on the market? You must count the cost of lost market share, a later return on investment, and less revenue. Delays have costs (paying the team for a longer time) but they also have expenses (less revenue and market share). With so many variables, decisions are tough.

    To simplify decisions, put on your accountant hat and create a profit and loss worksheet estimating costs, expenses, and income. And be sure to make them all in the same unit: dollars, euros, whatever. Next time you are debating whether it is better to deliver in two weeks with an extra feature are now without the feature, simply consult the P&L sheet and see which is more profitable. With an economic model, the team becomes empowered to look after the business’ best interests themselves. And data and knowledge-based decisions provide an easy-out when arguing with passionate product owners and developers. The Poppendieck text [1] provides some excellent templates to follow.

    Next Month: Respect People

    Fostering pride in workmanship is a key part of Respecting People, and the cultural change of allowing new ideas and languages like Groovy to flourish in your workplace can improve morale, attract new employees, and retain the ones you have. Next month we explore how using Groovy applies to respecting people and get back to some code examples with Grapes and GroovyWS.

    Learn more

    Several references were used to create this article, and all are enjoyable, informative reads in their own right:

    [1] Lean Software Development – Poppendieck

    [2] Implementing Lean Software Development – Poppendieck

    Need help with agile? Canoo is ready to help. It’s what we do. Call or email any time.

    Hamlet D’Arcy Canoo Engineering AG

    Hamlet D’Arcy has been writing software for over a decade, and has spent considerable time coding in Java, Groovy, and C++. He’s passionate about learning new languages and different ways to think about problems, and is keenly interested in code quality, testing, and static analysis. He’s a committer on several projects including Groovy, CodeNarc, and JConch, and is a contributor on a few open source projects (including Griffon, and the IDEA Groovy Plugin). He blogs regularly at http://hamletdarcy.blogspot.com and http://canoo.com/blog.


    css.php