This article originally appeared in the July 2010 edition of GroovyMag, the Groovy and Grails magazine. Parts 5 and 6 are currently available for download from the magazine’s site, and more will come each month. Previous articles in this series are on the Canoo website: Part 1: Eliminate Waste, Part 2: Build Quality In, and Part 3: Create Knowledge. Lastly, if you like this, you may want to check out some of my older blog posts from my personal site under the “craft” category. Enjoy!

The Groovy Programming Language advertises itself as an “agile and dynamic Language for the JVM”, but what does this mean exactly? This series of articles explains Lean Software Development, and shows how your choice of programming language can make your entire process remain nimble and adaptive. Each months article will cover one of the seven Lean Software Development principles and explain how Groovy and the associated ecosystem help eliminate waste, defer commitment, and build quality into your product.

About this Series

Groovy is an agile programming language. In order to explain what this means, these articles are structured around the seven principles of Lean Software Development. The previous installments included a short introduction to Lean, explained how an expressive language can eliminate waste in unit testing, showed how the easyb Behavior Driven Development framework and Groovy Builders can build quality into your life-cycle, and demonstrated how the Spock Testing Framework can create project knowledge. In later months we’ll explore how metaprogamming allows you to deliver fast, and see how Groovy’s Grapes module system, alongside modules like Groovy Web Services, streamline interactions between development and other groups like operations and QA (an important aspect of respecting people). Finally, we’ll look at optimizing the whole by applying Groovy to domains that cross department boundaries: how Gradle improves the build process and easyb serves as a collaboration mechanism between different organization roles. The series will end with a discussion on the best practices of mixing Java and Groovy.

Some of the code examples are basic while others are advanced. The articles explore why the features of Groovy are important rather than the mechanics of any one Groovy feature. Hopefully you’ll leave with some new ideas about how to use Groovy, how to convince your team that Groovy is worthwhile, or most importantly, how to increase your productivity.

Lean Revisited

Part 1 of this series c ontained a more indepth explanation of Lean, and I won’t repeat it here. But as a quick refresher here are the 7 lean principles along with a summary for those principles covered previously:
1. Eliminate Waste – Reduce work in progress and half done work
2. Build Quality In – Instead of settling for finding defects, optimize your process to create fewer defects
3. Create Knowledge – Create fast feedback cycle and encode project information in executable form
4. Defer commitment
5. Deliver Fast
6. Respect People
7. Optimize the Whole

Principle 4 – Defer Commitment

Deferring commitment is about keeping your options open. It may seem wise to make decisions early, and it does lead to a comforting illusion of progress. But decisions can be costly to unmake, and oftentimes new information will become available that affects the choices you’re willing to make. The best time to make a decision is at the “Last Responsible Moment”: the point in time where not making a decision eliminates an important alternative. By delaying decisions we are ensured that when new information does becomes available then we can factor it into our project. Just be sure not to wait too long because at some point a lack of decision will force your hand.

One of the most interesting agile family members to arrive recently is the Real Options community, which grew out of an understanding of financial risk management models. Given the recent world banking crisis, following the advice of financial risk modelers may seem as foolish as looking to oil giant BP for a good decision making framework. It is better to think of Real Options as a methodology based on both mathematical models and the basic principle of many agile processes: the freedom to choose.

As software developers, decisions are what we do: they are pervasive. We code decisions into unit tests and production code. We encode them into acceptance tests and test plans. And oftentimes, we discuss, rehash, and duplicate them endlessly in a merry-go-round of meetings and requirements documents. Of course, from the system analyst’s perspective the flow of value is reversed: decisions are made during analysis and the outcomes are duplicated and perverted by the developers in a merry-go-round of bug fixing. Regardless of your role, there are three types of decisions: a right decision, a wrong decision, and no decision. Real Options is about harnessing the power of “no decision”, which can be the best decision to make because it allows us to defer commitment until we have more information available. A better informed decision is a better decision and more likely to be right. There is a big value difference between making a premature decision today, making an informed decision tomorrow, and not making a decision at all and having an opportunity pass you by.

The fictional Doctor House from television is a master of leveraging the value of no decision. When faced with a mysteriously ill patient, his team masterfully narrows down the illness possibilities to either avian monkey pox, swine sars, or some other terminal infection; either way the patient will die in 24 hours. The important plot point is that treating an avian monkey pox patient with swine sars medicine results in immediate death, and treating a swine sars patient with avian monkey pox medicine also results in immediate death. Clearly, there is no good decision. Guessing the wrong illness could kill the patient (and end the show 30 minutes too early). So what do they do? They wait. And wait. And wait until more information is available. Their decision timebox is 24 hours, so they can spend 23 hours waiting before a decision must be made. Meanwhile, they poke, prod, and test the patient, trying to determine the true illness. Eventually they work to create knowledge, discover a new diagnosis, administer the penicillin, and end the show.

Real options are about the flow of information arrival; remember how developing software was a knowledge creation process? Creating knowledge in the system, and deferring decisions until this knowledge is available, leads to better decisions… making the “right decision” more frequently. We need information to arrive before making commitments, and we push out decision making until the last responsible moment. As the authors state, “The last responsible moment is after which execution of an option is too late. The option no longer exists or you’re not able to execute it any more.”[1].

Options contain three important aspects:
1. Options have value
2. Options expire
3. Never commit early unless you know why

In the case of House, deferring the decision to create swine sars was a valuable option: it left the patient alive. But that option expires: swine sars is deadly after 24 hours. By not committing to a possibly deadly treatment, House kept the patient alive until the proper treatment could be found. If there is no reason to make a commitment then do not make it.

This might sound like common sense. But it is very difficult to follow because of the psychology surrounding decision making. People hate uncertainty, so they naturally drive to make decisions early. This is even more evident in big meetings, where leaders need to be seen leading and progress needs to be made. Not only are big meetings costly, but they can lead to premature decisions, thus further driving the cost of business up. Counter-act this aversion to uncertainty by timeboxing decisions: pick an exact date or conditions to be met before making a decision. Replace the phrase, “We should not make a decision on this” with either “We should not make a decision until after the next iteration on Tuesday” or “We should not make a decision until after the prototype of Plan B is finished.” Giving people certainly over the time frame of a decision balances the natural angst of deferring the decisions. And as you can guess, keeping an up-to-date issue log visible in the team area is a best practice.

For more information on Real Options, download the free “Real Options at Agile 2009″ comic book by Chris Matts[4]. The illustration is rough, but the content, humor, and ambition are top notch. Now let’s get back to programming.

Do you remember the original Java white paper in 1995 from James Gosling[3]? Me neither, but you can find it on the Internet if you search hard enough. It contains a prescient passage on the value of both deferring commitment and dynamic languages like Groovy:

“Very dynamic languages like Lisp, TCL and Smalltalk are often used for prototyping… [One] reason given for these languages being good for prototyping is that they don’t require you to pin down decisions early on. JAVA has exactly the opposite property: it forces you to make choices explicitly.”

Perfectly well said, except that Groovy is missing from the list of good dynamic languages and JAVA is now written Java, as enforced by the divine writ of the Sun/Oracle legal department. As Java programmers, we are used to making commitments: up front and not deferred. We are asked by the compiler to make them early and make them often. I never noticed this until after I had spent a considerable amount of time in a dynamic language. Now, when programming Java, I feel that I’m constantly being asked the same question: what is the name of type you are using? It’s easy to notice this within the methods and code you write. Statements like “List<String> list = new ArrayList<String>();” are surely repetitive, but both IDEs and future Java versions are addressing this. A bigger problem is the repetition of types across unit tests. With Java, the compiler ensures that you commit to types in your API. If your “UserView” is a GUI for User objects, then the compiler is going to make darn sure that only User objects are allowed to be passed to it. Yet, in your test code for UserView, you must also make sure that Users are being passed; you’re being asked to commit to this type a second time, even though the compiler already enforces it! Have you ever refactored a production class only to find that a handful of unit tests now fail to compile, forcing you to update all the references before being able to run the system or run the tests? If so, welcome to premature commitment.

Groovy offers another way out. With Groovy, you can leave things dynamically typed and flexible for as long as you’d like, move them to be optionally typed when the need and commitment arises, or even convert them to Java and static typing. Consider the unit test example in Listing 1, which tests an imaginary UserView object, a user interface over a list of users.

public void testUserView() {
  def userList = [
    [getId: { 111 }],
    [getId: { 222 }],
    [getId: { 333 }],
  ]
  def view = new UserView(userList)
  assert view.rowCount == 3
  view.remove(userList[1])
  assert view.rowCount == 2
}

This is the test for the aforementioned UserView.Users can be added to and removed from the UserView, a behavior this test confirms. Does the UserView work off of Users? Or some sort of generic interface? Who cares. That information is specified in the production code, and there is no need to duplicate it in the test code; this would only serve to make your tests more fragile. How you decide to refactor is your own business. The only detail we have committed to is that the objects the UserView expects must have a getId() method. This test is flexible enough to withstand however you want to refactor your production types. As Taiich Ohno, inventor of Lean says, “The sturdier the human spine, the more easily it bends.” The same is true with unit tests. Break dependencies and maintain options with Groovy’s optional typing. If there is good reason to commit to types then do so. If not, then don’t. Groovy, and agile, are both about keeping options open.

How exactly does this example show the dynamic typing? The example’s userList object is a list of Maps, each entry is created using the [key: value] notation, and in this case is a map from String to Closure. In Groovy, a Map of type <String, Closure&rt; is a shorthand for implementing an interface. As long as the code being called is flexible enough then any method calls to getId() will be dispatched into this Map’s getID closure. This is a very handy way to implement partial objects for wide interfaces. As a word of warning, an API expecting User objects will require you to declare this partial interfaces as Users using the “as User” keyword. This requires only a small change and keeps coupling to a minimum.

What I like about dynamically typed unit tests is it allows me to focus on the outputs of a system instead of the inputs. Is the value of an IT system in the inputs or the outputs? Can you name a single IT system with value that has no inputs? The value is always in the outputs of a system; we need to make commitments on system output because that drives business value, but we need to not make commitments on inputs because that leaves the system flexible. On a programming level, being able to forget about static types for testing input can be a great aide in developing faster without any real loss of value.

Groovy helps you defer commitment and stay agile, at the “expense” of accepting a dynamically typed language. For a lot of organizations, the decision to use a dynamic language is a big one, and not to be taken lightly. Limiting dynamic languages to testing is a great way to introduce your group to these new ideas. The test tree is a place to experiment while having minimal impact on your production system. Should you eventually decide not to embrace Groovy, it is much easier to rip out Groovy tests than it is production code! Using Groovy in test is yet another way to defer commitment: it is an easily undo-able decision. However, I doubt you’ll end up doing that.

Sidebar: Structural vs. Nominative Typing

There is more to the debate than just whether a dynamically typed or statically typed language is better! Java developers are already familiar with Nominative typing: a type must be given a name, and two objects share the same type if they share the same name. Structural typing, on the other hand, does not require names. A type has a certain form, or structure. It does not matter what you name a type, if two objects share the same method or function signatures then they are of equivalent types.

Nominative typing can be crude and inflexible. Using a language with closures for any time will make you wonder why we have the types Runnable, Callable, Action, Command, Listener, and a host of other types that could all be replaced with a single, good function object.

Duck typing in dynamic languages grew out of this dissatisfaction with nominative types. If your method requires an object with an execute() method then you can pass it any old object you want, as long as execute() is present. This check happens at runtime, and only the part of the object invoked is checked for the proper structure. In comparison, languages with better structural typing like ML, OCaml, and F# will make this check at compile time and require that the entire structure be identical, not just the part that is invoked at runtime.

Groovy achieves flexibility by moving away from static, nominative typing and lives in a middle zone where both duck typing and optional typing are supported.

Next Steps

Put options thinking to work in your next meeting. Recognize when a premature decision is being made and then make like Doctor House and follow these steps to delay the commitment:
1. Identify the options and alternatives available
2. Identify when the option expires: when a commitment must be made
3. Identify concrete steps to seek new options until the expiry date
4. Wait to make decisions… and wait… and wait… until the conditions are correct
5. Finally, decide quickly and act with confidence, knowing you have made the best informed decision possible

This may require keeping an active, visible issue list and being accountable for options. But you can prevent premature decisions by strategically delaying and then tracking when decisions actually do need to be made.

Next Month: Deliver Fast

Delivering fast is about delivering less. Not just less code but also less features. Next month we’ll discuss how Groovy enables you to focus on small batches of features and avoid both big bang integrations and the typically resulting scheduling setbacks. This is available today from GroovyMag, the Groovy and Grails magazine. Other articles about agile and software craftsmanship can be found on my personal site under the “craft” category.

Learn More

Several references were made in this article, and all are enjoyable, informative reads in their own right:
[1] “Real Options” Underlie Agile Practices – http://www.infoq.com/articles/real-options-enhance-agility
[2] Implementing Lean Software Development – Poppendieck
[3] “Java: an Overview” – James Gosling, February 1995
[4] Real Options at Agile 2009 – http://www.lulu.com/product/paperback/real-options-at-agile-2009/5949485
Also, the MEAP pre-release for “Groovy in Action 2nd Edition” does not yet (at the time of this writing) contain an updated treatment of optional typing, but it should be coming out shortly.

Thanks for listening.

This article originally appeared in the June 2010 edition of GroovyMag, the Groovy and Grails magazine. Parts 4 and 5 are currently available for download from the magazine’s site, and more will come each month. Enjoy!

The Groovy Programming Language advertises itself as an “agile and dynamic Language for the JVM”, but what does this mean exactly? This series of articles explains Lean Software Development, and shows how your choice of programming language can make your entire process remain nimble and adaptive. Each months article will cover one of the seven Lean Software Development principles and explain how Groovy and the associated ecosystem help eliminate waste, defer commitment, and build quality into your product.

About this Series
Groovy is an agile programming language. In order to explain what this means, these articles are structured around the seven principles of Lean Software Development. The previous installments included a short introduction to Lean, explained how an expressive language can eliminate waste in unit testing, and showed how the easyb Behavior Driven Development framework and Groovy Builders can build quality into your life-cycle. In later months we’ll explore how a dynamic language lets you minimize coupling and defer commitment, and how metaprogamming allows you to deliver fast. You’ll see how Groovy’s Grapes module system, alongside modules like Groovy Web Services, streamline interactions between development and other groups like operations and QA (an important aspect of respecting people). Finally, we’ll look at optimizing the whole by applying Groovy to domains that cross department boundaries: how Gradle improves the build process and easyb serves as a collaboration mechanism between different organization roles. The series will end with a discussion on the best practices of mixing Java and Groovy.

Some of the code examples are basic while others are advanced. The articles explore why the features of Groovy are important rather than the mechanics of any one Groovy feature. Hopefully you’ll leave with some new ideas about how to use Groovy, how to convince your team that Groovy is worthwhile, or most importantly, how to increase your productivity.

Lean Revisited
Part 1 of this series contained a more indepth explanation of Lean, and I won’t repeat it here. But as a quick refresher here are the 7 lean principles along with a summary for those principles covered previously:
1. Eliminate Waste – Reduce work in progress and half done work
2. Build Quality In – Instead of settling for finding defects, optimize your process to create fewer defects
3. Create Knowledge
4. Defer commitment
5. Deliver Fast
6. Respect People
7. Optimize the Whole

Principle 3 – Create Knowledge

I prefer the term “Software Engineer” to “Programmer” because it evokes a sense of professionalism, adherence to standards, and attention to quality that sometimes seems lacking in our industry. But “Engineer” also carries negative connotations when applied to software. As my co-worker Mike Mannion says, “Software development has less to do with conveyor-belt-fabrication (the image unfortunately most likely to be to be conjured up when we use the word engineering) than knowledge acquisition.” Indeed, Software development is a knowledge creating process. The mostbasic thing we do in our jobs is to discover what needs to be done in order to delight our customers. In fact, your product will benefit by finding ways to speed up this knowledge acquisition. Specifically, Lean provides two practices for this: short, frequent learning cycles and delayed commitment. Delayed commitment, postponing decisions until the last responsible moment, is covered in next month’s article. Today we’ll focus on shorting learning cycles.

So where does all this created knowledge reside? Large requirements and specification documents are enticing because they provide a wonderful illusion of certainty. They can be agreed upon in meetings, verified with checklists, cross referenced with traceability matrices, and then enforced by lawyers and courts when projects fail. Great software has as much to do with rock solid documentation as a great marriage has to do with a rock solid prenuptial agreement. No thanks, I say.

The software industry has clearly turned towards executable solutions for knowledge creation and retention, any plausible description for a marriage related “executable solution” leaves me feeling quite uneasy. Executable acceptance test frameworks abound: Fit, Fitnesse, and easyb, and new unit test frameworks appear every few weeks. The code you write, and the tests that accompany it are the most basic persistent form of system knowledge, and it is important that this knowledge be verified in an automated way.

The problem with traditional unit tests is that the test framework imposes so many particularities that the important knowledge of how your system must behave gets lost in a noise of subclassing, type definitions, and annotations. Wiki and spreadsheet based solutions like Fitnesse are a nice idea in theory, but I have not yet found customers willing to edit test data directly. Instead it is left to the developer to do it. To make traditional unit testing more friendly, some developers try to cut through this noise by using strange naming conventions and commenting their tests with standard comments, as in Listing 1.

public void test_users_can_be_retrieved() {
    // arrange
    ...
    // act
    ...
    // assert
    ...
}

I’ve found word separating underscores in method names to bring no real value. You’re going to have to press the shift key one way or another, whether it be with underscores or camel case. The practice I do like is the arrange-act-assert pattern. This pattern originated with William Wake as a way to add clarifying structure to unit tests. The idea is to codify and standardize the way tests are written: First arrange all necessary preconditions and inputs, then act on the object or method under test, and finally assert that the expected results have occurred. Tests are easier to read because the setup and verification phase is clearly separated from the block exercising the system under test, and test smells like a too busy act phase or missing assert blocks rise quickly to the surface.

Encoding your tests with comments is an interesting idea, but in the end comments are not executed, grow stale, and become misleading. A better choice than using comments is to pick a testing framework that eliminates all this noise, one that natively supports the structure of unit tests. A testing framework like Spock.

Spock is an innovative framework, taking new directions on fixture setup, mocking, and data driven tests. The goal of Spock is to allow the developer to focus solely on the input and output of tests and then get out of the way. One type of test available in Spock is the data driven test, seen in Listing 2.

def """does the user service contain
        a known set of users?"""() {
    setup:
        def service = new UserService()
    expect:
        def user = service.get(username)
        user.firstName == firstname
        user.lastName == lastname
        user.id == userID
    where:
        username << ['user1', 'user2', 'user3']
        firstname << ['John', 'Jane', 'John']
        lastname << ['Doe', 'Doe', 'Smith']
        userID << [123456, 654321, 789987]
}

Did you know that question marks and new line characters in methods names, like in Listing 2, are completely legal on the JVM? The Java language just doesn’t allow you to use them! While this is far from the main productivity boost from Spock, it does lead to some very nice looking JUnit reports. Figure 1 is our actual JUnit report from my current project. Seeing a space separated test name sure beats deciphering a camel case or underscore laden test name.

The value of Spock being built on top of JUnit should be obvious: your reports, builds, IDEs, and other tools continue to work as before. To the tools, Spock is just another unit test.

There are more interesting questions to ask about Listing 2, however. Reading the listing from top to bottom may be confusing. Are those “==” statements assertions, and how can they be made before the variables have been initialized? What does this test do exactly?

Spock shows the great power available behind Groovy. Those goto labels (setup, expect, and where) are some of the meaningful tags you are allowed to use within Spock, and these tags are enforced within the compiler. They are not goto labels at all but a vital part of a Spock test specification. Just as test fixtures can have setup and cleanup phases, so can individual test methods. Spock supports this idea natively at the compiler level. In-line comments and funny method names are replaced by first class support for testing.

Within the expect: phase, the statements including == are indeed assertions. You may write out the assert keyword longhand if you desire, but the point of an expect: block is to run assertions, so Spock lets you drop out this bit of ceremony from your tests.

Another surprising feature is that the expect block will be executed three times in this test: one time for each row in the dataset. So the first execution is with data user1, John, Doe, and 123456, the second execution receives the second “column” of the dataset and so on. Once you are used to Spock, the data tends to be read in vertical columns, since each column is input to the expect block. For testers weaned on JUnit the test might seem backwards, raising questions like “Why is the data the last piece of the test, shouldn’t that come first?” The short answer is, not if you want to create knowledge about your system. The important part of the system is how it behaves, how the UserService retrieves users. This information is front and center communicating to any future programmers on the project. The data that ensures this behavior comes later, and is usually less important than the assertions. The Arrange, Act, Assert pattern is required in imperative, sequential programming. Luckily, a strong language and an innovative framework let you focus more on knowledge creation than satisfying the semantics of your chosen compiler.

Data driven tests are just one of the options in Spock specifications. As you explore Spock you’ll find many of the BDD concepts available to use, as well as a unique solution to working effectively with mocking frameworks. Spock is built on JUnit, so it works without effort for popular build tools and IDEs, and the online documentation is superb (how often can you say that about an open source framework?). If you want to take tests as a form of knowledge creation, then Spock is definitely worth investigating.

The new 0.4 release of Spock contains another option for data driven tests as well: the data table. Previously, Listing 2 showed how lists can be used within Spock so that assertion blocks are automatically called several times with different data. The downside is that I sometimes find myself tilting my head as I try to read the vertical columns embedded in the lists. Similar to the data tables features in Scala’s Specs framework, Spock now allows you to embed pipe delimited data tables in your tests. Listing 3 is the same test as in Listing 2 but uses data tables instead of the Spock data
driven format.

def """does the user service contain
        a known set of users?"""() {
    setup:
        def service = new UserService()
    expect:
        def user = service.get(username)
        user.firstName == fname
        user.lastName == lname
        user.id == id
    where:
        username| fname | lname | id
        'user1' |'John' | 'Doe' | 123456
        'user2' | 'Jane'| 'Doe' | 654321
        'user3' |'John' | 'Smith'| 789987
}

The result is a much more readable dataset, neatly divided into rows and columns.

The test driven community has a great interest in improving the framework landscape. Unit tests have been heralded in the past as a way to capture information about the system being developed, but the JUnit and Java syntax combine to under-deliver on this promise. Finally, with both the Groovy language and frameworks like Spock, we are seeing unit tests come into their own in their ability to capture, express, and execute the workings of the system.

Sidebar: Expressive vs. Terse
I’ve been very careful to call Groovy expressive instead of terse, which are two different things. Terse means showing less. Terse can mean anything from an abstraction to an abbreviation. Court stenography or writing shorthand is terse. All of the information is captured, but in as small amount of space as possible. And practically no one except the original author can make sense of it. Perl and regular expressions are terse.

Expressive is also about showing less, but expressive specifically means showing less of the inconsequential and more of the consequential. If something is important, then an expressive language shows it to you. If something isn’t important, then an expressive language should hide that detail.

Being expressive is an important part of staying DRY (Do Not Repeat Yourself). Everyone knows that DRY mandates that you do not say things twice: there should be a single source of knowledge for any fact. Hiding information, and terseness, often violates this. An expressive language helps you stay DRY by making sure knowledge is embedded in the system once without a lot of extra noise. Simple terseness and hiding information might leave you with zero sources of knowledge! Not a good place to be. Strive for expressive, and be wary of terse.

Next Steps
An important part of creating knowledge is maintaining a culture of constant improvement: every team can benefit from a feedback cycle where you experiment with a new practice, reflect and learn from the experience, and then accept or reject the practice as part of your team culture. The annual, year end reviews common in many companies are the antithesis of this. They are self-improvement theater, where feedback comes months after events occur and the details of how to improve are lost behind closed door meetings, never to be heard from again. As an alternative, think short term. Treat each two week iteration as a chance to conduct an experiment. Try Spock for two weeks and then discuss the results as part of your iteration close. If the team doesn’t like it then stop using it and try easyb for two weeks.

If that doesn’t work then try something else. A two week commitment is much easier to sell to your teammates than a long term framework decision, and two weeks shouldn’t be long enough for even the worst decisions to irreparably harm a project. Week by week your team will improve as some of the experiments work and others fail. Just remember, an experiment or new practice should try to solve a specific problem for your team.

Next Month: Defer Commitment
Deferring commitment is about your keeping options open. It may seem wise to make decisions early, and it does lead to the illusion of progress. But decisions can be costly to unmake, and oftentimes new information will become available that affects the choices you’re willing to make. Next month’s article explores how Groovy and dynamic languages allow you to make decisions when you are ready, investigating ways in which you can use Groovy’s optional typing to your advantage.

Learn More
Spock is available through Maven and Google Code at http://code.google.com/p/spock/. The web has some nice Spock tutorials, the project lead has a blog at http://pniederw.wordpress.com/ and Parleys.com has a great 20 minute presentation from the 2009 Devoxx conference available to subscribers. For more information on the very cool mock objects available then you may wish to read my own post on the topic: http://canoo.com/blog/2010/04/20/spock-and-test-spies-a-logical-choice/ For books on agility, I still contend these three cannot be beat:

This article originally appeared in the May 2010 edition of GroovyMag, the Groovy and Grails magazine. Parts 3 and 4 are currently available for download from the magazine’s site, and more will come each month. Enjoy!

The Groovy Programming Language advertises itself as an “agile and dynamic Language for the JVM”, but what does this mean exactly? This series of articles explains Lean Software Development, and shows how your choice of programming language can make your entire process remain nimble and adaptive. Each month will cover one of the seven Lean Software Development principles and explain how Groovy and the associated ecosystem help eliminate waste, defer commitment, and build quality into your product.

About this Series

Groovy is an agile programming language. In order to explain what this means, these articles are structured around the seven principles of Lean Software Development. Last month included a short introduction to Lean, and explained how an expressive language can eliminate waste in unit testing. This month we’ll see how the easyb Behavior Driven Development framework and Groovy Builders can build quality into your life-cycle. In later months we’ll explore how different testing tools like the Spock Framework help create knowledge, how a dynamic language lets you minimize coupling and defer commitment, and how metaprogamming allows you to deliver fast. In future installments you’ll see how Groovy’s Grapes module system, alongside modules like Groovy Web Services, streamline interactions between development and other groups like operations and QA… an important aspect of respecting people. Finally, we’ll look at optimizing the whole by applying Groovy to domains that cross department boundaries: how Gradle improves the build process and easyb serves as a collaboration mechanism between different organization roles. The series will end with a discussion on the best practices of mixing Java and Groovy.

Some of the code examples are basic while others are advanced. The articles explore why the features of Groovy are important rather than the mechanics of any one Groovy feature. Hopefully you’ll leave with some new ideas about how to use Groovy, how to convince your team that Groovy is worthwhile, or most importantly how to increase your productivity.

Lean Revisited

Part 1 of this series contained a more indepth explanation of Lean, and I won’t repeat it here. But as a quick refresher here are the 7 lean principles along with a summary for those principles covered previously:

1. Eliminate Waste – Reduce work in progress and half done work
2. Build Quality In
3. Create Knowledge
4. Defer commitment
5. Deliver Fast
6. Respect People
7. Optimize the Whole

Principle 2 – Build Quality In

If you want to build a high quality product, you could inspect your product after it is built to ensure that it conforms to standards. But it would be better to change how the product is made so as not to create defects in the first place. Picking the best of breed testing practices and using them before the system is completely built is of utmost importance to quality software. At one point in time JUnit and Java were the state of the art in software testing, and eventually Test Driven Development gained mind-share in the industry, pushing testing into the same phase as development. Today, some of the best tools and practices are coming from the Behavior Driven Development community. Groovy is lucky to have an excellent choice for BDD in the Jolt Award winning easyb framework.

Instead of writing tests, easyb focuses on writing “stories”. A story is a piece of functionality that is visible and meaningful to the user. The intent is different than unit testing, although it does occur frequently at the same level. Stories are expressed in a given/when/then format. For instance, the story for a web service might be: given a user web service, when a user is looked up in the web service, then the user’s record should be found. The example in Listing 1 might make this clearer.

using "xmlunit"
scenario "web service should return XML based users", {
  given "a user web service ", {
    service = new UserWebService()
  }
  when "a user is looked up in the web service", {
    webServiceResult = service.get(123456)
  }
  then "the user's record should be found ", {
    webServiceResult.shouldBeIdenticalTo """
        <user id= "123456">
        <firstname>John</firstname>
        <lastname>Doe</lastname>
        </user>"""
  }
}

To understand the low ceremony nature of easyb, understand that Listing 1 is the entire file. There is no setup, test class subclass, or magical annotation incantation required. The essentials of your test are what it is testing, not the mechanics of a test framework. Easyb works with you in achieving a test with only the essentials being communicated.

You may have also noticed a lack of variable declarations. Where are these variables declared and what are their scope? Again, declarations are rarely an essential part of a unit test, so easyb handles this for you. Variables are scoped within the scenario they are used and instantiated when referenced. Easy, and with a lot less fluff than Java. Hopefully you are wondering how all of this works (if not then skip to the next paragraph)! Remember, parenthesis are optional in Groovy method calls, and {} is used to create a closure. In this easyb story, scenario, given, when, and then are all just normal methods that take a String and a Closure as parameters. While it may look terse enough to be qualified as a domain specific language, this is for the most part just plain old Groovy. The implicit variable declarations are a little harder to explain, but suffice it to say that a little metaprogramming is happening under the covers.

At first glance, easyb might appear to be a lot different from the traditional JUnit approach. Some of it is quite different, which gives easyb its power. But for the most part, it is plain old Groovy and is a good JVM citizen. There is plenty of build system support for easyb (an Ant task and plugins to other systems), IDE support for at least IntelliJ IDEA and Eclipse, and other JVM tools like code coverage applications should still work with it. If you are serious about building quality into your product, then take a hard look at using BDD and easyb to drive down defects before they appear in the QA phase.

Behavior Driven Development

BDD bills itself as “Unit Testing Done Right “. So what was wrong with unit testing? Unit testing is a good start, but often addresses how objects behave at the microscopic level rather than focusing on what matters to stakeholders. BDD changes this by focusing testing back on the project and the value of the system. Developers are encouraged to focus on why tests should be created rather than adhering slavishly to a one test per object approach. It might sound more like functional testing than unit testing.

The way to focus tests back on business value is by developing and using a ubiquitous language. When your customer says, “The user should be able to change their password,” then that should be your test name. Calling your test fixture UserManagerTest, your test method testChangePassword, and your assertion assertEquals creates an artificial language barrier between you as a developer and the business. It is better to call the test User Features, the test method user must change password, and the assertion should be. A common and ubiquitous language breaks down barriers between you and the business and keeps you focused on why you’re writing the system in the first place.

Groovy Builders

Another Groovy language feature useful in achieving low ceremony testing is multi-line Strings. Strings may be triple quoted, like in the Listing 1, and then all special characters and whitespace may be embedded in the String without escaping or concatenation. This makes working with XML a breeze: no need to litter your test with long blocks of string concatenation, nor do you need to hide the XML snippets away in an external file where they aren’t visible when the test fails.

Building data, and lots of it, is a common occurrence in unit tests. Multi-line Strings are nice, but are definitely not the only way to build String based data. One of the most helpful classes in the Groovy library is the MarkupBuilder, as seen in Listing 2.

def writer = new StringWriter()
new groovy.xml.MarkupBuilder(writer).
   user(id: 123456) {
      firstname('John')
      lastname('Doe')
   }

<user id= "123456 ">
   <firstname>John</firstname>
   <lastname>Doe</lastname>
</user>

Listing 3: XML output from Listing 2

The MarkupBuilder is a dynamic object that treats method calls as requests to build markup (whether it be XML or HTML). Calling a method, any method, is going to create an XML Node with the tag named the same as the method. Passing a key/value pair is going to add an attribute to that node, and passing a value is going to write that value as the node value. In Listing 2, all of this is written into the StringWriter object. The real power of MarkupBuilder comes when mixing dynamic method calls with real, statically declared method calls like List.each().

In Listing 4, the users() and user() method calls are calls on the MarkupBuilder, and will have start and end tags generated correctly, while the names.each() method is an invocation on the variable called names that is in scope. Building ad-hoc test input and test control data is greatly simplified by understanding and working with builders.

def names = ['Alice', 'Bob', 'Carol', 'Ted']
def writer = new StringWriter()
new groovy.xml.MarkupBuilder(writer).
   users {
      names.each { name ->
         user(name)
      }
   }

Listing 4: Mixing MarkupBuilder and Lists

<users>
   <user>Alice</user>
   <user>Bob</user>
   <user>Carol</user>
   <user>Ted</user>
</users>

Listing 5: Output from Listing 4

Don’t just rely on this builder though. Write your own builder for working with any sort of structured data. Any tree like data structure is a good candidate, as well as APIs that require heavy usage of setParent() or setChild() method calls. All of this can be hidden behind a builder. There is even an abstract class called BuilderSupport that takes care of all the heavy lifting.

Before writing your own builder with BuilderSupport, it pays to understand how the object works to capture arbitrary dynamic method calls. Listing 6 shows all the possible ways to interact with a Groovy builder. There are four options: you may invoke dynamic method calls with no parameters, a
single value parameter, a Map parameter, or a Map and a value parameter.

new groovy.xml.MarkupBuilder().root() {
   valueNode('a value')
   attributeNode(a: '1', b: '2')
   mixedNode(a: '1', b: '2', 'a value') {
      nestedNode()
   }
}

Listing 6: Possible Uses of Markup Builder

All of these methods take an optional closure parameter that lets you nest nodes arbitrarily deep. To generalize the builder pattern, there are 4 possible ways to invoke a dynamic method. With this in mind, the abstract BuilderSupport class is a logic implementation for a base builder class. If you want to create a new builder, then you need to implement four abstract methods, one for each type of dynamic method invocation. The BuilderSupport abstract methods can be seen in Listing 7.

class BuilderSupport {
   ...
   abstract Object createNode(Object name)
   abstract Object createNode(Object name, Map attrs)
   abstract Object createNode(Object name,
   Map attrs, Object value) abstract Object createNode(Object name, Object value)
   ...
}

Listing 7: BuilderSupport Abstract Class

What BuilderSupport does for you is handle all of the dynamic dispatch and nesting with closure parameters. Writing a new builder, and enjoying the accompanying Groovy “magic” really is as easy as implementing these four methods. You’ll get the most value from a builder by taking a few minutes early in the project to write one as a wrapper around your tree-like API.

Next Steps

One particularly rigorous Lean practice is a zero tolerance or stop the line approach for defects. The idea is that no defects should exist in the system, and when a defect is present then all attention and effort must be focused on resolving the issue before normal work resumes. In the face of error conditions and ambiguities, the assembly line stops until the issue is resolved. The effect is that your daily process becomes fine tuned around producing a high quality product early in production as opposed to optimizing a defect cycle days or weeks later. Stop the defects before they happen. My team recently applied this approach for a single two-week iteration with interesting results. Based on our retrospectives, we decided to not continue with a strict enforcement of this policy, but we all agreed we learned a lot and the lessons learned were carried forward to our current process. A worthwhile experiment; I suggest you try it.

Next Month: Create Knowledge

Software Engineering is a knowledge creating process. It is important to find the correct balance between tacit, organizational knowledge and explicit, recorded knowledge. Next month’s article explores what this means, looks at ways to use Groovy to capture and codify knowledge, and discusses the differences between terse and expressive. Along the way we will see some cool new features of Groovy and the amazing Spock testing framework.

Learn More

Everyone learns differently, and for me the best way to learn about a new framework is to experiment with it. Easyb can be downloaded from http://www.easyb.org/ and BuilderSupport is a standard Groovy class. Otherwise, the web contains many tutorials on both subjects, and “Programming Groovy” by Venkat Subramaniam contains an in-depth BuilderSupport chapter. The MEAP pre-release for “Groovy in Action 2nd Edition ” does not yet (at the time of this writing) contain Builder information, but it should be coming out shortly.

For books on agility, I still contend these three cannot be beat:

• Implementing Lean Software Development – Poppendieck
• The Pragmatic Programmer – Hunt and Thomas
• Extreme Programming Explained, 2nd Edition – Beck and Anders

This article originally appeared in the April 2010 edition of GroovyMag, the Groovy and Grails magazine. Parts 2 and 3 are currently available for download from the magazine’s site, and more will come each month. Enjoy!

The Groovy Programming Language advertises itself as an “agile and dynamic language for the JVM”, but what does this mean exactly? This series of articles explains Lean Software Development, and shows how your choice of programming language can make your entire process remain nimble and adaptive. Each month will cover one of the seven Lean Software Development principles and explain how Groovy and the associated ecosystem help eliminate waste, defer commitment, and build quality into your product.

About This Series

Most of your agile team will spend the majority of their day working with a programming language, making it one of the most important tools to optimize in your project’s life-cycle. Yet many organizations are reluctant to experiment with newer languages and frameworks, instead preferring to apply what works in one domain to all domains. They are skeptical about the claimed productivity gains from polyglot programming, concerned about the educational investment of retraining their developers, and worried about the risk of using less proven technologies.

Some languages are truly better than others for certain types of problems. These articles exist to help you identify which domains in your project are most in need of a little Groovy magic as well as to help you sell Groovy to a skeptical or unconvinced audience. Hopefully you will see that Groovy is a great fit for your software life-cycle: its dynamic and expressive nature brings greater productivity to testing and operational support, its closeness to Java minimizes educational investment, and its tight integration with the JVM platform allows you to slowly phase in Groovy without an all-or-nothing decision, making its use a low risk decision.

Groovy is an agile programming language. In order to explain what this means, these articles are structured around the seven principles of Lean Software Development. This month includes a short introduction to Lean, and you’ll see how an expressive language can eliminate waste in unit testing. Next month you’ll see how the easyb Behavior Driven Development framework builds quality into your software development life-cycle. Later we’ll explore how different testing tools like the Spock Framework help create knowledge, how a dynamic language lets you minimize coupling and defer commitment, and how metaprogamming allows you to deliver your software fast. In future installments you’ll see how Groovy’s Grapes module system, alongside modules like Groovy Web Services, streamline interactions between your development team and other groups like Operations and QA. Finally, we’ll look at how Groovy can be applied to domains that cross departmental boundaries: how Gradle improves the build process and easyb facilitates collaboration and communication between team members and other stakeholders. The series will end with a discussion of best practices when mixing Java and Groovy

Some of the code examples are basic while others are advanced. This series explores why the features of Groovy are important rather than the mechanics of any one Groovy feature. Hopefully you’ll leave with some new ideas about how to use Groovy, how to convince your team that Groovy is worthwhile, or most importantly how to increase your productivity.

Lean Briefly Explained

Lean manufacturing grew out of the influential Toyota Production System, which was a set of both philosophies and practices designed to weed out waste, inconsistency, and overburden from a manufacturing process. While the Toyota brand recently took a beating for some public quality failures, it is hard to ignore several decades of successful engineering sensitive product development. In the late 1990′s, Mary and Tom Poppendieck took their knowledge of Lean and applied it to a software project, eventually resulting in 2003 with the publishing of the excellent book “Lean Software Development”. The Lean principles have been revised over time; for our purposes we will use the seven principles from “Implementing Lean”:

1. Eliminate Waste
2. Build Quality In
3. Create Knowledge
4. Defer Commitment
5. Deliver Fast
6. Respect People
7. Optimize the Whole

Lean has been an influential movement within the agile community, spawning a plethora of books, workshops, and articles. Besides being a meaningful set of practices in its own right, it has also directly influenced other agile family Members such as Kanban Software and Real Options. Lean is a set of principles first and a group of practices second. The principles are specific enough to align a team but broad enough to appeal across an organization. I once spent a three-hour car ride discussing eliminating waste with the senior manager and director of my development group, and then arrived at work to help an eager development team make practical changes to reach the goals we had discussed. Compare that to a methodology with principles of “Honesty” and “Communication”. I have
yet to find a use for vague principles beyond their ability to make me feel a little better about a crummy process.

Principle 1 – Eliminate Waste

Taken literally, it is easy to see how Groovy fulfills the Lean mandate to eliminate waste: list and map literals, a terse closure syntax, and even a lack of semi-colons means there is simply less keyboard typing to do for Groovy developers. But that isn’t what Lean means by eliminating waste. The real waste within the software process is in half done work. Work that has been started but not delivered eats up the time investment from developers but does not provide value to the client. Any work that has been started but not completed is a form of waste, and we must optimize our life-cycle to keep our user stories out of this state! This means moving stories to Done faster, and any good agilista will tell you Done isn’t done until the unit and acceptance tests pass. Having used Groovy as a testing language on several projects in the last few years, my team’s experience is that the same test coverage and quality can be reached with Groovy in less time than it takes using Java.

Good unit testing means testing not just the success scenarios, but also the failure scenarios and the edge cases where things just barely pass or just barely fail (bugs do tend to congregate in corners, after all). This means your unit tests have to create data, and lots of it. Much more data than your production code needs to create. And Groovy is a language better optimized to create data than Java. Consider the data driven test in Listing 1.

public void testSimpleMath() {
  [   4: { 2 + 2}, 
    6: { 2 + 2 + 2}, 
    8: { 2 + 2 + 2 + 2}, 
  ].each { expected, actual ->
    assert expected == actual()
  }
}

The example shows a few ways in which Groovy is different from Java. Should read: First, a Map is declared using the notation [key1: value1, key2: value2, ... ]. So in the above example there is a map of type Map<Integer, Closure> with three entries, which sure beats having to use new HashMap().

The values of the map are Closures, which are function objects. The closure can be executed and passed arguments just like formal methods. They can also be passed around to other methods, returned from methods, and assigned to fields. Very, very powerful. Think of them as Runnable or Callable objects without the awful syntax.

Finally, the each method is just a method on Collections and Maps that iterates over the collection, executing the attached closure once for every element. You can see how this closure runs a simple assertion, asserting that the integer key of the map equals the evaluated value of the map.

With a minimum of non-essentials, this test creates a dataset to be used as test input and then executes the test against that input; it is a data driver test, without a framework, without needing to hide any of the mechanics of the test, and without a bunch of annotations and factory methods. Data driven tests are so easy to write in Groovy that you end up using them far more often than in Java.

Listing 2 is another data driven test with a more useful assertion than simple arithmetic:

public void testUserService(userService) {
  [   user1 :  ["John", "Doe",   123456], 
    user2 :  ["Jane", "Doe",   654321], 
    user3 :  ["John", "Smith", 789987], 
  ]. each { username, userData ->
    def user = userService.get(username)
    assert user.firstName == userData[0]
    assert user.lastName == userData[1]
    assert user.id == userData[2]
  }
}

This time the data is a map of Strings to a list of values (within Maps, unquoted key value are treated as Strings, so user1 and user2 are Strings). Java purists will notice that the generics of this expression are not easily expressed in Java, while they can be ignored in a dynamic language like Groovy. In this example the assertion block is simply pulling elements out of the list using the subscript operator [0] and [1] instead of the List#get(int) syntax.

The point is that the Groovy language features (list and map literals, closures, good iteration functions) allow you to write data driven tests more quickly, which in turn means you cover more edge cases and more failure scenarios as well. The minimal syntax of Groovy allows you to get down to the essence of a unit test: clearly show the input, execute the system, and clearly assert the output. Sure, you can Extract Method in Java until you have something with minimal accidental complexity (maybe even “extracting�until you drop”), but as Alan Shalloway says, “There is a big difference between eliminating waste and not creating it in the first place.” With Groovy you will write tests faster and you will have your story cards spend less time in the “in process” phase. This is good for your project and good for your customers.

* “Extract Until You Drop” is a blog post from the inimitable Uncle Bob Martin about taking Extract Method refactorings to their logical, if absurd, conclusion. The piece could just have easily been titled, “Writing Clean Code in Java is a Complete Hassle”.

What happened to assertEquals?

Sure, using Groovy’s assert makes your tests more terse, but more importantly check out the exception message on a failing assertion in Listing 3.

def one = "1"
def two = "2"
assert one + one == two

Treating addition as String concatenation is endearing when your four-year-old is learning arithmetic but is an error in most mathematical systems. Listing 3 is interesting because of the exception message: each element of the expression is clearly expressed in the monospaced message: all three variable references as well as both the operators (+ and ==). Lower ceremony, clearer tests, easier debugging.

Next Steps

One particularly useful Lean practice for eliminating waste is Value Stream Mapping. Sit down at a whiteboard with your development team and write out your software process: the steps and time it takes to move an idea from inception to customer delivery. Where are the bottlenecks? What are the delays? Why does it take so long? Visualizing and quantifying your process helps everyone clearly see problem areas. And who knows, maybe a few Groovy scripts here and there can cut days out of your cycle. Stranger things have happened.

Next Month: Build Quality In

A lean, stream-lined development cycle with minimal waste is nothing if quality suffers. Next month’s article explores how Groovy helps you build quality into your development life-cycle. Instead of inspecting your product for defects after it is built, change your process to not create the defects in the first place. Easier said then done. Luckily, Groovy programmers can use great tools and techniques like easyb, Behavior Driven Development, and builders to build quality into their products.

Learn More

The world is overflowing with books and articles about agile development. These three books are classics and deserve a permanent spot on the bookshelves of discerning programmers.

• Implementing Lean Software Development – Poppendieck
• The Pragmatic Programmer – Hunt and Thomas
• Extreme Programming Explained, 2nd Edition – Beck and Anders

The newest issue of GroovyMag is hot off the presses, and it contains the first of seven articles on Lean Software and Groovy.

The title is “Slimmed Down Software – A Lean, Groovy Approach Part 1 – Eliminate Waste”, and from the article:

The Groovy Programming Language advertises itself as an “agile and dynamic language for the JVM”, but what does this mean exactly? This series of articles explains Lean Software Development, and shows how your choice of programming language can make your entire process remain nimble and adaptive. Each month will cover one of the seven Lean Software Development principles and explain how Groovy and the associated ecosystem help eliminate waste, defer commitment, and build quality into your product.

Enjoy!

Groovy Mag April 2010