8. A New War on Waterfall

"I'm concerned, because that looks very much like a waterfall!" a coworker of mine quipped. "We need to communicate across disciplines, and some times things discovered while implementing the solution might affect the final design of the product" he continued. The product manager had just finished outlining a sketch of a process he wanted us to follow when bringing new products or new versions of products to the market. My colleague (a very competent one, I might add) commented on what he perceived to be communicated as a linear step by step process. As he spoke, I felt an agonizing pain somewhere in my soul. As I've come to do ever so often lately.

The plan our product manager had outlined, contained steps of a data-driven process for product development. Where we every step of the way, from first concepts, then minimum value products (MVPs) and prototypes, could validate the product at different stages by interacting with the audience even before implementation and deployment. What my colleague failed to see was that the PM was picturing a "main-flow" - not a prescriptive "do this, then that, then finnish with this" type of plan. Colleague reacted as if saying that we want to do user studies and prototypes before we start working on the real thing, implies that lessons learned while implementing the product, or data from the live product, can not affect the direction the product takes. Which was obviously not what the PM had in mind. My colleague is not alone in making this mistake, though. An entire industry have made the same error for decades.

Even the person defining the waterfall process was clearly aware that any ventures in to software development would require feedback from parts lower "down" to various parts higher "up". The "waterfall process" is usually credited Winston W. Royce, although he did not use the term in the original paper. The figure he used to outline his basic process, or workflow, however, does indeed resemble a waterfall, hence the name. The waterfall is a flow of "steps" from System Requirements through analysis, design, coding and testing to deployment. The term Waterfall then, have come to mean a process where every activity in that list is performed to completion in isolation, where the result is handed over to the next box for processing. Below is the original canonical picture of the waterflow process.

From reading the paper of Dr. Royce, however, it becomes quite clear that this inflexible process is definitely not what he intended to communicate in his paper. On the general overview of the process he states, and I quote: "I believe in this concept, but the implementation described above is risky and invites failure." The paper is full of examples showing how, if applied blindly, such a "step-by-step" process might fail. He also state quite clearly, translated to more modern terminology, that it is usual to have iterations spanning multiple phases. And there is also modern sounding gold in the paper, like "Involve the customer - the involvement should be formal, in-depth, and continuing" - still a hallmark of good software development if you ask me! So the first paper that defined the Waterfall process, is actually the first loud critic of it. It actually talks about something that to a high degree resembles something closer to a modern agile iterative methodology. Unfortunately after Royce's paper, and contrary to his intentions, the term waterfall is cemented in to our minds to mean an inflexible step-by-step process that are doomed to fail due to lack of feedback loops. It is wrongfully used as a contrast by whoever have some new snake-oil a new process to sell. But that inflexible step-by-step process never existed.

This is the core of my pain: Most, if not all, people I meet in the software industry are (somewhat) grownup persons who are well aware that succeeding in software development require communication and feedback, and that things we learn underway almost invariably change what we end up building. Heck, it's those who are fresh out of school who tend to yell out the loudest about the need for feedback and communication. Do we forget this when we get older or something?

Could we please then, even if somebody for the sake of communication paints a simplified picture of a bunch of boxes with arrows between them take for granted that there are feedback-loops? And do that even if, taken literally, it resembles a waterfall? I'm not saying I believe that those feedback-loops will be popping up all by them selves. They must be nurtured and cared for as any other important aspect of the development process. But sketching a general flow from idea to finished product does not exclude it's existence!

It seems to me like yelling "waterfall!" has become the favorite guilt-by-association argument. Some feat, for a process that never really existed anyway! So then. Hereby, I declare war on the term waterfall it self. 

7. Test behavior, not methods

I recently revieved some code from one of my colleagues. A birds-eye view of his class looked like the following. There was nothing wrong with it - it looked perfectly fine to me.

// Foo.java
class Foo {
  public void Foo(Bar bar) {...}
  public void methodOne(String str){...}
  public String methodTwo(){...}
  public Integer methodFour(Boolean b){...}
}

However, I read the test for this class first, FooTest. Even though it looked like it tested what it should test, I had some immediate comments after just a quick glance in the file.

class FooTest {
  @Test public void testMethodOne(){
    ...code...
    assert...
    ..reusing and modifying state..
    assert...
    ...more code...
    assert..
  }
  @Test public void testMethodTwo(){..more of the same..}
  @Test public void testMethodThree(){..more of the same..}

As many others, he had written one test for every method in the system under test (SUT). The methods was long. State bled through from the start of the test-method to the end of it. Variables were reused and their state modified as the test went along. I usually prefer to test behaviors, not methods, and I felt being on safe grounds when I suggested for him to rearrange his test so that he had one test per behavior.

He thought what I said was a ridiculous idea. As an example he pointed out one of my own tests that follows this pattern and looked something like the following.

class FooTest {
  @Test public void methodOne_leaves_foo_with_clothes_on(){...}
  @Test public void methodTwo_calls_mom(){...}
  @Test public void methodTwo_writes_result_in_book(){...}
  @Test public void methodOne_after_methodTwo_lights_the_firecrackers(){...}
  ...more tests...

"I hear what you say, but I think it is ridiculous. What do you need all those damned tests for? You have twice the amount of code in your tests than what you have in the classes you write. Plus. You are violating our coding standard with those underscores! Think about maintainability, man! And after all, this is just the test-code. It doesn't need that much focus." Awh the pain! I managed to resist the temptation to slap the offending party in the face with a fresh cod I usually carry around for that exact purpose. Instead I resorted to a reasoned debate and conversation.

In some respects he's right. It becomes more code. Much more code. But I've come to realize that organizing tests by methods does not tell me anything about the behavior of the SUT. In order to understand the behavior of Foo from above, I need to read all the code in the tests. I probably need to read all code in Foo as well. And from that I might be able to deduce what it does or what it's supposed to do. But even if I get thus far, it's still very difficult for me to understand what aspects of that behavior actually have a test and which one have not.

One test per behavior, allows a quick glance of the behavior of the class or subsystem just by collapsing the body of the test-methods. The naming of the tests can read as a story. IMO the layout and cleanlyness of the test-code is more important than the production code it self. If we need more words to communicate, write more words - communicating the intents clearly is more valuable than writing less code.

As Einstein said, paraphrased, make it as simple as possible, but no simpler.

6. Efficient TDD: Start closer to the end user

Nobody gets it right every time. Especially not software design, where... well... everybody gets it wrong most of the time. Even when doing Test Driven Development (TDD). I remember a series of blogposts by Ron Jeffries where he tried to implement a Sudoku-solver using TDD and absolutely failed to solve the problem. If you read the posts from Ron Jeffries, notice how he jumps straight in and attack an internal design detail before even understanding the problem. He basically gets stuck there and end up without a solution.

I remember having had design-sessions that looks eerily similar. After listening to the product owner for two minutes outlining what's needed, we went on to discuss lots of internal design details. "We would need a Frobinizer." "Also, we need a FlexiPlug" "The Frobinizer and FlexiPlug cant talk to eachother directly - they need to post and read messages via the PlancCampfer." "But the semantics of a Planc dictates that it should not share component with the Camfer, they should be separated." At more than one situation, I've found my self preoccupied with the software components to a very high degree, and to a lesser degree about how it actually implements the features we wanted. And I do believe that the times I've ended up with a huge outburst of frustration and then scrapping everything and starting over, is strongly correlated with the times I've become to preoccupied with the internals of what I'm making.

The topic for this post is a strategy to approach design and implementation in a way that I experience steer me away from the problems Mr. Jeffries experienced. The key is to start with a test that is as close to the user as possible.

All applications originates with some user or some agent that interact with the application (not to much of a generalization, I presume.) If I write a GUI application (I've blogged about something slightly relevant previously)  I'm probably using a GUI framework, such as Qt, wxWidgets, GTK or Swing, so the actual view I'm putting together is composed of components from the framework. I'd like to avoid testing these framework-provided components. But the user clicks a button with the mouse, or he type some text in a text view, or he exercise a key-combo. I'd like to start as close to the user as possible, and for a GUI application this is where the user action leave the framework code - usually this involves implementing some kind of event handler. So the first test is for the event-handler that originates with the user interacting (and it's immediate surroundings.)

For another example, we can look at a web-service. When adding something new I like to start with writing a test for the network access-point. The test basically checks that the web-endpoint manages to get responses, and react appropriately to them. The endpoint is tested by relaying representations to an interface. The diagram below illustrates the  basic design of such a test.

I start by writing the WebServiceEndPointTest and the WebServiceEndPoint, create a interface that represent the real behavior, and translates calls to WebServiceEndPoint to something ServiceInterface can relate to (e.g. method calls and values.) The test then sets up the mock for the ServiceInterface, exercises the WebServiceEndPoint and validates that the interactions with the mock was correct. There is nothing revolutionary with this idea of course - the main point of this blogpost is that this is the right end to start working in. Get the user interactions defined first, before the design of the rest of the application get so set in stone that it can dictate how the user interaction can be.

You may view this as performing top-down design, as opposed to doing things bottom up. And you would be right in pointing out that this leads to a design that pretty much follows the Dependency Inversion Principle (DIP.) By following DIP, we get a design where "lover level" components and packages depend on abstractions exposed by "higher level" components, and that everything relates to each other via those abstractions. This in turn leads to a nicely decoupled and inherently testable codebase. The figure below is a caricature of the dependency inversion principle. If you let Foo in the picture below play the same role as WebServiceEndPoint above, notice how higher level components don't depend on lower level components, but that lower level components depends on abstractions from higher level components (higher translates to closer to the end user.)

It also turns out that the result of following this advice, becomes very similar to behavior driven development (BDD), without the domain language for specifying the tests. At the beginning I get an outer loop, where I write tests that closely capture the behavior and the interactions of the end user. These tests are also suitable for testing the overall behavior of the system, and can be 'wired' as integration tests (in addition to working as unit-tests for the outermost units.) I also get an inner loop, where I can refine and refactor the designs and tests of individual units that build the application. So from following a very simple advice - starting as close to the user as possible - we've discovered the dependency inversion principle and we've been doing BDD. Not bad for an afternoons work!

So, how would I do the sudoku thing then? When I solved sudoku by means of TDD, I started with a textual representation of a completed sudoku-board and wrote a test that verified that my solver recognized it as complete and valid. After that I progressed with leaving one, then two, then more fields blank. After a handful of tests, I had a solution that was not to dissimilar from this solution, based on search and constraint propagation. I felt satisfied with the solution, and I never needed any tests for the internal data structures. Starting closer to the user guided me in a good direction.

5. Ruthless with email

I'm using David Allens system for personal productivity called Getting Things Done. The system states some overarching principles, such as Inbox Zero - reduce the content in all inboxes to zero elements at least once every day, and a process in order to reach that state.

Roughly stated, the process for emptying all inboxes is as follows: For each item in the box, ask the question "Is it Actionable?" If yes, formulate a next action and place this in a system to keep track of it. If no, figure out if it's trash, if it's something you need for later, or something you want to keep for reference. Sounds like simplicity itself, right? See the picture below for details. There is one big problem with the application of this procedure. The first question - "Is it Actionable?" - generate too many false positives, and there are too many effing emails in this world.  If you open an email, and ask your self "Is this Actionable?" your brain will happily translate that to "Is there something I could possibly do with this thing?" and then happily continue to answer "Well, dude! Yeah, sure! Here's some random thing you could do!" "This commercial from the electronics store... You know, you should investigate a new stereo!" Bam. New useless project.

In the email-account I use for work, I get ~70 mails I need to relate to in some way or another pr. week. That's... 14 mails a day. There are spikes of perhaps 30 mails in one day. The actual number of mails that hits my mailbox is higher - maybe three times that number - but two thirds get sorted to designated folders I seldomly need to look at. I'm not on any mailing lists except those relevant at work. So 70 - that's the number of mails I need to open, read, and figure out if and how to act on. In addition I get ~30 mails a week for my private account. I don't know if 100 emails a week is a lot. It feels like a lot at times. And some times I get a bit on my heels, and they pile up. So how do I cope?

The solution to this email-problem is found in the parts of GTD that have not gained the same amount of attention as the mechanics around processing. Allen talk about six different horizons of focus:

1. Next Actions
2. Projects
3. Areas of focus and responsibility
4. Goals and intentions - things that should manifest in life in two to three years
5. Vision - how should life be
6. Purpose - principles, ideals and why I ultimately are here.

Most tools only handle 1, 2 and perhaps they pay lip-service to level 3. These are the high throughput artefacts. The things that should have turnaround-times in at most a few weeks. They pose interesting engineering challenges. They are easier to talk about. "Hey, what system do you use to track your next actions?" But in order to fix that mail problem, I find I need to play on more levels.

A more representational thought-procedure for my part might be something like the following - if I really spell it out.

Is it somewhat probable that this item does not relate to my current projects, my defined areas of focus or my goals?

If the answer is yes, I chuck it in the bin right away. This is not revolutionary, by the way. Judging how meaningful a piece of "stuff" is to you by aligning it with your areas, goals and vision is exactly the point of the GTD-system. This fact just seems to drown in the tool-mayhem.

I can intuitively answer that question quite fast. Often even without opening the mail. If I hesitate a little bit with deciding yes or no, I archive it instead of deleting it. Minor safety valve. Sure, I loose stuff. I quite possibly miss opportunities. I feel a bit like a ruthless bully when I munch through 20 mails in five minutes, and only two of them survived the ordeal. But I empty my inbox and feel good about it.

Didn't I get back to you on the email you sent? I guess you didn't align with my grand plan in life. I'm sorry. But my inbox is clean!

4. Group by function, not kind

Today I'd like to write a little bit about physical design. Physical design is all about placing the needed behavior in the right place, and not so much about objects and how they interact - a concept I first read about in "Large Scale C++ Software Design" by John Lakos. The functionality needed must reside somewhere, but it's not irrelevant where it ends up. Good physical design guidelines can be crucial for the long term success of a project.

Here's a very simplified abstract representation of the source-layout in a project I worked with a couple of years back:

myapp/dialogues 
myapp/shapes 
myapp/views 
myapp/caches 
myapp/preferences

For each new feature we added to the system, we wanted to add one or more shapes, one or more views, one or more caches, one or more preferences relating to the feature, and perhaps a handful of dialogues as well. So what was wrong with this?

It doesn't scale well with the size of the codebase. As long as we had a handful of features, this was ok. But the application grew fast - we were adding new features, and every new feature sliced through most if not all of the directories. In addition the team grew, and we collided with each others works more often. Build-times increased to an uncomfortable level (this was C++). At a point we wanted to separate things in libraries and we realized that we had a huge problem. Shapes depended on preferences, preferences depended on views, and views depended on shapes - all directories had dependencies to most of the other directories. Circular dependency galore. An entangled mess we spent a fair amount of time to try to tease apart.

The solution that materialized, was to group our files according to new features. So over time the directory structure morphed in to something that looked more like this:

myapp/jump/dialogues 
myapp/jump/shapes 
myapp/jump/views 
myapp/jump/caches 
myapp/jump/preferences 
myapp/slide/[dialogues, shapes, views, caches, preferences]
myapp/talk/[dialogues, shapes, views, caches, preferences] 
myapp/common/

...where the actions - verbs - represents different features or groups of somewhat tightly related functionality - features. When we started organizing the physical design of the application this way - by the feature rather than the type of the different software entities - we experienced several immediate benefits. If I worked with the jump-feature, for example, I would be pretty certain that my changes did not break anything relating to the slide-feature. This organization also made it easier to remove duplications by discovering and salvaging common functionality, and add that to separate utility libraries. In short, this physical design inspired loosely coupled designs. On the other hand, the original (and maybe more intuitive) physical design led us to make tightly coupled logical designs.

I use this as a rule of thumb. It's obviously not always true, but true often enough to make notice. If I fail to follow it, I notice that the natural evolution of general utilities that helps me continue to keep speed up while I'm adding new features, fails to manifest. I also experience that circular dependencies between packages is more prone to pop up. If I manage to group by function first, I find that keeping the codebase decoupled, clean and non-cyclic is much easier.

3. Talk about execution speed, not quality

I remember sitting in an all-day company meeting once. Everybody was there, and it was supposed to be sort of a pep-talk thing to get all of us back on track. It was important, as we had grown by aquicition a couple of years, business was not the best and there were some bad moods here and there (to put it mildly). We had a newly appointed CTO, a jolly nice fellow I liked a lot on a personal level. He was not from a software background, but had more of a domain background (this was in the oil and gas industry.) He stood up to hold sort of a speech, and he opened with a phrase that didn't sound to different from "Quality costs more, we must tone down our quality focus."

First of all - I'm a fan of W. Edwards Deming, and I believe that with a quality mindset, you produce better "stuff" for cheaper - whatever "stuff" happens to be. So how could this new CTO fellow stand up and say something that seemed to contradict that? I was furious. Who hired this idiot? Of course, nothing else from what he said stuck to my mind that day.

When I left the meeting, however, I remembered a lecture (or was it blog-post) by Kent Beck, where he discussed the notion of external and internal quality. I realized that what our CTO had talked about and what I reacted to was those two somewhat different and orthogonal entities.

If you have not heard about this before: External quality is relating to the end user observables. What features do we have, how complete are they, how do these features feel and how many bugs are there. Internal quality is almost entirely orthogonal to this (of course - they aren't entirely orthogonal, bugs are external quality observables that obviously relate to the internal quality of the software but humor me,) and is all about how the software behaves and feels when the software developers work on it. A low internal quality is often referred to as "Technical Debt" - a phrase coined by Ward Cunningham. Examples includes bad design choices, duplicated code and low test coverage. The theory goes, that if the internal quality is raised, new features can be added faster.

The next time I met said CTO, we had a chat about this and he concurred. He had talked about the size and scope of our deliverables and the amount of time we spent on polishing them. He meant we delivered more than what the customer payed for, or what the customer wanted to pay for. And he could absolutely understand the need to do some house cleaning in order to be able to deliver efficiently.

All in all this is boils down to a variant of a challenge we often experience as software developers. The stakeholders want more features they can see. The software developers want to clean up the bit-rot they're creating, caused by miscellaneous pressures to deliver features the stakeholders can see as fast as possible. I somewhat learned a lesson, and I've stopped talking about quality when communicating to stakeholders about needed cleanups. Instead I try to frame any discussion about when to do that cleanup in terms of how fast we can add other visible features. Further I try very hard to talk about all end-user observable external quality as scope stakeholders can rationally triage against estimates, established velocity and the calendar.

This perspective have made my life somewhat easier at least once. :)

2. A TDD Crux: Talking to others

I've been a fan of test-driven development (TDD) since I first learned how to do it my self, and at various times during the last ten or so years I've been asked questions that pretty much sounds like "How do you test THAT?" while the asker points to some source-code. The source-code being pointed to has almost always been a tightly coupled mess without tests. And of course it's nigh impossible to test THAT without investing a significant amount of work in both refactoring and a huge scaffolding.

The Bowling kata, for example, is a well known exercise everybody who learn TDD run in to. It is simplicity itself: An internal datamodel, with simple and well defined behavior. It's "Perfekt for TDD" (If you haven't done the bowling kata yet, check it out here.) It crosses over to the "How do you test THAT?"-realm in the second it sends a signal to an external system in order to play a video of a dancing turtle every time you roll a strike. If not doing TDD, some might probably just pass the "PlayAnimation" subsystem to the BowlingGame class, and call the "celebrateStrike" method every time the player roll a strike. And voila, it is evolving in to a tightly coupled unmaintainable mess. I'm thinking of this as the first stumbling-block, the first mental challenge, when learning TDD: How to test behavior that relates to other behavior not contained in the object being implemented.

I won't dive in to mocks, stubs, fakes and the lot - there's been plenty written about that already and this is by and large a well understood problem. Still, in order to hopefully make a condensate of how to get over that hump, I do want to describe a general pattern of thought that I often apply in such scenarios, outlined in the figure below:

A less direct path

The object we are aiming to develop with TDD is Foo, and it needs to call methods on some object Bar that is not under our direct control. Instead of interacting directly with Bar, Foo interacts with it's surroundings through FooActions. Foo gets a reference to FooActions via the constructor during application setup. FooActionsBridge is a very simple implementation of something that interact directly with the Bar system, preferably only relaying method calls without actually containing any other behavior so it can be checked for correctness through quick inspection. Notice also that Foo and FooActions is bundled together in the same package. FooActionsBridge is a part of the setup of the application - so quite possibly in a third package. There is of course one obvious simplification here. If Bar is under our control,  Bar can implement FooActions directly, eliminating the need for the FooActionsBridge. In this case I sometimes like to add the FooActionsBridge first while getting Foo sorted out, then change Bar afterwards. That might be overly cautious on my part though.

So - does this help in getting over the first crux while learning TDD? Please let me know!

1. Blogging every day

I want to form a habit of writing about software development and personal productivity. It's on my list of long term intentions, and I have blogged sporadically in the past. Small blips every now and then, and I hate to admit that publishing a blog-post have been much work to some extent. I've been baking on them, reading, rereading, re-rereading and moving paragraphs and - Egad! They're still not perfect! I do the same thing with emails btw. I'm fretting over the content way longer than I should. I think the proof is in the pudding, that's an unproductive approach. 

During the last week I've repeatedly thought to my self that if I, at some point in time, set out to blog something about what I'd like to write about, every day for, say, a month, that would be an attainable well formed goal that would help me form that habit.

I'm not one for new year resolutions. I don't believe they work. Never did. I believe, at least, that I have well thought ought reasons not to do so. The main reason is that it, in some way, represent postponing important things to some day in the future on the cost of more urgent matters. "Yes, I'd make it a new year resolution to quit smoking." - but not right now. I tried to put the idea in my Incubate-folder first. But the third time it showed up I decided that now was the time to do it. So - this is number one. Thirty to go. And I haven't even figured out what tomorrows topic will be yet. Kind of scary.

And this is the main topic of this exact post: I want to establish a new behavior. So I set out to exercise the habit I want to establish in a somewhat extreme manner. In a month, I'm changed. If I follow through with it. I've read about this somewhere. Not sure from where, but it's not my idea. I still think it's good. So I'm doing it anyway. Let's see if I manage to follow through, all the way.

Let me know if you manage to find a positive adjective to describe my writing. Until tomorrow, then.