Top-down vs. bottom-up design

I've been having a think about top-down (a.k.a. outside-in) design during my recent iterative development exercise. In the series I've been putting off testing from the client layer down, primarily because GUIs have a reputation for being hard to test and harder to test-drive, and I wanted to make some early, easy progress on the core logic of the game.

I started to think that this approach might be a mistake. I'm dealing with the model that I think we'll need, not one demanded from the primary client of the model -- the GUI. Chad and Ben mentioned in their recent screencast that bottom-up implementation tended to lead to mistaken assumptions about infrastructure required by the top layers. I saw a similar point made on the BDD mailing list by Pat Maddox. Pat wrote (emphasis mine):

"I find an outside-in style of development to be very helpful... It forces you to think of your objects at a high level, so your design is driven by real need, and then you apply your design skills as you go on. When I use a pure bottom-up style, I write more speculative code and go down the wrong path far more often than I'd like. That's not to say that it's a problem inherent with that style, but rather a problem that I've personally experienced, and have more or less solved by using an outside-in approach."

This is opinion is echoed in an unrelated post to the TDD mailing list by Olof Bjarnason:

"I've been using TDD [bottom-up] for 2 years now, and it's been mostly a _great_ experience. The thing that bothers me most with "classic TDD" is that sometimes I build too much functionality in my classes, which isn't used in the end application after all. Even whole objects are wasted in the worst case."

The view here is that bottom-up design can lead to speculation and waste. By having a design driven directly by the overall, required behaviour, you only implement (and test) things that directly serve that behaviour. This can help eliminate speculative implementations of lower-level behaviour based on what you think the overall required behaviour will be.

Not so fast...

Sounds great! So what about inside-out / bottom-up / middle-out design? Ron Jeffries recently stated on the TDD mailing list that he generally prefers to start with the model (unless the project is simply to build a viewer). Maybe there's a bit more to it?

Digging further into that thread on the TDD list, there are a number of great points of view on the topic. Some TDD-ists argue that bottom-up design lets you build in small, easy steps, and refactor your way to the required behaviour that you would otherwise start with in top-down design. Others state that this leads to waste -- writing code and tests that just get refactored away. Which resulted in a couple of great quotes on the difference between refactored code and waste:

"I suppose we could also call the scaffold we use when constructing a large building as waste, or the safety harnesses as waste." -- John Roth to TDD list

"The analogy with scaffolding for a house is an excellent one - there is a lot of "stuff" constructed when building a house, *just* to support the construction - it is then discarded." -- Casey Charlton to TDD list

Top-down design can also lead to a "mockist" approach to TDD, where you need to mock all the required dependencies to implement the high level behaviour. This isn't necessarily a bad thing, but over-reliance on mocking can result in fragile tests. Martin Fowler has a great article on the pros and cons of "classic" and "mockist" TDD.

Enough rambling already!

While planning for part 3 of my recent development exercise I was coming to the conclusion that top-down was the way to go. After looking into it some more I was reminded of a whole host of advantages of bottom-up design. Even more importantly it reminded me that there is no silver bullet, and there are times when either, or a mix of both, approaches are fine. All this started to sound familiar, so firing up Google I noticed that I had read something to this effect in Jeremy Miller's excellent (as usual) post on the topic (search for "Bottom Up versus Top Down", although the whole post is worth reading).

I think the most important conclusion I've reached during this ramble is that if you are working in iterations to deliver a complete slice of the application (top and bottom) then you're never going to go too far wrong. Any "waste" from a bottom-up approach will be minimal as you'll be working toward and implementing the top almost immediately. And you'll still end up with higher-level behaviour specified with unit tests. Likewise starting top-down you'll still get the advantages of designing in small steps, particularly as you drive down into the design.

BPM tools vs. developers

Was going to send email this to a former colleague that shared some negative BPM tool experiences, but I think it raises some interesting points that are also applicable to RAD tools in general.

Frank Sommers uncovers some very interesting views in this post (both in posts it links to, and in the comments) on why many developers rail against BPM tools:

Java Community News - Are BPM Tools Useful to Developers?

Two schools of thought seem to be that developers don't like BPM because of the threat (perceived or real) they pose to their livelihood (which I believe is true at least to some degree), or that they don't like it because it actually gets in the way of doing things efficiently (which is my main complaint).

One of the good quotes in the article is this one from Robert Cooper:

"These products are unworkable because they are based on the idea that “You won’t need programmers anymore!” at least at a core level. Once you make that assumption you start building things that get in programmers way, and still include enough abstract programming concepts that no non-programmer is ever going to be able to work with it proficiently."

Another good one is in a comment from James Watson, where he states "This idea that developers don't want to use drag and drop tools because it's boring is a load of crap... Most programmers are busy and if a tool can save them time and headaches they will use it". He then goes on to list the typical cycle of how these things are introduced, which would be hysterically funny if it wasn't so true. I have quoted the list here as there is no permalink to the comment:

1. The salespeople tell some suckers that they won't need developers anymore. Many lies are told.
2. The company bites. They might even layoff some developers.
3. Non-developers try to create their own programs with 'simple' drag and drop tools.
4. Complete meltdown ensues when the first bump brings the house-of-cards crashing down.
5. The developers are now told to use this tool to do their work and maintain the steaming pile that has been built. (a.k.a throwing good money after bad.)
6. The developers try to maintain the mess that the non-developers wrote and create maintainable software without the tools that developers normally use (because they work.)
7. Productivity and/or quality continue to suffer. Developer morale drops. Much more 'code' is created than would be needed in conventional development approaches.
8. "Experts" in the tool are contracted and/or hired at great expense. They teach the developers how to work around all of the product's deficiencies in order to be moderately productive.
9. The rising expenses cause managers to look for ways to reduce development staff. Outsourcing may be used.
10. A vendor comes in to tell some suckers that developers won't be needed with their new drag and drop tool...

Comment roundup: SharePoint as a development platform

Looks like a couple of interesting posts on using SharePoint / WSS / MOSS as a development platform. I was originally going to send this as an email to a former colleague (hi mate!) but thought I would post it instead because I find the debate quite interesting. Remember, the discussion is about SharePoint as a development platform, not as a CMS or whatever else.

I first found Ayende's post on the topic, which is a response to Sahil Malik's response to Jeffrey Pallermo's post on the SharePoint development platform :)

Jeffrey's post raises the (IMHO) valid issue of the friction of using the SharePoint platform for development, particularly for deployment and builds. Unfortunately his point gets a bit diluted by focusing on the non-issue of being unable to install SharePoint on Vista/XP.

I think one of the stars of the comment trail on Jeffrey's post is Andrew Connell (MOSS MVP, and pretty decent bloke if his blog, comments and presentation at the APAC SharePoint 2007 conference in Sydney are anything to go by), who has added some very constructive comments pointing out the strengths of the platform lie in the out-of-the-box navigation, security, search, Web Part framework, WF hosting etc, as well as acknowledging several weaknesses. One of the things I found really interesting (although not too surprising) is this part of Andrew's comment:

"Over hyped? Really... so that's why it's the highest growth product ever in the history of the company and closing in on $1B in sales for the latest version in just a few months from now? That's ALL marketing hype with the other portal solutions and other content management applications out there? Um... I think business decision makers are a bit more savy than that."

For years now I have had the strong suspicion that people that look to things like OpenOffice.org as potential Office-killers are missing the main point of Microsoft's Office strategy. I'm sure they are not too fussed about losing a few home users -- they are targeting the bigger end of town with volume licencing. The integration, no, dependency (they are part of the same platform after all) between Office 2007 on the client and MOSS 2007 on the server is where the action is. The complete platform is going to guarantee them a nice revenue stream from medium, big and giant businesses for as long as they can keep pumping up version numbers.

As a further aside, Andrew also writes a  "Not every application should be built on SharePoint... no one is advocating that (no reasonable person that is)." Marcus makes a similar observation:

"I'm not sure you realize how many people right now are investing in SharePoint thinking that it is a "no-code" solution to replacing custom ASP.Net apps, or in some way making managability of those easier. ... It doesn't do that.  Its a content management system (and a good one at that)."

(Sorry, gentle dig... bygones! ;) :P)

The main complaints in the comments revolve around the poor tool support (agreed by pretty much everyone), the poor / barely existent documentation (ditto), and testability, deployment and configuration (over which there is much disagreement). The latter issue brings us back to one of Jeffrey's original points: the friction of using the platform.

I have personally found SharePoint development (as a MOSS newbie anyway) to be very high friction. The minute you want to go "out-of-the-box" at all you run into all sorts of issues. Even talking to the experts at the recent APAC SharePoint 2007 conference in Sydney exposed so many work-arounds and compromises just to comply with the platform. The session on packaging and deploying custom applications (run by a bloke who really knew his stuff) was filled with advice like "this bit doesn't work properly, so you need to manually edit this XML, you can copy it from another file in the 12 folder, then adjust the package names and then...".

Andrew suggests that this is simply a cost of using any platform:

"So I want to build add-ins for Visual Studio. Is it easy? Nope... because I need this and that and have all this work to do just to get my environment ready to build a plugin.

To me the issue isn't with SharePoint, it is jumping into working within and integrating with platform unlike pure ASP2 development where you building from the ground up."

I am unconvinced. Yes, I agree there will be a cost associated with using and complying with any platform, but a platform really exists to make your life easier. SharePoint's big failing in this department IMHO is that the compliance costs are very high, and the platform is too broad. By trying to do everything, it does some things well and some things badly, and in the latter cases it can be very difficult to go outside of the platform. In these cases you start wondering if the whole thing would have been better off in regular ASP.NET.

There are a long list of great comments on Jeffrey's post, and they make a very good read, if you are interested in such things.