Wednesday, July 26, 2006


Evolving a Giant

I had the chance to work with a couple of different firms who have created software on which they run their businesses on. The software they created are not really for public consumption, but the service they offer using the software is for public consumption. They don't distribute their application to clients, but they provide services through the software created. Even the services they offer are written as software, but not in the traditional software service sense -- but rather as end-user experiences.

Now these companies are _not_ software development companies, although they employ people that know how to more or less write software. After a few years of existence, they decide to expand their service offerings: they do aggressive marketing campaigns, thinking that the software they already have will be enough for what needs to get done. Now here's the catch: they have a handful of developers and tons of different sometimes totally unrelated products/services to roll out. The deadlines are insane, and the competition in the industry is cut throat. What are the problems and what are the possible solutions?

The Giant Software
To set the table up, the companies have this piece of giant software -- or should I call it the giant slayer, because it taughts itself as the one that will bring the company to the top and in effect slay the giant already atop the mountain. But the software itself is huge -- too many components written by too many people who have gone and are being read/modified/maintained by new people every so often. So basically, you get a mish-mash of solutions in one giant piece of software.

The software is not trivial -- it is complex enough that only people with appropriate software experience with the programming language used will be able to change the software significantly, and effectively maintain it. It's complex and requires high performance, so you can imagine the programming language candidates for these applications.

The fun part is, the software is currently working. It delivers the required functionality within schedule and according to specifications.

Here's the Rub...
One day, one of the major features of the software will need to be changed -- meaning: code revision, testing, and deployment to production. Deadline: one week. So the developers go ahead and hack on the solution, to beat the deadline and deliver the service -- but that problem is one week is not enough for the giant software to change. Even if there are a couple of people working on the solution, the reality is that there is not enough time given the complexity of the system to make the necessary changes.

So what are the developers left with? Shorten the testing time, lengthen the code revision time, and deploy (and optionally cross their fingers).

The service is delivered on time, and the marketing machinery starts rolling -- more services come, and the deadlines get more and more insane. Until the inevitable happens.

A change one developer made to accomodate one feature now breaks at least one other service. As though it doesn't get any worse, the change was supposedly trivial -- just a couple of lines of code changed here and there, and suddenly everything starts breaking. The developers start bug hunting, and in the meantime some of the services go down.

Developers start losing morale, there's no solution in sight, and the bug hunting and debugging goes on longer than necessary. Eventually the problem gets fixed, and the team is exhausted, but the marketing is continuosly rolling -- there's no time to recoup the losses, and no time to celebrate the solution (not even enough time to fix the already gigantic mess the software is).

Lessons Learned
The first lesson learned goes something like this:

Any one piece of software should not reach gigantic proportions such that the resources required to maintain it is greater than what is currently available.

This talks about maintaining the software's size so that the company shouldn't require more than the same number of people that originally created the software to maintain it. That means keeping lines of code low, and keeping components isolated and self-maintaining.

Another problem encountered when the software written is already so big, is the fact that interdependency (or close coupling) is such a temptation just too hard to avoid. It's increasingly harder and harder to isolate the common parts of a large software project that it is most prudent to isolate the common parts early on before letting it grow out of proportion.

Listen to Developers
This means asking them what they think and how they feel about the problem and the solution. Ask them how long they will take to implement it, and how easy or hard it is compared to the actually pausing and fixing the software first before implementing something new.

This is usually a management issue -- we all would love to get the best developers for the price we're willing to pay. Sometimes though, they're just not available, already somewhere else, and what's left are the people submitting their resume's to your HR department. In cases like this, we either invest in them and make them the geniuses that we need for the business or deal with the fact that they can only do so much.

The lesson relates to managing both the resources and the software. It talks about fixing the software so that it becomes easier to extend/fix in the future. That's a whole different discussion altogether though.

At any rate, in order for the software to not turn into a giant piece of hell to touch, you make sure the people you have will be able to maintain it in its current form. Otherwise, any additions will make it less maintainable than it already is, without it getting fixed in the first place.

Don't fix what's not broken -- but if it's easy to break, it's considered broken.

Although it is a tough pill to swallow, continous maintenance of the software for bug fixes, performance enhancements, and issue resolutions need to come first before feature additions. If the software is too easy to break, then that's one bug that needs fixing. If the software is too slow for the requirement, then that's one bug that needs fixing. If the issues are recurring, then that's another bug that needs fixing.

Unless you can extend your software painlessly in its current form, then you should work harder to make your software easier to extend -- because extending it will be inevitable anyway, invest early and reap rewards late.

So if you face yourself with a giant, it's a good idea to turn that giant into a midget by making sure it's as nible as possible before adding the functionality you think you need. It won't only result to savings in the long run, but it will save you from the pain of a falling giant.

Standing on the shoulders of giants doesn't necessarily require that there be a giant to stand on the shoulders of. There are no nimble giants, so unless you don't plan on moving, you can grow yourself one.

This page is powered by Blogger. Isn't yours?