Third World Agility

The Swarm

2006-09-01T09:16:00.000-07:00

When we learn about the giant software and the quick and dirty, we have only a few solutions left: the swarm and the silver bullet. Let me first talk about "The Swarm".

Quantity Then Quality

To avoid the quick hapahazard solutions and the overgrown software systems, we create tiny little pieces and then make them try to work together. In this solution, we use the quantity of the parts involved compared to the quality of the parts involved.

This means writing a lot of small solutions that tend to band together in a loosely coupled manner to tackle one problem. We see this being employed a lot in the scripting solutions in most Unix systems where you have self-contained tools that seem to work together to solve a problem. This seems like the best way of going about things, and is usually sufficient to solve the problem being faced.

The problem then becomes seeking control of the individual parts of the whole solution. In a bee swarm attacking a bear, the bees act as individual agents part of a whole cause -- stinging each at their own pace. In a locust swarm, it's first come first served: each locust takes his pick and sticks with it.

United We Stand

The good thing about this behavior is that you can lose one bee (in the swarm) and it's alright -- you easily have a couple dozen of them attacking anyway. That gives you more solutions to one problem, but not necessarily effective solutions to that single problem.

Another good thing about the solution (the swarm) is that you don't have to worry about making many different parts -- you just create one "bee" pattern and just launch an army of it to solve a particular problem. I'm not talking about a cluster or a replicated solution, I'm talking about one piece of software that you just use to solve *almost* the same problem (looking at tools like sed, awk, grep, etc.).

Divided We Fall

We see that alone, these tools are just that -- tools. Them alone cannot be as useful if you had an army of these tools at your disposal. Of course the scale of the tool says a lot about the scale of the problem that it tries to solve. To be useful (or more useful) you'll be using them along with another tool or set of tools (usually it's called a combo -- the flex-yacc combo, the grep-sed-awk combo, etc.).

You then run into a chaining problem -- which is like a dependency problem, but not really exactly the same as a dependency problem. Whenever you chain solutions, you introduce more and more points of failures. Instead of reducing the moving and breakable parts, you increase them by adding more tools and more chains.

Remember what they say about chains? "It's only as strong as the weakest link."

John, John, and... John?

Sometimes the tools also look so much alike, it's too hard to distinguish really how they differ from each other. Have you ever wondered by toolboxes contain too many tools in too many sizes and that you almost always only needed one or two of them? Sure, they've made screws "idiot proof" but they made a toolbox that's "genius required".

So how do we avoid the swarm problem?

Reuse

We can re-use one solution, and make a variation to it -- but make that variation part of the original solution as an "option". It's so much easier figuring options out than wading through a number of solutions that all almost did the same thing. I've counted at least two instances when I hoped "sed and awk" was just one tool and felt that PERL was becoming more and more attractive as the times went by.

Consolidate

If the solutions looked the same, make them one tool/solution with minor differences. You can make libraries, frameworks, and even whole applications out of the process of consolidation -- and it's not a bad thing to put together related information in one place. Just make sure that the solution you're writing won't require more people to maintain it than it actually took to write it.

Conserve

Whenever you feel like wasting effort, take time to see what's already out there. Before you write another line of code look at what's already available. After all, you don't want the quick and dirty to become a giant anytime soon.

Conclusion

The swarm is one of the best ways of actually solving problems -- break it down into small autonomous pieces, and make them work together to a common goal. Keeping each unit of the solution self-contained is a good thing especially when maintainance time comes, as well as when bug hunting season is in.

Other than avoiding making a swarm of giants or a quick and dirty swarm, there are only a couple of things to remember: keep each unit relevant and each unit should be justified. Units that stand alone are great, and making units work with each other to solve a greater deal is great(er).

Look at Service Oriented Architectures and Distributed Computing to see how a swarm can bring down even the biggest bears and bulls round.

More heads are better than one.

The Quick and The Dirty

2006-08-13T14:26:00.000-07:00

Experienced software engineers and designers usually already have a bag of tricks to pick out solutions from -- ranging from the most trivial routines to the most complex designs. After the countless years and projects that they have worked on, the skills they've learned, and the solutions they've deployed, they usually are able to distill the learnings into easy to recall and easy to implement techniques.

Some software projects (see Evolving a Giant) can benefit from these experienced engineers and designers -- but then the same forces encountered in these situations applied to even the most experienced engineers and designers can lead to another type of problem. This problem is something I like to call "The Quick and The Dirty".

The Quick Solution
Experience is the best teacher, and lessons learned through experience are hard to forget. To avoid complex solutions, a lot of engineers have found that if you avoid over-engineering a solution, you avoid a lot of the complexities associated with that solution. If they kept it simple and it works, they get the job done quicker than if they spent a whole lot of time over-engineering it.

The quick solution is usually effective -- it solves the problems, and creates less problems that need to be faced at the moment. The quick solution is usually sufficient especially if the enhancement/addition to the software is trivial and can be addressed with tricks from the engineer's bag. Of course, this assumes that the "solution" works, otherwise it's not a solution.

The Dirty Solution
Another thing experience brings is confidence. Usually, the more experienced engineers are more confident with their solutions compared to the solutions of those with considerably less experience or exposure. That leads to problems especially if you have a team composed of experienced and not so experienced engineers.

The dirty solution comes from both the experienced and the not so experienced engineers -- the solution becomes dirty if it needs to get cleaned up even after the solution has been deployed. Experienced developers have gone through the nightmare of maintaining their own dirty code, and the dirty code of others. This leads to the same problems that are encountered when working with a humongous software project that has already grown out of proportion.

Dirty doesn't necessarily mean not-elegant -- on the contrary, a lot of the "elegant" solutions are so complex they end up being more dirty than the quick solution (for examples, look at the merge sort and quick sort algorithm and compare these to insertion sort and even bubble sort). Dirty solutions have any one or all of the following properties (please feel free to add to them if you feel like I'm missing out on some things):

Unnecessary elements

Repetitive elements

"Case Specific" elements

Unjustified existence

A solution can be a method, a class, an application, a framework, a design element -- or anything that is part of contributing to the solution of a certain problem. Dirty solutions work, but they are hard to maintain which is a reallistic problem that people in the software development industry face quite often.

Dirty solutions aren't really "bad", they're just dirty. If they're dirty, most probably they need cleaning -- so some people avoid dirty solutions at the cost of being slow.

The Quick and Dirty
Suppose the timeline requires a solution fast -- an experienced engineer under no circumstances will let the project run off track, because he knows how important keeping the project manager off his back is. But the same forces at work in "Evolving a Giant" are the same ones at play in almost all software projects.

These situations usually turn out quick and dirty solutions which are harder to find, and harder to fix.

Avoid the Quick and Dirty
Just like how you might avoid a black fly buzzing around you while eating your sandwich for lunch, you just might want to avoid these quick and ditry solutions. This section gives advice on how your organization can avoid these quick and dirty solutions, by putting in place some measures and techniques for minimizing the forces that bring about these solutions.

Experienced engineers are like artists --- they take their time when solving a challenging problem.

Haste makes waste, and experience teaches us this. Therefore, when planning and shceduling deliverables, leave ample room for leeway -- and plenty of time for Refactoring. Make sure that when they're in a productive zone that they are not bothered unless the building is on fire and they're not moving away from the workstation.

Being creative is easier than being harassed. You can't squeeze creativity out of a person, so don't even try.

If you're a project manager and are working on a timeline, make adjustments to the timeline as much as you can before harassing your developers. If you're a developer or a team lead, allocate enough resources (i.e. time and effort) to making sure that you stay as creative as you can be by taking enough breaks and keeping everyone fresh. Keeping someone fit to be productive is better than pulling a genius out of your hat.

If it doesn't look clean, then it's not.

If you find yourself writing a solution that doesn't look clean to you then it's not clean. Unless you yourself see it cleanly, you can't expect someone else to see it cleanly.

Making sure that your solution to the problem is clean will force you to stop worrying about the details and keep your eye on the overall challenge. Ask yourself if the solution you came up with is the cleanest possible solution you can think of given the time constraints -- as much as we want to have all the time in the world to figure out the cleanest solution, we just don't have it.

Conclusion
Making sure that your solution is quick and clean instead of quick and dirty takes a lot of effort. You will see that dirty solutions only postpone the agony of maintaining the system -- and like the wise people say, an ounce of prevention is better than a pound of cure.

Cliche's aside, the software industry relies on quick solutions to common problems -- so if your problem has been solved before, look at the solutions first before coming up with your own. After all, your limitation is a function of your experience and your knowledge: knowing this allows you to work on both and push your limitations further.

In an organization, it's important that you see your limitations and know how to work within them. So if you don't have enough experience, get as much as you can as soon as possible. If you don't have the knowledge, get as much of it as you can as soon as possible. That way, you'll be able to get quick and clean solutions like the experienced ones already do.

What's the use of being quick if you're going to the wrong place? What's the use of heading for the right direction if you don't start moving?

Evolving a Giant

2006-07-26T23:24:00.000-07:00

I had the chance to work with a couple of different firms who have created software on which they run their businesses on. The software they created are not really for public consumption, but the service they offer using the software is for public consumption. They don't distribute their application to clients, but they provide services through the software created. Even the services they offer are written as software, but not in the traditional software service sense -- but rather as end-user experiences.

Now these companies are _not_ software development companies, although they employ people that know how to more or less write software. After a few years of existence, they decide to expand their service offerings: they do aggressive marketing campaigns, thinking that the software they already have will be enough for what needs to get done. Now here's the catch: they have a handful of developers and tons of different sometimes totally unrelated products/services to roll out. The deadlines are insane, and the competition in the industry is cut throat. What are the problems and what are the possible solutions?

The Giant Software
To set the table up, the companies have this piece of giant software -- or should I call it the giant slayer, because it taughts itself as the one that will bring the company to the top and in effect slay the giant already atop the mountain. But the software itself is huge -- too many components written by too many people who have gone and are being read/modified/maintained by new people every so often. So basically, you get a mish-mash of solutions in one giant piece of software.

The software is not trivial -- it is complex enough that only people with appropriate software experience with the programming language used will be able to change the software significantly, and effectively maintain it. It's complex and requires high performance, so you can imagine the programming language candidates for these applications.

The fun part is, the software is currently working. It delivers the required functionality within schedule and according to specifications.

Here's the Rub...
One day, one of the major features of the software will need to be changed -- meaning: code revision, testing, and deployment to production. Deadline: one week. So the developers go ahead and hack on the solution, to beat the deadline and deliver the service -- but that problem is one week is not enough for the giant software to change. Even if there are a couple of people working on the solution, the reality is that there is not enough time given the complexity of the system to make the necessary changes.

So what are the developers left with? Shorten the testing time, lengthen the code revision time, and deploy (and optionally cross their fingers).

The service is delivered on time, and the marketing machinery starts rolling -- more services come, and the deadlines get more and more insane. Until the inevitable happens.

Downtime
A change one developer made to accomodate one feature now breaks at least one other service. As though it doesn't get any worse, the change was supposedly trivial -- just a couple of lines of code changed here and there, and suddenly everything starts breaking. The developers start bug hunting, and in the meantime some of the services go down.

Developers start losing morale, there's no solution in sight, and the bug hunting and debugging goes on longer than necessary. Eventually the problem gets fixed, and the team is exhausted, but the marketing is continuosly rolling -- there's no time to recoup the losses, and no time to celebrate the solution (not even enough time to fix the already gigantic mess the software is).

Lessons Learned
The first lesson learned goes something like this:

Any one piece of software should not reach gigantic proportions such that the resources required to maintain it is greater than what is currently available.

This talks about maintaining the software's size so that the company shouldn't require more than the same number of people that originally created the software to maintain it. That means keeping lines of code low, and keeping components isolated and self-maintaining.

Another problem encountered when the software written is already so big, is the fact that interdependency (or close coupling) is such a temptation just too hard to avoid. It's increasingly harder and harder to isolate the common parts of a large software project that it is most prudent to isolate the common parts early on before letting it grow out of proportion.

Listen to Developers
This means asking them what they think and how they feel about the problem and the solution. Ask them how long they will take to implement it, and how easy or hard it is compared to the actually pausing and fixing the software first before implementing something new.

This is usually a management issue -- we all would love to get the best developers for the price we're willing to pay. Sometimes though, they're just not available, already somewhere else, and what's left are the people submitting their resume's to your HR department. In cases like this, we either invest in them and make them the geniuses that we need for the business or deal with the fact that they can only do so much.

The lesson relates to managing both the resources and the software. It talks about fixing the software so that it becomes easier to extend/fix in the future. That's a whole different discussion altogether though.

At any rate, in order for the software to not turn into a giant piece of hell to touch, you make sure the people you have will be able to maintain it in its current form. Otherwise, any additions will make it less maintainable than it already is, without it getting fixed in the first place.

Don't fix what's not broken -- but if it's easy to break, it's considered broken.

Although it is a tough pill to swallow, continous maintenance of the software for bug fixes, performance enhancements, and issue resolutions need to come first before feature additions. If the software is too easy to break, then that's one bug that needs fixing. If the software is too slow for the requirement, then that's one bug that needs fixing. If the issues are recurring, then that's another bug that needs fixing.

Unless you can extend your software painlessly in its current form, then you should work harder to make your software easier to extend -- because extending it will be inevitable anyway, invest early and reap rewards late.

Conclusions
So if you face yourself with a giant, it's a good idea to turn that giant into a midget by making sure it's as nible as possible before adding the functionality you think you need. It won't only result to savings in the long run, but it will save you from the pain of a falling giant.

Standing on the shoulders of giants doesn't necessarily require that there be a giant to stand on the shoulders of. There are no nimble giants, so unless you don't plan on moving, you can grow yourself one.

Evolution

2006-06-23T16:25:00.000-07:00

The usual first problem with change is the question of "what happens to the old one?" Whether it's a process, a technology, a solution, a religion, a name, a pet, or anything. Comfort and familiarity with an existing entitiy or concept is something that prevents a lot of people and companies from change. Another thing that really prohibits change or jumping into change is cost and risk.

Cost is the amount of resources and effort required to perform or obtain a task or resource. It is usually monetary, but most usually it is more time than anything else.

Risk the likelyhood of loss or failure in acquiring a resource, investing time/money, and even emotional capital.

How do you introduce change so that it's 1) not so costly 2) not too risky or 3) more appropriate for what you need?

You evolve. For the next few posts, I will propose some ways of evolving your current processes, practices, and approaches so that becoming Agile is not as costly nor Risky as it sounds.

The only constant in the world is change.

Management

2006-06-05T16:14:00.000-07:00

Do you notice that management gets in the way of actually being able to effectively deliver a product? In traditional business models, managers tends to "over-manage" the resources: Gantt Charts, Planning Sessions, Status Meetings, Team Building, Status Reports, and all the management initiated activities tend to get in the way of delivering. Of course, there has to be a point to management, otherwise what is the need for a manager?

Here's an idea: let management do the work that needs to be done. What do I mean by "work that needs to be done"? It's really simple: make sure that there is progress, and then get back to doing something actually contributory to the project.

When I write "something actually contributory to the project", then that means something *tangible* or something (an action, or set of actions) directly contributory to a project's value creation model. And in software development, that's only one of two things: code or documentation.

When managers need to write code, then that means there are not enough developers writing the code that needs to be written. When managers need to write documentation, then that means that the developers aren't writing the documentation that's necessary in the project. These are cues to the manager, that there has to be something done about it. And if a manager has to write code to deliver a product on time, that means the developers aren't capable, aren't working, or aren't worthy of the pay they're getting; or of course the manager himself over-estimated the capabilities of the team, or undercalculated the risks of a tight timeline.

If there are other things that need to be done by a manager, that's budgeting the time spent on a project, and the available resources at his disposal.

Did you ever notice that the successful software development firms have competent managers who can do what the subordinates do, as well as do very little management as much as possible? Here's a hint:

The role of a manager is to make sure that there's little work for him to do and enough time for him to spend on more imporatant things by doing the things that need to be done so that things get done on their own.

This means getting the right people, getting the right tools, setting the right timeline, and keeping the resources utilized effectively to provide a high level of productivity. Here's a simple goal for a manager: do everything that needs to be done so that you don't need to do it yourself. Seems like a paradox? The point is getting others to do the work that you don't have to do yourself, but need to do otherwise.

Sounds shrewd? Not really... It's really more an advantage to the project and the firm if the manager continues to do only the necessary actions that a manager needs to do: like tracking the progress of a project and keeping the resources busy. That way, a developer can concentrate on developing, a manager concentrates on keeping the developers developing, and the firm can concentrate on delivering its products.

The more you manage, the less things get done.

Tomato

2006-05-22T11:34:00.000-07:00

Tomaytoe, Tomahtoe -- the same thing, referred to differently. Usually this is one of the major problems a non-tech person will encounter when talking with a tech person. Although techinical people are very precise (or very particular) with the words used to refer to things (server application as opposed to server hardware), there's a certain degree of assumed context whenever two technical people talk to each other:

Developer 1: "I built the version, and it ran without bugs. However, when you did it, the bugs came out!"
Developer 2: "Yeah, but I seem to have used a different version of the library from what you used, so I'll just get 2.0 then build-run-test it."

That's usually how developer talk happens within a team. The problem is, there's something missing: context. But they seemed to understand each other! Non-technical people hearing this will most probably say "WTF?!"

Now let's see another conversation between a business person and a developer (which happens all too often):

Business Person: "I just committed a delivery date on the 10th, we need it out by then..."
Developer: "It's not going to happen."
Business Person: "What do you mean, it's not going to happen?!"
Developer: "It's not going to happen."
Business Person: "What do you mean it's not going to happen?!"
Developer: "It's not going to happen."
Business Person: "It's not going to happen on the 10th?"
Developer: "It's not going to happen."
Business Person: "When will it happen?"
Developer: "It's not going to happen."

You can just imagine what will happen to this conversation... Absolutely nothing. The business person is looking for an explanation, and the developer isn't interested in giving one. Who's at fault here? Look at the first statement: I just committed a delivery date on the 10th, we need it out by then...

Does it look the least bit familiar? It's the PHB syndrome... If you don't know what PHB means, maybe you should start reading this too.

At any rate, the business people (especially management people) tend to keep their eye on the sale, and not on the product. While the developers on the other hand, tend to look at the product more than the sale. Here's a way of rephrasing the first statement:

Do you think we can get this out by the 10th?

This way of approaching developers will convey 2 important messages: 1) I'm asking for your opinion 2) I think I might be willing to listen. The last thing you want is a developer that doesn't listen, and will sabotage your project. We all don't want that.

Rephrasing your statements can go a long way especially if you want to make sure that some things happen as you expect them, while keeping everyone happy. Try it, then make sure you say it with sincerity. You might get a different answer from the developer.

Do you have communication stories? Let them be known by emailing me your comments and stories through dean [at] orangeandbronze [dot] com -- they might just well be listed here.

Communication is a process of conveying information while dodging the challenges brought about by media, experience, environment, and preference. Effective Communication happens only if the process is effective and the message clear.

Conversation

2006-05-12T02:26:00.000-07:00

How do you usually hold conversations with a technical person? Do you follow a plan? Or do you hold it like a normal one? If you're in the technology industry, communication is an important aspect of the operations. If you can communicate better with your peers/colleagues, you basically have 50% of your work done. However, not everyone has the best communication skills in the work place.

The perennial question in IT companies is how do you bridge the divide between the sales group and the tech group. What type of communication breakdowns usually occur? Are there any verbal or written cues that you should look out for? Is there more to it than meets the eye (or ears)?

In the next few posts, I shall discuss the many different issues interpersonal communication issues and problems in the technology industry especially in the small group sense. In the later parts, I shall outline the many pitfalls that non-technical people encounter when talking to technical people.

What you say and mean is not what I necessarily hear and understand.

Driven

2006-04-25T11:03:00.000-07:00

Strongly Motivated to Succeed.

When considering agility in terms of creating products, motivation is one very important aspect of being able to work on something effectively. Getting motivated to start working on something is easy, but keeping yourself motivated to succeed is another issue all together. Most of the times, getting motivated to start is easier than keeping at it.

Have you ever heard the phrase "keep walking"? It might be a rip-off of a liquor ad, but that's something easier said than done for a few reasons:

Starting to walk takes effort. The effort involved in motivating yourself to walk is usually trivial, but nonetheless exists. It's usually easier to start walking if you already have directions, and when you have a clear path to where you're going. Getting started usually is a very exciting time, when the anticipation of the outcome is the highest.

The first challenge is the most crucial. Let's say you're already walking, and you suddenly realize that the path you've taken gets a little steeper than the first few steps. You weren't able to anticipate that it would be this hard, and the response to the first challenge is the best indicator of the quality of the people involved. When you get the first challenge done with, it will define the whole mood of the journey.

Delays induce anxiety, and "are we there yet?" becomes annoying the first few hundred times. We know that patience is a virtue, and not all people are patient. Yet the driven are also people, and annoying them doesn't get you too far. Perhaps a better question would be "where are we now, and where are we going?" rather than just plain "are we there yet?".

Being there is never enough -- and be prepared to be never there. Directions can change over the course of time, especially if the journey has already taken a while. Maybe instead of the highest peak of Mt. Everest, you settle for the closest you can ever get because you're all losing fingers the higher up you go. Or in the case of software, all the features earlier requested seem to be too much for what is required, and the list is cut short -- or worse, the original features are removed and the project completely changes direction.

The challenge for IT companies (managers, supervisors, developers, and decision makers) is how to keep the work force motivated so that you achieve the goals already set. The even greater challenge is adapting to the change induced by a lot of external forces while contending with available resources and targets.

Just like altruism, being driven is something in-born to a person (or developed over a period of time) but it can be a quality which an IT company can adopt to not only project an image of passion and committment, but also as a means of motivating the work force to become productive as a part of the company.

The ends justify the means -- but when the ends change, the means change. When walking, keep an eye on the destination and the other on the path.

Altruism

2006-04-21T07:12:00.000-07:00

Altruism is unselfish concern for the needs or interests of others, providing gratification vicariously or from their responses. -- gotten from here . There are more definitions for altruism here.

When you're in the business of helping people, you should take pride in your ability to help. You should also take their interest and make it your own -- so that their success becomes your success. The fact that you're getting paid to do that is a big bonus. And when you focus on delivering your service to enable your customer rather than focusing on getting paid, then your work becomes more about delivery than revenue.

The problem with being fixated on revenue (too much) is that you lose sight of the precursor which is delivery. When you deliver and are reliable, you (should) get paid. However, the satisfaction you gain from delivery should be the driving force for revenue. The point in travelling is not the destination, but the journey.

When your business requires that you deliver a product on time and within specifications, you should focus on that instead of (just) the revenue you generate. This sounds like a foolish ploy to go broke fast, but look at it this way: when your work is your advertisement, it's better than hype of hot air. No matter how much advertisement and marketing you do, it's no substitute to good service and reliable delivery.

How about value add? When you have great service and reliability, value add is now hinged on the relationship with the customer. If you can build lasting relationships with your customers by making them feel important and by keeping their interest in mind, then that should be the value add of your business. I'm not talking about just being friends with your customers, it's something else -- it's being a good friend to them that matters more.

Altruism is not something you can practice, because it's usually built into a person -- however if you're in charge of making decisions for a business, then you have the power to give your enterprise a personality. An altruistic personality fits well for businesses that make stuff that their customers use to build better products.

Is your IT company altruistic enough?

It's not what makes me happy... It's what makes you happy.

Empowerment

2006-04-20T13:03:00.000-07:00

As a consultant, one of my goals is to empower clients to become self-sufficient, successful, and effective. Empowerment is one of the key aspects of modern businesses -- it is what allows consultants to become successful with the client, and an enterprise to become successful with their customers. This post is about how an enterprise can empower the customer to make them more productive with them and their product/service than without them.

Being indispensable to your customer is a good strategy. Being indispensably helpful is yet another. Empowering and Enabling your customer is yet even better. This is what makes the largest IT companies in the world the most successful -- these companies don't just sell software, they allow you to do more things with their products/services. This works not only for IT companies, but also more traditional businesses: if you sell something that allows the customer to become happy, empowered, and productive, then their success is translated to your success.

Throughout the next few days/weeks, I will tackle some aspects of Agile Development which will emphasize on the empowerment of project stakeholders, especially software developers and managers.

This is a considerably broad and tricky topic, and I hope I can do justice to the different advice that other people before me have already suggested.

Knowledge is Power. Height is Might. Knowing more allows you to see what you don't know yet. Growing more allows you to see how much bigger you can grow. But knowing enough and staying high enough breeds stagnation, and there is no power without change. Only through progressive change can results be found and measurements taken.

The Interstate Highway

2006-04-06T08:21:00.000-07:00

When building roads, engineers take a lot of things in consideration: expected volume of traffic, expected load on the roads, expected length of the roads, and special terrain conditions. Much like software that spans a lot of domains -- like enterprise systems -- a solution shoud be chosen based on the expected volume of transactions, how much processing will need to be done, how much data will have to be dealt with, and special domain requirements.

For systems that are meant to serve a lot of users at any given time, and perform as robustly as possible, and minimize down time certain architectures fit the bill better than others. Let me enumerate some of these architectures that in a lot of ways, are like Interstate Highways -- high reliability, high scalability, and robust.

Three-Tier Web Applications

There are a lot of these enterprise solutions that use this architecture where the front end of the application (usually a web site, or a special client application) interfaces a middle tier holding the "business logic", and the back end usually a database management system. This architecture proves to be robust and scalable since each of the three tiers can scale as necessary -- based on demand and actual circumstances. These systems are managable too, since the different roles of the solutions are made self-contained and de-coupled from each other to a certain extent.

However, this architecture proves to be single minded -- or a better term would be domain directed. When you create web applications that use the three tier approach, usually the functionality required is programmed in the middle tier. This means you have a bungalow in between two bungalows tied together with different connectors.

This solution is attractive for clearly targeted applications such as e-commerce systems, online community sites, and content management systems. They make sense because they're essentially a three part house (or a three layered house), or a two lane highway which expands to the sides only.

Distributed Systems

A distributed application or distributed system is one where the functionality of the solution can be located in different non-dependent locations/systems where each other the functionalities may or may not be related. This allows the solution to scale well as new functionality is required, and a well-designed system for service discovery and registration is used to coordinate these different services as well as made available to different users of the system. Distributed application design allows developers to access a central service brokerage system to which service providers (which may replicate the services of each other, or provide completely different services) can register.

The main down-fall or issue in this system is the central brokerage system which becomes a crucial point of failure for the whole design. This may be side-stepped or circumvented by directly accessing the service provider in an end-to-end fashion, however this approach will make clients completely dependent on a single service provider. This end-to-end coupling is a potential point of failure which is not a solution to the original problem.

This system however is most effective in non-mission critical applications where flexibility is valued more than reliability, and maintenance is covered by different domains of influence. This system can be likened to a system of streets and side streets where a common avenue runs through all possible connections. In these systems, the avenue may be circumvented but ultimately they serve a purpose of keeping the system of streets connected.

The Hybrid

Combining the best of both the Three-Tier Web Application and the distributed system allows users to enjoy the flexibility of the distributed design, and the robustness of the three-tier design. This basically works by making the tiers distributed systems -- allowing for flexible service usage and registration -- at the same time clearly delineating an interface between the tiers in the original design.

Another hybrid of this system is making the service providers in a distributed system part of a three-tier web application architecture. This means each service provider is essentially a front end in a three-tier system where the middle tier can scale to the demands of the front end. This also allows the service providers to replicate the front end such that it can serve more users at any given time.

Only by determining the requirements in terms of flexibility, maintainability, reliability, and robustness will the right architecture for systems be determined. These decisions are usually made before hand, especially when the requirements are being determined. Making the design decisions at the start also allows the developers to gauge how much work will be done, and thus how much time it will take to come up with the system.

Knowing what to do first allows you to decide how to do it. What you need will dictate what you should be done, and what you know or don't know will dictate how you'll do it.

The Bungalow

2006-04-04T09:53:00.000-07:00

A house with one story and divisions to create rooms is called a Bungalow. When you hear the term Bungalow, you get the idea that it's a house with one story. Here is a listing of the many definitions for a Bungalow.

There are a lot of solutions where a single self-contained application can do the trick. There are also a lot of solutions where there may be other available applications, but unfortunately a lot of the software written in the 90's were meant to stand alone and are meant to be self-contained. They didn't require a lot of other things to operate, and don't really care whether there are any other applications installed in the computer.

Bungalows have a property which can aptly be seen in a lot of these tools: they are small. Back in the early days of personal (and enterprise) computer systems everybody had limited resources -- so the applications needed to be small and self contained. Usually, these applications only do small simple things -- and usually don't require a lot of other applications to perform its duty/purpose.

A Bungalow also has all the rooms in one floor -- so you have only one layer. You have everything you need in one floor, so you don't need to go too far to get something. This is true for self-contained applications where all the functions they need are mostly found in a single binary file (sometimes some services are supplied by external sources like the operating system, or a software library -- think utilities like water, sewer systems, gas pipes, etc.)

So what's wrong with a Bungalow?

It doesn't scale with the people using it. You can only house so much people comfortably in one bungalow. And usually, as the bungalow grows, it's harder to maintain. Not to mention that you can't really grow a bungalow too much, because you're limited by the real estate.

In the same light, standalone applications are like Bungalows, in that they don't scale too well, and as they grow larger they get harder and harder to maintain. Monolithic solutions where all the units of functionality are confined in one application are limited by the computer resources on which the single application is running in.

When does a Bungalow make sense?

A Bungalow makes sense for a low budget small family. They do the job of housing the people, and usually come at a considerably cheaper price tag compared to other house types. It's easy to build, and can last a long time (when built right).

Much like monolithic applications, a low budget firm who doesn't need a scalable solution for a particular problem is a very attractive solution. Keeping things small and simple (and cheap) is just right, especially if they don't expect the requirements to change too much (or even at all, how rarely the case may be). Monolithic software is also easy to build, and can last a long time when built right.

So What's the Point?

When you need a quick solution to a small not so recurring problem (like finding out the square root of a number), you can use small effective tools that come cheap and are functional (like a calculator). However, when you start needing the roots of a transcendental exponential polar system of non-linear equations in N dimensions, then you might not be able to solve that so easily with a simple calculator -- especially if you do it more than once, and need to do it quickly many number of times.

The trick here is recognizing the problem, and the scale of which you're going to be solving these problems. Simple problems usually need simple solutions. If you find a complex problem and see a simple solution, then you're in luck -- otherwise, prepare to come up with a complex solution nonetheless.

Complexity defines size.

Architecture

2006-04-03T11:11:00.000-07:00

When do you determine the architecture of your whole system? Do you go with an architecture you've already seen and used, or one that you know will be most effective for your solution? Do you go with the best and feature full, or the fastest and easiest to deploy?

All week I will be writing about making architectural decisions -- when to choose robust feature full solutions, and when to use the light, quick and "dirty" solutions. It's all in the light of making design decisions when you're already working on the user stories.

So when do you choose the architecture?

In Agile settings, you choose the architecture for the solution everytime you need to -- which translates to, almost everytime you write code. When talking about architecture, the discussion covers the hardware and software components of the whole solution -- and how the system should look like from afar. That is, from a higher level -- higher than the code.

This seems like a simple task, but it really isn't all that simple -- there are a lot of things that need to be considered: budget, timeline, problem domain, scale, core competencies, among many other things. It also assumes that you have a host of architectures already in mind -- otherwise, that's yet another discussion.

In the next few days, I will be outlining criteria I use to determine which architecture (infrastructure wise, containing both software and hardware solutions) to deploy in different settings. I will be providing hints and insights into different domains that I've had the privilege in working with.

A good design looks good and gets better when it's actually built. A bad design just breaks everything. So design it as good as you can, and hope it doesn't break too much in the end.

Then

2006-03-31T08:21:00.000-08:00

And then it hits you. All the stories are important! The customer says I want everything done yesterday! Then you start thinking that writing the stories down are a waste of time, and consider solving the problems on your own. Then you start thinking that you shouldn't do Agile anymore because you can't do it right the first try.

Hold your horses there... Giving in to all your customers whims all the time is not really good business. But the customer is always right! Right? Technically, yes... But nothing is stopping you from telling your customer: "Look... We can do this given enough time. Now yesterday's gone, and we only have today. And tomorrow will be gone, because it will become today tomorrow. Now what do you want done first?"

Being firm is not bad business -- but being inconsistent and unreliable is. When your customers know that you're working on it and that they see constant and verifiable progress then you now have chips that you can play with. If you've given your word before and you pushed through, then you have a reputation of delivering -- and that's what you want. The problem is when you have given your word before, and failed to deliver: then that's another point of discussion.

Allowing your customer to evaluate what is of most value to him alleviates the risk from the technical team of telling the customer what is important (and being wrong). The User Stories that have already been written should have the priority as an attribute -- meaning, the most important things are done in the beginning of the project. However, the priority doesn't have to be written down on the index cards -- though it would be nice to put them down somewhere else.

This post is about prioritizing stories which is not the job of the technical team. The customer should tell the technical people what they need, when they need it, and which needs should be addressed first. When the technical team already has all this information, then they can start working on providing the solutions and working with the customer to fulfill the needs.

If your customer still wants things done yesterday, then tell them the reality of things -- or charge them a little more so that you can put in more resources, but nonetheless involve them in the solution process. Building software should not be like a fast food drive through service -- where people just line up, tell the programmers what they want, and wait at the end for the delivery. It's more like a "I'll build your house" shop, where you tell the architect and engineers what you want, and be there every step of the building process -- that way, when your house sucks, then it's not the builders fault because you made the decisions on how it should look.

Allowing technical people to be creative on solving the problem doesn't mean you let them determine the priority of the deliverables. It means you allow them to solve the right problem at any given time. It's like leading a pack of wolves and directing them to the right flock of sheep.

How do you lead a pack of wolves? That's another discussion.

Continuing on to the prioritizing of stories, it also allows the technical team to solve the harder problems first. You ride the productivity wave by allowing the most productive people to do the most demanding tasks first, and not worry too much about the plateau in productivity later in the project.

The last thing you want in a project is delivering the non-important features of the software while leaving out the more important ones. I as a customer would rather have the important parts there than have the extras first without the core functionality. I wouldn't mind too much if I see that the extras will come later, but the really core functionalities are already there.

Prioritizing stories allows project managers to determine a lot of things at the start of the project, as well as while the project is on-going. It allows customers to communicate exactly what they want in the order that they will be satisfied in. It also allows the technical team to be productive and creative in the most efficient manner.

First things first. One after another. You don't get to the next one if you don't start with the first one.

Now What?

2006-03-30T09:04:00.000-08:00

Okay, you have the stories written down containing enough details. You've told the developers about the necessary details, without setting them on stone. Is there anything else to do?

A good story is one that usually leaves the reader satisfied -- may it be a fairy tale, a suspense novel where the bad guy is revealed, or something that ends in "happily ever after". But we all know that there are some stories that need a sequel. Think of a story when you were a kid that you loved to extend, and expound upon, or make up stuff about.

Role playing games rally around the notion that stories are built around quests. When you finish a quest, you're sucked into another one. It's no different from stories that we write for the software developers -- since fulfilling a story leads to another story.

In the previous days, I wrote about writing clear simple stories, and writing complete simple stories. Today, I'm writing about sequencing these simple stories.

Defining which stories are related to each other is nice -- you can make a great big story built of many different small stories. You can mix and match these small stories and reorder them to make completely different big stories. But you'll run into a brick wall when you find out that these stories don't necessarily need to be told in sequence.

What?! You just told me we're going to put them into a sequence?! I did. But that doesn't mean that the sequence of the small stories to form the big story will determine the sequence in which they will be built. It does however tell us about which stories are related, to allow us to determine which set of stories are most important.

How does this happen? It's important for the customer to determine which big story to realize first. This is again so that we can determine which set of stories (not necessarily related stories) we want realized first. Important decisions like these come into planning sessions, especially when developers are falling behind in delivering stories for the latest development iteration.

Imagine this (bear with me here, I don't know a lot of fairy tales): Cinderella and Sleeping Beauty are living in different parts of the continent. They both have the same prince (let's just imagine, please) but the prince has only until midnight to slay a dragon and wake up Sleeping Beauty, or do the last dance with Cinderella. This wouldn't be too much of a problem if he had all day to slay the dragon, but unfortunately it's 10 pm and he has to make a decision.

In this situation, the prince can continue on with the "Slay the Dragon" story and do the "Dance with Cinderella" story after that (and leave Sleeping Beauty until the next day). Or if the director wants to finish the "Wake Up Sleeping Beauty" story first, then Cinderella will have to wait until the next midnight taping, and reschedule the bippity-boppity-boop magic for next time (which might cost her a lot).

The point is, someone has to make a decision. In this case, the director (or producer, or someone else) has to make a choice and tell the prince what to do, and which story to finish first. And the prince doesn't have to decide for himself, so that he doesn't get in trouble.

Aha! Yes, that's the point. The developer doesn't have to get in trouble (or too much trouble at least) for decisions that the customer has to make. This way, the developers will know what they should be working on when the end is near -- so that the little time they have will be channeled to the more important things to be delivered.

Grouping related stories together, or those that might depend on each other is a good way of determining which things may or may not come first, as well as attribute priority to these stories. This exercise is usually done to keep everyone's sanity and let everyone in on the big picture -- which is always a good thing.

Important decisions made by people make people important. Keeping important decisions to the important people gives people importance -- and value of the decisions affect the value of people.

2006-03-29T08:19:00.000-08:00

Now you've got them stories written down, and made really nice and simple to understand. They seem like simple stories, but somehow you feel like they're a little too concise. You feel this compusion to get into detail and get into the technical aspects of things. It feels like you can't stop at telling more about the story, that you find many different ways of putting in more detail than necessary in the tiny index cards you have been given.

This is a common problem encountered especially by customers who want to tell the technical personnel what to do. It's like the time when you have your mechanic looking at your car, and you then telling him what should be done -- instead of telling him what the problem is. This is a symptom of orderitis or a showing of a primal need to order someone around.

This thought process basically impedes the thinking process of most software developers -- telling someone how to do his job leads to the "why don't you do it yourself?" response. And you don't want your software developers telling you that. Or at most, you don't want them not doing their job, which is basically creating software to fulfill the needs.

What do you do then when you have a compulsion to give more than necessary details regarding a story? It's really pretty simple. Just tell your developer(s) about it.

Yes, don't write it down on the index cards, don't write it down in your emails, or print them out on paper. You may print them out on paper, but shredding them afterwads would be great (but wasteful).

It's that simple? Yes. You don't need to bombard your developers with painstaking details to let them do their jobs -- unless you're the lead developer/architect (but even so, it's not healthy). Even if you do know what you're saying, stick to words not written down on paper. That way, it's easy to say them, and it's easy to forget them. The last thing you want is your developer telling you that the way you do things is so much more complex that the reason they're behind schedule is because you wrote down/emailed them to do it a certain way.

Sometimes, the best people who can decide how to solve a problem is the ones facing it. You can give advice, you can lead them somewhere, or you can give them books -- but it's essentially their job to solve the problem. Supporting them in the problem solving process is so much better than telling them what to do.

So the next time you feel the need to direct developers (when you're not a developer yourself, or if you're a developer but not assigned to the project), then just tell them about what you think. Don't tell them what to do but do tell them about the details that may be helpful to their problem solving endeavor.

Learning is not a matter of hearing it, reading it, or seeing it. It's about experience, exercise, and retention. You don't learn if you don't make mistakes. Making mistakes involves doing something.

One Time...

2006-03-28T08:05:00.000-08:00

When you're telling a story about something that's supposed to happen just once -- or something that happened already, once in your life, someone else's life, or something you saw in a movie -- how do you do it? Do you talk generally, or do you talk in specifics?

The answer could be it depends on who you're talking to. If you're telling a story to a friend, then you'll be telling the relevant parts and even the details that are needed for her to understand the story. If you're telling a stranger about it, then maybe you'll tell them about the really general things -- and maybe even exaggerate or obliviate some aspects of the story. Or maybe when you're telling someone about something that you already told that someone about, then you'll use a reference to your earlier story.

It's true that it really depends on the audience -- but there are undeniable reasons why there are some really important parts of the story that have to be there. There are elements to a story that give it a concreteness, even if the details aren't mentioned literally. There are a few important things that all stories will require:

Setting. A story should be set in some location or circumstance for things in the story to make sense.
Timing. A notion of time will make a story more believable -- words like meantime, while, after, and before give you a sense of order so that the sequence of events will make sense (or not make sense).
Actors. Stories without participants or actors wouldn't make too much sense, since there are no participants to the story. It will be hard to visualize a story, and even construct a sentence (that makes sense) without using nouns or pronouns.
Action. Stories wihout action can't be stories -- action doesn't necessitate violence, but rather an occurence involving actors. Action that doesn't make sense doesn't make a good story.

These are the same elements that User Stories should contain. Usually the Setting is well defined in the context of the project, and each story should contain a sequence (if it's required), the actors, and the action in short concise 3x5" index card scribblets. The simpler and understandable (and clear) a story is, there's a less chance of confusion -- and it enables the customer to communicate precisely what they require.

If you look at the stories that need to be made reality, usually these are functional stories. A short example follows:

User Registration

A User enters his personal information into a form. The information is validated, so that the required information is entered. The data is then stored, and made available for future use.

This story is short, concise, and complete in its own right. This can be part of any project, and it doesn't have to be a web based project -- it can be a paper form, a verbal form, a virtual form, or some other form I don't know about. What this allows the customer to do, is define the setting of the whole project -- so that these stories are set in a common setting.

Keeping stories really short and simple allows the developers to ask the most important details from the customer, so that this information can be noted properly, and communicated and discussed in detail. This is done so that all the developers involved in the development process will have a common reference, and so that the customer and the developers are on the same page.

Creating these stories allows the customer to define the functional requirements from the system without having to burden himself of the technical details. This also allows the developers to isolate the customer from the technical details of a system and focus on delivering the functional requirements.

A well said story is one that usually gets told again. A well understood story is one that doesn't need repeating.

Stories

2006-03-27T04:11:00.001-08:00

It's not rocket science really... It's just about telling someone how your product works. And what better way to do it than with a story?

It's a simple game really of making something (a unit of functionality) sound simple so that the requirement sounds really simple to understand. When it's been understood correctly, it now involves action and careful planning to actually fulfill the requirement.

But what if you're talking to people who do rocket science for a living? What if they already go about it like it's astrophysics or string theory? How do you tell someone who's been doing it for a certain way already to simplify things?

It's actually hard (though it may sound really simple) to get someone to simplify a solution when they're so used to thinking complex terms. It's not really just about simplifying a solution -- simplifying the thinking process and solution formulation process is like trying to unlearn a lot of habits.

I just found out how really complex experienced programmers who have been so used to 'architecture' and 'hack-up' and even 'design-centric' approaches can think. And with the focus on flexibility, speed, and adaptability, the agile methodologies (like XP and TDD) might have to require the rocket scientist to go back to the good old simple arithmetic days. And in this regard, you might (and would most probably) encounter resistance.

Listen to a child when you ask one "how do you think a rocket will get to the moon?" You might get answers like "of course, it will fly!" or "someone in the rocket will push a button and it blasts off into orbit" or even "I dunno, maybe my dad knows...". Okay, the last response is the worst kind of response because it shows a lack of curiosity and a dependency to some "higher power" or "authority" hampering the creative process.

The first response is that of simplicity -- there's an assumption on the functional requirement to reach an end, no matter how that happens. This kind shows the abstract and simplistic reasoning behind problem solving, which sadly gets lost when we start amassing knowledge. People who respond this way are usually people who simplify things as a matter of functional units -- and that's what you want when solving problems. Notice that the response lacked detail, but contained enough information to count as an explanation?

The second response is what you usually get from people who already have a preconceived notion on the solution -- usually these responses come from people who have seen something about the topic, or have read about the topic, or maybe even know a lot about the topic and try to simplify. Usually, children who respond this way already have some idea how a rocket works -- and thus will be more or less boxed in when a new rocket design deviates from his already preconceived notion.

In the next few days, I will be outlining some of the general methods I employ to get people (especially software developers) to simplify the solution to a problem. However, I borrow a lot from the experiences and writings of other authors and colleagues in the field. Some may be new, and others might be prviously proposed/done.

Remember someone telling you a story that dragged on and on that you lose the point?

The simple solution is usually the most elegant solution. Elegance comes at a cost, and usually this elegance leads to considerable value.

Show Me

2006-03-24T04:57:00.000-08:00

Have you ever heard the phrase "talk is cheap, I want to see results"? There has been a lot of changes in the way software is being made and being used at least in the past 10 years -- it used to be about creating the latest and greatest one application that dominated the market; now it's about playing well with others and being invaluably useful alone.

In the nineties, a lot of money went into marketing hype and "sales talk" as well as the dreaded press releases. They're all fair in business, and of course you've got to find a way for your product to get noticed. But is all this effort really worth it? Let me put it another way, let's say you get everyone hyped about a product you're working on -- and then suddenly a competitor comes out with a product that's half as hyped as your product but it sells like hotcakes fast now. What do you think will the impact be on your business' bottom line by the time you release your hyped product?

A lot of developers I've talked to can explain the whole phenomenon as "crap". You usually hear the word crap right after marketing. You want to know why? Because these developers know, that unless they see something then there's nothing. It can't be valuable unless it's tangible (or at least, virtually useful -- which is an oxymoron).

So how do you create value with hype? Business has many terms for this -- market share, name recall, mind share, branding, etc. -- but all these have something in common: they all refer to nothing. Not exactly nothing, but something that exists with the white unicorns in the world -- something you make believe in.

Now that's alright in business, but you have a problem when you have a developer telling you the same things: "you know, the software I wrote will do your thesis, his thesis, and the theses of other people for the next 200 years... but I'm still in alpha" or perhaps "I have forseen these problems and have provisioned solutions already in the software I'm writing... but I'm still working on it". Again, talk is cheap but managers and customers want to see results.

The worst thing that can happen is you hype yourself up for the fall. In this situation, you're only going to be disappointing your customers, if your product isn't up to their expectations. And the worse part even, is when they start saying:

Show Me.

Now wouldn't it be better if you just showed your customer a working product and then asked "what do you think?" Your "marketing" is already part of your value development, and feedback will let you build on whatever value you've already produced. It might be a cool new framework that was derived from your initial product, or a cool new user interface which made things a lot easier. At any rate, your initial product already has value and you can build on that value -- and create even more value -- when you just show the customer something they can gauge.

This is the crux of the agile methodologies: solve the problem now, figure out design later. While you're solving the problem, you might stumble upon great design. Maybe designing the solutions is the solution, so it's a cyclic process that benefits from something: results.

The two words show me can also be used by the developers when they're confronted by a customer that says the product doesn't work. And the only way to show that something works or doesn't, is by testing it.

Now the next time you hear a developer tell you "The system I designed is fool proof, it's not my fault that it's not doing what it's supposed to do...", just ask him:

Show Me.

Something that doesn't do what it's supposed to do, is something that needs fixing.

Rationale

2006-03-23T09:03:00.000-08:00

I have decided to come up with another blog just for keeping tabs on my consulting services for Third World IT Companies. The clients I have aren't really third world companies, but they are companies that are here in the third world.

Now just to set the rationale of the blog, I've written down the bullet points and main issues I will be tackling in this blog. These are the basic assumptions (and constraints as well) that will lay down the foundation of subsequent posts and articles on this blog.

Software Development is not an easy vocation. Creating software is never easy, no matter what field you make software for -- but you can make it fun. Thus, the neverending quest for a way of creating software that is easy, effective, and enjoyable is not a losing battle -- mainly because of the many success stories out there.
Building a business revolving around creating software and providing solutions based on software is dynamic and challenging. The traditional business models of old may not work as well as the more dynamic and adaptive approach of customer oriented businesses. Several industries like the telecommunication industry and the entertainment industry are very dynamic and challenging fields that attract and reward adaptable players. Therefore, the business model (the way of making money) should be dynamic, adaptable, and scalable.
A responsive business satisfies customers. Guaranteeing customer satisfaction is all about being responsive to the customer's needs. This means a business that delivers something is better than a business that delivers nothing. Thus allowing a business to be responsive to the customer's needs will help the business be more competitive.
A business is built around creating value and reaping revenue. A business that doesn't make money is a business that's failing. To make money, you should create value -- and value the business can build upon. Thus, a way of making value fast and effectively will translate to better business.

This blog is about experiences, tidbits, tips and tricks, and bite-size articles about Agility in Business -- especially in the business of making software.

You may email questions, suggestions, stories, and comments to dean [at] orangeandbronze [dot] com -- input is most appreciated.

Dean Michael Berris is a resident C++ consultant for Orange and Bronze Software Labs and is doing software development and project management consulting for a number of IT companies in the Philippines. He pushes Agile Software Development Methodologies along with his partners Calen Martin Legaspi and Butch Ladingin to various IT companies in the Philippines.