|
|
Building Blocks
2007 Jul 28 I've decided to move my blog over to Blogger. I like the interface, and I suspect that it will be easier for readers (if I have any) to comment. I've taken all of the original Building Blocks postings and placed then into a new book called Building Blocks: The First Fifty Posts that can be downloaded for FREE or printed. The new blog is called The Programmer's Paradox (after the book), and will hopefully continue to explore key issues within software development. If your reading this, download the original postings, or drop by the new site. Comments and feedback are always welcome. Thanks,  2007 Jul 13 A process is a set of rules and a sequence of steps used to guide some work to a final conclusion. It is not, by its very nature, necessarily good or bad; that distinction depends on the specifics of the process. Either explicitly or not, a process has one or more goals that are supposed to be achieved by following the steps and rules. Presumedly, the goals of a process are positive. Things that in some way make sure the conclusion of the process is better than it might have been if the process were not used. For instance the process might be trying to organize the work and insure that it meets a certain level of quality. Or it may try to speed up the work. A process could also have a number of other positive effects that were not intentionally stated, but come as a side-effect for free. A process for quality, might also make the workers feel more secure about their output for instance. This reduces anxiety, and by doing so likely increases the effort put in by the workers.
Along with positive side effects come their opposite. Depending on the process, there can be a number of negative things that come directly from the way a process works. One example could be a process that is so onerous, that it literally sucks the will to get work done from its workers. This is a well known negative side effect associated with an overzealous bureaucracy for example. Things crawl along because there is no other alternative. Morale is dismal and people are cranky. Another example is a process that it is so strict that it forces the workers to find ways to avoid it or find a way around it whenever possible. This type of side effect usually comes with the excuse that it is somehow increasing quality, when in fact its true nature is to penalize official effort. If you can't get the work done within the process, then you either find a way around it, or more likely you just don't do it. Either circumstance is undesirable.
Interestingly enough there are also many examples of negative side effects in processes where the desire is to cross-reference different pieces of information to make sure that all of the work really gets completed. In this type of step there is a dependency on how the initial list of work gets created. The process acts as a multiplier on any work initially submitted. Everything submitted just causes more and more work. The effect is to make it extremely desirable to minimize the amount of initial work that is actually known and covered by the process. Again providing a very strong incentive to go around the process or not do the work in the first place; either way it is causing a huge negative result.
The effects of applying a process are always visible, so that when negative side effects do occur, everyone using the process is eventually aware. Certainly, one aspect of this, is that many people choose to voluntarily wear blinders so as to ignore the negative aspects. A bad process in general causes severe morale problems making it harder and harder to want to participate. If you don't want to leave, then pretending that you don't see the problems is a common option.
Given that processes are just sets of rules and sequences of steps, they are always open to being changed at any point. Adapting the process to fix its glaring flaws is usually the initial direction that most people take when they first implement a new process. Counter to this, over time, the rules in the process multiply and morph into some else, such that the process becomes less and less positive. The negative aspects increasingly cancel out the positive ones. As well, it is easy for well meaning people to enhance the negative aspects accidentally without fully understanding the consequences of their changes. Overall, a process is never really stable, and there should always be some ongoing work to analyze and improve it. Over time all processes degenerate.
Recently I have had the chance to closely observe two very different software development processes at work. One is on the far right — a heavyweight process concerned about producing quality, and the other is on the far left — a lightweight process concerned about getting functionality out quickly to the clients. Retrospectively, both processes are extreme examples in their specific categories. Interestingly enough, both of them are broken due to excessive negative side effects; although it is very different problems they are similar in many ways.
The heavyweight process was intended to insure a very high quality output, but because it is so inflexible and rigid, examples of it working have shown that the actual quality of the output is abysmally low. Barely passable, and often not. The key problem is that the process is so strict and rigid, that there is a huge incentive to avoid it. The process then doesn't insure good work, but it does act to throttle any existing work and insure that any cleanup work that should have been done is definitely "not" done. Software gets messy and rusts over time, so cleanup isn't an optional part of the process it is mandatory. This particular process is a variation on many of the large heavyweight processes that are common in the larger development shops in the software industry. Generally they all suffer from serious defects because they reward counter-productive behavior; while fixing things and cleaning up poor work are highly penalized. This process in particular is exceptionally strict and is far worse than a normal bad example. Because of this, it is easy to observe how that increased rigidity is the cause of the marginal quality. It doesn't need an objective study when the process is followed so diligently but the output is so obviously substandard.
The lightweight process does not attach enough significance to finding and correcting problems before releases, so consequently the quality of the output is low. While it is lower than most of its heavyweight cousins, it is surprisingly not that much lower than a marginal result, showing that intensive testing follows the law of diminishing returns. No testing is not that much worse than lots of strict, yet poor testing. The point of the process was to get out code quickly, but the low quality of the releases cause enough serious support problems that quickly eat up the resources. Ultimately the effect from this is that the functionality is not being released in a timely manner. In fact the delay between changes is nearly as long as a heavyweight process, mostly owning to the necessity to wait between releases for the furor from the last release to die down. Ultimately then, while it certainly is less work, and less rigorous, this example of a lightweight process is no better or worse than the heavyweight one, with the exception of allowing more bugs to be released.
Regardless of what people say, process is always within our scope to modify and fix. It is the people who use and rely on the process who should be interested in trying to fix it. If everyone pulls a 'not my problem', then the process gets locked in by external parties. If they pull together, then while it may be slow in some organizations, pressure can be brought to bear that the only answer is to correct the obvious problems. It may take a while, but any process can be changed, and every process can be fixed. There is no magic to it, but you do need to be realistic about the side-effects in order to properly evaluate their contributions. The goal, of course is a good working process that does what it intends and it does so with a minimal amount of negative side effects. For software, at either end of the extreme, left or right, the processes quickly run into trouble and fail to deliver. We have enough evidence by now that shows that my above observations are quite common in the software industry. There are always trade offs to be made, but that is not the same as ignoring the negative side effects. A balanced process — somewhere between light and heavy weight — is likely the only one that is well rounded enough to deliver on its promises. But irregardless of the starting point, if they are not carefully maintained all processes degenerate into ineffective swamps. That much is beyond certain.
2007 Jul 04 I suppose you should forgive a few flaws. Sometimes, particularly with high technology we need to show some patience. Sometimes not. The general assumption with software flaws is that they come from some part of the process were the developers missed noticing that there was a problem. A large number of inconsistencies or bugs then is an indication that their development process is not working. A good well-rounded process wouldn't perpetuate the problems. It would fix them rapidly, and by definition should prevent many new ones from appearing. After all, that is the whole point of having up'ed the complexity and work by applying a process to development in the first place. The process must pay for itself by helping to ensure that the quality of the results is better, either faster or better quality, but hopefully better quality. Without that, following the process is just a pointless waste of energy.
So, if you were to look closely at the output of a development project, and in that you were to see that there was a consistently high number of bugs not being caught, and continuing to exist, then you can infer that the process is absolutely under-performing. More importantly, if there is essentially nothing to 'tweak' in the process to change the results, then you can be sure that the problem lies with the process itself. The process is ineffective, or perhaps even useless.
At this point, there would be many that could easily point to the people in the project, and blame them. "It is not the process, it is just that specific team" might be the defense. However, if the people — no matter how lazy and undereducated — follow the process, according to the rules of the process itself, then while the people may not be ones you would like to employ, the fault still lies with the process. A good process must ensure some type of minimum result, no matter who is involved. If they do not follow the process, then you can be less sure of the cause of the problems. But, mostly the process should have some way of validating itself, so developers continuing the disregard the process can in many ways be attributed as a problem to the process.
There is a great deal of information to be learned from a web site. If you use the site frequently, you get a real sense of its underlying flaws. How stable is it? Are there lots of weird links? Are the functions consistent? Do people ask a lot of different strange support questions? You can easily tell whether the foundations are stable, or if they have severe problems. You do need to differentiate between coding and operational problems. Is the code weird, but the administrators are working around it. Do the bugs halt the system or are they just inconvenient. When tied to an organization, you can also find out about the underlying development philosophies, particularly if you look at the company want ads or job descriptions. Often programmer job ads are very specific about the processes used for development. You can determine most of the technologies and any of the major techniques used for building their systems.
Mixing these two pieces of information can give you an interesting example system for a given methodology. What you can't tell is if the developers followed — correctly or not — the process, but you do end up with some sense of whether or not the development itself has problems. Beyond the base functionality, what becomes important is that sense of how well things are going over a period of time. Has the development been getting better, or worse? Do the old problems ever get fixed? Are the developers overwhelmed by adding functionality or have they kept pace with the technological challenges?
When we sense that a site is degenerating, we must come to understand that there is a mass amount of code involved. Over time that builds up and gets worse, it is inevitable. That momentum should be considered, but again the real issue is whether or not the development process is working. A relatively slow development, with a large number of reoccurring annoying problems and inconsistencies tends toward showing that the development process is severely broken and needs to be fixed.
We can forgive an off-day, or a few flaws, but I find it hard to be patience with an organization that sticks to a bad process that is repeatedly causing problems. I also find it hard to be patience with people that turn away from their own examples of a bad process. Things work, so you leave them, but if not you should choose to fix the problem. Not fixing defects with anything other than spin is a flawed approach. Sticking to a visibly broken process in light of enough public evidence that it is not working is tad amount to burying one's head in the sand to avoid any dangers.
What makes software so frustrating, is the sheer amount of publicly available proof that our development processes are dysfunctional. The web is the perfect example. The bigger the web gets, the more ASPs that come to market with big flaws in their systems. Yes, we do learn to live with the flaws, and even manage to work effectively around them, but in so many examples the flaws themselves just are not an acceptable part of the solution. They don't need to exist, and if the underlying process weren't broken, not only would then not exist, but the cost of building the software would be less as well. But, people being who they are, we so often choose to stick with the familiar process even when the results aren't impressive, or even obviously bad. So now — lucky us — we are surrounded by lots of interesting applications, but trusting them to work correctly is a problem. And, investing lots of time into them is wasted, as the likelihood that that effort is supported, even 4 years from now is slim at best. The technology that we built to leverage our abilities instead burns through our resources. Those small software flaws are the symptoms of much larger problems.
|
|
Download for Free
Download for Free
