Creativity is stifled by a manufacturing process. Project managers perceive the development process to begin by identifying and analyzing requirements in full, until they are perfect for feeding to designers. Then, designers will satisfy those requirements by detailing designs, until they are perfect for coding. Then, code will be written and tested, until all the pieces can be integrated and QA tested. No matter how often this process fails to deliver innovative solutions, software companies continue to blame the lack of compliance to the process, rather than the process itself.

Modern software methodology suggests that the disciplines operate concurrently. Requirements anaysis, design, coding, integration, and testing all happening in parallel and continuously. The process is iterative, attacking problems incrementally, and refining rapidly with every cycle. Project managers claim to support this approach, and yet they continue to define gargantuan iterations (several months in duration) with entire sets of features fully implemented. Within each iteration, they plan with distinct milestones for completing of requirements, design, coding, unit testing, integration, and QA.

This pattern seems almost inevitable, when the project team is large (>20) and composed of individuals, who are highly specialized in their discipline. Designers have expectations of receiving good requirements from the analyst. Coders have expectations of receiving good designs, if they were not the designer. Testers have expectations of receiving good code. Unfortunately, the software assembly line does not produce good software, because development is a creative process, not a manufacturing process.

You do not produce good software based on dreaming up a perfect design. You need to start with a great deal of unknowns, some gross assumptions (likely overly naive), and a poor understanding of the problem. Proceed with the knowledge that you will make mistakes, and these will facilitate learning how to solve the problem better. Build it quickly and screw up several times, improving with every attempt. Eventually, the problem will be understood, and it will be adequately solved by the solution at hand, despite its imperfections. Project managers do not understand how to plan for a process that involves spinning around cycles of two steps forward and one step back. They think that an assembly line is the model of efficiency, and iterations are too chaotic and unpredictable, because of the constant rework. A software assembly line is certainly orderly and predictable. It predictably produces little innovation at a very high cost and over a long duration.

There is a real difference between those, who develop software as a job, and those, who do it for the love of it. It is easy to distinguish between them. The former believes that only straightforward problems can be solved in a timely manner, and the less to be done, the better. The latter believes that straightforward problems are mundane and tedious, and hopes to spend time on more challenging problems. True hackers are a rare breed.

The biggest difficulties in software are non-technical. They are psychological. When a group of people have produced a body of work, they become emotionally invested in it. They feel that they know something. It becomes their anchor to reality, and every future possibility is considered in relation to that knowledge. With a sense of security comes malaise. It impedes the child-like motivations to reach out to the unknown and discover. Legacy knowledge within accomplished veterans becomes toxic to the culture, because the perceived value of legacy knowledge eclipses the potential value of further discovery, which has more risk attached. Cultural attachment to legacy kills innovation. It blunts the entrepreneurial spirit.

A true hacker does not give a rat’s ass about what is established by what came before. Everything is open to destruction and rebirth. The risk of expending effort towards a failed attempt is irrelevant, because a true hacker understands that every failure is a positive step towards ultimate success.

part 2 of infinity – reuse and replacement

(Continued from part 1 – introduction.) Organizations frequently become enamored with building software infrastructure and frameworks for reuse. They see a great deal of common problems being solved again and again in different ways by different people. They see duplication of effort as a business inefficiency that can be optimized by factoring the commonality into reusable components, so that wheels are not reinvented. Centralized control through corporate standards is another popular activity.

One sign of successful reuse is the proliferation of users. With each new user there is a new dependency. The dependency is manageable so long as the requirements are stable. If new requirements must be satisfied, as is typical of rapidly evolving technology, these dependencies become a serious impediment to productivity.

A fundamental enabler of component reuse is the software contract. The interface that isolates the underlying implementation details from its users must remain stable to allow the two to evolve independently. However, changing requirements often result in interface changes. Any change to the software contract will break its users. As the number of users grows, the impact of this breakage becomes more severe. Thus, the desire to scale software functionality linearly with development effort is not viable because of changing requirements.

Only with perfect foresight can we avoid changing requirements. If we could employ prescience so that all new requirements resulted in wholly new software components, leaving existing software components in tact, we would have a predictable and scalable development process. Unfortunately, human fallibility prevents us from having good visibility into the future, and our software designs suffer.

We must be honest and recognize that software, no matter how well designed, has limited ability to evolve to meet new and changing requirements over time. We must build into our development processes the practices that can accommodate the retirement and eventual replacement of components that have collapsed under the weight of their outdated designs. A fatal flaw in many development organizations is the denial that software can grow old and die. This flawed thinking allows younger and healthier software from the competition to eventually win out.

Therefore, any reuse strategy must anticipate this component lifecycle to ensure that the entire ecosystem of dependent users does not die a horrible death tied to aging components. A reuse strategy without a recurring replacement strategy is a path of certain destruction.

Another thing to recognize is the distinction between infrastructure versus application code. The view that some components are foundational, while others are fundamentally not, is short-sighted. The entire premise of reuse is that software should be built to enable an evolution towards building more functionality using the components already constructed. In effect, every component eventually becomes infrastructure for other components, which are often not anticipated.

part 1 of infinity – introduction

One of the strongest motivations that is espoused for promoting object oriented programming is reuse. While object oriented programming languages provide features, such as encapsulation and inheritance, that enable better reuse, OOP has gotten a bad rap, because the discipline that is required to produce good software must address many concerns aside from reuse in order to be successful.

Reuse is a buzzword software professionals bandy about, when they promote certain software practices. It refers to the ability to incrementally grow software capability by building additional functions to augment what was already built previously. Development organizations like this idea, because it implies predictability, which is the holy grail of software methodology (quality process). If the additional effort required to build and test functionality can be made proportional only to the additional code being incrementally developed, then development can be highly predictable. This would be the key to projects being delivered on time and within budget.

Although reuse in this sense is a good motivation, nobody knows how to make it work. Very intelligent and highly respected individuals will say how to apply principles of reuse, but either they don’t know what they are talking about, or the people who apply their principles don’t understand what they should be doing. Or both. The state of the software industry is a testament to the fact that reuse strategies are being widely applied with a high degree of (and near total) failure. No one achieves anywhere close to the linearity of effort, predictability of process, and scalability of outcome that reuse is meant to achieve.

In an series of articles, of which this is the first, I would like to evaluate various reuse strategies, practices, and object oriented programming techniques. Stay tuned for more in this series.