Software development methodologies such as the Unified Process are intended to institutionalize a quality process.

Often, the manner in which a quality process is implemented will focus on quality of manufacturing, in order to achieve predictable schedules and outcomes. In the name of predictability, rigorous up front planning attempts to predetermine tasks, resources, and schedules to estimate costs and deliverables. Putting constraints on the size and shape too early will necessarily require gross assumptions about the requirements and design. In so doing, we have traded off quality of innovation to achieve quality of manufacture. We are able to build it within budget and on time, but what we are capable of building is not innovative.

When faced with designing objects, many developers immediately try to identify the abstractions. They have a preconception that the world must be organized as a hierarchy of abstractions. They quickly run into situations that don’t fit nicely into their hierarchy. They struggle to force-fit things into this model. After a while, they ask for help.

The world is not a tree. There may be forests. But generally the world is what it is. Do not force things to be a part of a tree, when that is not what they are. That should be obvious enough. It is always the most obvious things that are difficult to realize, when one is blinded by misconceptions.

Step 1. Identify the concrete types. Start from the objects that will actually exist, not from higher level abstractions. The concretes have definite characteristics that can be analyzed. These characteristics are where the abstractions will be found later through refactoring (unit economy). Concretes are characterized by their behavior. A concrete has roles and responsibilities relative to other objects with which it collaborates. It may be necessary to decompose a concrete into smaller parts that take on those distinct roles and responsibilities (separation of concerns) more clearly.

Step 2. Refactor the characteristics into higher level abstractions. Identify characteristics and behaviors that are common across concretes. Often behaviors can be seen as actions performed on the concretes (rather than actions performed by the concretes), and there is a distinct characteristic that each action depends on. I frequently call such a characteristic a capability, and the action is known as a generic algorithm. As this refactoring proceeds, the generic algorithms and the capabilities are identified as higher level abstractions shared or applied across concretes.

Step 3. Refactor the algorithms and capabilities. As the generic algorithms are implemented against the capabilities of the concretes, the methods will often benefit from factoring out commonality (duplicate code). Parts of the algorithm may benefit from being specialized through pluggable policies. These motivations identify additional abstractions at finer granularities. The finer grained parts of algorithms will likely act upon finer grained capabilities on the concretes. Broader generalizations and narrower specializations appear in the model as a consequence of this iterative approach due to refactoring to reduce complexity (the complexity is refactored to become encapsulated within objects, which are more manageable than messy algorithms).

The resulting model is forests of trees, along with lions and tigers and bears, as well as all the other concretes that exist in the world. Things as they really are. The way it ought to be.

Fear of failure is paralyzing. It is the one source of impotency that can prevent all forward progress. Success becomes the most imperative, when there is a great deal to lose. It is also when it is the most difficult to act towards achieving success.

We should not forget that success is a strategic vision. There will be tactical failures along the way. Often, failures are necessary in order to gain the requisite knowledge that leads to ultimate success. An unwillingness initially to go forth and learn from failure would be the critical error that would make success impossible.

There is a tendency to mitigate risk by introducing rigid processes. If we go through these tried and true steps, then we will have satisfied ourselves that we did everything in our power. As an organization matures, its processes become more rigid and ingrained into the culture. They also become more invasive and fine grained. Every thought and action becomes regimented. The more procedures to follow, the less we rely on individuals to think. When individuals do not think for themselves, innovation ceases.

Institutionalized processes are put in place to ensure that best practices are documented. Organizations talk about continuous improvement and point to the procedures that allow for innovation to feed back into the quality system. We must realize that quality is something that we trade for at the expense of innovation and time to market. A quality system also seeks to prevent failure from occurring. Stifling the opportunity to learn from our mistakes is a fatal error. That can be a detriment to long term success.

This article is a follow-up to cost-value entanglement. Product management is notorious for being risk averse. This often comes from a history of dealing with frequent failures to deliver on time and with quality due to chronic cost-value entanglement. This initial architectural failure cripples a product forever, unless the root cause of the problem is recognized and corrected. Risk aversion grows as the product becomes brittle, and development becomes unwieldy due to ever-increasing code complexity.

Architecture is often thought of as a design function, but this is far from accurate. Use case and requirements analysis are specification activities, which are central to product architecture. It is most important to identify how its users interact with the system and what functions a system performs. These aspects of the system should be encapsulated by its facade, the boundary between the externally visible behavior (interfaces) and its internal implementation. Poor product specification and poor separation between interface and implementation are the architectural manifestations of cost-value entanglement.

This leads to product management demanding a meticulous “evolutionary” approach to development, meaning only small patchwork enhancements are permitted. Significant redesign and technological improvements are impossible, because internal changes will disrupt the externally visible behavior, breaking things for the installed base of users. Such unreasonable constraints can be alleviated by disentangling the facade from the internals. Clearly identify the externally visible concepts in a precise model to support human understanding and interfaces for programmatic access. This enables evolving the facade independently of radical redesigns to the internal implementation. Without this flexibility, revolutionary change is impossible, if quality and time to market are to be maintained.

I have noticed that there are those leaders who blaze the trail. They lead the world kicking and screaming into the frontier. Then, there are those who are willing to follow, but have little to blaze for themselves. Finally, there are those who refuse to follow, but have no skills to lead either.

What do we do, when we want to learn something? Research what others have done before. The vast majority of things can be accomplished without a deep understanding of the problem or solution; it just requires emulating success. Someone else did the hard work of understanding, and documented a procedure and a simplified explanation that others could absorb and reproduce. That is how humans work. To become a successful trailblazer requires a deep understanding for oneself, but also the ability to distill that into simple explanations and instructions that ordinary people can absorb. The sophisticated knowledge will probably be taken to the grave, but it is the idiot’s guide that will endure the ages.

Ordinary men need traditions and procedures to emulate. Otherwise, they wouldn’t be able to even feed themselves. Imagine the vast knowledge and investment in thought that was required to invent the cooking recipes that we know today. Many people would starve today if they were responsible for acquiring that knowledge themselves through creative thinking, rather than by emulation. Traditions, rituals, and procedures bring comfort to us, because it relieves our burden to think. Most people are incapable of advanced contemplation; emulation is their only recourse. It also instills a common denominator in a society’s culture, and it binds them together psychologically.

We grow up emulating our parents and neighbors by decorating a tree, putting up lights, singing songs, exchanging gifts, and gathering with loved ones. There is comfort in these mindless activities, because they are familiar and safe. This manner of blind emulation leaves us vulnerable to less benign inclinations, like smoking or religion. Ritualistic activities give me no comfort at all. Tis the season to celebrate the most basic survival technique: mindless emulation.

With software, the value is in the concepts (model), brand (reputation), people (experience), and customers (installed base). Not so much the design or implementation artifacts. Often we place too much focus on the code, because it is the most tangible manifestation of our investment.

This is a grave error. The code is the result of the sunk cost, but it is not the true value of a software product. The value lies in the capabilities enabled by the software. The software is the means, not the ends. The code must be free to evolve rapidly and radically. If it is not, it will not be able to survive in this ever-changing business environment.

Software organizations that are code-centric have long product release cycles, slow response times to changing requirements, and poor agility to scale the business to expand its market. Code-centric organizations rely heavily upon a skilled development team’s intimate and long-standing relationship to the code. This leaves them vulnerable to competition and employee turnover.

We must be value-centric, not cost-centric in identifying goals. The value is the ends, whereas the costs are the means. The costs (code) must be very flexible, adjusting with agility to the times. The value must be durable. The value must be identified, disentangled from the costs, represented tangibly and separately, and communicated widely. Cost reduction implies discounting investment in code. Decoupling value from code as much as possible allows costs to be reduced without impacting the value.

Complexity grows with code size. Costs grow nonlinearly with code complexity. When you suffer from cost-value entanglement, then code complexity will sink your entire product in time. Entanglement takes away the ability to significantly redesign and reorganize the code, because it puts the value at tremendous risk. Without the freedom to significantly redesign, eventually the product will collapse under the ever-growing weight of its code complexity.