In my previous article, maturity as the enemy of innovation, I identified the destructive forces of organizational maturity. As an engineer, I cannot allow a problem to go unsolved.

To stem the tide of chronic risk aversion, we must imprint the entrepreneurial spirit into the culture as a core value. The courage to initiate calculated risk-taking needs to be admired and rewarded. The stigma associated with failure must be removed by appreciating the knowledge gained through lessons learned.

The cost of research and experimentation must be reduced. The pace of development must be rapid. These are achieved by keeping the team size small. The most risky and unproven ideas should be attacked by an individual, not a team. Functional teams need alignment on direction and a degree of agreement. Revolutionary advancement is frequently the result of radical thinking. New directions, different techniques, and breaking well-established rules are unpopular, especially to those with emotional investments in past innovations. There are fewer disagreements in a small team, and none at all in a team of one.

Uniformity of thinking leads to stagnation. Diversity must be encouraged, and this is promoted through a free market for ideas. Organizations that rely on a command-and-control style of management to set direction eliminate the competition that nurtures a diversity of ideas. Management should empower technical decision-making to be done bottom-up rather than top-down.

Once new ideas have been adequately proven, there must be ways of teaching them to others and incorporating them into product. Product development is often done sequentially, one release at a time. The immediate release focuses on short term commitments, and the delivery is usually constrained by schedules and costs that can ill afford risk. Riskier development must be allowed to proceed concurrently on longer-term schedules, without overly constraining the process. Developments from the unstable branch should be merged into the main product release, as they become ready. This provides an environment to accommodate both risky development and risk-averse development.

The above techniques can be effective in remedying the organizational impediments to innovation. However, there are also commercial impediments that must be overcome. That will be the topic of my next article.

As a software development organization matures, its culture institutionalizes practices that promote quality. Maturity is for quality. The most obvious measures of quality are in terms of defects. The quest for lower defect counts is a noble one.

At the same time, complexity grows as the weight of larger feature sets and lofty requirements are incorporated into the software. With increasing complexity comes a larger development team. A larger team means more division of labor, higher cost of management overhead, and more careful planning to be able to utilize those resources. Division of labor requires the delegation of distinct responsibilities to specialized roles. Decision-making becomes more dispersed and distributed, and a decision will have wider impacts on team members, so good communication becomes a critical success factor.

Unfortunately, decision-making, communication, and the distribution of responsibility are never perfect. The more complex the software, the larger the team, the more distributed the responsibilities, the more specialized the roles, the higher the cost of development. This is not a scalable organizational model. Once a software organization has reached this stage of maturity, it is doomed by inefficiency. Creative individuals are no longer empowered to quickly realize their ideas. Two key requirements to motivate and nurture a creative mind are independence and freedom. The degree to which these environmental factors are blunted is the degree to which innovation is impeded.

Because decision-making is distributed among many people, it becomes impossible for innovative ideas to be realized without winning consensus. Innovative ideas are unconventional ideas. A novel approach often needs to break well-established rules and go against practices that have worked in the past. Innovation is destructive to the status quo. It must be disruptive in order to rise and overtake older practices. Innovation is the product of experimentation and risk-taking, usually resulting in failure and rework. Without an environment that encourages this, there will be no innovation.

The quest for quality through institutionalized practices and disciplined management are in many ways counter to the independence and freedom that motivate a creative mind. Careful project management and planning discourages experimentation with risky new ideas (technical, organizational, and commercial), where many unknowns make results difficult to predict. Risky ideas need experimentation so that many failed attempts can be discarded quickly, allowing successful ideas to be taken to production. Mature organizations respond to higher risk by applying stronger management practices for mitigation, thereby increasing the cost and slowing the pace of failure towards success. In fact, once management smells failure, they immediately work to eliminate it, thereby destroying its very value as a vehicle to learn what will be needed to succeed. This is how many mature organizations have killed off the entrepreneurial spirit. This is the reason why most innovative ideas come from small groups of individuals working in a garage or a small start-up company.

We spend an extraordinary amount of time planning our development, because it is so costly to get any amount of code written. Developers are asked to give estimates. They demand to know what is in scope down to the individual operation. It’s crazy how much design work and detail needs to be thought through before reaching that point. Then, they estimate maybe 10-20hr of work per method. Even a tiny piece of functionality results in estimates in several person-weeks of effort. This is how software professionals get their jobs done.

Where have all the hackers gone?

I came up with a cool idea on the weekend. I decided to code it sunday. I spent about four hours in total coding the entire thing, analyzing and designing, as I went. It turned to be over 2000 lines, which I proceeded to unit test. I put in another three hours tonight debugging a rather obscure problem (actually wasting 2.5 hours chasing a non-existent bug due to my own stupidity), which I discovered in my sleep the other night.

I simply cannot understand why experienced software professionals cannot hack out 3000 lines of unit tested code on a good day, and average 1000 lines of unit tested code per working day, with low bug counts.

I’m almost glad to see that many of North America’s grunt coding jobs are being outsourced to India. The cost of our software professionals in combination with their low productivity and high maintenance attitude is unbearable.

Software development methodologies such as the Unified Process are intended to institutionalize a quality process.

Often, the manner in which a quality process is implemented will focus on quality of manufacturing, in order to achieve predictable schedules and outcomes. In the name of predictability, rigorous up front planning attempts to predetermine tasks, resources, and schedules to estimate costs and deliverables. Putting constraints on the size and shape too early will necessarily require gross assumptions about the requirements and design. In so doing, we have traded off quality of innovation to achieve quality of manufacture. We are able to build it within budget and on time, but what we are capable of building is not innovative.