Category Archives: innovation

notes on quality process

Software development methodologies such as the Unified Process are intended to institutionalize a quality process.

Often, the manner in which a quality process is implemented will focus on quality of manufacturing, in order to achieve predictable schedules and outcomes. In the name of predictability, rigorous up front planning attempts to predetermine tasks, resources, and schedules to estimate costs and deliverables. Putting constraints on the size and shape too early will necessarily require gross assumptions about the requirements and design. In so doing, we have traded off quality of innovation to achieve quality of manufacture. We are able to build it within budget and on time, but what we are capable of building is not innovative.

follow the rules

I have an engineer who works for me. His responsibilities include ensuring design consistency and best practices across the components in the system. We establish design guidelines and rules for the team to apply in their work, and he performs the grunt work of reviewing and editing everyone’s designs. He is well suited to the role, because he is very methodical, and he has an affinity to documenting and following rules.

Engineering can be divided into two different modes of operation: manufacturing and invention.

I use the term manufacturing in reference to ordinary development. That is, designing based on a strategy of incremental improvement over past designs. This strategy has served the Japanese well in automobiles and consumer electronics. A disciplined approach to engineering with focus on continuous improvement and attention to market forces is the key to success in manufacturing. Manufacturing is about quality of implementation. Manufacturing is about applying best practices and quality process.

Invention is completely different. Invention is not about responding to market demand. It is about using imagination to create new markets, where they never existed before. Invention requires innovative thinking to formulate new ideas and different (hopefully better) practices. Invention is not about applying today’s best practices and rules to incrementally improve. Invention requires the understanding that today’s best is flawed and impeding progress towards a superior possibility that may be highly risky, unproven, and difficult to achieve – but worthwhile to pursue, because it would be revolutionary.

A business needs disciplined engineers, who are skilled at manufacturing. This is where the money is.

My grunt is definitely a manufacturer, not an inventor. He has mediocre design creativity, because he is unable to intuit across a mass of contradictory information and conflicting motivations (resolving the forces), while paying attention to detail. He needs rules to govern his thinking and everyone else’s. Without rules, he is unable to function. He has no intuitive understanding of good design principles. He follows the rules.

An inventor (and every good designer) does not follow the rules. He does not break the rules either. He defines the rules that are appropriate, and he uses them to aid his thinking. Above all, he uses his brain to produce good designs, given the facts in evidence. He leads, where few have the courage to follow.


I am currently reading Three Roads to Quantum Gravity. I have just completed the first part of the book, which is a lengthy introduction. The author felt a strong need to prepare the reader’s mind for thinking about the universe differently from whatever misguided education and personal experience was filling his head.

Perhaps if I had not already read The End of Time: The Next Revolution in Physics, I would have needed this preparation. However, being already convinced that space and time are invalid concepts for properly describing reality, I felt perfectly at home with the author’s perspective.

Something I was unprepared for was the realization that objectivist epistemology may be overly constrained by its man-qua-man perspective. Scientists, who practice the Scientific Method, impose constraints on their thinking by considering themselves an objective observer. Those who do this cannot effectively practice the science of cosmology, because no such artificial (mystical) separation can be established between observer and universe.

Only man-qua-man can allow himself to be so easily fooled into thinking the world is as he himself sees it, and be convinced that this is an accurate view of reality. Only man-qua-ordinary-man would so easily accept space, entities, and their attributes to represent the true state of things, while changes in state are viewed as snapshots in time. Most men still believe that reality is organized into a three dimensional grid representing space, and an extra dimension of time. Even their inability to sense these alleged characteristics does not bother them. Ask them to show you space and time one day, and see how far they get trying to use existents to describe non-existents.

Man-qua-ordinary-man is in fact a mere fool. In order to understand cosmology, we must become man-qua-universe and think likewise.

[It might seem strange that a software architect would spend so much effort on cosmology. Cosmology and software architecture are actually similar. Cosmology and software architecture are both about studying the universe. Just as every ordinary cosmologist has failed to gain a proper understanding of the universe, by being unable to accept that he himself cannot be separated from it as objective observer, ordinary software architects are in the same bind.]

forest for the trees

When faced with designing objects, many developers immediately try to identify the abstractions. They have a preconception that the world must be organized as a hierarchy of abstractions. They quickly run into situations that don’t fit nicely into their hierarchy. They struggle to force-fit things into this model. After a while, they ask for help.

The world is not a tree. There may be forests. But generally the world is what it is. Do not force things to be a part of a tree, when that is not what they are. That should be obvious enough. It is always the most obvious things that are difficult to realize, when one is blinded by misconceptions.

Step 1. Identify the concrete types. Start from the objects that will actually exist, not from higher level abstractions. The concretes have definite characteristics that can be analyzed. These characteristics are where the abstractions will be found later through refactoring (unit economy). Concretes are characterized by their behavior. A concrete has roles and responsibilities relative to other objects with which it collaborates. It may be necessary to decompose a concrete into smaller parts that take on those distinct roles and responsibilities (separation of concerns) more clearly.

Step 2. Refactor the characteristics into higher level abstractions. Identify characteristics and behaviors that are common across concretes. Often behaviors can be seen as actions performed on the concretes (rather than actions performed by the concretes), and there is a distinct characteristic that each action depends on. I frequently call such a characteristic a capability, and the action is known as a generic algorithm. As this refactoring proceeds, the generic algorithms and the capabilities are identified as higher level abstractions shared or applied across concretes.

Step 3. Refactor the algorithms and capabilities. As the generic algorithms are implemented against the capabilities of the concretes, the methods will often benefit from factoring out commonality (duplicate code). Parts of the algorithm may benefit from being specialized through pluggable policies. These motivations identify additional abstractions at finer granularities. The finer grained parts of algorithms will likely act upon finer grained capabilities on the concretes. Broader generalizations and narrower specializations appear in the model as a consequence of this iterative approach due to refactoring to reduce complexity (the complexity is refactored to become encapsulated within objects, which are more manageable than messy algorithms).

The resulting model is forests of trees, along with lions and tigers and bears, as well as all the other concretes that exist in the world. Things as they really are. The way it ought to be.


business idea for enterprise collaboration

(Continued from 2002-07-08.)

One of the biggest impediments to productivity is the lack of tools to facilitate collaboration. This was not as noticeable, when workers tended to be collocated with those they needed to collaborate. Physical collocation makes less sense today, when globalization requires companies to be distributed throughout the world. Expertise is becoming so specialized that it is impossible to develop a broad range of skills in house and locally, so this creates a need to outsource and recruit teleworkers. Email, telephony, instant messaging, and intranet Web sites only go so far to facilitate communication with these remote sites. These technologies need to be integrated into business processes to facilitate wide scale real-time collaboration.

In a software development organization, there are very specific requirements that are unmet by today’s information technology. Software development tools tend to focus either on the relationship between individuals and code artifacts (e.g., integrated development environment) or the relationship between the code artifacts and the quality assurance processes for the software product (e.g., software configuration management). Over the past decade there has been a greater focus on analysis and design tools (e.g., UML modeling). However, modeling is a very small part of analysis and design. The majority of analysis involves the capture of requirements and use cases in the form of natural language (e.g., English) descriptions. This is a very iterative and interactive process among experts and analysts, and the information flows into design. Design is a very creative process among analysts, architects, and programmers in various areas of expertise (e.g., user interface, application server, database). The collaboration tools support for analysis and design are wholly inadequate today.

The software development community needs a Web based collaboration system for analysis and design. In the analysis space, Rational Requisite Web almost completely misses the mark for a requirements database. In the design space, there is nothing to facilitate the whiteboard-style interactions that are necessary during the creative process. Such interactions work toward evaluating design options and formally capturing the final design decisions. UML modeling is merely a visualization technique. UML diagrams are not in a format that is appropriate for communication between all stakeholders. Ultimately documents must be produced for the software architecture and the design of each subsystem. Collaboration tools are required for producing these artifacts as a team.

I have looked at using a Wiki to facilitate collaborative content creation. This is not a bad mechanism for authoring, reviewing, and revising content. However, it is wholly inadequate for taking that content and producing documents for publishing. Moreover, a Wiki alone is insufficient for handling all the various kinds of content that is needed for analysis and design (e.g., models, diagrams, code, user interface prototypes), and for managing the workflow.

I believe that this is a tremendous business opportunity. There is clearly a need for this kind of collaboration tool for software development. One can imagine this need extending into other engineering activities, and perhaps even into other disciplines (e.g., sales and marketing) within the enterprise. This is likely a $100M idea with a market window that is likely to remain open for quite some time. If I had the guts, I would probably pursue this one as my baby. I’ve got the perfect skillset and experience to make this happen.

language design 101

Language provides us with the framework with which we express ideas. Numbers, arithmetic, and calculus are the language of mathematics. C, C++, and Java are languages for computer programming. This is only partly true, as Guy Steele Jr. explains in Growing a Language. This perspective on language design extends far beyond programming.

I will focus on software related language design. It is an exercise for the reader to extrapolate how the same principles apply to literature, music, and other forms of human expression.

Just as Java is the most basic vocabulary and grammar with which more expressive languages can be designed, English is similar. The language required to produce a book is far more advanced than English vocabulary and grammar. Literary devices, plot structure, and character development are patterns that are designed into the language of literature. Similarly, design patterns make Java a useful language for programming.

The language of object-oriented analysis and design (OOAD) is the Unified Modeling Language (UML). However, UML is not very expressive by itself. It was not designed to be as precise as programming languages, because it was motivated by the needs of conceptual modeling. They thought that conceptual modeling is not precise. That is the tragedy of imprecise thinkers.

Now, they have caught on to the idea of using models to generate code. This requires precision; a machine only understands 1 and 0. There is no room for ambiguity. It is now a huge challenge to apply UML to this model-driven architecture (MDA).

We are constantly grasping for ways to improve how we express ideas. A thousand words are better expressed with a picture. The concepts represented by the geometric shapes are better expressed with tags. Patterns are recognized as we organize those concepts into more advanced ones. As a consequence, many thousands of instructions are captured in a blueprint with a small number of symbols. At each step, we seek to be more expressive with compact representations. This is language design.

change is destruction

I am reading The New New Thing. It is more captivating than I anticipated. It was given to me as recommended reading, but I had no idea what it was about. It is a biography of Jim Clark, the founder and inspiration behind SGI, Netscape, and the present commercialized dot com era of the Internet.

One characteristic of Clark’s appeals to me. He paid no respect for history. His only care is towards how to invent the future. The key word is “invent”, as opposed to predict. People who work at prognostication are merely an audience. Innovators aren’t just interested in looking forward, but in engineering it into existence.

I have noted before the difference between builders and breakers. However, creators are also destroyers in a sense. The reason for creating something new is to impose change upon that which is the source of dissatisfaction. The intent is to obliterate something, as it exists today, in order to replace it by something better. Engineers love to destroy things as we know it, and substitute a new world order that matches our vision. The great destroyer is change, and this weapon is wielded by original thinkers. They have no respect for authority. They show no reverence for what came before them.

The opposing force to change is the people trying to hold onto what they have, desperately fearing that the shape of their environment is being altered beyond their control. A person’s reaction to innovation is revealing. It shows one’s appreciation for creation, the power to destroy and reinvent. Or opposition, if that is the case. What do you call “visionaries” who are unable to recognize the future, even when it is put in front of them? Tourists are there to see things that others have built. I have no interest in seeing the world. I want to rebuild it.

road to utopia

into the fast lane

In the long term, I still firmly believe that it is modestly priced, high volume consumer services like telephony, television, video on demand, music, gaming, and e-commerce that will be the Utopian goal. There are many obstacles that must be overcome first: political reforms (regulation, laws, licenses, spectrum), radical changes to content provider business models, network infrastructure investments (fiber, packet radio, and IPv6), cost reductions to high end gear (optical CPE routers, IP enabled home theaters), and a critical mass of adopters. There are too many factors to accelerate artificially; it must happen as a natural progression.

Consumers are notoriously frugal. They generally purchase the minimum to satisfy their needs. Anything beyond what is good enough becomes a luxury. To succeed in this market, it is crucial to provide services perceived to be essential rather than merely luxurious. Without that economy of scale, a provider cannot recuperate the cost of building out the infrastructure to neighborhoods.

In the meantime, there are high value services that can address immediate needs. Businesses are always in search of productivity gains and improving efficiencies.

  1. Collaboration – With enterprises extending their global reach, workers in remote offices will need to collaborate more effectively. Video-conferencing, remote white boarding, groupware, and other real-time collaborative work tools will significantly reduce the need for in-person meetings. Expensive, time-consuming travel will become largely unnecessary.
  2. Teleworking – Urban traffic congestion, costly office facilities, and a diversely distributed global workforce are all forces that will encourage greater use of telework. Effective online collaboration and improved broadband access will enable workers, who want the convenience of working from home, the ability to do so without any compromise in productivity.
  3. Business Integration – This is already underway. Supply chain management and other forms of business-to-business (B2B) integration are improving the ability to outsource. Concentrating on core competencies implies contracting out non-core work to business partners, who do consider that work to be their core competency.

Enterprises are willing to invest in incremental improvements to business efficiencies, because there are measurable returns. The capital markets exert great pressure for such improvements to be demonstrated each and every fiscal quarter. This market can be penetrated without having to incur sizable short-term losses, unlike the pricing that is necessary to penetrate the consumer market. Whereas information technology is often considered a luxury to consumers, it is clearly essential to many businesses.

The productivity gains at work become catalysts for workers desiring similar lifestyle improvements at home. Online collaboration with remote coworkers leads to online collaboration with family and friends. Telework leads to home workers applying the technology already installed towards personal entertainment. General availability of the technology tends to stimulate new opportunities as users discover new ways to apply it. It is this manner of cultivating markets that will lead us to the end goal.

success and failure

Fear of failure is paralyzing. It is the one source of impotency that can prevent all forward progress. Success becomes the most imperative, when there is a great deal to lose. It is also when it is the most difficult to act towards achieving success.

We should not forget that success is a strategic vision. There will be tactical failures along the way. Often, failures are necessary in order to gain the requisite knowledge that leads to ultimate success. An unwillingness initially to go forth and learn from failure would be the critical error that would make success impossible.

There is a tendency to mitigate risk by introducing rigid processes. If we go through these tried and true steps, then we will have satisfied ourselves that we did everything in our power. As an organization matures, its processes become more rigid and ingrained into the culture. They also become more invasive and fine grained. Every thought and action becomes regimented. The more procedures to follow, the less we rely on individuals to think. When individuals do not think for themselves, innovation ceases.

Institutionalized processes are put in place to ensure that best practices are documented. Organizations talk about continuous improvement and point to the procedures that allow for innovation to feed back into the quality system. We must realize that quality is something that we trade for at the expense of innovation and time to market. A quality system also seeks to prevent failure from occurring. Stifling the opportunity to learn from our mistakes is a fatal error. That can be a detriment to long term success.

revolutionary in an evolutionary world

This is a follow-up to cost-value entanglement.

Product management is notorious for being risk averse. This often comes from a history of dealing with frequent failures to deliver on time and with quality due to chronic cost-value entanglement. This initial architectural failure cripples a product forever, unless the root cause of the problem is recognized and corrected. Risk aversion grows as the product becomes brittle, and development becomes unwieldy due to ever-increasing code complexity.

Architecture is often thought of as a design function, but this is far from accurate. Use case and requirements analysis are specification activities, which are central to product architecture. It is most important to identify how its users interact with the system and what functions a system performs. These aspects of the system should be encapsulated by its facade, the boundary between the externally visible behavior (interfaces) and its internal implementation. Poor product specification and poor separation between interface and implementation are the architectural manifestations of cost-value entanglement.

This leads to product management demanding a meticulous “evolutionary” approach to development, meaning only small patchwork enhancements are permitted. Significant redesign and technological improvements are impossible, because internal changes will disrupt the externally visible behavior, breaking things for the installed base of users. Such unreasonable constraints can be alleviated by disentangling the facade from the internals. Clearly identify the externally visible concepts in a precise model to support human understanding and interfaces for programmatic access. This enables evolving the facade independently of radical redesigns to the internal implementation. Without this flexibility, revolutionary change is impossible, if quality and time to market are to be maintained.