Category Archives: innovation

Enterprise Collaboration – how we work

Over the past few years we have seen Web 2.0 technologies being established in our personal and professional lives. Social networking tools are being used to keep us connected with friends, family, and colleagues across space and time. In business, technology has ignited a new era in collaboration and social interaction to bring customers closer to those who market, sell, and support their products and services. But we have still not seen this bring about a new era in collaboration among workers within the enterprise.

In January 2011, Tomi Ahonen wrote the Fortune 500 CEO Guide to Mobile, which brought to my attention the on-going technology revolution that we are currently experiencing—the steady move to mobile. It was not just the speed and scale of adoption that is impressive (and it is extremely impressive). It is the way that mobile is permeating every aspect of our personal and professional lives. It is the way that it is transforming almost everything that we do, and the tools that we use to do them.

However, I still believe that we are only seeing this transformation in its infancy. It has started with the adoption of platform technologies (mobile, Internet, Web 2.0), but we have yet to see a cultural transformation to accompany the technological revolution at the scale we are seeing. We have yet to see Enterprise 2.0 yield gains in productivity at work that are comparable to the improvements in our interpersonal relationships on Facebook, Google+, and LinkedIn. Blogs, Wikis, Instant Messaging, and the gradual enabling of tools to become more social are adding incremental value to our business processes, but they are not fundamentally changing the way we do business.

We still drive to the same office and largely interact with our colleagues face-to-face or by audio conference and with web conferencing (and accompanying presentation tools) as a visual aid. The manner in which we facilitate work between remote participants remains awkward and inconvenient. Awkward and inconvenient in that audio and video conferencing provide a very poor user experience in the following ways.

  • Only a single person is able to speak at once. Interruptions are inevitable. Often the individuals who speak the most do not yield sufficient time for others to contribute their ideas and opinions. It is sometimes not clear whether the most valuable insights have actually been discussed, because the person moderating the discussion has not structured it to give time to key participants at key points in the conversation. Time is allocated according to the aggressiveness (and courtesy) of the speakers, usually in proportion to rank, as opposed to the merit of their content.
  • Forcing conversations to happen in real time limits the forum due to participant availability (i.e., conflicting schedules, time zone differences), attention span (i.e., often our thoughts are occupied by other priorities), and preparedness (i.e., expecting participants to have reviewed material beforehand in preparation to reach consensus). Some people do not perform well in this setting; they perform better when given time to collect their thoughts. Some people respond better in writing.
  • Although audio and video can be recorded for later playback, it is not a digital transcript whose content can be indexed and searched. A scribe may take notes and publish minutes, but this is labor intensive, and the result is not a full-faith representation of the conversations.
  • Often a key participant is not included in a meeting, because the organizer was not aware of that person’s capabilities until later.

For the preferred mode of collaboration in the enterprise to evolve from real time conferencing to a format that overcomes the above deficiencies, we need a culture that relies less on real time collaboration and more on online tools that bring people together to work more closely, while allowing them to be more decoupled in space and time.

Clearly we need better tools to fulfill this need. We have seen tools like Google Wave appear and quickly disappear. Google Wave failed to integrate with enough supporting tools to enable collaborative editing, peer review, and post-processing (interchange) of a broad range of content (documents, images, audio, video, presentations, conferences, models, charts). It failed to integrate with enough supporting applications to participate in real world business processes. But worst of all, it failed to provide a compelling paradigm or inspiration for a cultural shift to collaborate more effectively in real time and otherwise.

Where do we go from here?

Recognize that workers need to collaborate with greater decoupling in space and time. Mobile computing with always-on smart phones and tablets is a fundamental enabler for such decoupling. Enterprise applications that run in the cloud are another element that enables the work force to collaborate using these devices. What we are missing is the right mix of applications and integration into business processes that support the new ways of doing work.

As the corporate office, desk, desktop computer, and telephone become extinct, we must anticipate how a mobile always-on work force will need to work. We must develop applications to facilitate this new work paradigm where activities are often time-shifted; and real-time collaborations are between remote workers, who may be anywhere (e.g., at home on the couch, at the coffee shop, in the airport, in-flight on an airplane, driving an automobile). We will know that the revolution has arrived, when the culture in the enterprise considers it more natural to do work using mobile devices from anywhere and at all hours rather than at the office during office hours.

vorlons and shadows – philosophy

Babylon 5 is a story about two differing philosophies. The Vorlons promote a life of order, stability, and peace. The Shadows promote a life of chaos, destabilization, and conflict. There is a similar philosophical difference in software development.

Hubris of Prescience

Most commercial software projects lean toward a philosophy of order, stability, and peace. Optimism leads to anticipating stable requirements, which would allow for a stable design. This outlook influences management and developers to behave in a particular way. We gain an expectation for requirements and design to be orderly, stable, and for a peaceful progression of events to ensue. Successful methods and techniques are expected to continue to be viable. The products of past investments into research are expected to retain their value with the passage of time. Knowledge of the present incubates confidence in being able to anticipate future needs. This confidence is reinforced by the belief that momentum has longevity. This belief is correct, but not in the way that we would hope for a healthy technology business.

The technology market relies upon continuous innovation. Innovation is about disruptive change. A strategy based on incremental improvement bets on a steady pace of change, where one’s own product is the market leader. Trailing competitors are constantly seeking to impose revolutionary change upon the market to overtake the market leader. If one is not in the lead, then it makes sense to disrupt the market to put oneself in an advantageous position. New markets are created, where old ones are destroyed. Competitive innovation radically alters requirements, thereby invalidating entrenched designs. Disruptive change accelerates demand for technology through obsolescence.

The lure for the market to adopt innovative products is efficiency gain. Gains in efficiency increase productivity or reduce costs – or both. A methodology promoting the entrenchment of a status quo is unable to adapt to an environment of unrelenting disruptive change. In technology, the status quo is precisely what must be destroyed in order to sustain a healthy market. A strategy of maintaining an entrenched design is a strategy of certain failure in the technology market.

Creativity and Renewal

While a life of order, stability, and peace breeds comfort, it also leads to stagnation and complacency. Creative individuals will recognize where the status quo is not good enough. A culture that institutionalizes the status quo cripples creativity, ensuring a disadvantage in relation to the competition, who seeks disruptive change.

Innovation is nurtured by an entrepreneurial spirit. There are several factors that are required to facilitate creative thinking: freedom, motivation, inspiration, and courage. Creativity needs a culture that promotes creative ideas by nurturing independent thinking, not obedience to instructions delivered by management or designated “thought leaders”; individuals will be willing to think if they are entrusted to do so. Motivation comes from achievable goals, incentives, and the rewards of a job well done. Inspiration comes from the growth and enjoyment of working with peers, who demonstrate competencies, which become a source of knowledge. Finally, courage is provided by the willingness to incur risk to achieve the rewards that come from unconventional thought; people who are confident enough in their ability to attempt great things should be encouraged to do so, because overcoming technical problems is daunting enough without the burden of an unsupportive culture.

The key characteristic of a technology business that is prepared for innovation is agility. It must be adaptable quickly to change. It must embrace disruptive change, and use change to its advantage, rather than being resistant or vulnerable to it. This includes a high degree of tolerance for risk, because change is inherently risky, and the more disruptive the greater the risk (and potentially the reward). Innovation opens opportunities, but a business must be able to pounce on an opportunity to benefit.

Until Moore’s Law no longer holds, we can expect innovative growth to continue. Technology innovation and obsolescence is the dominant trend. A technology business that does not institutionalize a culture that embraces disruptive change will not be a business for long.

innovation as the enemy of maturity

In my previous article, maturity as the enemy of innovation, I identified the destructive forces of organizational maturity. As an engineer, I cannot allow a problem to go unsolved.

To stem the tide of chronic risk aversion, we must imprint the entrepreneurial spirit into the culture as a core value. The courage to initiate calculated risk-taking needs to be admired and rewarded. The stigma associated with failure must be removed by appreciating the knowledge gained through lessons learned.

The cost of research and experimentation must be reduced. The pace of development must be rapid. These are achieved by keeping the team size small. The most risky and unproven ideas should be attacked by an individual, not a team. Functional teams need alignment on direction and a degree of agreement. Revolutionary advancement is frequently the result of radical thinking. New directions, different techniques, and breaking well-established rules are unpopular, especially to those with emotional investments in past innovations. There are fewer disagreements in a small team, and none at all in a team of one.

Uniformity of thinking leads to stagnation. Diversity must be encouraged, and this is promoted through a free market for ideas. Organizations that rely on a command-and-control style of management to set direction eliminate the competition that nurtures a diversity of ideas. Management should empower technical decision-making to be done bottom-up rather than top-down.

Once new ideas have been adequately proven, there must be ways of teaching them to others and incorporating them into product. Product development is often done sequentially, one release at a time. The immediate release focuses on short term commitments, and the delivery is usually constrained by schedules and costs that can ill afford risk. Riskier development must be allowed to proceed concurrently on longer-term schedules, without overly constraining the process. Developments from the unstable branch should be merged into the main product release, as they become ready. This provides an environment to accommodate both risky development and risk-averse development.

The above techniques can be effective in remedying the organizational impediments to innovation. However, there are also commercial impediments that must be overcome. That will be the topic of my next article.

maturity as the enemy of innovation

As a software development organization matures, its culture institutionalizes practices that promote quality. Maturity is for quality. The most obvious measures of quality are in terms of defects. The quest for lower defect counts is a noble one.

At the same time, complexity grows as the weight of larger feature sets and lofty requirements are incorporated into the software. With increasing complexity comes a larger development team. A larger team means more division of labor, higher cost of management overhead, and more careful planning to be able to utilize those resources. Division of labor requires the delegation of distinct responsibilities to specialized roles. Decision-making becomes more dispersed and distributed, and a decision will have wider impacts on team members, so good communication becomes a critical success factor.

Unfortunately, decision-making, communication, and the distribution of responsibility are never perfect. The more complex the software, the larger the team, the more distributed the responsibilities, the more specialized the roles, the higher the cost of development. This is not a scalable organizational model. Once a software organization has reached this stage of maturity, it is doomed by inefficiency. Creative individuals are no longer empowered to quickly realize their ideas. Two key requirements to motivate and nurture a creative mind are independence and freedom. The degree to which these environmental factors are blunted is the degree to which innovation is impeded.

Because decision-making is distributed among many people, it becomes impossible for innovative ideas to be realized without winning consensus. Innovative ideas are unconventional ideas. A novel approach often needs to break well-established rules and go against practices that have worked in the past. Innovation is destructive to the status quo. It must be disruptive in order to rise and overtake older practices. Innovation is the product of experimentation and risk-taking, usually resulting in failure and rework. Without an environment that encourages this, there will be no innovation.

The quest for quality through institutionalized practices and disciplined management are in many ways counter to the independence and freedom that motivate a creative mind. Careful project management and planning discourages experimentation with risky new ideas (technical, organizational, and commercial), where many unknowns make results difficult to predict. Risky ideas need experimentation so that many failed attempts can be discarded quickly, allowing successful ideas to be taken to production. Mature organizations respond to higher risk by applying stronger management practices for mitigation, thereby increasing the cost and slowing the pace of failure towards success. In fact, once management smells failure, they immediately work to eliminate it, thereby destroying its very value as a vehicle to learn what will be needed to succeed. This is how many mature organizations have killed off the entrepreneurial spirit. This is the reason why most innovative ideas come from small groups of individuals working in a garage or a small start-up company.

planning and estimation

We spend an extraordinary amount of time planning our development, because it is so costly to get any amount of code written. Developers are asked to give estimates. They demand to know what is in scope down to the individual operation. It’s crazy how much design work and detail needs to be thought through before reaching that point. Then, they estimate maybe 10-20hr of work per method. Even a tiny piece of functionality results in estimates in several person-weeks of effort. This is how software professionals get their jobs done.

Where have all the hackers gone?

I came up with a cool idea on the weekend. I decided to code it sunday. I spent about four hours in total coding the entire thing, analyzing and designing, as I went. It turned to be over 2000 lines, which I proceeded to unit test. I put in another three hours tonight debugging a rather obscure problem (actually wasting 2.5 hours chasing a non-existent bug due to my own stupidity), which I discovered in my sleep the other night.

I simply cannot understand why experienced software professionals cannot hack out 3000 lines of unit tested code on a good day, and average 1000 lines of unit tested code per working day, with low bug counts.

I’m almost glad to see that many of North America’s grunt coding jobs are being outsourced to India. The cost of our software professionals in combination with their low productivity and high maintenance attitude is unbearable.

notes on quality process

Software development methodologies such as the Unified Process are intended to institutionalize a quality process.

Often, the manner in which a quality process is implemented will focus on quality of manufacturing, in order to achieve predictable schedules and outcomes. In the name of predictability, rigorous up front planning attempts to predetermine tasks, resources, and schedules to estimate costs and deliverables. Putting constraints on the size and shape too early will necessarily require gross assumptions about the requirements and design. In so doing, we have traded off quality of innovation to achieve quality of manufacture. We are able to build it within budget and on time, but what we are capable of building is not innovative.

follow the rules

I have an engineer who works for me. His responsibilities include ensuring design consistency and best practices across the components in the system. We establish design guidelines and rules for the team to apply in their work, and he performs the grunt work of reviewing and editing everyone’s designs. He is well suited to the role, because he is very methodical, and he has an affinity to documenting and following rules.

Engineering can be divided into two different modes of operation: manufacturing and invention.

I use the term manufacturing in reference to ordinary development. That is, designing based on a strategy of incremental improvement over past designs. This strategy has served the Japanese well in automobiles and consumer electronics. A disciplined approach to engineering with focus on continuous improvement and attention to market forces is the key to success in manufacturing. Manufacturing is about quality of implementation. Manufacturing is about applying best practices and quality process.

Invention is completely different. Invention is not about responding to market demand. It is about using imagination to create new markets, where they never existed before. Invention requires innovative thinking to formulate new ideas and different (hopefully better) practices. Invention is not about applying today’s best practices and rules to incrementally improve. Invention requires the understanding that today’s best is flawed and impeding progress towards a superior possibility that may be highly risky, unproven, and difficult to achieve – but worthwhile to pursue, because it would be revolutionary.

A business needs disciplined engineers, who are skilled at manufacturing. This is where the money is.

My grunt is definitely a manufacturer, not an inventor. He has mediocre design creativity, because he is unable to intuit across a mass of contradictory information and conflicting motivations (resolving the forces), while paying attention to detail. He needs rules to govern his thinking and everyone else’s. Without rules, he is unable to function. He has no intuitive understanding of good design principles. He follows the rules.

An inventor (and every good designer) does not follow the rules. He does not break the rules either. He defines the rules that are appropriate, and he uses them to aid his thinking. Above all, he uses his brain to produce good designs, given the facts in evidence. He leads, where few have the courage to follow.

man-qua-universe

I am currently reading Three Roads to Quantum Gravity. I have just completed the first part of the book, which is a lengthy introduction. The author felt a strong need to prepare the reader’s mind for thinking about the universe differently from whatever misguided education and personal experience was filling his head.

Perhaps if I had not already read The End of Time: The Next Revolution in Physics, I would have needed this preparation. However, being already convinced that space and time are invalid concepts for properly describing reality, I felt perfectly at home with the author’s perspective.

Something I was unprepared for was the realization that objectivist epistemology may be overly constrained by its man-qua-man perspective. Scientists, who practice the Scientific Method, impose constraints on their thinking by considering themselves an objective observer. Those who do this cannot effectively practice the science of cosmology, because no such artificial (mystical) separation can be established between observer and universe.

Only man-qua-man can allow himself to be so easily fooled into thinking the world is as he himself sees it, and be convinced that this is an accurate view of reality. Only man-qua-ordinary-man would so easily accept space, entities, and their attributes to represent the true state of things, while changes in state are viewed as snapshots in time. Most men still believe that reality is organized into a three dimensional grid representing space, and an extra dimension of time. Even their inability to sense these alleged characteristics does not bother them. Ask them to show you space and time one day, and see how far they get trying to use existents to describe non-existents.

Man-qua-ordinary-man is in fact a mere fool. In order to understand cosmology, we must become man-qua-universe and think likewise.

[It might seem strange that a software architect would spend so much effort on cosmology. Cosmology and software architecture are actually similar. Cosmology and software architecture are both about studying the universe. Just as every ordinary cosmologist has failed to gain a proper understanding of the universe, by being unable to accept that he himself cannot be separated from it as objective observer, ordinary software architects are in the same bind.]

forest for the trees – abstractions

When faced with designing objects, many developers immediately try to identify the abstractions. They have a preconception that the world must be organized as a hierarchy of abstractions. They quickly run into situations that don’t fit nicely into their hierarchy. They struggle to force-fit things into this model. After a while, they ask for help.

The world is not a tree. There may be forests. But generally the world is what it is. Do not force things to be a part of a tree, when that is not what they are. That should be obvious enough. It is always the most obvious things that are difficult to realize, when one is blinded by misconceptions.

Step 1. Identify the concrete types. Start from the objects that will actually exist, not from higher level abstractions. The concretes have definite characteristics that can be analyzed. These characteristics are where the abstractions will be found later through refactoring (unit economy). Concretes are characterized by their behavior. A concrete has roles and responsibilities relative to other objects with which it collaborates. It may be necessary to decompose a concrete into smaller parts that take on those distinct roles and responsibilities (separation of concerns) more clearly.

Step 2. Refactor the characteristics into higher level abstractions. Identify characteristics and behaviors that are common across concretes. Often behaviors can be seen as actions performed on the concretes (rather than actions performed by the concretes), and there is a distinct characteristic that each action depends on. I frequently call such a characteristic a capability, and the action is known as a generic algorithm. As this refactoring proceeds, the generic algorithms and the capabilities are identified as higher level abstractions shared or applied across concretes.

Step 3. Refactor the algorithms and capabilities. As the generic algorithms are implemented against the capabilities of the concretes, the methods will often benefit from factoring out commonality (duplicate code). Parts of the algorithm may benefit from being specialized through pluggable policies. These motivations identify additional abstractions at finer granularities. The finer grained parts of algorithms will likely act upon finer grained capabilities on the concretes. Broader generalizations and narrower specializations appear in the model as a consequence of this iterative approach due to refactoring to reduce complexity (the complexity is refactored to become encapsulated within objects, which are more manageable than messy algorithms).

The resulting model is forests of trees, along with lions and tigers and bears, as well as all the other concretes that exist in the world. Things as they really are. The way it ought to be.

collaboration – a business idea

business idea for enterprise collaboration

(Continued from 2002-07-08.) One of the biggest impediments to productivity is the lack of tools to facilitate collaboration. This was not as noticeable, when workers tended to be collocated with those they needed to collaborate. Physical collocation makes less sense today, when globalization requires companies to be distributed throughout the world.

Expertise is becoming so specialized that it is impossible to develop a broad range of skills in house and locally, so this creates a need to outsource and recruit teleworkers. Email, telephony, instant messaging, and intranet Web sites only go so far to facilitate communication with these remote sites. These technologies need to be integrated into business processes to facilitate wide scale real-time collaboration.

In a software development organization, there are very specific requirements that are unmet by today’s information technology. Software development tools tend to focus either on the relationship between individuals and code artifacts (e.g., integrated development environment) or the relationship between the code artifacts and the quality assurance processes for the software product (e.g., software configuration management). Over the past decade there has been a greater focus on analysis and design tools (e.g., UML modeling). However, modeling is a very small part of analysis and design. The majority of analysis involves the capture of requirements and use cases in the form of natural language (e.g., English) descriptions. This is a very iterative and interactive process among experts and analysts, and the information flows into design. Design is a very creative process among analysts, architects, and programmers in various areas of expertise (e.g., user interface, application server, database). The collaboration tools support for analysis and design are wholly inadequate today.

The software development community needs a Web based collaboration system for analysis and design. In the analysis space, Rational Requisite Web almost completely misses the mark for a requirements database. In the design space, there is nothing to facilitate the whiteboard-style interactions that are necessary during the creative process. Such interactions work toward evaluating design options and formally capturing the final design decisions. UML modeling is merely a visualization technique. UML diagrams are not in a format that is appropriate for communication between all stakeholders. Ultimately documents must be produced for the software architecture and the design of each subsystem. Collaboration tools are required for producing these artifacts as a team.

I have looked at using a Wiki to facilitate collaborative content creation. This is not a bad mechanism for authoring, reviewing, and revising content. However, it is wholly inadequate for taking that content and producing documents for publishing. Moreover, a Wiki alone is insufficient for handling all the various kinds of content that is needed for analysis and design (e.g., models, diagrams, code, user interface prototypes), and for managing the workflow.

I believe that this is a tremendous business opportunity. There is clearly a need for this kind of collaboration tool for software development. One can imagine this need extending into other engineering activities, and perhaps even into other disciplines (e.g., sales and marketing) within the enterprise. This is likely a $100M idea with a market window that is likely to remain open for quite some time. If I had the guts, I would probably pursue this one as my baby. I’ve got the perfect skillset and experience to make this happen.