oops reuse misuse – part 2

part 2 of infinity – reuse and replacement

(Continued from part 1 – introduction.) Organizations frequently become enamored with building software infrastructure and frameworks for reuse. They see a great deal of common problems being solved again and again in different ways by different people. They see duplication of effort as a business inefficiency that can be optimized by factoring the commonality into reusable components, so that wheels are not reinvented. Centralized control through corporate standards is another popular activity.

One sign of successful reuse is the proliferation of users. With each new user there is a new dependency. The dependency is manageable so long as the requirements are stable. If new requirements must be satisfied, as is typical of rapidly evolving technology, these dependencies become a serious impediment to productivity.

A fundamental enabler of component reuse is the software contract. The interface that isolates the underlying implementation details from its users must remain stable to allow the two to evolve independently. However, changing requirements often result in interface changes. Any change to the software contract will break its users. As the number of users grows, the impact of this breakage becomes more severe. Thus, the desire to scale software functionality linearly with development effort is not viable because of changing requirements.

Only with perfect foresight can we avoid changing requirements. If we could employ prescience so that all new requirements resulted in wholly new software components, leaving existing software components in tact, we would have a predictable and scalable development process. Unfortunately, human fallibility prevents us from having good visibility into the future, and our software designs suffer.

We must be honest and recognize that software, no matter how well designed, has limited ability to evolve to meet new and changing requirements over time. We must build into our development processes the practices that can accommodate the retirement and eventual replacement of components that have collapsed under the weight of their outdated designs. A fatal flaw in many development organizations is the denial that software can grow old and die. This flawed thinking allows younger and healthier software from the competition to eventually win out.

Therefore, any reuse strategy must anticipate this component lifecycle to ensure that the entire ecosystem of dependent users does not die a horrible death tied to aging components. A reuse strategy without a recurring replacement strategy is a path of certain destruction.

Another thing to recognize is the distinction between infrastructure versus application code. The view that some components are foundational, while others are fundamentally not, is short-sighted. The entire premise of reuse is that software should be built to enable an evolution towards building more functionality using the components already constructed. In effect, every component eventually becomes infrastructure for other components, which are often not anticipated.

oops reuse misuse – part 1

part 1 of infinity – introduction

One of the strongest motivations that is espoused for promoting object oriented programming is reuse. While object oriented programming languages provide features, such as encapsulation and inheritance, that enable better reuse, OOP has gotten a bad rap, because the discipline that is required to produce good software must address many concerns aside from reuse in order to be successful.

Reuse is a buzzword software professionals bandy about, when they promote certain software practices. It refers to the ability to incrementally grow software capability by building additional functions to augment what was already built previously. Development organizations like this idea, because it implies predictability, which is the holy grail of software methodology (quality process). If the additional effort required to build and test functionality can be made proportional only to the additional code being incrementally developed, then development can be highly predictable. This would be the key to projects being delivered on time and within budget.

Although reuse in this sense is a good motivation, nobody knows how to make it work. Very intelligent and highly respected individuals will say how to apply principles of reuse, but either they don’t know what they are talking about, or the people who apply their principles don’t understand what they should be doing. Or both. The state of the software industry is a testament to the fact that reuse strategies are being widely applied with a high degree of (and near total) failure. No one achieves anywhere close to the linearity of effort, predictability of process, and scalability of outcome that reuse is meant to achieve.

In an series of articles, of which this is the first, I would like to evaluate various reuse strategies, practices, and object oriented programming techniques. Stay tuned for more in this series.

Business vs Software Use Cases

Let us examine how Use Cases are defined in the context of a business, who uses a software system, and a developer (vendor), who supplies that software.

Use Cases are intended to capture the software requirements. This is done by identifying and describing the units of work (Use Cases) that are performed by the users. Users are who are known as Actors. In this way, the behavior of the system is captured by analyzing how the Actors interact with the software in achieving their goals. In layman’s terms, I sometimes describe Use Case analysis as understanding the problem.

The business wants to describe Use Cases in terms that are familiar to it. Its Actors are named according to the roles within its organization. Its Use Cases are described according to the organization’s business processes. The actions and entities are named according to verbs and nouns that are specific to the business.

Meanwhile, the software developer wants to describe Use Cases in terms that can be generalized to the larger market for such software systems. Its Actors must be generalized to accommodate the many organizations that are typical within its market. Its Use Cases are described according to generalized business processes that are typical within the industry. The actions and entities are named according to abstract verbs and nouns that are not specific to the one particular deployment.

This difference in perspectives causes a communications gap. The business speaks in concretes that are specific to its organization. The developer speaks in abstractions that are generally applicable to the market. There may be a many-to-many relationship between the concretes identified by the business and the abstractions identified by the developer. The perspectives must be reconciled to allow the two parties to communicate effectively.

The abstractions provided by the developer become design elements to realize the business Use Cases.

forest for the trees – abstractions

When faced with designing objects, many developers immediately try to identify the abstractions. They have a preconception that the world must be organized as a hierarchy of abstractions. They quickly run into situations that don’t fit nicely into their hierarchy. They struggle to force-fit things into this model. After a while, they ask for help.

The world is not a tree. There may be forests. But generally the world is what it is. Do not force things to be a part of a tree, when that is not what they are. That should be obvious enough. It is always the most obvious things that are difficult to realize, when one is blinded by misconceptions.

Step 1. Identify the concrete types. Start from the objects that will actually exist, not from higher level abstractions. The concretes have definite characteristics that can be analyzed. These characteristics are where the abstractions will be found later through refactoring (unit economy). Concretes are characterized by their behavior. A concrete has roles and responsibilities relative to other objects with which it collaborates. It may be necessary to decompose a concrete into smaller parts that take on those distinct roles and responsibilities (separation of concerns) more clearly.

Step 2. Refactor the characteristics into higher level abstractions. Identify characteristics and behaviors that are common across concretes. Often behaviors can be seen as actions performed on the concretes (rather than actions performed by the concretes), and there is a distinct characteristic that each action depends on. I frequently call such a characteristic a capability, and the action is known as a generic algorithm. As this refactoring proceeds, the generic algorithms and the capabilities are identified as higher level abstractions shared or applied across concretes.

Step 3. Refactor the algorithms and capabilities. As the generic algorithms are implemented against the capabilities of the concretes, the methods will often benefit from factoring out commonality (duplicate code). Parts of the algorithm may benefit from being specialized through pluggable policies. These motivations identify additional abstractions at finer granularities. The finer grained parts of algorithms will likely act upon finer grained capabilities on the concretes. Broader generalizations and narrower specializations appear in the model as a consequence of this iterative approach due to refactoring to reduce complexity (the complexity is refactored to become encapsulated within objects, which are more manageable than messy algorithms).

The resulting model is forests of trees, along with lions and tigers and bears, as well as all the other concretes that exist in the world. Things as they really are. The way it ought to be.

red shift in an expanding universe

As we look into the distance, we see deep into the past towards the big bang. It is mind boggling how unintuitive this is. If you look far enough into the distance, you should be able to see the earliest moments of the universe. This is the cosmic background radiation. Quarks condensed into the first protons and neutrons.

What is more confusing to me is the interpretation of the evidence that suggests the universe is expanding at an increasing rate. This conclusion is based on observing red shifts. The farther an object is from us, the faster it appears to be moving away. Unless I’m misunderstanding something, doesn’t this tell us that the expansion of the universe is actually slowing down? Let me explain.

Wouldn’t an accelerating expansion demonstrate higher differences in velocity (red shift) for objects nearer to us? Since they are near, we see them as they were relatively recently as compared to far away objects. If the rate of expansion is accelerating, then observations of more recent events should show higher velocities than observations of events in the more distant past. On the other hand, if the velocity of distant objects is higher than the velocity of nearer objects, then shouldn’t it be logical to conclude that expansion is slowing rather than quickening?

quintessence – dark energy

I have begun to read Quintessence. It starts out rather dull. I am hoping that it picks up soon. Unlike End of Time, the author’s style is not provocative; he seems very methodical — though I’m only at the beginning, which contains all the introductory context.

Dark matter and dark energy have become a favorite topic for my curiosity. What better subject can there be to help stimulate the imagination? Observations suggest that there is something incredibly important throughout the universe that dominates its behavior, and no one knows what it is. These things cannot be sensed directly. They are completely different than anything we know today.

Free your mind.

Many people are unable to free their minds of what they know to be true, so that they can evaluate ideas from a different perspective. It is debilitating to be a rigid thinker. They are weak at evaluating alternative mental models, because they cannot apply their limited imagination in constructing such models.

What kind of image does the expanding universe invoke? Does your mind picture the universe itself as a ball growing larger, its boundaries stretching outwards? Or does it picture, as an observer within the universe itself, all of the galaxies moving away from eachother? Or does it picture a static universe without any notion of time or motion, and the expansion is merely a path in configuration space? Does it recognize the false notion of being an outside observer, while allowing this fiction to be imagined?

How do you picture the void of empty space? Is it a three (or higher) dimensional grid? Is it curved by the massive objects? Is it absolutely nothing at all, except for the void between things that actually exist? Or is it ethereal, filled with virtual particles that flash in and out of existence beyond our ability to perceive?

At many points in history, fundamental questions as simple as “what is space?” and “what is time?” have been asked and answered. Still we do not know the true answers; we only have intuitive notions based on teachings that have been socialized. We go through school learning mostly by rote the laws of the universe, such as those related to motion, momentum, and energy — notions that depend on understanding space and time. We are taught to treat space and time as intuitively obvious, because they are understood ostensibly. Yet, there may not be a single human being that has ever truly understood the nature of space and time — if those ideas have any correspondence to reality at all, and the jury is definitely out on that one. Despite this dilemma, the vast majority of students finish school with confidence that they know a great deal about space, time, motion, and many other important things. They don’t have an appreciation for the context of their knowledge — the limited scope of understanding that is encapsulated. They are taught the current state of understanding, which is good enough for most practical purposes in ordinary life. They are lies, or reasonable facsimiles of the truth, good enough to get you through most days without going too awry.

Just as the earth is not flat (though from a person’s viewpoint, it may appear that way), and the sun does not orbit the earth (though from a person’s viewpoint, it may appear that way), these ideas were once at the pinnacle of human knowledge, only to be invalidated later. We accept theories like general relativity and quantum mechanics because there isn’t yet any better explanation. They can be applied to good effect as a practical matter for a limited set of problems. But let us not forget that the accuracy of current theories has much room for improvement.

mda – model driven architecture

I explore Model Driven Architecture (MDA). This approach uses modeling for programming. Models drive the generation of code according to patterns.

We tend to write a lot of similar code over and over. If you have ever written code to persist a Java object into a relational database table, you know what I mean. It is similar for every object. Although the characters you type are different, there is a definite pattern, and you know deep down that an intelligent person should not have to endure such tedium day in and day out. The same goes for a lot of user interface code, like data entry forms. The same code gets written again and again; only the names are different.

I believe very strongly that model driven architecture is the way of the future for software development. Objects led to interfaces. Then, collaborations. Then, patterns and pattern languages (families of patterns). Model driven architecture is a natural extension of this trend. After identifying a pattern language, it should be possible to instantiate those patterns (write a software program) by expressing that intent through modeling. Modeling in the abstract sense is the act of capturing concepts in a notation.

I consider programming in Java to be a form of modeling. A programming language is not a very compact modeling language. Programming languages are designed to have a lot of redundancy to allow for error checking. Patterns in the code also indicate a lot of information that could be expressed in a more compact form. I view MDA as expressing software concepts in a very compact form (the model).

I think we need an example. I could write a program to represent pictures formed from geometric shapes. The software itself would probably require thousands of lines of code to perform the editing, rendering, and other functions. However, the application is really only dealing with a few basic concepts: ellipses, polygons, line segments, and a few others.

It should be possible to model each shape with a minimum number of parameters; e.g., an ellipse would be modeled by three points. In this manner, we could express any diagram understood by this application very compactly in a modeling language that captures the parameterization of shapes.

Now, let’s take that graphics package and reuse it as the foundation for constructing an application that renders telecommunications networks. This application deals with concepts like location, network elements, ports, and connections. Each concept is rendered with symbols, which are merely recurring patterns of our familiar geometric shapes.

The parameterization of shapes is no longer the most appropriate modeling language for capturing the essential concepts of this application. A more compact representation can be achieve by parameterizing the concepts that are directly understood by the application. This is the essence of model driven architecture.

Not everyone views MDA as I do. Actually, I have yet to encounter anyone who has expressed it as simply as that. Most proponents of MDA ramble on about UML and MOF and platform independence and all kinds of other non-essential mumbo jumbo. The key idea that is lost among the OMG UML/MOF folks is the notion of unit economy in the theory of concepts; that is the need to represent concepts in the most compact form. This is necessary within the mind, and this need is reflected in language.

UML makes perfect sense as a language for representing analysis and designs for software development. However, when one is looking to model concepts within any other context, the absurd verbosity and complexity exhibited by UML/MOF make as much sense as using geometric shapes as the modeling language for telecommunications networks.

collaboration – a business idea

business idea for enterprise collaboration

(Continued from 2002-07-08.) One of the biggest impediments to productivity is the lack of tools to facilitate collaboration. This was not as noticeable, when workers tended to be collocated with those they needed to collaborate. Physical collocation makes less sense today, when globalization requires companies to be distributed throughout the world.

Expertise is becoming so specialized that it is impossible to develop a broad range of skills in house and locally, so this creates a need to outsource and recruit teleworkers. Email, telephony, instant messaging, and intranet Web sites only go so far to facilitate communication with these remote sites. These technologies need to be integrated into business processes to facilitate wide scale real-time collaboration.

In a software development organization, there are very specific requirements that are unmet by today’s information technology. Software development tools tend to focus either on the relationship between individuals and code artifacts (e.g., integrated development environment) or the relationship between the code artifacts and the quality assurance processes for the software product (e.g., software configuration management). Over the past decade there has been a greater focus on analysis and design tools (e.g., UML modeling). However, modeling is a very small part of analysis and design. The majority of analysis involves the capture of requirements and use cases in the form of natural language (e.g., English) descriptions. This is a very iterative and interactive process among experts and analysts, and the information flows into design. Design is a very creative process among analysts, architects, and programmers in various areas of expertise (e.g., user interface, application server, database). The collaboration tools support for analysis and design are wholly inadequate today.

The software development community needs a Web based collaboration system for analysis and design. In the analysis space, Rational Requisite Web almost completely misses the mark for a requirements database. In the design space, there is nothing to facilitate the whiteboard-style interactions that are necessary during the creative process. Such interactions work toward evaluating design options and formally capturing the final design decisions. UML modeling is merely a visualization technique. UML diagrams are not in a format that is appropriate for communication between all stakeholders. Ultimately documents must be produced for the software architecture and the design of each subsystem. Collaboration tools are required for producing these artifacts as a team.

I have looked at using a Wiki to facilitate collaborative content creation. This is not a bad mechanism for authoring, reviewing, and revising content. However, it is wholly inadequate for taking that content and producing documents for publishing. Moreover, a Wiki alone is insufficient for handling all the various kinds of content that is needed for analysis and design (e.g., models, diagrams, code, user interface prototypes), and for managing the workflow.

I believe that this is a tremendous business opportunity. There is clearly a need for this kind of collaboration tool for software development. One can imagine this need extending into other engineering activities, and perhaps even into other disciplines (e.g., sales and marketing) within the enterprise. This is likely a $100M idea with a market window that is likely to remain open for quite some time. If I had the guts, I would probably pursue this one as my baby. I’ve got the perfect skillset and experience to make this happen.

timeless – thinking outside the box

I am sitting here in a Boeing 737 on the tarmac, as maintenance personnel check over the aircraft to determine why some indicator lights are showing a problem. What has been occupying my mind is the first few pages I’ve read of The End of Time – The Next Revolution in Physics by Julian Barbour. This book is a great example of thinking outside the box.

Barbour suggests that there is in reality no such thing as time. It is merely a figment of our misinterpretation of the facts, which result from the limited scope of our personal experiences. This idea is very provocative, because it goes against intuition and it seemingly contradicts everything we know to be true. There is something extremely violent and destructive in this notion. Just the possibility of setting fire to something that we hold as fundamental is exciting.

This is very serendipitous. My life’s work seems to revolve around creative thinking. One of my fundamental responsibilities is to think outside the box to solve problems that are beyond the normal capacity of others. I must imagine what lies outside of common knowledge, and embrace possibilities that may be seemingly absurd or false, in order to discover that something out there is in fact precisely what must be true (or brought into reality). Thinking outside the box requires that in fact there is no box. Or perhaps that all those poor souls, who thought they were in the box, were actually floating outside a bubble. Or something much more bizarre.

language design 101

Language provides us with the framework with which we express ideas. Numbers, arithmetic, and calculus are the language of mathematics. C, C++, and Java are languages for computer programming. This is only partly true, as Guy Steele Jr. explains in Growing a Language. This perspective on language design extends far beyond programming.

I will focus on software related language design. It is an exercise for the reader to extrapolate how the same principles apply to literature, music, and other forms of human expression.

Just as Java is the most basic vocabulary and grammar with which more expressive languages can be designed, English is similar. The language required to produce a book is far more advanced than English vocabulary and grammar. Literary devices, plot structure, and character development are patterns that are designed into the language of literature. Similarly, design patterns make Java a useful language for programming.

The language of object-oriented analysis and design (OOAD) is the Unified Modeling Language (UML). However, UML is not very expressive by itself. It was not designed to be as precise as programming languages, because it was motivated by the needs of conceptual modeling. They thought that conceptual modeling is not precise. That is the tragedy of imprecise thinkers.

Now, they have caught on to the idea of using models to generate code. This requires precision; a machine only understands 1 and 0. There is no room for ambiguity. It is now a huge challenge to apply UML to this model-driven architecture (MDA).

We are constantly grasping for ways to improve how we express ideas. A thousand words are better expressed with a picture. The concepts represented by the geometric shapes are better expressed with tags. Patterns are recognized as we organize those concepts into more advanced ones. As a consequence, many thousands of instructions are captured in a blueprint with a small number of symbols. At each step, we seek to be more expressive with compact representations. This is language design.

Insights into innovation