Reliable Messaging with REST

Marc de Graauw’s article Nobody Needs Reliable Messaging remains as relevant today as it did in 2010, when it was first published. It echoes the principles outlined in Scalable, Reliable, and Secure RESTful services from 2007.

It basically says that you don’t need for REST to support WS-ReliableMessaging delivery requirements, because reliable delivery can be accomplished by the business logic through retries, so long as in the REST layer its methods are idempotent (the same request will produce the same result). Let’s examine the implications in more detail.

First, we must design the REST methods to be idempotent. This is no small feat. This is a huge topic that deserves its own separate examination. But let’s put this topic aside for now, and assume that we have designed our REST web services to support idempotence.

If we are developing components that call REST web services for process automation, the above principle says that the caller is responsible for retrying on failure.

The caller must be able to distinguish a failure to deliver the request from a failure by the server to perform the requested method. The former should be retried, expecting that the failure is temporary. The latter is permanent.

The caller must be able to implement retry in an efficient manner. If the request is retried immediately in a tight loop, it is likely to continue to fail for the same reason. Network connectivity issues sometimes take a few minutes to be resolved. However, if the reason for failure is because the server is overloaded, having all clients retry in a tight loop will exacerbate the problem by slamming the server with a flood of requests, when it is least able to process them. It would be helpful if clients would behave better by backing off for some time and retrying after a delay. Relying on clients to behave nicely on their honor is sure to fail, if their retry logic is coded ad hoc without following a standard convention.

The caller must be able to survive crashes and restarts, so that an automated task can be relied upon to reach a terminal state (success or failure) after starting. Therefore, message delivery must be backed by a persistent store. Delivery must be handled asynchronously so that it can be retried across restarts (including service migration to replacement hardware after a hardware failure), and so that the caller is not blocked waiting.

The caller must be able to detect when too many retry attempts have failed, so that it does not get stuck waiting forever for the request to be delivered. Temporary problems that take too long to be resolved need to be escalated for intervention. These requests should be diverted for special handling, and the caller should continue with other work, until someone can troubleshoot the problem. Poison message handling is essential so that retrying does not result in an infinite loop that would gum up the works.

POST methods are not idempotent, so retry must be handled very carefully to account for side-effects. Even if the request is guaranteed to be delivered, and it is processed properly (exactly once) by the server, the caller must be able to determine if the method succeeded reliably, because the reply can be lost. One approach is to deliver the reply reliably from the server back to the caller. Again, all of the above reliable delivery qualities apply. The interactions to enable this round trip message exchange certainly look very foreign to the simple HTTP synchronous interaction. Either the caller would poll for the reply, or a callback mechanism would be needed. Another approach is to enable the caller to confirm that the original request was processed. With either approach, the reliable execution requirement needs to alter the methods of the REST web services. To achieve better quality of service in the transport, the definition of the methods need to be radically redesigned. (If you are having a John McEnroe “you cannot be serious” moment right about now, it is perfectly understandable.)

Taking these requirements into consideration, it is clear that it is not true that “nobody needs reliable messaging”. Enterprise applications with automated processes that perform mission-critical tasks need the ability to perform those tasks reliably. If reliable message delivery is not handled at the REST layer, the responsibility for retry falls to the message sender. We still need reliable messaging; we must implement the requirement ourselves above REST, and this becomes troublesome without a standard framework that behaves nicely. If we accept that REST can provide only idempotence toward this goal, we must implement a standard framework to handle delivery failures, retry with exponential back off, and divert poison messages for escalation. That is to say, we need a reliable messaging framework on top of REST.

[Note that when we speak of a “client” above, we are not talking about a user sitting in front of a Web browser. We are talking about one mission-critical enterprise application communicating with another in a choreography to accomplish some business transaction. An example of a choreography is the interplay between a buyer and a seller through the systems for commerce, quote, procurement, and order fulfillment.]

OLTP database requirements

Here is what I want from a database in support of enterprise applications for online transaction processing.

  1. ACID transactions – Enterprise CRM, ERP, and HCM applications manage data that is mission critical. People’s jobs, livelihoods, and businesses rely on this data to be correct. Real money is on the line.
  2. Document oriented – A JSON or XML representations should be the canonical way that we should think of objects stored in the database.
  3. Schema aware – A document should conform to a schema (JSON Schema or XML Schema). Information has a structure and meaning, and it should have a formal definition.
  4. Schema versioned – A document schema may evolve in a controlled manner. Software is life cycle managed, and its data needs to evolve with it for compatibility, upgrades, and migration.
  5. Relational – A subset of a document schema may be modeled as relational tables with foreign keys and indexes to support SQL queries, which can be optimized for high performance.

The fundamental shift is from a relational to a document paradigm as the primary abstraction. Relational structures continue to play an adjunct role to improve query performance for those parts of the document schema that are heavily involved in query criteria (WHERE clauses). The document paradigm enables the vast majority of data to be stored and retrieved without having to rigidly conform to relational schema, which cannot evolve as fluidly. That is not to say that data stored outside of relational tables is less important or less meaningful. To the contrary, some of the non-relational data may be the most critical to the business. This approach is simply recognizing information that is not directly involved in query criteria can be treated differently to take advantage of greater flexibility in schema evolution and life cycle management.

NoSQL database technology does not meet the need, because they do not support ACID transactions and they do not support relational tables and relational queries in the manner described above.

Perhaps the next best thing would be to provide a Java persistent entity abstraction, much like EJB3/JPA, which would encapsulate the underlying representation in a document part (e.g., as a XMLType or a JSON CLOB column) and a relational part, all stored in a SQL database. This would also provide JAXB-like serialization and deserialization to and from JSON and XML representations. This is not far from what EclipseLink does today.


innovation [noun] – context violating exaptation.

Ever since I read this tweet in 2012 by Fast Company, I have redefined innovation in this way.

Here is the first definition of exaptation from

noun, Biology
1. a process in which a feature acquires a function that was not acquired through natural selection.

By taking something or a combination of things and applying it to a purpose to which it was not intended (violating its original context), one may discover that it is well-suited to perform a different function. This discovery becomes an innovation.

Ignorance of the law

Ignorance of the law is no excuse. That is the principle we are expected to live by. If we embrace the full implication of this principle, it may merit being adopted as a Constitutional principle that places the most effective constraints on government over reach.

If ignorance of the law is not an allowable excuse, it is imperative for government to enact laws that people can read and comprehend to remain in compliance. Moreover, the laws for crimes and misdemeanors, as well as the regulations that every person must comply with must be readable and comprehensible in totality for the average person without requiring professional legal counsel. This requires that all crimes, misdemeanors, and regulations must not exceed a certain maximum number of words in their totality. That limit should be established to be what an average student can read and comprehend by investing one hour per day during four years of high school. The government is forbidden from writing laws and regulations that exceed this limit, so as not to instigate ignorance of the law.



faster than light travel

The article NASA May Have Accidentally Created a Warp Field is getting people excited about faster than light travel.

You don’t need to travel faster than light to go arbitrarily far in arbitrarily less time. All you need to do is travel closer to the speed of light. As you get closer to c, time dilation and space contraction will contribute to bring arbitrarily distant destinations within reach. Although the travelers will experience relatively manageable passages of time, it is their friends observing from home who will age much more quickly. Travelers moving at nearly c in space have most of their velocity contributing to movement through space dimensions and almost none through time. At home, we are moving at c almost entirely in the time dimension, remaining motionless in space. The laws of physics give everything no option but to move at c through spacetime; we can only choose what part of our motion is through the space dimensions and the remainder is through time.

The benefits imagined from warping space are to alleviate this huge difference in the passage of time, so that travelers can go places and return without generations dying off before they return home. The “faster than light” travel is about how outside observers perceive the traveler’s motion, so that they can share in the experience within their lifetimes. Travelers have no need for FTL motion to reach any destination within their own lifetime, with enough acceleration to move at close to c through space. The desire for FTL motion is for non-travelers who don’t want to die waiting for the travelers to return.

The search for intelligent life

The search for intelligent life outside of our solar system is a difficult one. We tend to think that if we expand the scope of our search to include more galaxies, this is sufficient. But we must accept that even if we had the technology to examine every galaxy exhaustively in perfect detail, we are only covering a minuscule part of the search space, which is almost entirely inaccessible to us by the laws of physics.

We can only see something in the current snapshot in time. Let’s try to imagine a search for human radio signals on Earth from the perspective of a distant alien civilization. The Earth is 4.4 billion years old. Humans started producing radio signals in 1894, so these radio signals have been transmitting for the past 121 years. These signals have only had the opportunity to propagate 121 light years away from Earth in that time. Beyond that distance, no alien civilization would be able to detect these signals. Moreover, an alien civilization would have to coincidentally have developed at a pace in which their technology was at least as advanced at exactly the right time to detect such signals during this tiny window in time upon their arrival. This is a 121 year window out of the 13.82 billion years in which the universe has existed.

Universal rest frame

Sometimes we see stories about searching for the origins of high energy particles called cosmic rays. These are massive particles like protons, which have been accelerated by something in deep space to nearly the speed of light. The usual suspects are black holes, neutron stars, supernovae, and other exotic phenomena. The puzzling thing is that some of these particles seem to have traveled great distances, farther than thought possible without losing momentum (slowing down by bumping into things like photons).

What I wonder is whether the human perspective on Earth is far too biased. Einstein’s theory of special relativity says that there is no preferred rest frame in the universe. A fast moving particle is moving fast relative to us, but it is equally valid to say that the particle is at rest, and it is we who are moving fast relative to it.

If indeed there is no preferred rest frame in the universe, shouldn’t there be a uniform distribution of velocities for distant galaxy clusters? Because of the strong influence of gravity, galaxies within a cluster would be bound to move together. But galaxies that are not close enough together will move independently. Wouldn’t one expect that two galaxies separated in space and time by 12 billion light years have an equal probability of moving at any speed between zero and c relative to each other?

However, indeed our picture of the universe seems to be of a relatively organized structure like a web of filaments, possibly with a flow in a particular direction. It is far more accurate to describe the structure as static than it is to say that it is randomly moving with a uniform distribution of velocities. This means there is a definite bias for a rest frame, where the relative motion of the large scale structure of the universe is minimized. Am I wrong?


Modular home construction

I wonder if one day we will build homes like we do the space station—in prefabricated modules. Perhaps rooms can be built in standard dimensions and standard interconnections to adjacent rooms for electricity, networking, coaxial cable, HVAC, hot and cold water, natural gas, etc. Each room would be somewhat over-engineered, but this extra cost is offset by savings from the economies of scale due to mass-production. A home builder would simply assemble a chosen configuration of modules, and provide some finishing touches, such as the exterior facing, roofing, and utility hookups.

This approach would benefit from guaranteed quality of workmanship, replacing skilled labor (e.g., carpenters) with robots and 3D printing, and rapid construction. Moreover, the big innovation comes years later. As technology improves, and the homeowner wants to uptake improvements, it becomes a simple matter of replacing modules, and possibly reassembling them in a different configuration.

Cosmic inflation unnecessary

There is no time without clocks. There are no clocks without mass. There is no mass without the Higgs field. There was no Higgs field before it was frozen out from being unified with other fields. Massless particles travel at the speed of light. When traveling at the speed of light, all components of its motion are through space and none are through time.

Wouldn’t these conditions of the early universe by themselves explain the homogeneous and isotropic qualities without needing cosmic inflation? If energy can radiate arbitrarily far in space without time passing, there is no need for esoteric explanations of how that happened so quickly.


Non-Aggression Principle

The Non-Aggression Principle (NAP) is the only criterion for libertarianism. One might expect libertarians to be contemplating deeply and writing the most scholarly articles on the topic. One might expect libertarians to be forming the most precise definition of aggression and non-aggression. Alas, a search of the most prominent libertarians yields only passing references on the topic. Libertarians believe that the NAP must be self-evident, despite the lack of agreement on the precise definition of what constitutes aggression and what does not.

This matters not only as an academic endeavor. More importantly, it is the libertarian position of non-intervention in foreign policy that depends on a clear understanding of what the principled libertarian view is on foreign aggression. Presidential candidate Ron Paul was perceived by many conservatives, who otherwise supported his libertarian positions, to be off putting for his foreign policy, not because they believed so much in US military intervention, but because the libertarian position of non-intervention is effectively silent on when it is appropriate to deploy military assets in the face of foreign aggression to defend American interests. A libertarian candidate for Commander-in-Chief cannot evade this essential topic, because it is the foremost qualification for the job. It is not enough to hand wave a general remark about supporting a strong national defense. Because of the libertarian position on non-intervention it is incumbent upon a libertarian candidate for POTUS to assure the citizens that with restraint also comes an impassioned intolerance for foreign aggression and no hesitation to deploy overwhelming military force and unspeakable violence in retaliation to any foreign power that initiates force against America. Non-intervention does not mean pacifism or appeasement or isolationism or weakness.

Now what exactly constitutes aggression? And what is the proper response to various acts of aggression? Silence on these questions is what is losing libertarians nominations, because a void in leadership on matters of defense is catastrophic to the preservation of liberty at home.