ESBs again

I've posted in the past about what ESBs really are (and why we need them). As evidenced by this entry in InfoQ the discussion seems to be widespread in the industry. I sense that we are entering one of these "grand" debates like SOAP vs. REST. This time it's the question "do we need an ESB or are WS standards enough".

BPEL4People - The Architecture

As I explained in my last post, the new BPEL4People draft standards are intended to integrate people into those business processes that are being controlled by a “requesting application” ( I think it’s pretty safe to assume that in most SOA implementations this would be a process engine running BPEL). The draft spec itself is split into two parts. The first (WS-BPEL Extension for People) explains how BPEL can be extended with specific process constructs to deal with human interactions and lists a number of ways (called constellations) that the management of these tasks can be implemented. From an architectural point of view the most interesting one is when the task infrastructure is decoupled from the process infrastructure and a separate spec (WS-Human Task) has also been released to cover this. I’ve tried to convert this last “constellation” into a more detailed architecture diagram:

How does this all go together? Let’s start off at the simpler end. The basic assumption is that people provide services by acting on tasks which have been assigned to them. To see and handle the tasks which have been assigned to them is done using a task client. This is not specified by the specification, but it is a relatively safe bet that will take the form of a user interface with some kind of task list which the user can interact with (for instance, in a SAP environment this could be a universal work list). The user interface interacts with a task management component which stores the tasks assigned to the users. Again, it’s pretty likely that the task manager will be a central component that can hold the tasks for a number of users each of who has their own user interface. The spec defines the services which the task manager component exposes to a user interface. I’ve grouped them into three categories:

• Life Cycle, i.e. claiming a task (i.e. saying you’ll do it), starting, suspending and stopping tasks etc.
• Query, i.e. finding tasks based on certain criteria. The most important one here is getMyTasks.
• Administration services such as explicitly nominating someone to a task

To do this the task manager needs to access some information regarding the people. The spec assumes a people directory with an interface for queries, but doesn’t actually fix what this looks like.

So how do the tasks get into the task manager? Conceptually this is very simple. The task manager exposes a service for the business function (or functions) which need to be performed by a human. The service can then be called from another application or a process engine. Once the service has been invoked, the task manager creates a task which, as far the users are concerned, appears in a task list and can be dealt with over the task client. The user can then decide to act on the task until it’s completed. I’ve sketched a simplified scenario (using as an example a credit approval service) in a UML sequence diagram:

I must admit, I find the interface between the requesting application (the process engine in our scenario) and the task manager rather specific to the business logic required. The task manager has to expose specific services such as ApproveCredit rather than use a more general approach with services such as TaskCreate. This implies that any service developer will have to use an API which is specific (i.e. proprietary) to the task management component being used.

The above only sketches some of the main features found in the draft specification. Other features (such as notifications and placing service calls in a context) I haven’t included. The spec also defines in detail the messages between the main components. I look forward to more articles detailing what the specs actually mean for architects.

SOA: The Human Angle

Most discussions on SOAs assume an architectural model in which services are provided by software and these services are in turn consumed by clients (e.g. user interfaces) or are orchestrated by a process engine. A good example of this type of thinking has been recently published by a set of authors from IBM in IEEE IT Pro. In their article, “S3: A Service-Oriented Reference Architecture” they describe a metamodel which:

“ … captures architectural building blocks … as well as the relations among these blocks and among layers, interaction patterns, options, and architectural decisions.”.

However , although most of these SOA models talks about business processes they give scant attention to the role of humans within those process. For instance, at one stage in a credit process we might need someone to approve the loan. Effectively this means that the approver is providing a service – deciding on loans using their experience and intelligence. In the IT Pro article we come across this statement:

“An SOA ... would support that process by creating a composite service that choreographs the information flow among a set of services and human actors”

But that’s it. How the human actors are actually integrated into the SOA is not explained. This is fairly typically for most of the literature in this area which assumes an architectural model such as:



I was therefore interested to see that the BPEL4People draft specification has been recently published by a consortium of companies with the intent of getting this ratified by OASIS as a WS-* specification. BPEL4People is a set of extensions to the BPEL specification that address scenarios in which humans are involved in business processes. How it does this from an architectural point of view I’d like to explore in my next entry.

German government specifies SOA

The German federal government has released an IT Architecture for the administration. Surprisingly (or perhaps not, depending on your PoV) it's service oriented.

Currently it is only in German, but often such documents are translated into English and published on the same web site later.

More on ESBs

Just to muddy the waters in trying to explain what an ESB is (and why we need one) I came across this (dated 2005) presentation from Jim Webber (you have to display it in presentation mode to see it all). Jim doesn't hold much with ESB vendors and believes that the "bus" should just be SOAP and WS-Addressing.

The slide set reminds of a colleague (I wish I could remember his name) who once said "it is almost criminal to let a vendor determine your architecture".

Point-Point, Hub-and-Spoke, ESB

My colleague Mark Masterson asks in his blog about the “fitness of the classical "hub and spoke" architecture in a modern enterprise systems landscape” and “what, exactly, is the purpose of an enterprise service bus (ESB)?”

As he points out there is a lot written about this, but nothing seems to be conclusive. When confronted by such situations I attack them by asking “want do we really want” and then work backwards to “what have we got”.

So, the first question is; do we want a “hub and spoke” architecture? Well, this was proposed as an answer to the point-to-point interconnection style. Continuing on with this ruthless analysis (so beloved of Dr. House) let’s ask the seemingly dumb question; do we need a point to point interconnection style?

Point to point came about because people needed to transfer data between applications. As time went on, more and more applications were written and, with them, more and more point to point interconnections were produced. This lead to a nightmare IT landscape which nobody could understand, let alone maintain (I once did an application inventory for a client in which the interfaces were also documented – the result filled four sheets of DIN A0 and made for interesting wallpaper!).

To solve this the hub-and-spoke architecture was proposed, which lead to EAI tools etc. etc.. But that was then and this is now and I think we have to ask again: why point-to-point?

In the brave new world of SOA we don’t have applications which need to pass data between themselves. Instead we have autonomous services. These are arranged in a hierarchy (although not a strict one) . Services are consumed by other services (e.g. composite services call atomic services) or, at the top nodes, by service consumers such as user interfaces. If we start building systems to implement this architectural style, we soon start to realize that certain functionality is required time and time again. We need to implement small processes which call services in predefined sequences, data needs to be aggregated from many sources or needs to be distributed to other services. The data needs to be transformed, enriched, split up. In addition we need build low level coordination between the services, so that we can implement concepts such as transactions and security. We also need to ensure that data is really transfered to the services (i.e. reliable delivery) etc..

All of this is what we really need to implement an SOA. But instead of implementing this all ourselves, we buy these things from somebody who has packaged it all together and given it a name - ESB.

Using this analysis means that the original question of fitness of a hub-and-spoke system is a question that does not need to be answered as a hub-and-spoke architecture is irrelevant for a modern systems landscape if an SOA style is used.

Designing Interfaces and REST

I've blogged in the past about how the design of interfaces in distributed systems is an important, but often overlooked topic. Instead the current debate seems to be around the technology used - SOAP, REST - rather that the interface design itself.

I was therefore interested to come across this blog entry from Joe Gregorio in which he applies REST technology to the design of the Apache DayTrader benchmark. What I find striking about this entry is how much care he has given to the interface design, reducing the original, and somewhat ad-hoc interface specification, to simpler principles based on a few entities and operations.

Tools for Globally Distributed Development (GDD)

Working for a company that routinely spreads its development out through the world I was interested to come across this report from Ryan Bagueros on how a small software development company can set up the tool support for GDD on the cheap by using tools available on the web. As he says:

"With some careful setup, any group of developers can compete with the budgets used by multinational software development firms to distribute their application engineering ..."

Of course, these tools do not need to be used on the scale of global development, but can be used on smaller scale developments where the developers do not all sit together.

Using BPMN

As an architect I often have to sketch the processes to see how everything goes together. Over the last year I been increasingly using BPMN to do this, either on paper or with Visio. Unfortunately there doesn't seem to be much out there in the way of books which explain how BPMN should be used . Instead, I use a couple of tutorials written by Stephen White from IBM. Those with patience can use the textual version, for the impatient the slide show is the best.

As to sketching processes in Visio a number of stencils (with various levels of completeness and features) are available on the bpmn.org site.

Developing rich internet apps - what to use

A quick round up of what seem to be the current major development libraries and/or environments for developing RIAs.

Flash is the well-known browser plug-in from Adobe as well as the development platform. Adobe have also announced in the mean time - but not released - Apollo as a cross platform run-time that handles Flash, Flex, PDF, plain HTML or Ajax based user interface. Effectively a type of universal client instead of the browser.

DoJo is an open source javascript library. As well as the usual UI components, one feature I find interesting are components to store data on the local drive. I first saw this feature used in TiddlyWiki.

script.aculo.us is also a javascript library. It is based on the Prototype javascript library.

Google Web Toolkit (GWT). The GWT lets developers write code in Java which is then converted to JavaScript apparently apparently freeing the developer for learning JavaScript (Java developers are assumed here, not a bad assumption for most enterprise development work). This has been nicely written up in a IBM developer world article.

Are there any more?

Loosely coupled interfaces

I posted a short while back an entry about designing generic service interfaces. By chance I happened to come across this (rather old) blog entry by Carlos E. Perez in which he proposes extending speech acts to define loosely couple interfaces. He bases it on a paper by researchers at Stockholm University in which they try to provide a universal framework for human communication in an economic domain, i.e. the business world (the paper is still marked as work-in-progress, so I guess it hasn't been peer reviewed). To do this they use a a triple:

(intention, action, object).

The values each element can take on are:

	Intention		Action
	propose		create
	accept		change
	reject		cancel
	declare		none
	query
	reply
	assert

So, to create a new customer record the triple could be:

(propose, create, aCustomer)

the service could then return with:

(accept, create, aCustomer)

or, in the case of an error could return:

(reject, create, aCustomer)

You can have a lot of fun thinking up new combinations. One thing to notice is that services themselves don't seem to have names. Instead, one could envision a dispatcher which takes each received triple and then finds the correct service implementation.

If this would work out in practice, I do not know. But it seems to aid support to my argument that the REST vs SOAP debate is about which technology should be used, and not about designing interfaces to achieve loosely coupled services. How to achieve this should be the real debate.

Quantum Computers

Really?

The next n-tier architecture

It seems that the now traditional three tier architecture may well be slowly coming to an end. For specialised high performance applications, the bottlenecks between the tiers have already been pushing developers in this area to turn to specialised data caching platforms such as Tangosol, or to unified application platforms such as Intersystems Ensemble. In the data caching area the architecture seems to be consolidating to a cached data pool which is accessible from many machines in a cluster. This trend is now being followed by vendors of "traditional" application servers such as BEA and IBM. For SOAs this has the advantage that each service accesses the same data in the data pool, rather than having to aggregate data from different data sources (see diagram). The application platform deals with all the hard problems of data caching such as data consistency, cache actualization etc.


Not so long ago I saw a presentation from GigaSpaces who also provided a cached data pool. What is interesting about the product architecture is that they have revived the old, and hardly adopted, JavaSpaces concept. The data pool (or space) is a set of Java objects which are effectively in a shared memory accessable from each machine in a cluster. Each object can be accessed by providing a "template" which matches the contents of the objects and returns one of more of the matching objects. Templates can also be set as triggers, so that creation of new objects with particular data characteristics can notify another Java object. If you're still interested, a good tutorial on Java Spaces can be found here.

Building Virtual SOAs

Doug Kaye has written about how he used Amazon web services to build the infrastructure for his application, i.e.

• elastic compute cloud to run the apps
• simple storage services to store the content
  
• simple queue services to connect the apps.

And talking about building applications, Coghead (which has recently gone into limited beta) lets users define screens which can trigger user defined orchestration processes and data mappings. Just about what every ESB/EAI/SOA tool can do except that this uses a SaaS model.

Using these tools, what stops you defining a "virtual" SOA, in which the Coghead defined UI triggers the orchestration which calls up web services implemented using the Amazon virtual infrastructure?

On Generic Interfaces

In his comment on my last post , Steve Vinoski warns against getting into “pseudo-religious technical debates like "REST vs. SOAP”. I agree.

However, for an enterprise architect implementing (or planning to implement) an SOA, the decision still has to be made as to what technology should be used – a decision which could have a considerable influence on the costs and success of any SOA initiative. Such a decision needs to be based on arguments which corporate management understands, rather than hearsay about what apparently works or does not.

The difficulty in making such architectural decisions is not on seeing what simply works. SOAP works, so does REST. Way back in the mists of time (i.e 1999) I worked on a project which used a REST style of communication between the client (an applet - remember those) and the server. I made the decision to use a simple HTTP interface as no other alternative existed then (according to Don Box, the SOAP spec only shipped in 4Q1999). It actually worked surprisingly well (and is still working to this day). REST can work in an enterprise environment.

The problems start when we come to architecture in the large. In large architectures other factors come into play, such as how to manage the interfaces in the face of changing requirements. As Steve mentions in his article "... some of the architects and developers I know who work on large SOA systems ... figured out the uniform interface constraint on their own, without ever hearing of REST". Developing generic interfaces does help, but how generic does an interface need to be?

In the REST model the generic interface supports the four interaction verbs described in the HTTP spec, i.e. GET, POST (equivalent to create), PUT (equivalent to update) and DELETE. However, taking one extreme it is possible to reduce POST, PUT and DELETE to one verb - MAINTAIN - and split the data payload into a before image (BI) and an after image (AI). Specifying the BI and leaving the AI empty is equivalent to a DELETE. Specifying the AI and leaving the BI empty acts like a create. Specifying both is an update, with the advantage that the system can check if the data has changed in the meantime by looking at the BI. Apart from database interfaces I haven't seen anything written about this technique in creating SOAs. I have, however, seen one system which implemented it in a web service context and with SOAP.

As the above example shows, you can still provide generic interfaces with the SOAP/WSDL combination as well as with REST. It seems to me that the SOAP vs. REST debate is arguing about the wrong things. It should about how to design interfaces rather than the technology used for their implementation.

REST vs. SOAP Reloaded

It seems that Google have decided to drop its SOAP interface in favour of a REST style interface. This again raises the REST vs. SOAP debate. I was therefore interested in reading this article from Steve Vinoski of IONA Technologies. Although he tries to "explain REST from the viewpoint of someone steeped in SOA..." I had the sneaking feeling that he came down on the side of REST, which is even more surprising when you realise that IONA was one of the original exponents of CORBA and has been heavily pushing SOAP over the past years.