In fact, SOA appears to have become such an IT necessity that it is the new focal point of CIO thinking and the ‘juicy,’ resume-enhancing project that technicians have been awaiting since CRM projects fell from favour. The rationale behind SOA projects is largely understood and well thought out. Justification often stems around the need to consolidate technologies and simplify support, enhancement and development processes through the use of a standards-based, contemporary technical architecture. The end goal of the initiative is normally to reduce the overall cost of running the IT operation and to provide a better technical framework for servicing the ongoing needs of the business.
But what does SOA actually mean to the broader organisation and how can the business ensure that it realises the benefits that were likely used to cost-justify the not insignificant investment in SOA?
Firstly, let us take a look at the history behind the SOA snowball and try to understand the rationale behind the journey to SOA heaven.
In the modern world, business has morphed and evolved significantly over the last 20 years in response to an ever changing consumer market and higher degrees of regulation and scrutiny. Mergers and acquisitions have
significantly altered the business landscape and one-time household names now trade under the brands of foreign parents or worldwide conglomerates. This is illustrated quite poignantly in the UK financial services sector where High Street banks such as Williams Glynns, the Midland and the Trustee Savings Bank (TSB) are now parts of the much larger RBS, HSBC and LloydsTSB groups respectively. Whilst this sort of M&A activity presents all sorts of well documented challenges on the business side of an operation, the technology implications are often not understood until the deal is done. Too many times, the newly formed company wakes up to find it is now running multiple lines of business applications on disparate platforms and technologies. It has its new customer base distributed across different sources of data and, in many cases, the same customers duplicated on both the “old” and new systems. Another common problem is a duplication of roles and responsibilities, not only at the operational level but within senior management also.
It is not atypical for organisations to have multiple CTO’s following such a merger with a stated management strategy of waiting for the “cream to rise to the top.” Such situations obviously run contrary to a coherent and aligned technology strategy. …and to generating the business benefits used to justify the acquisition. But we’ll get to than later in this paper.
Whilst SOA cannot solve the HR challenges described previously, it can help to address the technology issues encountered when operating a business running multiple disparate applications. This is not a new phenomenon but one that has been addressed in different ways over the last 15 years. Some technologists argue that SOA is just Enterprise Application Integration (EAI) with XML. In a most simplistic sense, that is true. But it is the standardisation of the whole approach that ‘ups the game’ in terms of flexibility and ease of deployment.
EAI was first coined as a phrase in the mid 90s, giving rise to a number of application vendors specialising in integration technology. The products tended to fall into two specific categories, although there were certainly some hybrids and fringe players with more ‘novel’ approaches to EAI.
The two main categories were/are hub and bus technologies. The hub, as suggested, is logically positioned in the centre of the architecture and point to point integration is built between the individual applications and the hub. All existing systems communicate via the hub and any new applications being introduced to the organisation often have to demonstrate their ability to integrate with the hub as part of the product selection process. The absence of any specific integration standards within the early EAI vendor community made life difficult, and costly, for application developers trying to please everybody all of the time.
The bus approach involves the provision of a messaging or service layer above the underlying applications with all applications communicating with each other via the bus’ messaging interface. Buses also tend to be more comprehensive in their functionality, supporting (amongst other features) transactional integrity, rollback and fault tolerance. Buses are generally deployed as the integration mechanism of choice for an SOA framework and the term ESB (“Enterprise Service Bus”) is often viewed as being synonymous with a Service Oriented Architecture.
In early EAI implementations, a degree of ‘clunkiness’ was often a by-product of the inelegant integration methods required to circumvent the closed, proprietary nature of many legacy applications. In the financial services world, many key line of business systems were originally developed in the late 70s and early 80s at a time when standards were focused on coding practices. In those days, the entire IT department often existed because of the company mainframe. Even today, many seemingly ‘up to the minute’ financial services companies rely on application and database technology originally developed 30 years ago. New university graduates who grew up in the Internet age are surprised to learn that massive mainframes running COBOL applications are the heart of the elegant web sites and wireless connectivity provided by many apparently “modern” companies. Also remember that it wasn’t long ago when aging COBOL developers were blowing the dust off their IBM 3090 MVS manuals to supplement their pension funds from the impending year
2000 calamity. Those systems still haven’t all retired, even if many of their creators have.
This clunkiness came in a variety of forms and often invited amusing terminology to describe its characteristics. The ‘Lipstick on the Pig’ term was often used to describe the practice of screen scraping or network spoofing a legacy application to provide a slicker, apparently modern user interface. (Today’s recent use in the American political scene is humorous to many older technicians.) For the most part this was a successful means to a temporary end, but only with consistent communication between the legacy system developers and the technicians applying the virtual lipstick.
Additionally, while this practise delivered “prettier” user interfaces, the end user rarely benefited from any additional functionality and the business was still hamstrung by the limitations of the aging legacy system.
This phenomenon also exists in some EAI and SOA deployments where the implementation has been solely focused on the creation of a modern technical infrastructure without a strong, parallel objective on driving business benefit from the new technology.
The term “Swan Syndrome” is alive and well today as a description of many technical architectures. To the user the system appears elegant and flawless as it operates with seemingly effortless efficiency, hence the part of the swan above the surface that one sees. However, underneath the “waterline,” (where the swan’s legs are feverishly kicking and manoeuvring) there is often a variety of manual or outdated operations and inefficient processes which are keeping the entire operation afloat (to complete the overuse of the swan analogy). Even with many modern SOA projects, the difficulty encountered by the technologists revolves around the age old issue of integrating with computer systems that were not originally designed to be integrated with. So various ‘devices’ are either purchased or constructed to sit in front of the legacy applications in order to make them easier to ’hook up’ to a Service Oriented Architecture.
To be fair, it is certainly accurate that some organisations have replaced their legacy systems with shiny new technology that fits well in an SOA, however the reality for most companies is that budgets do not exist to “rewrite” systems that, simply put, still work.
So where does all that bring us in terms of delivering a successful ‘business friendly’ SOA implementation that delivers the tangible business benefits that the CTO promised the CEO when the project started? Many CTO’s might prefer that we ask that question without the pesky “business friendly” and “tangible business benefits” parts.
The reality is that a well implemented Service Oriented Architecture can bring significant benefits to the IT side of an operation. These may include:
But, what about that business-friendly implementation with those tangible benefits? Unfortunately, that’s not as common as the technical achievements without the addition of another more business-focused component.
It can be easily gleaned from the discussion so far that SOA is typically introduced as an IT lead initiative. In many cases the whole rationale behind investing in such a programme is to simplify and bring down the cost of running an IT operation. What many organisations fail to recognise is that by embarking on that very process, all sorts of opportunities to reduce business complexity and costs open up as well.
For example, let’s explore the experience of a typical High Street bank. As mentioned earlier, acquisitions and growth often result in a customer dealing with a large organisation whose businesses (and impact on that customer) may go well beyond retail banking. That same bank also offers mortgages, credit cards, life insurance and other financial products. As with many companies, those business units often operate almost as separate companies with their own systems, personnel, IT departments, administrative operations, management structure etc. In fact, the situation can become even more confusing to the consumer when that bank has a branch that specialises in servicing retail bank customers situated adjacent to an office that specialises in providing mortgages. Generally companies recognise this issue as part of their reorganisation programmes, but the strength of certain brand recognition means that phasing out duplicate operations is not an instant activity.
This then brings us to processes; the mystical world in which companies interact with their customers and their IT applications.
The whole subject of process improvement strategies and associated technologies has had an interesting history. Beginning with imaging and routing in the early 1990’s, the subject has evolved through workflow, work management and process automation to what is often called Business Process Management (BPM) today. However, even BPM has evolved into a term that is used so broadly that it fails to accurately define the many powerful business solutions that are now available.
An enormous amount of effort has been spent over the last 15 years in the reengineering, streamlining, automating and outsourcing of business processes. There are a plethora of application and toolkit vendors out there with a whole range of solutions available to address organisations’ process management requirements. Most of the main suppliers in the field have demonstrated an ability to deliver varying degrees of business benefit as part of a bigger process improvement process, and while there are a number of ‘horror stories’ surrounding these projects, the problems generally result from poor implementation practices rather than inherent problems with the technology.
Many organisations who have run workflow, process automation or BPM programmes for 5 years or more have reached a stage at which the technology appears to be rolled out across all eligible business operations. This leads to the perception that that technology has reached its natural end of life given that there is no obvious extra benefit to ‘squeeze out’. This is where SOA comes into the game. Typically, process management deployments were constrained by the ability to efficiently integrate with the legacy systems and technical processes in an organisation. Whilst human processes can often be changed with little monetary cost, the effort and impact of replacing a legacy mainframe application can be enormous. By implementing a comprehensive and well thought out SOA strategy, processes can be implemented in a more agile fashion without the need to commission wholesale changes to IT applications.
Let’s now return to our High Street bank analogy to illustrate how SOA and process management can be complimentary. Consider an operation with a branch network that wishes to sell and service a varied portfolio of financial offers. This may include mortgages, credit cards and traditional ‘High Street’ banking products. However, that bank’s retail customers have become accustomed to the highly convenient manners-branches, Internet, ATM, mobile phone, etc. – in which they interact with the banking side. If the bank wishes to provide similar access to the IT systems that support those other products, the inevitable integration challenges arise. Without an SOA (or even modern EAI) architecture we are back to our Swan Syndrome and Pig Lipstick situations. This becomes exacerbated when the systems that deliver the functionality to the branches have to be modified for the inclusion of additional acquired business units and compounded further when one part of the enterprise is expected to pay for a modification that will also benefit another. Does this sound familiar?
So now we look at how to approach an enterprise wide SOA process management programme and how to leverage the investment in a way that all stakeholders in an organisation can benefit from the monetary and personal investment.
Knowing where to start is the key, and often external help in the form of experienced credible consultants with industry, in addition to technical, expertise can accelerate the process in the right direction. Adopting the right SOA and process improvement framework is also imperative. Whilst the IT department is closest to the nuts and bolts of the line of business applications, the business management are the ones closest to the operation of the company and the expectations of an ever more demanding customer base and ‘cut throat’ market competition. Fortunately, there are service definition frameworks available for purchase (at not inconsiderable cost) that define a model SOA service profile for particular industries. One example is IBM’s IFW for Financial Services which defines the service profile for a financial services company. It is a good example of a key reference model on which an SOA programme can be built or, at a minimum, validated against. There are other models emerging in multiple markets, which should serve to ease the entry cost for many companies who are just starting to
focus on SOA and process improvement.
Temptation comes in the form of the tendency to architect the service layer to suit today’s business processes. What would appear on the surface to be a business-friendly approach has proven to be erroneous in that it results in tying the processes and the service layer too tightly together. This eliminates the flexibility that should be a hallmark of a well-developed SOA. It is also a mistake that is repeated with alarming frequency. (Even as you read this article, I assure you that someone, somewhere in the world is beginning to define a service layer exclusively to support an existing set of business processes. And their boss is praising them for promoting “IT-business synergy.”) The usual outcome of this approach is a set of high level services delivered through an SOA architecture that have little or no reusability. The next project comes along, perhaps to deliver an interactive online solution or enhanced call centre operation, and a whole new set of services must be implemented based on the needs of that business process. This can continue until the SOA-based applications are in the same state as the disparate legacy systems that they once sought to eliminate.
The missing link is the piece of the solution which translates the low level granular services (perhaps “Get Customer Address”) into information that the high-level business processes (perhaps “Call Center customer search”) can usefully consume. This is normally achieved using graphical orchestration tools which enable the binding and execution of multiple services together in a manner that delivers discreet business functionality.
Consider our High Street bank again, the SOA service layer may expose a set of low level services to fetch a customer’s account information from the various disparate applications (retail banking, life insurance, credit card, etc). If the process (let’s keep the call centre desktop example) wishes to retrieve all customer accounts, then the orchestration layer is responsible for invoking the individual services and aggregating the results in an appropriate manner to the initiating process. If additional services are added to the layer then the orchestration component is responsible for the invocation of the new service call and major surgery is not required to the SOA infrastructure (and the call centre desktop application) to accommodate the change. Similarly if parts of the business operation are to be outsourced, then new orchestration jobs can be produced to support the new operation without the dependency on IT making changes to supporting software.
The diagram below illustrates this abstracted approach to deploying process management across an enterprise
SOA framework.
At the top we have a ‘transaction’ which is used to drive the process through the various required steps in the process which are also illustrated. (Just so I don’t ruffle the feathers of the technical community, I acknowledge that the process depicted is oversimplified in reality, and for illustrative purposes only.)
At the bottom of the picture we represent our various systems of record. Included in this level is a CRM component, which in reality is likely to be duplicated across the various operating units but has been separated out, again for illustration. The ESB provides the service interface as described earlier in the document and contains the definitions of the specific profile defined services.
The important piece in the middle provides the orchestrated sub-processes which connect to and aggregate across the individual service calls. This is the piece that is often overlooked at the start of an SOA initiative.
Once we have this model in place, then there is an opportunity to consolidate the disparate processes across a typical enterprise. Again using our real life High Street bank example, a change of address process is fairly consistent across the different business operations. (Again, let’s exclude obvious complexities like a mortgage account from this example.)
There exists a major opportunity for large organisations, particularly those that still operate silo based operations, to leverage significant business benefits and cost savings from an investment in SOA and process management technologies.
Various divisions within an organisation will argue that they are in some way unique and that they cannot be
bundled into some consolidated process initiative. However the experiences of several large corporations
suggest that this is not necessarily the case. The traditional argument about the uniqueness of the processes that interact with line of business systems has been put to bed by the combination of process management and SOA deployment.
If there are opportunities for business operations to leverage benefit out of SOA investments today, then what other current trends may benefit also? Well, since 2006 the emergence of SaaS (Software as a Service) technologies has been gathering pace. Indeed, an impediment to enterprise SaaS-based solutions is the challenge of integrating the service-based architecture at the SaaS provider with the legacy applications that reside at the customer’s premises.
Early SaaS deployments either deployed workstation based adapters or avoided the thorny subject of host application integration as a whole. In the new SOA world with the right levels of network connectivity and security, the SaaS option for more comprehensive application integration becomes more viable.
Additionally, we have seen the development of outsourcing and off shoring operations that have managed technology logistics through the use of VPN or ndedicated networks. If business potential truly exists for concepts such as mass collaboration and “crowdsourcing,” secure service-based access to organisations’ systems of record will need to become more common.
Techniques will need to be developed to ‘anonomise’ human tasks so that secure information is not compromised. The use of process management and SOA technologies will need to evolve to support these and other potential trends in both customer expectations and workforce deployment.
I have a favourite quote from the much heralded US investor and financier Warren Buffet: “In the business world, the rear view mirror is always clearer than the windshield.” To a large extent, that quote can be applied to the world of software applications. IT history is littered with the remains of impressive technologies that never lived up to their expected business value. This does not have to be the case with Service Oriented Architectures. Large organisations are already proving that a well thought out investment in SOA and process management solutions can bring business benefits that we won’t be ashamed of five years from now.
Boosting efficiency by enabling third-party administration without worries about compliance issues.
Offering key benefits for insurers and brokerages in a proven, scalable process management solution.
Delivering the right work to the right person in the right priority to make the most of resources.
MBA's CREF 12, February 5-8, Atlanta, GA