Friday, December 10, 2004
Today most B2B services are of an Asynchronous nature. They involve state-full processes and are usually based on a model of establishing an initial business request followed by a series of milestones indicating how the request is being handled. In most cases the timescales for these processes are based in hours rather than minutes.
Our current model is based on message passing that when they are sent are purely bits on a wire. It does not have a URL, it can not be addressed even when it arrives at its destination. Here it is consumed and fed into the receiving organisation. If the destination is not responding the result is bits fading in the ether. Should the destination system come back to lie it does not know that the message ever existed. It can not of itself resynchronise to a state that allows it to continue.
Another model is to use the idea behind publish and subscribe. In this model the person with some information that it wants to make available publishes this on to a web site. The expected destination has already subscribed to know about this information. The destination will then be informed that a change has happened. The result of this is that the destination goes to the publisher’s site at collects the information. If for some reason the destination has not received the prompt it can reset itself by going to the publisher’s site and picking up the set of information that they have not previously received.
There is a set of simple technology that is being used in large volumes today by millions of people to receive information from the internet. These solutions based are based on simple web technologies, of HTTP, XML/HTML and RSS. These technologies are being used for major news organisations such as the BBC, Reuters and also by small publisher’s who are writing Weblogs.
This technology has now become common place and is regarded as rather low tech. This is a good thing. It is tried and tested and is becoming ubiquitous on the various computing platforms. In most cases the technology is very simple to manage and the development tools are available to make the coders job very simple.
What I envisage is an amalgam of the event based protocols such as Web Services, ebXML Messaging, etc and the publish and subscribe model from news syndication. The aim of using both messaging and static publishing is to allow for a platform that can support high volumes using simple technology and offering high resilience.
Taking a simple transaction such as raising an order. The default mechanism for sending the order is via a messaging gateway based solution. At the same time the message is despatched over the network it is place in a publicly facing web site (secure if necessary) and an entry in an appropriate syndication feed is updated (a new RSS Item entry). The URL for this stored entity is referred to within the message. This could be directly, the URL is in the message, or indirectly, the URL is derivable from information held within the message.
On receiving the message a receipt would normally be sent back indicating that message had been successfully transmitted. With the use of the Stored Message this is no longer necessary as the receiver now take the message and also stores it at a known location and updates their RSS entry. The reason for this approach is that there is not a receipt to a receipt and if a receipt fails to get through or be sent there is some ambiguity as to the state of the process associated with the messages. In this case it is clear that a message has been received by simply enquiring of the RSS feed. This is a step that the sender can choose to check or not. The sender is free to continue.
When the Receiver processes the order they reverse the process except that the message being sent back is linked to the original message. This could be by using new directory structure for each new conversation or by making sure that the items are linked in the RSS items. When the company who raised the order gets the reply they also link it to the original message on their web site and update their RSS entries.
In the unfortunate event that something goes wrong and the messaging interfaces fail. The party wishing to send messages still attempts to use the messaging route while at the same time keeping the web site based message store up to date. When the Gateway that has failed returns, it will be able to resynchronise with the remote system by using the RSS feed and the Web Message Store. Once it has resynchronised its Web Message Store should be the reverse of the remote systems.
Within this scenario the two ends of the interface have been operating according to the published process. However, there are times when things occur that need to be flagged to the remote system. This might be the fact that a crucial back end system is down or that a series of messages have been sent but were not part of the formal process. Using the Web Message Store and the RSS feed it is possible to keep the other parties systems informed of things that are not directly related to the process but may be service affecting.
Using XML messages and storing them increased some companies storage by six times over previous EDI mechanisms. In order to mitigate this the Web Message Store would not store raw XML but compressed XML in a format agreed by what ever mechanism delivers the best compression and can be used by both sides. Typically XML compresses extremely well and the amount of storage required could be similar to that of a reasonably busy Web Forum.
The reason for the Web Message Store is that it moves the messaging interface into something visible and something that can be view both by humans and machines and can be monitored by machines and humans. There are several tools that can aggregate RSS feeds and make them appear as one feed. This would allow a B2B gateway site to have access to all the remote stores for their partners using very low technology solutions. What is more using the amalgam of both Messaging and RSS you allow the point of entry for customers to be a great deal lower. Messaging solutions typically take time to test and integrate. Web Sites are now commonly managed by even the smallest enterprise and publishing information on them is fairly straight forward. This would allow an enterprise to enter the market of B2B simply by placing XML messages in the Message store and making sure the remote system knows to pick them up. With this simple agreement a set of B2B interfaces could be up and running in days. If the interfaces need to become more real time the introduction of the messaging feed could be developed, but is not necessary for most solutions deployed today.
There is another scenario which is the possibility of using a common Web Message Store supplied and managed by a third party. In this model both the sender and the receiver would simply use the Web Message Store as a simple Web Site to which they publish their XML messages. This would allow the third party to take the messages and either despatch them to the other party using a mechanism agreed with them or simply just manage the logistics of the common platform.
Internet vs Intranet
All the discussion above has been around the B2B space, But should not this model be employed within an organisation. Does integration work so well inside enterprises that the need for resynchronisation between systems is not required? Do hubs get out of step? Do we always know the destination of the information that a system publishes.
Inside an organisation the use of the publish and subscribe model could allow for multiple client to consume information using very low technology solutions. The ability for most applications to read XML and to attach to an HTTP stream makes it probable that within an organisation this method might actually have greater effect. No longer would messages have to be sprayed around the company they could be published once and picked up by whoever needs them. The only major discipline in this environment is when the interface being implemented is bidirectional, there is a need for both parties to register with the others site. Once this is done the results can flow unrestricted by downtime.
With a messaging system it is impossible to withdraw a message should the circumstances of the original transaction no longer apply. With the Web Message Store it is possible to remove entries should they be causing problems or if the due to failure of the receiver the transactions are no longer applicable. This means that when the roll forward takes place at resynchronisation time only the messages that still apply need to be rolled forward.
As the Store is a simple Web Site model removing entries is as simple as deleting web pages. This might be regarded as a good thing in the event of trouble but might be an issue if access right to delete are too freely given.
B2B Gateway that implement messaging solutions can be complex and difficult to manage in error scenarios. By employing an open approach to what messages have been sent and received it is possible to offer a low technology based solution to manage failure and possibly offer a lower entry barrier for users of B2B solutions.
Within enterprises the same issues occur with regard to message based solutions and using a simple store of messages sent and received could allow for basic solution to a complex integration problem.
Thursday, December 02, 2004
Recently we have been seeing a push towards the use of an extreme programming methodology. This is based on the notion that it must be possible to build things quicker than we do. At the same time there has been an initiative to put together a consistent architecture based on the SOA model. Alongside this is a realisation that there is an amazing amount of useful high quality code out there in the Open Source market. The emerging picture is also beginning to include the idea of Corporate Source where the techniques of making the source available in public for Open Source is used within a company.
The question that arises from all this is can this all work. Is it possible to gain the apparent efficiencies of the Open Source, eXtreme Programming model within a Corporate environment that thinks it needs a strong yet flexible architecture.
As I thought about all this it became clear that the Open Source model has some important things to offer to a company above just the savings. The main thing it does and XP backs this is that it enables people to relinquish ownership of code as others have to look at it and the result can only be better. If you add the JUnit style test harnesses that would be available integration testers you end up with a completely different atmosphere. No longer is one individual key to any one project, as their code is available for other to decipher and probably re-factor. This alone in my view would be enough to encourage companies to not only use Open Source code in their projects but also insist that developers, even external companies make their code available as part of the deliverable.
Having been down the Open Source route myself there is nothing better than knowing other will see my code and other may use it to make me try to make it good. This encourages one to talk to others about how to do things and to look at others code to solve problems you are experiencing. There is also nothing better than to know that others are using your code either through being asked what it can do or by receiving bug reports. The buzz from having produced something useful is what motivates a great number of programmers. The other buzz that the new programming models produce is that using the Test driven method I have written for the first time complex code that I know works. The relief that this has given means that I can relax when I hear that others are using my code.
So as you can see I am very keen to see the traditional corporate programming model of hiding the code within the team broken and allow for ALL code within a company to be open. It is also worth noting that the Open Source projects are both well managed but also fairly relaxed. Sourceforge.net allows thousands of projects to be created knowing that only a few will be successful and move to a version 1.0 product. This does not matter, it still enables a developer to borrow code from someone if it does the job even if it is slightly rough at the edges.
To enable this corporate sourcing it would be necessary to supply a free to use platform similar in nature to the Sourceforge platform that supplies both places for managing code and the ability to publish web pages. It also have processes for publishing new versions and any associated broadcast and bugs etc. The key to the success of this kind of platform is the relaxed fairly free manner in which it is provided. If corporate policies such as those applied to official web servers were to be applied the result would be less use and a poorer take up of the facilities.
Before going down a wholesale Corporate Source route it is worth asking the question about how much a corporation benefits from hiding its resources in house. If a company published its code for all it did would it be giving away so much that it would in fact be vulnerable in the market. I am not convinced that the average corporation is better off hiding its code. I suspect that in most cases the reason for hiding it is a fear of embarrassment not a genuine competitive advantage. In most industries the desire to reduce costs has led to the introduction of industry standards bodies. If companies put their efforts into producing code for their own use in a standard way and then publish that code, the adoption of the standard would be more widely taken up and the results would be a more widely used standard and probably improved code.
The final point on the Open Source debate before turning to look at SOA is the use of COTS packages and external outsourced programming shops. May be it is time for corporations to ask for the source code to a product when they buy a license. In this way they are able to monitor quality and possibly even temporarily fix bugs themselves, sending them to the supplier in the same manner as Open Source patches. The other effect of this is that the COTS product would no longer be subject to problematic integration points. If the corporation wanted to access data in a none COTS manner they would have the code to be able to test the effect and also build in the hooks they need to do the work. It has been this reluctance of the COTS vendors to support genuine SOA interfaces that has meant slow delivery timescales and poor integration solutions. It would also mean that any extensions the corporation made to the COTS product could be done in such a manner as to not affect the underlying COTS product and therefore improve the likely hood of being able to migrate to new version of the product. It would also provide the corporation with a better understanding of the product and therefore prevent lock in as we see today.
Outsourced programming shops should be encouraged to publish their code in the same manner as the Corporate Source model. In fact it should be a clear advantage of the model as it would enable the quality to be more easily seen and also would allow the outsourced agency to reuse other pieces of the infrastructure more easily.
SOA and all the above some have felt to be mutually exclusive. What is the benefit of Open Source and reuse if there is only going to be one instance of a service. For example if a corporate Main Frame makes a particular function available through a web service, where would publishing the source help? In most cases the Services that would become useful within a corporation are in fact amalgams of previously engineered capabilities. Even the simplest Service might require logging, security and persistence resources. Publishing the code for one service might enable a new version to be built more easily.
For sometime now I have been pushing the model of interfaces that Microsoft have been using for years in building the Windows Platform. In this model a Service is never deleted. API for Windows 95 are still available should ancient programs require them. This model means that new services are built from and deployed alongside old one. This model seems to work fine for a PC but would it work for a corporation? The main issue is access to underlying data and resources. If the data changes under a service, for example and database schema is modified to allow new functions, the service no longer works. In this case the link between resources and services needs to be maintained. By having regression suites such as proposed by Test Driven programming the services that fail after the change would be very obvious. It would then be necessary to reprogram the old service until it worked. By adopting this model the clients of a service would be unaffected by the new introduction and therefore the deployment of new services becomes much less dramatic to the corporation.
By making the code to the original service available to all, the people responsible for the change to the underlying data source can see the effects their change will have and also they can take responsibility for making sure the old code continues to work without having to establish permission from the original owners. So now they not only deploy the data changes and the new services they also redeploy the old service in a new form.
One of the other effects of encouraging openness amongst the development community even if the code is expected to be deployed only once is that it may allow other parts of the business who operate in a similar fashion to build on the previously existing service code. For example if a business is split across countries or divisions, it is very likely that similar functions are replicated. Allowing these other entities to have access to the code would allow them to reuse it even if they have different underlying capabilities. It is my experience that picking up some one else code that works enables me to stand on their shoulders and provide a better solution. Examples abound in the public Open Source environment and the establishment of projects such as the Apache Commons shows that reuse can be a useful bi-product even if the original aim for the code was very specific to one function. IN fact the majority of coding that takes place today is based on copying existing examples and making them fit your needs, there is almost no other way to understand complex libraries that have proliferated over that last few years. Knowing the Java language is easy knowing the right library is the key skill difference between the novice and the expert.
SOA needs to have fixed numbers of resources deployed in a controlled manner. To date successful Open Source projects have tended to be platform based and not specific to a particular data source. However, the benefits of Open Source to the mind set of corporate development are too crucial to not pursue. SOA is also critical to allowing large enterprises to flex the resources underlying services and to enabling heterogeneous clients to access the capabilities offered by the service. Therefore the two models should be seen as supporting each other rather than being mutually exclusive.
There is a clear need for enterprises to move on both fronts. Spending time looking at current screen based operations and providing underlying API to the resources in question is a key enabler for the SOA. This is one of the functions that Microsoft’s API have been very successful. Allowing other applications to call API within another applications space allows reuse at a level never expected when the original application is built. For example the Word spell checking capability is just as useful when designing flowchart as producing email as producing this document.
Providing a free at the point of use platform for code sharing and publication similar to the Sourceforge model is crucial to encouraging developers to share their code. Alongside this is the encouragement for in house developers to use Open Source code from outside business is also needed. This will encourage the right mind set for sharing and seeing the benefits of sharing. This platform needs to offer both simple sharing models such as a repository of file and the use of complex code management tools such as CVS and it like. Before spending begins on such a platform the enterprise should discuss if the sharing in a public manner might have some benefits especially where the desire to use standards across the industry is seen as important.
Lastly, making sure that a suitable simple ‘Google’ like search engine of an enterprises intranet is very important. If this could be combined to return results not just fro the intranet but also associated internet resources the better. The key to making this work is to adopt a decentralised model for most of the key functions. For example allowing the search engine to roam over the whole intranet would enable small projects to maintain islands of information they saw as necessary without hiding their resources from the masses. It is interesting that functions such as Wiki and other free to use capabilities actually encourage a voluntary centralisation and by adopting an open policy to information on such sites the mindset of the enterprise can move from empires with walls to a much more community based environment where sharing is seen as being key to being a good corporate citizen.
Tuesday, May 25, 2004
Jon Udell explain's in his article The challenge of partial trust that he experienced trouble with mixed permission's on XP. I have also experienced similar frustrations. The main culprit seems to be Office and on two machines with repeated installs I have failed to make this work, where I can control the machine and my family can also use it safely without knowing how it works.
It seems to me that the analogy of the car works here. The mechanic keeps it running and I do not have to know what they do, just pay the bills. The computer needs to get to this point where I can do all I need without having to resort to sledge hammers! It is interesting that the Audi A2 has a special hatch for all the functions that a user is expected to interact with, oil, water and fuel. The mechanic has access to the engine compartment, but the user does not need to ever enter the hallowed bay. Would it not be excellant if when I install new code, it could act as the mechanic and I could sit back and relax knowing it would still all work after the event.
Thursday, May 20, 2004
XML template files
Why do so many applications that say they are XML compliant actually use simple fill in the blank templates. Is this an issue? There seems to be a problem with empty versus optional in XML. A number of apps I have com across seem to take a best case template and then fill in the fields. If a filed such as a date is not supplied they leave the tags in with no content. This then trips up on schema checks and other format checks as an empty date may be interpreted as incorrect, rather than optional.
What seems to lie behind a great deal of these type of apps is the fact that XML is tacked on and not XML parsers or DOM tools are used to create the XML output.
Thursday, May 06, 2004
Is the Post Office the best custodian of Addresses
For a number of years now the Post Office in the UK has been the custodian of Address data. This is based upon the fact that they produced an encoding of an address known as the Postal Code. This code is designed to aid sorting of mail down to the level required by the individual postman's round. For postal items this is a very clever and useful solution to a problem. However, it suffer from a clear computing problem - As a primary key for an Address the Postal Code does not conform for Codd and Date's definition. This is because, a) it does not uniquely identify a property b) it changes over time c) it does not cover the entire set of properties and buildings to which other might need to refer.
Identification of a Property
All properties that will have mail delivered to them are associated with a postal code. This code may cover many properties within a street. or even a particular company working within a property
This means that it is not possible to uniquely identify a single property from a postal code. This means that the properties number or name has to be added to the postal code
Changing Postal Codes
For the Post Office the requirements for a code are based around their needs for the idenitifying appropriate delivery routes. If a route has a property that receives large amounts of mail this is identified by its own Postal Code. Therefore when a new compoany moves into a street or premises, the postal code for that property may change. These changes are sent out regularly with the Post Office Address Finder CD-ROM. Most applications that update addresses at key transaction point rarely update this information leading to future problems when transactions are initiated using the new Postal Code which is not recognised.
Not Every property has a Postal Code
The Post Office quite rightly does not identify properties to which they do not deliver mail. However, other companies do wish to identify these premises. This means that when trying to identify an address in the UK any technical solution needs to be able to handle both postal Code to Premises matching and also some other mechanism such as GPS ref. or grid ref.
As I contemplate this complexity I begin to wonder how much UK plc losses because business up and down the line do not have a satisfactory way to unique identify buildings and locations that is generic across industries and is as accessible as the Postal Code solution.
The Ordanance Survey have a solution based on uniquely identifying every building in the Uk using a 16 digit number. Associated with this number is extra information such as apartments and possible the postal code information. The main draw back is it is completely inaccessible to the person wishing to send a post card to Aunt Milly, but is is great for getting that Aerial site on that remote hill side if all parties use the OS software.
I have come to the conclusion that the government of the UK should oil the wheels of commerce by introducing a system that is more comprehensive that the Postal Code system, but is utilised by the Post Office and other delivery companies to manage their businesses.
Why should the governemt be interested in this? Virtually every transaction carried out by the utily companies in the UK requires an address match to check the premises are correct. This is probably the single most complex step in the transaction and it subject to the most failures and require most manual intervention. Reducing the cost of these transactions across the whole of teh country would benefit every one involved.
Therefore I believe the Post Office is NOT the best custodian of Addresses - here in the UK. Is this the same in other countries?
Workflow versus Process Automation
When does a workflow engine become a process automation tool and vice verse?
This seems to be a non-question in my mind but it seems to exercise people greatly, especially vendors and the various standards groups that are working in the web services area.
When I first used a workflow tools (Oracle Workflow 2) I was implementing a solution that was designed to connect our front portal applications through to our back ends. As happens from time to time in work, I found the exercise immense rewarding in that it forced me to rethink the way in which I imagined a solution. No more monolithic designs, not even the sweetness of a OO but a simple set of modules that could if possible stand alone and be played with by a process designer at will.
As the solution involved both routing work to humans as well as robots I did not understand that this was not normal for most 'Workflow' applications. In fact I designed the application as 90% manual with the aim of getting stats to say which parts should be automated. In this way the application itself along with suitable humans actually drove the requirements process. Oracle's Workflow tool was very suited to this type of work particularly as most of the robots were database centric.
So where does this question arise? It seems that the next generation of tools based on forthcoming BPEL based technology have simply overlooked the benefits of dove tailing Workflow solutions with Process Automation. When a Process fails where do you need to route the fault to? Normally a human - so why do most tools make this a cumbersome task? Why do these so called next generation tools find dealing with people such an alien idea? I believe the answer lies in the fact that most of these emerging tools have been built by people used to handling classes that rarely touch humans directly. They tend to be focused on the J2EE/.Net like frameworks which are low level in the inspirations and have failed to take into account the gains of the 4GL world of the early 1990's.
This inability to see the bigger picture has meant that large investments in Process Automation and EAI technology has failed to benefit from the humdrum, low tech solutions that are sidelined as being manual.
Part 3: Architecture - we need a better analogy
It has been a couple of months since I published parts 1 and 2 of my ramblings on Architecture in IT. The original article was an internal document that I have tried to make more generally applicable. Most of the comments in my expected Part 3 were linked with the various internal architectural initiatives which I can not comment outside of my company.
Here are some of the conclusions I can share
The Structural plan should be based on an expression as to how we want to do business and not constrained by our current creaking (yet working) system infrastructure. The changes in the atmosphere of London over the last few years are tangible and have come about through careful thought of the overall vision and the implementation of key yet sometimes small changes. The new bus map in London has moved me from a rat in the underground to a top floor tourist enjoying and appreciating the landscape of London.
On the detailed level I would very much like to see Architects involved in developments in the same way the architects are involved in major developments. The name Norman Foster is associated with many key amazing buildings and developments. It is his name that is linked with the result. This is because not only did his team design the buildings but also they maintained control of them through to the final opening. This means that the concepts put to planning authorities are seen through to fruition by the same people who expressed the plan. Without this continued involvement and enthusiasm you would find a set of mishmash results, which fail to express the real solution and exude an air of missed opportunity.
In this view of Architecture, I am an Architect. I am not a Town Planner. I do not have that grasp of the trends of change, or the business needs to construct the Plan, but I along with the majority of 'Solution' people are capable of understanding such a Plan. In fact when expressed in visionary language I feel that most people would find it more inspiring than the current 'Architectural' language. You will notice that there are key areas left to architects to decide - the technology of Systems outside of basic infrastructural pieces is left in their hands. This freedom to innovate within a framework is key to any respectable architect. Without this the enthusiasm does not get expressed in the designs and instead of beautiful landscapes of juxtaposition designs you get the homogenous repetition of the authority housing estate.
Thursday, February 26, 2004
Part 2: Architecture - we need a better analogy
It struck me that what we call architecture is more a kin to town planning. Town planning involves various skills that any architect would not claim to have. It involves taking into account what is currently in existence, forecasts of certain trends of change in a particular locality and the production of a 'Town plan' to provide a framework for those changes subject to either local authority funding or speculative development. In this way a town or city can remain flexible for longer term trands such as population changes, age of current infrastructure and quality of life issues of its citizens. When a town plan is put out it is never expected to proscribe the colours of building or even the style of building beyond broadbrush categories such as industrial or residential. It is left to the local planning authority to authorise actual plans and to use the 'structural plan' to decide if a particular submission is conforms and also if it will enhance the location, through some defined criteria - will it bring jobs, is it esthetically pleasing, etc.
So who are the players involved in these activities:
Enterprise CIO are the Town Planners - they should draw up and maintain the structural system plan. This is a schematic map of the business as if it were a town detailing both what is there, what is available for change and what is not (brown field sites, conservation areas, etc) and any aspirations for the locality (attract tourist, reduce car traffic, better schools, etc). There should be no illusions that this structural plan will be ever completed, or that the detailed results can be anticipated from it - for example an initiative to attract work may cause ugly factories to be built.
The Enterprise CTO are the advisers to the Town Planners - they advise as to what is possible so that the Structural plan is scoped within the art of the possible rather that completely off the wall. For example they may advise on the forms of Public Transport that the CIO might sanction.
In larger companies the Divisional CIO/CTO have a dual role as in the current Local Authorities - They are involved in the detailed structural plans where they affect their local area. They would lobby for changes as they see the need and they would be responsible for making sure that any developments conform to that plan. This dual role of setting some of the detail and authorizing the change can be seen in Wholesale where the CIO employs both the Architect and the Design Assurance teams.
Part 3 to follow
Tuesday, February 24, 2004
Part 1: Architecture - we need a better analogy
I have been puzzling for some time as to why Architecture in Software and Systems terms is treated with such respect by Senior Management yet is treated with scant regard by most day to day projects and developments. Alongside this question, is the scorn that the word 'Strategic' causes and the complete lack of expectation that any architecture will in fact become a reality.
If you examine Architecture in terms of buildings you see that it is aimed at providing both the conceptual impression of what buildings will look like and also going all the way through to supervising all the detailed design of the building and its components. For example an Architect friend of mine who designed the Norwich Swimming Pool recently opened, not only put the models and pictures to win the competition, but he also specified the quality of the door handles and other fitting used in the building. This happened over the time of the project. This picture of Architecture is very different from our rather ephemeral beast.