Category Archives: SOA

Spaghetti Grows in System Architectures – not an April Fools’ Day joke

A replay on breakfast TV this morning of the well known Panarama hoax (1st April 1957) reminded me of the mission we’re on at Bristol to “turn spaghetti into lasagne”. This mission is number 7 on the JISC 10 pointer list for improving organisational efficiency: spaghetti refers to the proliferation of point-to-point (tightly coupled) integrations between our University’s many IT Systems and lasagne refers to the nicely layered systems and data architecture we’d like to achieve (see elsewhere in this blog).

However, transforming our data architecture overnight is not achievable, instead we’ve developed a roadmap spanning several years in which reform of our data architecture fits into the wider contexts of both Master Data Management and Service Oriented Architecture.

In November last year our senior project approval group (now known as the Systems and Process Investment Board) agreed to resource a one year Master Data Integration Project. We will return to the same board early in 2015 with a follow on business case, but this year’s project is concerned with delivering the following foundation work.

The establishment of Master Data governance and process at the University (the creation of a Master Data Governance Board and the appointment of Data Managers and Data Stewards as part of existing roles throughout the University – responsible for data quality in their domains and for following newly defined Change of Data processes),
Completion of documentation of all the spaghetti (aka the integrations between our IT systems) in our Interface Catalogue, and also the documentation of our Master Data Entities (and their attributes and relationships) in an online Enterprise Data Dictionary (developed in-house),
Development of a SOA blueprint for the University, including our target data architecture. This with the help of a SOA consultant and to inform the follow on business case for SOA at Bristol, which we hope the University will fund from 2015.

We are undertaking this work with the following resources: Enterprise Architect (me) at 0.3FTE for a year, a Business Analyst (trained in Enterprise and Solutions Architecture) at 0.5FTE, a Project Manager at 0.3FTE, IT development time (both for developing the Enterprise Data Dictionary and for helping to populate the Interface Catalogue with information) and approximately £60K of consultancy.

We had some very useful consultancy earlier this year from Intelligent Business Strategies: several insightful conversations with MD, Mike Ferguson, and a week with Henrik Soerensen. From this we were able to draw up a Master Data Governance structure tailored to our organisation, which we are now trialling.
This work also helped us to consider key issues around governance processes and how to capture key information – such as including business rules around data – in the online data dictionary.
Later this year we will be working for an extended period with an independent SOA consultant based in the South West, Ben Wilcock of SOA growers. We have already worked with Ben in small amounts this year and I am very much looking forward to collaborating with him further to develop our target data architecture (most likely a set of master data services, supporting basic CRUD operations) within the context of a SOA blueprint for our enterprise architecture.

SOA Knowledge Exchange 2013 – a beacon of light or the blind leading the blind?

Some time ago, at a UCISA Enterprise Architecture event, I discussed whether we needed a workshop for Higher Education (HE) sector technical architects to get together and share knowledge of Service Oriented Architecture (SOA). Whilst enthusiastic, colleagues expressed concern at the time that possibly the sector was too immature in this area for any individuals within it to be able to contribute to a worthwhile discussion. However I was not keen to host a vendor-led event as sometimes these can introduce an unhelpful level of product bias and not necessarily highlight the pitfalls to avoid when pursing a SOA roadmap in an HE institution.

Since that time I bumped into several colleagues from other HE institutions at various events and began to realise that there was a lot more SOA activity going on around the sector than I’d maybe thought. The people I spoke to were very encouraging about the possibility of a SOA knowledge exchange and some obvious speakers began to emerge. So I organised the SOA Knowledge Exchange at Bristol. I only had 40 places and they were filled within five days of advertising by UCISA and the JISC, clearly demonstrating that this is a hot topic for many of us. The workshop took place last month and here is a summary report of what went on.

I presented the first session, briefly describing our SOA roadmap at Bristol (much of which is detailed elsewhere in this blog) and explaining how I managed to reach the ears of our senior level decision-making body last year. I talked about how, as Bristol’s Enterprise Architect, I am able to take an holistic, enterprise-wide approach, and to articulate to non-technical, key strategy-makers, the problems that our complicated, point-to-point architecture cause us. Taking a SOA appproach should reduce the costs of maintaining our data architecture, reduce data redundancy and duplication as well as save us the embarrassment of system breakages caused by data changes in a master system rippling in an uncontrolled way through the systems architecture and causing data failure. However, implementing a SOA solution will require investment and at Bristol we see the road to SOA maturity taking years not months. We are beginning with a year-long foundation stage. I talked about this and the work we are doing to document our current integration architecture: the interface catalogue and the enterprise data dictionary (I also talk about these elsewhere in this blog).

Next up was Simon Bleasdale from Cardiff. He talked about how Cardiff is taking an iterative approach to enterprise application integration. Cardiff are building in-house skills through the creation of data integration and application integration teams and their approach is project-focussed. Simon described their range of data services so far developed using Restful Web Services via a combination of Grails and Mule. Having previously invested in IBM software, Cardiff are now conducting a review of this and plan to explore whether Talend or WSO2 would be a good future option for their University. They are committed to taking an agile, open source, simple and cost effective approach to data integration.

Allister Homes from Lincoln followed Simon to describe their current technical architecture in terms of their early work on master data management (MDM) and deploying Enterprise Service Bus (ESB) technology using Microsoft Biztalk. He discussed how a combination of MDM and ESB is enabling Lincoln to distribute data via their Sharepoint and Customer Relationship Managment systems to a range of application services across the institution. He also read out a highly entertaining and very apt paragraph that Lincoln put in their SOA business case for senior management, likening the current experience of their digital communities to that of dwelling in a primitive, unintegrated town settlement and describing the vision of the future as analogous to a thriving ‘metropolis’ with properly integrated services.

Martin Figg and Russel Gibson from Imperial gave a tour of the issues currently faced at their University and progress to date. Imperial have implemented SOA solutions for sharing HR organisation hierarchy data and research data with other IT systems and are now working on building data security into the architecture by integrating with their Authentication As A Service (AAAS) solution. They have worked on building SOA infrastructure in a standardised way to support generic error handling, reporting and resubmission. They are continuing to try and build strong SOA governance to stop the proliferation of point-to-point interfaces between systems and to promote the standardised reuse of data services to replace them. They have at least 6 skilled staff working in this area, with other staff in various supporting roles, including a SOA consultant to help answer particular queries. Some really interesting questions were raised in this presentation about the implementation of Enterprise Business Objects (EBO’s) and Enterprise Business Services (EBS’s), and the hard design decisions they’ve been facing in configuring the generic Oracle Business Suite to work best for their HE-specific purposes.

Cal Racey brought us up to lunchtime with a presentation on Newcastle’s SOA and Institutional Data Management work to date. He described Newcastle’s institutional data feed service (IDFS) which delivers data to over 30 of their systems via 70 different feeds. This came out of the JISC funded IDMAPs project and is currently supported by the equivalent of two full-time staff. They are using Talend. Cal advocates a light-touch data dictionary approach, avoiding going down rat holes with data structuring and modelling and also warns against the distractions of endless battles around whether to choose a SOAP versus REST approach, XML versus JSON data formats and so on. He also recommends continuously identifying where master data is held and what new master data an application produces. Cal described the importance of measuring the data problems right at the start so that it is possible to evidence the success of SOA-based solutions and thus build the case for continued SOA investment.

After lunch Jim Phelps joined us virtually from the States where it was 8.30am his time! He is Enterprise Architect at UK-Madison University and also chair of ITANA. He gave us a presentation that first described the Push Notification architecture they have introduced at Madison and then went on to talk about ITANA’s work. Some interesting stats on technologies being deployed by US Universities in a recent survey: the majority are using WSO2’s open source stack, probably because of the fairly large developer skillset in this area over in the States. The survey picked up just one Oracle SOA Suite institution, one IBM institution and some Fuse source. For more information on Jim’s excellent work, see ITANA’s SOA API Wiki: http://goo.gl/evbpk5 and the W-Madison AIA Wiki: http://goo.gl/xblJS4

David Rose opened our final speaker session, joined remotely by Chris Harding, Director For Interoperability at the Open Group. The pair gave a very useful, wider context to the whole day, talking about completed and current projects being undertaken by the SOA Work Group. Current projects include SOA for Business Technology, SOA Reference Architecture V2 and SOA Certification. Resources from completed projects can be found online (http://www.opengroup.org/). They also talked about Cloud readiness and the relevance of SOA to this. They went on to discuss whether the HE Sector could create a Reference Architecture that would evolve over time and enable both a language for conversations (and leveraging power) with suppliers as well as help individual universities to develop their SOA architectures. They finished with a thoughtful slide on the potential relationship between UCISA (with its Enterprise Architecture group), ITANA and the Open Group (shown above).

And the conclusion? This event didn’t feel like the blind leading the blind by any means and the feedback has been excellent (a selection of which I’ve recorded here). If we can keep up the momentum I think something very constructive could come from this initial effort to collaborate. Hopefully we can work on a reference architecture for our sector, share emerging SOA design patterns and perhaps we can begin to influence key vendors by clearly specifying our common data architecture requirements… Many thanks again to all the speakers and for the great input from those who joined us as participants. It was a very enjoyable day!

SOA Knowledge Exchange, 1 day event, Bristol, 27th September 2013

About the event

******** SORRY, THIS EVENT IS NOW FULL **********

We are holding a workshop in Bristol this month for Enterprise Architects, Technical Architects or those in similar roles at Universities in the UK. The general aim of the event is to gather and discuss the SOA roadmap strategies we are developing within our institutions, to share success stories and to alert others to potential pitfalls along the SOA adoption route. This is not an introduction to SOA, but a chance to share real experiences of adopting a SOA approach within an HE institution. There will be presentations and discussions throughout the day, mostly from colleagues working on SOA initiatives in a UK University setting, but also with contributions from Jim Phelps of the ITANA group (serving HE institutions in the USA) and Chris Harding of The Open Group (dedicated to developing global IT Standards).

If you would like to attend, please compete the registration form at: https://www.survey.bris.ac.uk/it-services/soa_workshop_registration

Location

The Bergerac Room, 8-10 Berkeley Square, Bristol: Map and directions

Agenda

Subject to change

10.00 Refreshments

10.30 Welcome and introduction to the day

10.40 Talk 1 The business case for SOA at the University of Bristol: developing the roadmap and getting buy-in from senior management, Nikki Rogers, Enterprise Architect, University of Bristol

10:55 Q&A

11:10 Talk 2: SOA Strategy at Cardiff University: taking an open source, iterative approach, Simon Bleasdale, Senior Integration Developer and Myles Randall, Integration Developer, Cardiff University

11.25 Q&A

11.40 Talk 3: The early stages of implementing an ESB and deploying a Master Data Management strategy at Lincoln, Allister Homes, Senior Systems Architect, University of Lincoln

11.55 Q&A

12.10pm Talk 4: Institutional Data Management at Newcastle, developing and governing data services, Cal Racey, Systems Architecture Manager, Newcastle University

12.25: Q&A

12:40: Lunch & opportunity for breakout discussions

1.30pm: Talk 5: Building Student Services at Imperial – making SOA a Reality, Martin Figg, Software Development Manager and Russell Gibson, Technical Project Services Manager, from Imperial College London

1.45: Q&A

2pm Talk 6: Chris Harding and David Rose from the Open Group: Emerging Open Standards in SOA and the new ‘Open Platform 3.0’ area.

2.20 Q&A

2.30pm Talk 7: Update from Jim Phelps, joining us remotely from the US, on the ITANA group’s SOA API initiative

2.45 Q&A

3pm: Coffee and Tea, Open discussion, Are there steps we can take at HE Sector level for example to develop common data standards and encourage IT Suppliers to comply with SOA approaches that we will require? (noting the Higher Education Data & Information Improvement Programme (HEDIIP which is developing a Higher Education Data Language) and its potential relationship to the growing number of SOA initiatives in UK Universities).

4pm close

Looking back on Year 2

As a follow on to my blog post that reflected on year 1 of EA at Bristol (https://enterprisearchitect.blogs.bristol.ac.uk/2012/04/17/looking-back-on-the-first-year-of-my-ea-role-at-bristol/), here’s a summary of the top three key things I covered in year 2:

Raising the profile of EA: a two-hour workshop with the Portfolio Executive (http://www.bris.ac.uk/planning/programmesandprojects/portfolioexecutive/). This senior decision-making group within the University were interested to know more about Enterprise Architecture, so I ran a 2-hour workshop with them last summer. Enterprise Architecture covers a very large range of concepts so it is hard of course to hand-pick the right things to discuss in a session like this, especially where the audience is largely non-technical. I decided to concentrate on two things: firstly on how an Enterprise Architect needs a clear University strategic framework within which to work and secondly on the importance of a well-designed data architecture at Bristol. The workshop went very well indeed; the first part resulted in follow-on Benefits Mapping work – I have blogged about that here: https://enterprisearchitect.blogs.bristol.ac.uk/category/benefits-maps/ and the second part resulted in high-level recognition and endorsement of the master data management strategy I launched at Bristol – see http://coredataintegration.isys.bris.ac.uk/2012/10/22/explaining-the-importance-of-a-good-data-integration-architecture-to-senior-management/ for how I introduced this concept to the group.

Embedding EA within the Governance Structure at Bristol. One problem I wanted to tackle was that of how to ensure that EA is a formal consideration whenever new IT-related projects are proposed at Bristol, whether that should involve in-house IT development or the procurement of third-party systems. Until this second year, EA was being acknowledged as valuable at Bristol, but there was no place in the project approval process at which the Enterprise Architect could make a clear recommendation based on an appraisal of some new project’s fit with Bristol’s business architecture, information architecture, and technical architecture. When I started in my role I was invited to be part of a group called the System Programme Managers Group (SPMG) which met monthly to review current and proposed projects, and which was comprised of managers leading various University programmes (IT, Education, Planning etc). I ran a workshop with this group entitled “Capacity for Change” and I talked about how we should review potential projects in terms of what priority they should have in the already packed project portfolio that the University was funding. This, in combination with some senior level initiatives culminated in the renaming of the SPMG to the “Portfolio Management Group” (PMG). Its new remit is now to support the higher level Portfolio Executive group more directly by making recommendations on any new project’s fit with the strategic priorities of the University. We have aligned the strategic priorities with the benefits mapping work I mention above, and each new business case is now submitted with a PMG recommendation, describing whether we have existing resources to support the proposed project, its priority level, and a statement of its fit with the enterprise architecture – hurray! So, this is not to say that Enterprise Architecture considerations now dictate decision-making, just that it has become formally recognised as part of the project approval governance process; a positive step forward.

Developing the SOA roadmap for Bristol. After the workshop for the Portfolio Executive, mentioned above, I submitted a Stage 0 business case (entitled Master Data Integration Framework) which was approved. The next step is to develop the Stage 1 business case, for presentation to the Portfolio Executive this Autumn, and for this I am developing the roadmap, with an indication of costs and benefits along the way, in consultation with others. For this work I have maintained a completely separate blog, see: http://coredataintegration.isys.bris.ac.uk/category/final-project-report/ for more information. The initial roadmap I produced for Bristol can be summarised as follows,

Step 1. Make the initial business case for Service Oriented Architecture (SOA) (even if it is not explicitly called ‘SOA’ at this stage) to achieve senior level understanding of the importance of a good data architecture and why investment in this apparently invisible middleware layer will bring benefit to the institution.

Step 2. Complete the data dictionary and the interface catalogue to at least 80% for all master data system integrations.

See my separate blog for why this is important: http://coredataintegration.isys.bris.ac.uk/2013/06/16/important-documentation-for-soa-the-interface-catalogue-and-data-dictionary/. The benefits at this point are:

Faster (cheaper) implementation of new IT systems (quicker to assess impact of a system change on other systems)
Faster (cheaper), more reliable Business Intelligence/Operational report production

Step 3: Introduce data governance.

By introducing data stewards who will be responsible for ensuring that no master data system changes in data structure ‘go live’ before a check has been done against the interface catalogue and the data dictionary (through a formally defined process), then the benefit we introduce at this point is:

A more sustainable IT architecture and a higher guarantee that business processes will function without disruption. This is due to data being managed as an institutional asset and ongoing changes to master data structures being managed in a controlled way as they propagate throughout our IT system landscape (avoiding system and process failure and/or damage to our organization’s reputation)

Step 4: Analyse the interface catalogue and develop a logical data model from the data dictionary. Our interface catalogue already reveals the high percentage of point to point interfaces that replicate very similar data synchronisation tasks between IT systems. We can at this point make recommendations regarding how we can reduce this percentage dramatically and refine the integrations into a minimal number of key data services to share master data across the systems that require it. This is where we relate SOA planning to the business architecture of the University, with an appreciation of how data is used across the student and research lifecycles. This step therefore requires business analysis as well as technical systems analysis.

The benefit of doing it is that we are able to describe more specifically the cost-savings and agility that the University stands to gain if it invests in consistently applied SOA technology. We can note that Gartner’s research demonstrated that 92% of the Total Cost of Ownership – http://www.gartner.com/it-glossary/total-cost-of-ownership-tco/ – of an average IT application, based on 15 years of use, is incurred after the implementation project has finished. A significant part of those costs will be concerned with maintaining the application’s seamless integration within the organisation’s application architecture, so by simplifying this we should save costs over time.

Naturally, the other purpose of doing this step is so that we can analyse the “As Is” architecture and design the “To Be” architecture – a precursor for step 5.

Step 5: Adopt a SOA technology solution, train IT developers to use it.

The analysis from Step 4 should have provided us with the basis from which to design the data integration architecture we need. The ‘blue ring’ diagram is not a formal systems architecture diagram by any means, it is simply used to roughly illustrate the concept of a master data systems architecture through which we decouple systems requiring master data from connecting direct to the master data systems themselves (avoiding the proliferation of point to point systems integrations that we suffer from currently). We should have learned through step 4 where we need real-time data, near real-time data and where real-time data is not essential. We can therefore look for the most key processes that we need to support via ESB technology and where, in other places, nightly refreshes of data will ‘do’. We can prioritise which master data services we want to implement in which order, using our favoured SOA technology (whether we go the opensource or commercial route). We can decide if we want to keep and improve our operational datastore and we can determine carefully whether we deliberately want to maintain some point-to-point, database-database integrations (i.e. analyse where in fact, web services, say, will not offer us greater benefit if we acknowledge the trade off between maintaining existent Oracle database to Oracle database connections versus the possible reduction in performance if we deploy data service intermediaries). We need a dedicated team of trained developers to monitor and operate the data integration architecture according to standards we define (SOA governance), to understand how to optimise data caching within it and so on. At this point the benefits we expect to achieve are:

Cheaper, less complex to maintain, standardized master data integration architecture (blue ring).
Higher guarantees of data security & compliance (data services to have Service Level Agreements and data security to be built in from the outset)
Agility to respond quickly to Cloud and Shared Services opportunities
Quicker to respond to changes in external reporting data structures (KIS, HESA, REF etc.)

Step 6: Train our IT developers and Business Analysts to work together using a standard set of skills and tools based around BPEL, BPM etc.

At this point we expect to reach the level of SOA sophistication where we have the ability to orchestrate and optimize end to end processes that share and manipulate data, creating efficiencies and business agility across the institution.
Suffice it to say, we are some years off Step 6! We are currently working on steps 2-4 above. This Autumn I am running a knowledge exchange workshop for HEI’s working on SOA roadmaps to gather and compare and contrast their plans, successes and lessons learned to date. If you are interested in coming, please contact me directly.