Code Branching gone mad!
March 23, 2009
Question : What happens in the last year of a multi-year, multi-million dollar, multi-release program of development and business change???
Answer : All the hard stuff that has been left to the end has to be delivered in parallel, in overlapping releases and in double quick time before the money runs out…oh, and if you’re really unluck you’ll have to upgrade some of the applications from the earlier releases too.
This is a reality for us at the moment on our project at one of Australia’s largest fund management companies. Despite shifting requirements and architecture we have managed to deliver 2 successful major releases over the last 2 years and yet our biggest challenge lays ahead of us.
So far we have built a powerful Investment Data Management Platfor using the Eagle PACE product as a base, into which we have integrated accounting feeds from Hiport, market data feeds from Bloomberg, Factset and S&P. On the other end we have integrated 3 trading platforms - Charles River (for Equities and Enterprise Compliance), Imagine (for Derivatives) and BlackRock Aladdin (for Fixed Income). On top of all that we have integrated a Performance and Anlaytics system (StatPro), Recs System (Smartstream TLM) and Corporate Actions system (Smartstream Coacs). To really finish things off we have added a Data Mart layer to support Internal reporting via Business Objects and External Reporting via SS&C Pages - PHEW!!
How can there be more you’re asking yourselves? Well now we have to feed Registry Data into the mix too from multiple registries and switch off several legacy systems ( not forgetting remediation of a reasonably long list of small access and excel based apps).
Finally we are upgrading Eagle, Charles River, StatPro and SS&C Pages….all before December 31st 2009!
As you can imagine we have alot of issues to overcome, and there’s not enough room on this page to discuss them all, but I thought I’d share with you one of the many issues that just get overlooked when these gran plans are hatched - code branching!!
For the uninitiated code branching (and merging) is the situation you find your self in when you have multiple release dates of the same code base running concurrently. In a perfect world you would only have 2 lines of code - Development (known also as ‘THE TRUNK’) and Production Support (known also as ‘THE PROD BRANCH’). In this perfect world when ‘hot fixes’ are applied to the ‘PROD BRANCH’ they are then merged into ‘THE TRUNK’ to keep it in line with with what is in Production, that way when a major release is deployed to Production from THE TRUNK it includes all the hotfixes that have been accumulated since the last major release.
Our problem is we have mul;tiple teams working on multiple ‘major’ and ‘minor’ releases of the same application (The Data Management Platform). As a result we have THE TRUNK, THE PROD BRANCH and 2 additional branches - one for an upgrade release and another for a parallal major release. Confused? I’m not surprised! Here’s a photo of the whiteboard after our team meeting this morning..
The release order runs left to right. The PROD BRANCH is always on the left as a deplpoyment to Production by the support team can happen anytime, next comes ‘91′ which represents an upgrade of our data management platform toolset to version 9.1, then there’s 3A - the current major release we are working on - due to go to Prod in June. Finally there ‘4′ that is due to go to Prod in Septemberbut development has to start now - hence another branch!!!
The arrows at the top indicate where all the overhead exists with this situation. Changes in earlier branches must be merged into later branches to keep them up-to-date. The can be quite time consuming - particularly as some of our components (in the ETL space) are binary and can’t be auto-magically merged by our source code repository tools (Subversion).
We aren’t exactly sure what the impact will be on our project plan - we intend to perform the merge tasks as we go to avoid breaking huge numbers of our automated unit tests. A cursory estimate is indicating about an additional 20% to each development task - a total of 1 man month in total. I’ll be talking to our PM today - hope he’s in a good mood!!
Scrum Master
