Code Branching gone mad!
March 23, 2009
Question : What happens in the last year of a multi-year, multi-million dollar, multi-release program of development and business change???
Answer : All the hard stuff that has been left to the end has to be delivered in parallel, in overlapping releases and in double quick time before the money runs out…oh, and if you’re really unluck you’ll have to upgrade some of the applications from the earlier releases too.
This is a reality for us at the moment on our project at one of Australia’s largest fund management companies. Despite shifting requirements and architecture we have managed to deliver 2 successful major releases over the last 2 years and yet our biggest challenge lays ahead of us.
So far we have built a powerful Investment Data Management Platfor using the Eagle PACE product as a base, into which we have integrated accounting feeds from Hiport, market data feeds from Bloomberg, Factset and S&P. On the other end we have integrated 3 trading platforms - Charles River (for Equities and Enterprise Compliance), Imagine (for Derivatives) and BlackRock Aladdin (for Fixed Income). On top of all that we have integrated a Performance and Anlaytics system (StatPro), Recs System (Smartstream TLM) and Corporate Actions system (Smartstream Coacs). To really finish things off we have added a Data Mart layer to support Internal reporting via Business Objects and External Reporting via SS&C Pages - PHEW!!
How can there be more you’re asking yourselves? Well now we have to feed Registry Data into the mix too from multiple registries and switch off several legacy systems ( not forgetting remediation of a reasonably long list of small access and excel based apps).
Finally we are upgrading Eagle, Charles River, StatPro and SS&C Pages….all before December 31st 2009!
As you can imagine we have alot of issues to overcome, and there’s not enough room on this page to discuss them all, but I thought I’d share with you one of the many issues that just get overlooked when these gran plans are hatched - code branching!!
For the uninitiated code branching (and merging) is the situation you find your self in when you have multiple release dates of the same code base running concurrently. In a perfect world you would only have 2 lines of code - Development (known also as ‘THE TRUNK’) and Production Support (known also as ‘THE PROD BRANCH’). In this perfect world when ‘hot fixes’ are applied to the ‘PROD BRANCH’ they are then merged into ‘THE TRUNK’ to keep it in line with with what is in Production, that way when a major release is deployed to Production from THE TRUNK it includes all the hotfixes that have been accumulated since the last major release.
Our problem is we have mul;tiple teams working on multiple ‘major’ and ‘minor’ releases of the same application (The Data Management Platform). As a result we have THE TRUNK, THE PROD BRANCH and 2 additional branches - one for an upgrade release and another for a parallal major release. Confused? I’m not surprised! Here’s a photo of the whiteboard after our team meeting this morning..
The release order runs left to right. The PROD BRANCH is always on the left as a deplpoyment to Production by the support team can happen anytime, next comes ‘91′ which represents an upgrade of our data management platform toolset to version 9.1, then there’s 3A - the current major release we are working on - due to go to Prod in June. Finally there ‘4′ that is due to go to Prod in Septemberbut development has to start now - hence another branch!!!
The arrows at the top indicate where all the overhead exists with this situation. Changes in earlier branches must be merged into later branches to keep them up-to-date. The can be quite time consuming - particularly as some of our components (in the ETL space) are binary and can’t be auto-magically merged by our source code repository tools (Subversion).
We aren’t exactly sure what the impact will be on our project plan - we intend to perform the merge tasks as we go to avoid breaking huge numbers of our automated unit tests. A cursory estimate is indicating about an additional 20% to each development task - a total of 1 man month in total. I’ll be talking to our PM today - hope he’s in a good mood!!
Scrum Master
A Lesson From Kalashnikov
November 26, 2008
Ok, don’t panic, we haven’t gone gun crazy, but we picked up an interesting analogy today from one of our business sponsors. It went something like this : “The Kalashnikov AK47 only has 13 moving parts and is the most successful weapon ever invented, remember this when you start designing your next software delivery. Keeping things simple is the key to building something that is reliable, will accept the abuses imposed by our operating environment and will stand the test of time”.
This is a more colourful way of repeating our mantra “Keep IT Simple Stupid”. Applying this ‘KISS’ principle has enabled us to deliver robust and reliable software into a production environment over and over again. More importantly that software is Maintainable and Extensible - qualities that are often forgotten by the project team, most of whom have no intention of hanging around after ‘Go-Live’.
If I were to pick the biggest challenge we have faced on our current project it would be the transition of project to ‘Business As Usual’. Despite a large comms budget and endless workshop sessions in an attempt to engage the future custodians of our systems it has been an uphill struggle. Our team has suffered less than others, despite being one of the larger elements of the overall program, and the one thing that has made the process more palatable is the fact that we have tried to reduce complexity at every opportunity. If I were to list the top 5 tips for ‘Keeping It Simple’ they would be :
- Don’t accept the early version s of the solution architecture without thorough review. It has probably been conceived from a 60,000ft view and is likely to be over engineered. Review it, Challenge it, Simplify It
- Any steps that result in writing less code is good, as long as it is not at the expense of true requirements. Less code means less testing and fewer points of failure.
- Keep the number of ’systems’ involved in an end-to-end process as few as possible. Each step in the chain is a potential point of failure, and with different systems using different logging and exception mechanisms means the process of diagnosing issues becomes a phorensic excercise of huge proportions.
- Standardize your development process and avoid ‘Specialists’. Each member of your design, development and testing team should be as interchangeable as possible. They should subscribe to a common set of goals and processes, and should collaborate regularly. If a team member is working in isolation for long periods of time you can bet he/she is building a problem.
- Integrate small chunks of functionality at a time. It’s much easiler to resolve issues and refactor components if you and introducing small changes to the code base regularly rather than attempting large integrations. Your options are severley limited if you find problems after a large integration (in terms of time and resources), especially if a signisifcant redesign is needed.
Scrum Secret #2 - low tech rocks!
August 1, 2008
There are a number of elements to SCRUM that really work well for us. One of the most obvious features is our ‘SCRUM Wall’, where we ‘publish’ all the work items that form part of the current sprint to ‘The World’.
The Wall is primarily for the team, but it’s surprising how many ‘outsiders’ take a real interest. Initially there’s the comments about ‘nice wall - but a bit low-tech isn’t it’, to which we respond - ‘that’s the point’. The great thing about the Wall is that each card is a tangible thing, representing a tangible deliverable. Rather than being locked away in a spreadsheet or web app (though we do use TACKLE to track our tasks electronically to - more on that later) the cards can be accessed easily by everyone.
The cards typically begin life with a one-liner describing a deliverable, and estimate of how long it will take and the initials of one of the developers. However it doesn’t take long before the card has evolved with the scribblings of several members of the team as more information is discovered.
Quite often one card will become two or three as the iterative process of understanding more about the task unfolds. The pure simplicity of the wall means that things don’t get forgotten or fall off the radar - every thought and new discovery is captured quickly and easily.
The wall only represents 2 weeks of work at a time - a Sprint. This means that every 2 weeks the wall gets cleared. The completed tasks are archived. The items that are not complete or blocked are taken down and form the basis of the ‘Backlog’ for the next sprint, in addition to new scope items that we take from our ‘product Backlog’.
We are currently running a one-hour planning session with the whole team every 2 weeks, which gives us ample time to review where we are with the existing work items and allows us to estimate how much new scope we can bit off to keep us occupied for the next 2 weeks.
This process is incredibly tranparent and efficient. The constant replanning and course correction means that: we rarely build anything that is not part of the final solution; everyone on the team is cross skilled and across the big picture; we can easily slot in refactoring tasks to increase the robustness of components we have already built but need some ‘love’; reporting upwards to management on progress is easy due to the high visibility of the methodology (lets face it - who could miss our scrum wall!).
The best thing about this methodology is that the team is completely behind it. The core team members don’t want to work without it and new recruits accept and join in with the processes eagerly.
LONG LIVE SCRUM!
Application virtualization with xenocode
July 14, 2008
Application is the future of virtualization. Xenocode allows you to package up an application and deploy it in a virtaul exe to run anywhere, (anywhere on windows). I came across it when I wanted to create a portable application using the ,net framework. Xenocode is amazing. It allowed me to package up a winform application written in C# to produce a stand alone exe that could run on any windows box even without the .net framework installed. I have not tried it under wine/linux but it is on my todo list.
While we have yet to use it in a project think how easy a deployment process will be when have xcopy deployment without any need for a framework to be installed on the destination machine.
So for all you .net coders out there start pumping out the usb portable apps with visual studio.
turn your dotnet exe into a portable usb app
Subversion
July 14, 2008
Subversion is our prefered tool for version control. With the latest version you have merge tracking and can compare history of word and excel documents, (when using the tortisesvn client). With this new functionality I can see it being our document management system as well. It is fast, easy to set up and transactional, (commits all your changes or none at all unlike pvcs). As a added bonus it is opensource and free to use.
There are many clients for subversion the two we prefer are
http://tortoisesvn.tigris.org/
This is one of the core bits of software you want running on day one of a project. We had a architecture team try to force us to use pvcs which is non transactional and pay for the privilege. What a step backwards that would have been. It was only that we had it running in our build process from day one that stopped this happening. It was a bit of a lesson for me, if you want your key bits of technology supporting the development process have them running on day one. I wish we did the same with our bug tracking tool. Unfortunately we were forced to use trackrecord which I would not recommend to anyone, (especially when there are some many excellent open source bug tracking tools).
The New Developer
July 14, 2008
We believe that when a new developer starts work in our team they should be able to start work on real project tasks by lunch. To achieve this you need a number of things to be in place namely
- A Buildprocess
- Automated unit testing
- Developers virtual image
- Project tasks that can be allocated to a new developer
- A developer allocated to pair with the new developer in the initial stages.
Added to this we have a number concepts to keep in mind
Developers are drip fed into the team. New developers should be assimilated into the team one or two at a time. This is to avoid old school or negative views around the benefit of automated builds and unit testing. By having one or two developers come in at a time it is easy to get them coding and testing the way we want.
Egoless programming. By this we mean no developer owns a specific piece of code and you should always be able to listen to a different view. Don’t be defensive when someone when someone offers a different approach to something. I have met very few developers I could not learn something from, (they do exist but usually will not fit into ego less programming because they have big ego’s).
Everyone is Expendable. If you find someone in your team that can not be replaced easily start to arrange so their knowledge gets shared around. The reality of the IT industry is people change jobs all the time. The culture we have built up in our team has kept this to a minimum as everyone enjoys the success and development processes we follow. From a management point of view we have done a good job if on any given day we can loose an experienced resource and the team keeps working as usual. Having one person own a price of code can have disastrous results. If one person is left to be the only developer on a piece of the application they generally fall back to their core skills to solve a problem. A recent example we encountered was when we inherited an etl process. We had no influence over the development and were lead to believe that ssis was used to implement it as prescribed by the project architecture. After we looked at the code we soon realized that just about all of the etl was implemented in tsql which was the developers primary skill. All the out of the box functionality which makes ssis a useful etl tool was ignored as it was not in the skill set of the developer. This left us with a very complex solution to something that should have been a lot easier if it was implemented by someone with ssis experience. In the end we had to make the call to rewrite the etl layer using ssis functionality and keeping the design simple.
Be wary of people who tell you they are an architect every chance they get. While some architects are worth their weight in gold they are few and far between. Make sure you check out their cv references through people you know instead of who is on their cv. I onced asked a architect who was running a data warehouse project if their database was a typical data warehouse schema expecting something like start or snowflake schema but got “Yes third normal form”.
Application support should be involved in the development process. Instead of getting new resourses it is better to get application support staff involved in the development process. How better way to ensure your legacy of automated builds and deploys will be continued when the project moves into the application support phase.
Keep It Simple. Every project is a battle to keep technology decisions simple. I have worked on very few large projects that have not been over engineered to some degree. You are always battling architecture groups, enterprise teams, empire builders and people who can talk well in meetings but are light on technical ability to implement a simple solution. You loose your fair share of these battles due to political influences. The only thing you can do in this case is to make sure you have a review at the end of the development cycle where you point out to upper management how a simple job was made complex to the detriment of the project. If you can hire someone off the street with 2+ years experience, they can sit down and understand your code, maintain it and deploy changes to production you have done your job.
So when a new developer arrives on site what happens first. During their first hour
- They meet the team during our morning 10 minute scrum. During the meeting they are assign a task.
- We pair a developer with them
- They copy our developers vmware image to their computer
- Start the developers image
- Run a script to personalise the image
- Start our build process and run the local clean step. This will download the latest databases, code and binaries that successfuly built on the build machine, (compiled and pasted unit tests).
The developer now has everything they need to complete their task and become a valued contributor to the team in no time at all.
Eagle PACE Development
June 21, 2008
The Buildmasters development Team has two years of Eagle PACE development experience having built the Operational Data Store (ODS) components of a massive front and middle office change program at one of Autralia’s biggest funds management companies. We have applied our Continuous Integration and SCRUM development methodology to great effect, delivering an incredibly robust solution, which includes over 1000 unit tests and a build/deploy environment that compiles, tests and deploys our solution on average 10 times a day.
This implementation of Eagle PACE involved over 80 data interfaces between the ODS, the internal trading, performance and rec systems, external data providers (Charles River, Bloomberg, Factset, S&P) and our outsources back-office service providers (one in Sydney and one in Shanghai).
We have refined the development of Eagle PACE uploader and exporter components to reduce lines-of-code and provide a robust suite of unit tests to ensure effective QA. We have also built a comprehensive ETL framework, allowing us to build generic re-useable interfaces to Eagle PACE.
Though the project has been a considerable challenge due to complex requirements and dependencies, the resulting system is now running in production 24/7 processing market data, portfolio positions and transactions and security details for an increasing volume of investment portfolios.
Quality Assurance - QA all the way!
June 15, 2008
I am amazed at how often organisations will not place a strong enough focus on QA. For the small amount of time spent on this task there are a number of major benefits. These include:
- Bugs can be found earlier, where it is much cheaper and quicker to fix them
- It is easier to enforce coding standards to give the code base a common style and pattern usage.
- Poor programming techniques or inadequacies such as not sufficient unit tests can be fixed earlier.
Whilst it is probably a good idea to not go too overboard with extremely strict QA enforcement, I believe that every line of code developed should be reviewed, or walked through with someone other than the developer of that code. To help with this goal, our SRUM methodology again comes in very handy. This is due to the short 2 week sprints in which we develop. It means QA’s are occuring frequently thoroughout the development cycle, and prevents it all having to occur at then end, with the various issues that that entails.
Software Coding Standards
June 15, 2008
Coding standards are a very important aspect of software development. Coding standards is not just about naming standards, but also includes using a standard pre tested and robust set of common core libraries, where appropriate using simple tested and tried design patterns, and quality control, to ensure standards are enforced.
Some of the main benefits of using set of coding standards, is to improve the quality of software delivered and uniformity of code which assist with maintainability. It is often said that if you cannot tell who has written code in an applications code base then that is a good start. Defining standards early, appropriate to the specific project and software and tools used, is important to help reduce the amount of refactoring that will need to be done later in the project at a greater cost.
There are many pre-defined examples of coding standards, which can be used as a starting point for your own standards, applicable to your own specific project. Tools can assist in implementing standards, such as Resharper and to enforce these standards, such as FXCop.
Do you WANT it or NEED it?
June 9, 2008
