Buildmasters

Market Data Monopoly? - Part One

adam — Thu, 07 May 2009 11:21:43 +0000

The provision of accurate and timely market data is truly the lifeblood of almost every financial services organisation worldwide. Every day these orgaisation rely on the latest stock prices, financial instrument attributes, corporate action events, rates and analytics in order to make trading decisions, reconcile their accounts and track their performance.

These requirements for a steady stream of vast quantities of data have give rise to a number of extremely successful market data organisations who collate their data from the exchanges and brokers and package it up so it can easily be consumed by banks, investment managers and anyone else who can afford to sign up for an account.

Two of the major players in the market data business are Bloomberg and Reuters. Their market data products are very similar and allow their clients to subscribe to market data feeds via several mechanisms, including ‘Pay-As-You-Go’ whereby you can make requests for information (eg prices and attributes) about a particular financial instrument (eg BHP Ordinary Shares) and they send back the details, for which you pay ‘Per Security’ - currently around $2(USD), depending on security tyep (Bond, Equity, Derivative, etc). Alternatively you can subscribe to entire segments of security data in a ‘batch’, for instance ‘Asian Equities’, and receive the entire universe of those securities daily. The choice of mechanism really depends on how many securities of a particular type the client needs data for and what the most cost effective mechanism is at the time.

The most common problem that subscribers to these market data services face is the degree to which ther are tied in to their provider and their ability to switch their services between providers when a ‘better’ deal can be struck with a competitor. Quite often the internal systems within the subscriber’s organisation have been designed to interface with a particular provider and mechanism - probably the provider/mechanism that offered the best deal when the systems were developed. Over time as more cost effective services become available or when data volumes increase to the point that the original mechanism is no longer cost effective the subscribers may wish to ‘replumb’ their feeds but cannot do so due to the high costs involved.

Lets illustrate this problem with a real life example….

Back in 2006 a large Australian investment management organisation in Sydney began a program of work to replatform their entire Front and Middle Office systems in order to ready themselves for their planned business strategy. This program involved integrating market data from Bloomberg and a number of smaller market data providers.

The initial release of the new platform involved relatively small volumes of financial instrument (security) data and therefore the Bloomberg Datalicense ‘Per Security’ mechanism offered the most flexible and cost-effective solution, allowing us to request the 1000 or so securities that we needed each day, incurring costs of around $2000/month - well within our budget.

One year later, during the next phase of the program we increased our security volumes to 8000, resulting in a significant increase in the month costs for market data from Bloomberg. At this point we were getting numerous calls from Reuters who wanted to pitch their market data services to us and were also looking at an alternative service from Bloomberg that would allow us to bulk buy data rather than paying ‘Per Security’.

ShareThis

Tales from the War Room

adam — Wed, 29 Apr 2009 07:09:46 +0000

With out 3rd major software release due in 40 days success is really reliant on some pretty effective collaboration between the multiple moving parts that need to come together on the day like one big well-oiled machine. That’s why I’m so glad that we setup our ‘War Room’ meeting. Each morning at 9am 20 indivduals descend on Meeting Room One from their various teams and take stock of progress over the previous 24hrs.

From the top we have : the Delivery Manager, ultimately responsible for the successful implementation of the release into Production; the Project Manager, reponsible for coordinating the campaign and keeping us all revved up; the Test Director and test team leads, responsible for executing 500 test cases with 100% success rates through System Test, Joint System Test and Joint Acceptance Testing; the Trading System BA, responsible for design and development of the BlackRock Trading Platform component; the Data Management System team lead, responsible for the 40+ integration components they feed data throughout the solution; the Enterprise Services team lead, responsible for scheduling of data feeds; the Environments team lead, reponsible for servers, hardware and dealing with the spate of City-wide power-cuts we’ve had recently!!! Finally there’s the ‘Stats Guy’, responsible for compiling an impressive array of graphs and statistics that boil down the infite number of variables that all the other members generate during any given day into something that can meaningfully be discussed in under 45 minutes.

All up a motley crew and some might wonder how a large group like that can be productive. Well lets start with the ‘why?’ question - Why do we need to sit down EVERY morning - how much can happen in 24hrs to make it worth while??

The answer is that an enormous amount of things happen in 24hrs - mainly due to the fact we are now running ‘Prod-Like’ - ie. we are simulating in out Joint System test environment exactly what will happen in Production when we go live. This means running 2 batch cycles per day loading data from our Administrator and Trading Systems and send thing data downstream - TWICE!

We are also trading on 30 different security types - exactly as will be the case when we release the system to our Front Office users.

We are fixing 10-15 defects daily and executing a similar number of test cases.

On top of all that are the numerous daily issues, risks and surprises that make this such an interesting place to work.

Top issues, defects and risks are tracked, discussed and resolved on our ‘War Board’ - see picture :

The ‘health check’ is our ‘Spider’ which shows which interfaces between systems in the current code line are running ‘clean’ :

The session is usually over in less than an hour and everyone leaves the room with the right direction for the day. Without this process our ability to keep on track would be significantly more difficult - so if you don’t have a War Room of your own then it’s time to start thinking about it - Let Battle Commence!

ScrumMaster

ShareThis

Buildmasters dataBRIDGE

adam — Mon, 20 Apr 2009 14:23:16 +0000

dataBRIDGE brings you flexibility and freedom to choose the combination of market data services that best fit your business. Historically financial services organisations have been limited in their ability to shop around for market data and acheive value for money from their service providers. Users of Bloomberg Datalicense ‘Per Security’ find it difficult to migrate to their ‘BackOffice’ product when the volumes of security data they need make it more economical to do so.

Ideally you need to be able to subscribe to a blend of services from a number of providers in order to meet your market data needs in th emost economical way possible. Buildmasters dataBRIDGE provides this capability and more.

For more information email us by filling out our form here: Buildmasters dataBRIDGE

ShareThis

Code Branching gone mad!

adam — Tue, 24 Mar 2009 04:30:43 +0000

Question : What happens in the last year of a multi-year, multi-million dollar, multi-release program of development and business change???

Answer : All the hard stuff that has been left to the end has to be delivered in parallel, in overlapping releases and in double quick time before the money runs out…oh, and if you’re really unluck you’ll have to upgrade some of the applications from the earlier releases too.

This is a reality for us at the moment on our project at one of Australia’s largest fund management companies. Despite shifting requirements and architecture we have managed to deliver 2 successful major releases over the last 2 years and yet our biggest challenge lays ahead of us.

So far we have built a powerful Investment Data Management Platfor using the Eagle PACE product as a base, into which we have integrated accounting feeds from Hiport, market data feeds from Bloomberg, Factset and S&P. On the other end we have integrated 3 trading platforms - Charles River (for Equities and Enterprise Compliance), Imagine (for Derivatives) and BlackRock Aladdin (for Fixed Income). On top of all that we have integrated a Performance and Anlaytics system (StatPro), Recs System (Smartstream TLM) and Corporate Actions system (Smartstream Coacs). To really finish things off we have added a Data Mart layer to support Internal reporting via Business Objects and External Reporting via SS&C Pages - PHEW!!

How can there be more you’re asking yourselves? Well now we have to feed Registry Data into the mix too from multiple registries and switch off several legacy systems ( not forgetting remediation of a reasonably long list of small access and excel based apps).

Finally we are upgrading Eagle, Charles River, StatPro and SS&C Pages….all before December 31st 2009!

As you can imagine we have alot of issues to overcome, and there’s not enough room on this page to discuss them all, but I thought I’d share with you one of the many issues that just get overlooked when these gran plans are hatched - code branching!!

For the uninitiated code branching (and merging) is the situation you find your self in when you have multiple release dates of the same code base running concurrently. In a perfect world you would only have 2 lines of code - Development (known also as ‘THE TRUNK’) and Production Support (known also as ‘THE PROD BRANCH’). In this perfect world when ‘hot fixes’ are applied to the ‘PROD BRANCH’ they are then merged into ‘THE TRUNK’ to keep it in line with with what is in Production, that way when a major release is deployed to Production from THE TRUNK it includes all the hotfixes that have been accumulated since the last major release.

Our problem is we have mul;tiple teams working on multiple ‘major’ and ‘minor’ releases of the same application (The Data Management Platform). As a result we have THE TRUNK, THE PROD BRANCH and 2 additional branches - one for an upgrade release and another for a parallal major release. Confused? I’m not surprised! Here’s a photo of the whiteboard after our team meeting this morning..

The release order runs left to right. The PROD BRANCH is always on the left as a deplpoyment to Production by the support team can happen anytime, next comes ‘91′ which represents an upgrade of our data management platform toolset to version 9.1, then there’s 3A - the current major release we are working on - due to go to Prod in June. Finally there ‘4′ that is due to go to Prod in Septemberbut development has to start now - hence another branch!!!

The arrows at the top indicate where all the overhead exists with this situation. Changes in earlier branches must be merged into later branches to keep them up-to-date. The can be quite time consuming - particularly as some of our components (in the ETL space) are binary and can’t be auto-magically merged by our source code repository tools (Subversion).

We aren’t exactly sure what the impact will be on our project plan - we intend to perform the merge tasks as we go to avoid breaking huge numbers of our automated unit tests. A cursory estimate is indicating about an additional 20% to each development task - a total of 1 man month in total. I’ll be talking to our PM today - hope he’s in a good mood!!

Scrum Master

ShareThis

A Lesson From Kalashnikov

adam — Wed, 26 Nov 2008 23:45:59 +0000

Ok, don’t panic, we haven’t gone gun crazy, but we picked up an interesting analogy today from one of our business sponsors. It went something like this : “The Kalashnikov AK47 only has 13 moving parts and is the most successful weapon ever invented, remember this when you start designing your next software delivery. Keeping things simple is the key to building something that is reliable, will accept the abuses imposed by our operating environment and will stand the test of time”.

This is a more colourful way of repeating our mantra “Keep IT Simple Stupid”. Applying this ‘KISS’ principle has enabled us to deliver robust and reliable software into a production environment over and over again. More importantly that software is Maintainable and Extensible - qualities that are often forgotten by the project team, most of whom have no intention of hanging around after ‘Go-Live’.

If I were to pick the biggest challenge we have faced on our current project it would be the transition of project to ‘Business As Usual’. Despite a large comms budget and endless workshop sessions in an attempt to engage the future custodians of our systems it has been an uphill struggle. Our team has suffered less than others, despite being one of the larger elements of the overall program, and the one thing that has made the process more palatable is the fact that we have tried to reduce complexity at every opportunity. If I were to list the top 5 tips for ‘Keeping It Simple’ they would be :

Don’t accept the early version s of the solution architecture without thorough review. It has probably been conceived from a 60,000ft view and is likely to be over engineered. Review it, Challenge it, Simplify It
Any steps that result in writing less code is good, as long as it is not at the expense of true requirements. Less code means less testing and fewer points of failure.
Keep the number of ’systems’ involved in an end-to-end process as few as possible. Each step in the chain is a potential point of failure, and with different systems using different logging and exception mechanisms means the process of diagnosing issues becomes a phorensic excercise of huge proportions.
Standardize your development process and avoid ‘Specialists’. Each member of your design, development and testing team should be as interchangeable as possible. They should subscribe to a common set of goals and processes, and should collaborate regularly. If a team member is working in isolation for long periods of time you can bet he/she is building a problem.
Integrate small chunks of functionality at a time. It’s much easiler to resolve issues and refactor components if you and introducing small changes to the code base regularly rather than attempting large integrations. Your options are severley limited if you find problems after a large integration (in terms of time and resources), especially if a signisifcant redesign is needed.

ShareThis

The search for a C# DAL code generator that actually works. Part 3: the solution.

tubby taylor — Thu, 04 Sep 2008 08:28:44 +0000

part 1
part 2
part 3

In part 2 of this series, I outlined my requirements for an ideal DAL generator.

Well, not having found any tool out there that meets my criteria, I ended up writing the tool myself, meeting all but one of my objectives [I dont handle stored procedures yet].

Sql Server 2005 is the target DBMS for now, using VS2005 as the compiler.

Lets walk through how to use the tool.

From the command line, the tool is invoked like this.

mycodegentool.exe config.xml

The simplest config.xml file is



 D:\development\codegenExample\myDatabase

 Data Source=.;Initial Catalog=myDatabase;

    Integrated Security=SSPI;

 update_date

This will give you a DAL dll with every table and view of the database in it, and handle concurrency for any table that that has an update_date field.

To get a unit test dll as well, add this xml fragment to the config file.



 D:\development\codegenExample\myDatabase\CodeGenUnitTest

This takes care of the plumbing. Now reference the DAL dll from your business layer dll, and happy porcelain coding.

Here are a few use case scenarios in actually using the DAL from your business layer.

Load up a table.

ProductsFactory factory = new ProductsFactory();

List rows = factory.LoadAll( connection ) ;

Console.WriteLine( rows.Count ) ;

Quicker way to see have many rows in a table.

ProductsFactory factory = new ProductsFactory();

int rowcount = factory.GetRowCount( connection ) ;

Load up a specific row from a table using the primary key.

int id = 17 ;

ProductsFactory factory = new ProductsFactory();

Products product = factory.LoadByPrimaryKey( connection, id ) ;

Load up a set of rows from a table based on custom search criteria.

bool discontinued = true ;

ProductsFactory factory = new ProductsFactory();

List rows = factory.LoadByDiscontinued( connection, discontinued ) ;

To add custom search criteria to a table or view, add this xml fragment to the config file.

where the contents of the sql file are

where discontinued = @discontinued order by productname

Get a row’s parent row.

Here we are walking the employee hierarchary using ootb generated methods - cool.

EmployeesFactory factory = new EmployeesFactory();
// robert - the pleb

int? employeeId = 7;

Employees robert = factory.LoadByPrimaryKey( connection, employeeId);

Assert.AreEqual( robert.FirstName, "Robert" ) ;
// robert reports to steven

Employees steven = robert.parentEmployeesByReportsTo( connection ) ;

Assert.AreEqual(steven.FirstName, "Steven");
// steven reports to andrew

Employees andy = steven.parentEmployeesByReportsTo( connection);

Assert.AreEqual(andy.FirstName, "Andrew");

// andrew is the big cheese Employees nevilleNobody = andy.parentEmployeesByReportsTo(connection); Assert.IsNull(nevilleNobody);

Get a row’s children rows.

Here we start with an order, then go up to get its customer, then go down to get all the orders of this customer - powerful pattern in a few lines of code.

OrdersFactory factory = new OrdersFactory();

int orderId = 10289 ;
// get an order

Orders order = factory.LoadByPrimaryKey( connection, orderId); 
// the customer for this order

Customers customer = order.parentCustomersByCustomerID( connection) ;

// all the orders of this customer List orderList = customer.childOrdersByCustomerID( connection ) ;

Load up from a view.

Same as loading from a table.

Alphabetical_list_of_productsFactory factory = new Alphabetical_list_of_productsFactory();

List rows = factory.LoadAll( connection ) ;

Load up from a custom query.

Same as loading from a table, with some extra config.

CustomerOrderFactory factory = new CustomerOrderFactory();

List rows = factory.LoadByRegion( connection, "Quebec" );

List rows2 = factory.LoadByCustomer( connection, "quick" ) ;

The config in this case would be

Here we have a customer order view, for display on a web page data grid for example.

The CustomerOrder.sql file contains

select * from customers c, orders o

whilst the LoadByCustomer method has the following criteria in the config sql file reference

where c.customerid = o.customerid and c.customerid = @customerid order by c.companyname, o.orderdate

and the LoadByRegion method’s where clause is

where c.customerid = o.customerid and isnull(c.region, '') = isnull(@region, '') order by c.companyname, o.orderdate

Insert a new row

OrdersFactory factory = new OrdersFactory();

Orders order = new Orders() ; // create a new row in memory

order.CustomerID = 123;

order.EmployeeID = 7 ;

order = factory.Save( order, connection ) ; // does the insert

Update an existing row

OrdersFactory factory = new OrdersFactory();

int orderId = 10289 ;

Orders order = factory.LoadByPrimaryKey( connection, orderId); // get an order

order.ShipPostalCode = "2565"

order = factory.Save( order, connection ) ; // does the update

Insert then update a row

OrdersFactory factory = new OrdersFactory();
Orders order = new Orders() ; // create a new row in memory

order.CustomerID = 123;

order.EmployeeID = 7 ;

order = factory.Save( order, connection ) ; // does the insert

order.ShipPostalCode = "2565" order = factory.Save( order, connection ) ; // does the update

Delete an existing row

OrdersFactory factory = new OrdersFactory();
int orderId = 10289 ;

Orders order = factory.LoadByPrimaryKey( connection, orderId); // get an order

order.MarkForDelete = true ; factory.Save( order, connection ) ; // does the delete

Negative use case - get a concurrency error.

TestTableFactory factory = new TestTableFactory();
TestTable alice = factory.LoadByPrimaryKey( connection, 12345 ); // get a row

TestTable bob   = factory.LoadByPrimaryKey( connection, 12345 ); // get it again
alice.update_date = DateTime.Now ;

bob.update_date   = DateTime.Now ;

alice = factory.Save( alice, connection ) ; // does the save ok bob = factory.Save( bob, connection ) ; // will throw a concurrency exception

Negative use case - get a save error when saving a stale object

OrdersFactory factory = new OrdersFactory();
int orderId = 10289 ;

Orders order = factory.LoadByPrimaryKey( connection, orderId); // get an order
order.ShipPostalCode = "2565"

factory.Save( order, connection ) ; // does the update - ok

order.ShipPostalCode = "2000" factory.Save( order, connection ) ; // will throw a stale row exception

Many users simultaneously inserting rows into a table.

In this example, a total of 5000 rows will be inserted into a table by 50 threads running in parallel. The table in question has a monotonically increasing non-identity integer ID primary key field, and therefore it is a contention hotspot.

In the first version of this use case, the inserts fail, as the code and sql that determine the next primary key value are not protected against concurrent threads. The test fails with the error “Violation of PRIMARY KEY constraint ‘PK_TestTable3′. Cannot insert duplicate key”.



protected void InsertLotsOfRowsInParallel()

{

 using ( MyThreadPoolManager tpm = new MyThreadPoolManager( 50, 5000 ))

 {

  for (int i = 1; i <= rowsToInsert; i++)

   tpm.Queue( InsertOneRow() ) ;

  tpm.WaitUntilAllFinished() ;

 }

}

// every invocation of this method runs in its own thread

protected void InsertOneRow()

{

 using (TransactionScope transaction =

         new TransactionScope(TransactionScopeOption.RequiresNew))

 {

  using (SqlConnection connection = new SqlConnection( _connectionString ) )

  {

   connection.Open();

   TestTable3Factory factory = new TestTable3Factory();

   TestTable3 obj = new TestTable3();

   // BAD: get the next available ID = max(ID) + 1

   obj.ID = select max(ID) + 1 from the table

   obj.update_date = DateTime.Now;

   // insert the row - this will fail with PRIMARY KEY constraint violation

   obj = factory.Save(obj, connection);

   transaction.Complete(); // commit - wont get here

  }

 }

}

In the correct version of this use case, the codegen tool has ootb capability to handle this common scenario. The tool will escalate the lock around the next primary key value and serialise access to the table, giving everyone a warm fuzzy feeling.



// every invocation of this method runs in its own thread

protected void InsertOneRow()

{

 using (TransactionScope transaction =

         new TransactionScope(TransactionScopeOption.RequiresNew))

 {

  using (SqlConnection connection = new SqlConnection( _connectionString ) )

  {

   connection.Open();

   TestTable3Factory factory = new TestTable3Factory();

   TestTable3 obj = new TestTable3();

   // GOOD: the tool will get the next available ID, the safe way

   obj.ID = factory.GetNextId( connection, TestTable3.Column_ID, true );

   obj.update_date = DateTime.Now;

   // insert the row - now it works

   obj = factory.Save(obj, connection);

   transaction.Complete(); // commit - aha

  }

 }

}

Here are a few .NET Reflector screen dumps of the DAL layer that the tool generates for Northwind.

This is what the Orders row object looks like.

And the corresponding factory object for the Orders table.

Here we have a custom query object that is a join between Customers and Orders tables.

And the corresponding factory object.

This screen shot shows some of the generated code involved in unit testing the DAL. Here, in the insert phase, each table is being populated and saved in referential integrity order, with pre and post row counts, object state, and field values being asserted.

One of the databases that I have recently codegened is the popular DotNetNuke database.

You can download my DAL for DNN here and also download the corresponding unit test dll here.
I recommend .NET Reflector to inspect the generated code.

Contact Buildmasters if you would like a DAL generated for your database.

This is the to do list for the tool

Make sure it compiles and works under Mono and Debian Linux.
Make sure it compiles and runs under VS2008.
Extend to target SqlServer 2008, MySql, Oracle databases.
Extend to handle stored procedures.

So far I have used the tool against 13 corporate, vendor and public databases, and it has my made my programming experience easier, faster, and more reliable.

I do not think about the DAL layer at all now - it just works.

ShareThis

The search for a C# DAL code generator that actually works. Part 2: the requirements.

tubby taylor — Thu, 04 Sep 2008 07:57:02 +0000

part 1
part 2
part 3

Now that I have outlined the reasons why current DAL tools do not satisfy me, I will list my requirements for the ideal DAL code gen tool.

The tool should:

Generate the DAL for the whole database in one hit, not table by table, generating all non-system tables and views out of the box.

Have a simple pleasant configuration file experience.
Ideally, the minimum configuration required would be a connection string to the database, and a directory path to which files are generated.

Be able to support the addition of custom query objects and custom criteria clauses, and to do so in an easy manner.

Create the DAL in a separate dll, so that the source code of my business rules layer contains only business rules code.

Be a command line tool, so I can run it as part of my build process, and via cruise control, regenerating the DAL every night and on-demand during the day, keeping it in synchronised with the underlying database schema.

Create objects that do not hold any database locks.
Therefore all objects returned from the DAL are to be disconnected.

Be designed with concurrency in mind from the ground up.
No joke, the vast majority of systems that I have encountered over the last two decades have not taken concurrency into account.
Consider this scenario: Alice and Bob both read row A from a table in the database.
Alice modifies her copy of row A and saves it back to the database.
Bob then changes his copy and successfully saves it back.
Congratulations, your database is now corrupt.
This is incorrect behaviour, as Bob should have received a concurrency exception when he attempted to save his stale row.
The DAL tool should support this out of the box.
This is plumbing - the programmer should not have to worry about it.
Concurrency handling is especially important now that we live a disconnected data set world.

Uses nullable types. I do not want to explicitly check for nulls before I get or set a field.
The tool should do it.

Not use events.

Be designed with database transactions in mind from the ground up, and should work with implicit transactions via TransactionScope, or explicit BYO transactions.

Generate clean, understandable code that is easy to follow and debug.

Generate deterministic predictable code, so that small schema changes do not cause large diffs.

Use the referential integrity of the database itself to automatically generate accessor methods on objects that allow you to walk the database tree.
These accessors to parent and child objects are to be lazy populated, ie when the accessor is invoked..

Have a simple object model using the factory pattern.
TRow : RowBase , List, TFactory: FactoryBase.

At buildmasters, we have a very strong test ethic.
Even generated code needs to be tested.
The tool should auto generate unit tests to code cover the generated DAL code.
The unit tests should perform the following tasks in one database transaction:
- Insert a row into every table, populating each field, read the rows back, and assert the row counts and field values.
- Update the inserted rows, changing the values of each field, read the rows back, and assert the row counts and field values.
- Delete the inserted rows, and assert the row counts.
- Rollback the transaction [unit tests should leave no footprint in the database].

Have the following persistance patterns:
1. Successfully persisting a row object invalidates the object, so it cannot be saved again, and returns the latest version of the row in a new object.
2. Persisting an unmodified row object does nothing.
3. If a table does not have a primary key, its row objects cannot to be persisted.
  This is a classic case of bad database design. Every table needs a primary key.
4. A standard persistance mode that follows the rules of database integrity:
  A new memory row object can only be inserted, not updated, based on primary key.
  A row object that is read from the database can only be updated or deleted, based on primary key and concurrency cookie, and only if the concurrency cookie is still valid.
5. An ETL persistence mode:
  Persisting a row object involves trying an update, and failing that, an insert is attempted, all based on primary key.
  The concurrency flag is ignored.
  This mode of operation is used in batch ETL type tasks where you just want to transfer or inject a whole bunch of rows in to a table, whether that table is in the same database or not, and maybe on a different connection.

Provide a DAL wrapper around legacy stored procedures.
Im not a big fan of TSQL [see my post here ].
I have seen many instances of whole systems that have been written in TSQL, where the business rules have been buried away in unmaintainable several-thousand-line TSQL stored procs, as opposed to having the business rules in a proper SOA/business tier written in a high level language like C#.
It is a pain to debug TSQL and it does not scale.
But sometimes you will inherit such a mess, so at least the DAL tool can hide it behind a nice C# interface.

In part 3 of this article, I will outline a tool that meets these requirements.

ShareThis

The search for a C# DAL code generator that actually works. Part 1: the problem.

tubby taylor — Thu, 04 Sep 2008 07:55:34 +0000

part 1
part 2
part 3

Besides developers, the database is your company’s greatest asset.

The data in your database, your ability to get at it, shape it, analyse it, report on it, turn it in to understandable meaningful information, is what gives your company competitive advantage.

I have been designing, using, optimising and programming against relational databases for over 20 years. DB2, Oracle, SqlServer, Sybase, Access, and MySql are some of the major relational databases engines that I have worked with, and DAO, ADO, ADO.NET, visual studio dataset designer, nHibernate, CSLA, codesmith, and LINQ are some of the tools and DAL technologies that I have used to get at the those databases.

In all that time, I have been frustrated with all of the above DAL methods of accessing data, for various reasons that I will discuss below.

I found that I was spending too much time stuffing around trying to placate or work around the DAL technology and not enough time just getting the job done.

Developers should spend their time writing the porcelain (the business rules that are specific to solving a business problem that cannot be generated by a tool) and very little time writing and working around with the plumbing (the repeatable monkey code infrastructure that can be generated by a tool).

Let us take the visual studio dataset designer for example.
The visual studio DAL tool is fairly good, but I have several issues with it which have caused me a great considerable frustration .

For one, it does things one table at a time. This is very cumbersome when one has 100+ tables in a database that needs to be DALd.

It is also very slow. When I right click on the designer, my pc freezes for several seconds until the context menu pops up. FWIW I have a high spec dev box.

The generated code does not compile in SQLCLR safe mode.
Why can’t something I generate in visual studio work in SQLCLR safe mode?
Hello Microsoft. Visual studio and Sql Server are both your products.
The problem is that the designer inserts a bunch of delegates and events into the generated code, and events use synchronisation objects, which are forbidden in SQLCLR safe mode, and there is no way to prevent generation of these events every time you touch the designer.
Who actually uses those events anyway?
Am I the only person that thinks that events are overrated and over used.
For app support teams, events are a nightmare to debug.
One needs innate familiarity with the code to visualise the execution flow, even when using a debugger.
In my experience, events have been more trouble than their worth over the years.

Null checking. The code generated does not use nullable types for some reason, so every time I get or set a property, I have to write code to first test for null, otherwise the dataset code will kindly throw an exception.
I just want to set a property – 1 line of code, not 5, please.

Also, after your program is complied, you don’t know if or when the underlying database schema has changed. Guess when you find out – runtime!
I want the DAL layer to be in sync with the actual underlying database schema at all times.
If I had a command line interface to the visual studio dataset designer, I could regenerate the DAL layer through my continuous build process and have confidence in the accuracy of my DAL.

Another annoyance is that sometimes the designer changes column orders, seemingly at random, and a small change in the designer causes illogical reordering in the designer.cs file, making diffs hard to read.

The typed data adapter classes do not expose their contained SqlCommand objects publicly, so I cannot associate an explicit transaction to them.

The other data access methods don’t do the job for me either:

CSLA: the version I used was based on XSLT templates – OMG -–enough said.

nHibernate: getting closer, doesn’t handle my requirements see below.

linq2sql and EF4. Microsoft’s latest attempts at ORM. Yep that’s right - plural - there’s 2 of them (to be fair linq2sql is sql server only). Linq2sql’s DataContext cache-based architecture will cause you to pull your remaining hair out, especially in n-tier scenarios, as here. EF4, the first real version of entity framework that works, in visual studio 2010, is the long term best bet, but you’ll need to invest quite a bit of time and energy to get to EF4 nirvana, to learn it philosophy and quirks, which are not obvious at first glance, and if you don’t keeps things simple, you can really get your knickers in a knot.

In part 2 of this article, I will outline my requirements for an ideal DAL code generator.

ShareThis

Legacy MetaData Analysis

adam — Fri, 22 Aug 2008 06:09:00 +0000

Do you have complex legacy systems with extensive database components? Do you need to map out what you have in order to move forward efficiently?

This kind of analysis is time consuming and error prone if attempted manually. We have a fast and efficient automated solution that will save you time and money during this type of project.

Some of the issues your Business Analysis Team will face are:

Legacy System with 000’s of Tables, Views, Stored Procedures and Functions
Complex interdependencies between them
No documentation and no original developers
No-one knows what is being used and what isn’t
Fragile environment as small changes in one place can have large impacts elsewhere

The Buildmasters MetaData Analysis process uses a combination of industry standard tools and our expertice to generate a comprehansive and easy to navigate resource that will enable your own team perform their analysis.

SEE A SAMPLE REPORT HERE

An engagement with us involves the following non-intrusive steps:

1. Proof Of Concept

This is a low cost exercise that allows us to sample the source database systems and produce a report covering a fragment of the databases for appraisal by the client. The process will typicall involve:

Extraction of database structure from source database(s)
Parsing and Cleansing of Metadata
Load into the Buildmasters Metabase for integrity checking and further cleansing
Processing via the Buildmasters Database Documentation Tool
Publishing of Documentation either online or in a portable help file

____________________________________

2. Comprehensive Data Analysis Report

Having refined the output during the Proof-Of-Concept the full report is generated and cleansed by the Buildmasters Team before being delivered to the client for final acceptance. An example of one of our meta-data analysis reports is available here, generated using the sample Northwind database.

The Proof-Of-Concept (POC) step guarantees our clients expectations are met before having incur significant analysis costs. If you could like to find out more about this service please contact us using the link below.

ShareThis

Scrum Secret #2 - low tech rocks!

adam — Sat, 02 Aug 2008 03:12:34 +0000

There are a number of elements to SCRUM that really work well for us. One of the most obvious features is our ‘SCRUM Wall’, where we ‘publish’ all the work items that form part of the current sprint to ‘The World’.

The Wall is primarily for the team, but it’s surprising how many ‘outsiders’ take a real interest. Initially there’s the comments about ‘nice wall - but a bit low-tech isn’t it’, to which we respond - ‘that’s the point’. The great thing about the Wall is that each card is a tangible thing, representing a tangible deliverable. Rather than being locked away in a spreadsheet or web app (though we do use TACKLE to track our tasks electronically to - more on that later) the cards can be accessed easily by everyone.

The cards typically begin life with a one-liner describing a deliverable, and estimate of how long it will take and the initials of one of the developers. However it doesn’t take long before the card has evolved with the scribblings of several members of the team as more information is discovered.

Quite often one card will become two or three as the iterative process of understanding more about the task unfolds. The pure simplicity of the wall means that things don’t get forgotten or fall off the radar - every thought and new discovery is captured quickly and easily.

The wall only represents 2 weeks of work at a time - a Sprint. This means that every 2 weeks the wall gets cleared. The completed tasks are archived. The items that are not complete or blocked are taken down and form the basis of the ‘Backlog’ for the next sprint, in addition to new scope items that we take from our ‘product Backlog’.

We are currently running a one-hour planning session with the whole team every 2 weeks, which gives us ample time to review where we are with the existing work items and allows us to estimate how much new scope we can bit off to keep us occupied for the next 2 weeks.

This process is incredibly tranparent and efficient. The constant replanning and course correction means that: we rarely build anything that is not part of the final solution; everyone on the team is cross skilled and across the big picture; we can easily slot in refactoring tasks to increase the robustness of components we have already built but need some ‘love’; reporting upwards to management on progress is easy due to the high visibility of the methodology (lets face it - who could miss our scrum wall!).

The best thing about this methodology is that the team is completely behind it. The core team members don’t want to work without it and new recruits accept and join in with the processes eagerly.

LONG LIVE SCRUM!

ShareThis