Not so fast.

Nassim Nicholas Taleb defined black swan events thus:

First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme 'impact'. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.

Thibodeau notes in his article:

It turns out that project's 55 contractors had only two weeks to conduct end-to-end testing of Healthcare.gov prior to launch.

Okay.  I've been doing this for twenty years, though I've never led a project with so many stakeholders, or one so mired in regulations.

This is no black swan.  This project was doomed from the start.  It doesn't even matter directly that it's a government IT project, all you need to know is that 55 contractors were involved.

The two weeks allowed for integration testing, and the fact that they got anything resembling a working web site, actually gives me a new-found respect for those contractors.  With a project of this size, an integration test run would take around three months - two months preparation, and one month of everyone working together testing and gathering information.

And you'd typically need about four test runs to get things working smoothly.  And even that could only be achieved if none of the rules changed during that time, and none of the components were found to be toxic - by which I mean that fixing the problems they caused for the overall system would cost more than throwing the component out and rewriting it.

With 55 contractors, you're pretty much guaranteed to have a few toxic components.  Even if everyone involved is basically competent and honest, simple statistics tell us something is bound to go wrong somewhere - a key project lead gets hit by a bus, or hired away by Google; or a new software library causes unforeseen performance problems that require a rewrite and blow out the schedule for the component - and since the deadline is fixed, that component comes in with only minimal testing and passes malformed data on to the rest of the system.

And even then, we're assuming that the overall project management are not just competent, but very talented indeed.  Pulling 55 teams together like that is hard; if the top management are merely experienced and competent, the timeframe for integration would expand from a year to at least two.

And even then we're still ignoring an issue I hadn't thought of myself* - the problem of data schemas.  Or more precisely, the lack of data schemas.  I don't know exactly what systems the site needs to hook into, but we can take it as read that some of them involve decades-old hierarchical databases with no formal schema - no strict definition of the layout of the data - and instead the rules and regulations are enforced by forty years of accumulated Cobol.  And if you try to import that data directly, you'll need to replicate that 40 years of work precisely.

I bring projects in on time and within budget.**  But I have a tool that these people simply did not have - I can go to the client and change the requirements.  In the real world, project requirements are a mix of necessities, hopes, and dreams, and the task of the project manager is to deliver the necessities and to work with the client to turn the hopes and dreams into something realisable.  When I'm presented with a proposal, I can usually say within minutes that the combination of A, B, and C simply cannot be achieved within the desired timeframe (or sometimes, at all), and that we either need to drop one of them, or substitute D for C - where D achieves much of what C would have done, but without the combinatorial blowout of complexity or performance that would have doomed C.  And real-world clients are usually very receptive to that.

In the case of Healthcare.gov, the number of stakeholders, the functional requirements, the interoperability requirements, and the deployment schedule were all nailed down.  With no relief valve in the project definition, there were only a couple of ways this could go.  A major cost blowout would be one way, but that runs head-on into Brooks's Law - someone would have needed to realise the impossibility of the project before it began, and tripled (or whatever) the budget and workforce for this to have any chance of succeeding.

More likely (and this may be exactly what happened) the schedule problems would have been realised half-way through the project.  That's pretty typical; half way through is where you end up taking a serious look at your progress, and while you can convince yourself (usually falsely) that if you're halfway done at the halfway point you're in good shape, if you're a quarter of the way done at the halfway point, even the most optimistic manager is going to hear alarm bells.  Then you push the problem upstairs, and you get assigned more resources, and, as we've known for 40 years, that makes things worse.

Basically, as defined, this was probably a five-year project.  You could have stripped out one requirement - made this an advisory site rather than a registration site - and a good mid-sized development team could have knocked it out inside a year.

And in the commercial world, that's what would have happened.  As a government project, well, the only reason this thing is still sitting on its perch is because it's been nailed there.

Posted by: Steven Den Beste at Saturday, October 26 2013 03:06 AM (+rSRq)

Posted by: Pixy Misa at Saturday, October 26 2013 11:09 AM (PiXy!)

Posted by: Tommy at Wednesday, November 13 2013 05:21 PM (70H8v)

Hide Comments | Add Comment