Oct 30 2013

Healthcare.gov: 10 Lbs Of “Stuff” In A 3 Year Schedule

Published by at 10:54 am under All General Discussions,Obamacare

I have been doing more research on the programmatic and technical management of the Healthcare.gov program, and have identified the main reasons ObamaCare’s roll out was such a disaster. Bottom Line: Too much scope in too short a time. And the root cause of that was government taking on more than it knew how to do and not delegating enough technical management to contractors.

But before I get into those details, I must address this idea that it was the contractors’ fault and not the fault of HHS and CMS.

Health and Human Services Secretary Kathleen Sebelius will tell a House committee tomorrow the site’s botched rollout was the result of contractors failing to live up to expectations – not bad management at HHS, as the contractors suggested.

It most definitely WAS the fault of the government, who mismanaged this thing from the beginning. Not only did they not understand what they were doing, they took on the technical lead role to design and specify the solution, then integrate and test it. From what I have seen their intentions and some strategies were OK, but their execution was naive and clumsy.

Of course, the Sebelius CYA effort begs the question: if these contractors screwed up, why did HHS/CMS just appoint one to take over for them as technical lead? Got an answer to that?

And as for the idea HSS/CMS was not aware of the problems as their amateur effort began to fall apart this summer, the evidence is clear this was not the case. I noted this a while back in documents from state exchanges who were growing more and more concerned that CMS was losing control of the situation.  From this initial post on the subject [click to enlarge] is this fascinating report from the Colorado Exchange on Sept 13, 2013:

The 3rd issue notes The [Federal Data Services] Hub had just completed stand alone testing! End to End testing was yet to begin – and this is Sept 13? They ran out of time, and that is poor technical management. And then they deployed anyway – and that is political stupidity.

But an even earlier risk assessment from the Washington State Exchange effort from Jul 2013 pointed to the sane problems:

This “urgent” risk notes the interfaces between the State Exchange and the CMS Federal Data Services Hub [FDSH]. The conclusion (or warning to everyone state and federal) was:

These interfaces and associated systems are critical to the ability of the [Washington State] Exchange to be operational on the October 1 Go-Live date. … HBE [Washington’s Exchange] application processing has a high dependence on the [CMS FDSH] interface. As we noted in our June month-end assessment, CMS (verbally) re-confirmed a mid-August delivery of fully operational FDSH services.

OK, stop right there. In June 2013 Washington State (and probably all the 25 other states with exchanges being deployed) were pushing HHS/CMS to ensure HHS/CMS could meet schedule.  HHS/CMS promised an August delivery of the Data Hub (FDSH).  But we now know from the report from the Colorado Exchange from September that promise was never met.

Washington State goes on to conclude:

HBE will not be able to conduct full end-to-end testing of services until CMS makes this delivery. The probability of an on-schedule delivery of all FDSH services is, at best, questionable – and worrisome.

Look, Washington State is not creating and documenting these risks in isolation. They are being passed to HHS/CMS as well. And they were predicting failure in July.

Conclusion: HHS/CMS knew about these issues and their ramifications.

The one piece of this mess that HHS/CMS defends as ‘working’ each and very time is their precious Data Hub. Sebellius goes out of her way to make it clear the Data Hub is working. But the Data Hub is clearly the problem most noted here! There is something about the Data Hub, it seems to be the HHS/CMS Holy Grail.

As can be seen in this schedule that came out on Sept 13 in that Colorado State report, the Hub was being patched and tested all the way up to “Go-Live”

The Data Hub is on the fourth row under “CMS Required Testing”. Click to enlarge and you will see FDSH testing going on in parallel with end-to-end (E2E) testing. This is a sure sign of a program heading for a crack up.  If you are doing E2E in parallel with stand alone testing of your center element, you are in serious trouble. And it is apparently during this period that this happened:

Days before the launch of President Obama’s online health ­insurance marketplace, government officials and contractors tested a key part of the Web site to see whether it could handle tens of thousands of consumers at the same time. It crashed after a simulation in which just a few hundred people tried to log on simultaneously.

The Centers for Medicare and Medicaid Services (CMS), the federal agency in charge of running the health insurance exchange in 36 states, invited about 10 insurers to give advice and help test the Web site.

About a month before the exchange opened, this testing group urged agency officials not to launch it nationwide because it was still riddled with problems, according to an insurance IT executive who was close to the rollout.

How can HHS/CMS credibly claim they had not clue about the situation? CNN has blown the lid off that excuse.

So why do they continue to insist the Data Hub is the one piece working well?

This recent reporting would indicate just the opposite:

Connecticut residents trying to sign up for subsidized health care coverage through Access Health CT, the state’s health insurance exchange, won’t be able to complete the process because of an outage involving the federal system used to verify applicant information, Access Health announced Sunday afternoon.

The system, known as the federal data services hub, is used by the exchanges to verify information including citizenship and income of applicants. Access Health said in a statement that it received notice of the outage from the Centers for Medicare and Medicaid Services, which oversees the systems used for people to enroll in coverage made available as part of the health law commonly known as Obamacare.

While Connecticut does not use the problematic healthcare.gov website to enroll people in its exchange, as 34 other states do, the state’s exchange must still use the federal data hub as part of its enrollment process.

“Customers will not be able to access Remote Identity Proofing, Social Security Administration or Advanced Premium Tax Credits also referred to as subsidies. As a result, consumers will not be able to complete the sign up process for Medicaid or Advanced Premium Tax Credit determinationsm,” the Access Health statement said.

It is so bad Alaska has just put the stop to the whole mess until HHS/CMS can get the damn thing to work right:

Enrollment is suspended until a major issue is resolved, said Tyann Boling, the chief operating officer of Enroll Alaska. The issue is that the premium subsidy calculator for the Alaska exchange is underestimating the amount of the federal government’s portion of the total premium payment made to the insurance companies.

Friday Boling said she received an email from Susan Johnson, Region X director for the DHHS saying that the government was working to fix the Alaska subsidy calculator.

The Department of Health and Human Services expects to have the site fixed by the end of November.

There are 51 state Medicaid/CHIP systems (add DC) to interface with, half a dozen federal data systems such as the IRS and SSA (to name two), and 26 state Exchanges each implementing a subset of the Federal Exchange. That totals about 85 major and complex interfaces to work the ‘glitches’ out of.  Not to mention the hundreds of insurance companies trying to get valid data out the back end. Medicaid enrollment data to the states does even not “Go-Live” until this week!

I would wager a lot of the recent crashes have been attempts to fix the Medicaid Enrollment piece – which are adding more ‘glitches’ at each attempt.

So now we get down to the core problem. The wonders of this precious Data Hub were laid out by its author, Henry Chao, Deputy Chief Information Office and the CMS Technical Architect for what has happened. In the initial documents (see here for a primary one) you see two things: this wondrous ubiquitous system seamlessly (never a good word to see in a spec) tying to together around 85 state and federal data systems, and from there producing tailored products (insurance applications) for individuals by state – easy as pie.

The arrogance of such an endeavor is mind boggling. Tie together 85 data systems that are all independently  developed and implemented? Just mapping the data model across these would take 2 years! Each data system uses some ID to identify some attribute (e.g., name, given name, first name,  etc). All this information has to be collated in the exchange so that it can be used to calculate what options each individual has. But of course, it is not just individuals, it is families too, so don’t think one individual per application.

The first step HHS/CMS took in trying to tackle this entire effort all at once – was to tailor their process and technical documents to try and squeeze this 10 year effort into a 3 year program (section 3, page 11):

Given the need to respond quickly to the health reform requirements of the Affordable Care Act, CMS will use the Exchange Life Cycle as a streamlined model to guide and coordinate the development and implementation of the Exchanges. Various organizations will be involved in developing the Exchange IT solutions. Toward those ends, the Exchange Life Cycle accommodates all forms of systems development methodologies. CMS encourages the use of agile systems development methodology and will accommodate agile processes in the Exchange Life Cycle stage gate reviews as appropriate.

Full stop.  This is an enormous undertaking with lots and lots of moving parts. Each of the external systems is under independent development, many are legacy systems. You cannot use an Agile SW Process to tackle this job.  First and foremost you need to define all interfaces and get them on a coherent schedule.  Then you have to model all state and federal sources of data (i.e., define how each one labels common information and create a Rosetta stone to related each existing label in those data sources to a common definition, like “first name”) and then you need to model all data products (each state and each insurer in the state has unique definitions for data as well). Just understanding the format for “DOB” or “Date Of Birth” in terms of whether the day or month comes first is tedious, time consuming work.  But it must be CHECKED and verified if you want to get it right from the applicant’s input through onto the insurance enrollment form.

And of course, data modeling and translation is one of those things HHS/CMS tailored out to save time [click to enlarge]:


R = Recommended, which in tight programs means it won’t get done.

If you look at the above diagram and process, you also need to mentally apply the timeline across the top to see how ridiculous this plan was/is. The source document is dated March 2011.  All four of the purple-colored phases have to be done by October 1 2013 – around 30 months.  And most of the time (more than half) was dedicated to Concept definition through Requirements and Design. Development (3rd phase) did not begin until what looks like 20 months in.  Give it a year at best.

85 interfaces to build and test in 12 months (not including insurance providers)? With no overarching data model to provide a managed Rosetta stone between applicant data, federal databases, state Medicaid/CHIP databases and State Exchanges?

HHS/CMS developed a fictional program, that’s the problem here. They threw in everything they could possibly think of, used the sure-to-fail big bang approach (all at once), shunned critical aspects of the waterfall approach over agile wanderings, and left no time for testing and tuning. Worse yet, I found statements were this was patchwork funded across state and federal programs. And there was no way to hold states to any schedule (says so in the Statement of Work).  From top to bottom this was a fantasy.

If Congress does not hold someone accountable, then all our tax money can and will be thrown down the toilet because even obvious monstrosities of ineptitude like this will go unchallenged. Congress needs to independently investigate what happened and document ALL the lessons unlearned (and ignored). They need to make an example of programs like this which fly on good intentions yet fail on incompetence.

9 responses so far

9 Responses to “Healthcare.gov: 10 Lbs Of “Stuff” In A 3 Year Schedule”

  1. TomAnon says:

    I have been watching this Healthcare.gov problem unfold with absolute fascination. A bunch of things keep echoing around in my mind.

    1. I wonder if we will ever see an EV run on this project? (Earned Value) The government constantly asks for this on every project I am a part of.

    2. Until HHS/CMS is willing to openly evaluate and improve their “precious” data hub this will never get fixed.

    3. Who made the decision to develop this big massive all encompassing beast in one step? (the arrogance of such a decision, then again, what do you expect from a bunch of central planners with no accountability)

    4. I wonder who inserted the language that this had to be 100% functional on day one into the Law. (pretty impressive poisin pill)

  2. WWS says:

    This is going to be like taking apart an onion – each layer that is fixed is going to reveal even greater failures underneath; for example, Information Security, as in, there is none.

  3. momdear1 says:

    It doesn’t take a genius to figure out what went wrong. Incompetence. This project reminds me of the moron who mounted a sheet on a fishing pole on his row boat and dragged it around town behind a car thinking he had a sail boat. There is a reason why certain people don’t make it to the top. It is because they don’t’ have sense enough to devise programs that will work, or do anything that is worth some one paying them to do it. Because of white guilt, which has been taught in our public schools for the past 60 years, Our whole country has been turned over to incompetents, malcontents and also rans as reparations for real and imagined past discriminations against their ancestors. Whatever happened to “cream will rise to the top” and “You can’t keep a good man down.” The fact is that this idiotic ill devised plan was thought up and instituted by an affirmative actioned, self esteem enhanced, socially promoted, jive talking malcontent who turned it over to his associates who had less than sterling reputations and they have F-ed it up. Paying nearly a billion dollars to a outfit to set up an outdated website that doesn’t work, a company that is under investigation for fraudulently over charging the British Govt. just because it gave a big campaign contribution and one of the executives attended college with Michelle is malfeasance of the worst kind. It looks to me like Obama and his crew think every day is party time at tax payers expense. The Clintons entered the White House so poor that they got involved in the White Water trailer park deal to get money for Chelsea’s college. and left 8 years later worth over $100 million. I don’t think Obama has sense enough to set anything up so he gets his share. I think he is letting the Pillagers, con men and marauders take the money and run with it. So the tax payers are out a couple of trillion dollars and have nothing to show for it. History repeats itself. Think “First Reconstruction,” the last time incompetents were put in charge of governments, which left the Southern states in debt for Billions in bonded debt for projects that were never built and nobody knows where the money went. It took the South 100 years to pay off that debt. In this Second Reconstruction, the whole country is in debt, not for billions, but trillions for projects that have accomplished nothing and nobody knows where the money went. Obama Care is just another nonworking boondoggle designed to create pool of money to be looted by today’s carpetbaggers.. Welcome to the Third World.

  4. momdear1 says:

    I’d like to amend the above comment. It isn’t just incompetence and dishonesty that has caused this fiasco. It’s downright stupidity. After listening to the Oaf in Chief reiterate his claim that he didn’t lie when he said people could keep their insurance and it isn’t his fault all those canceled plans were lousy plans and his plans are better. You know those plans that didn’t require men and women past child bearing age to pay for maternity care, they are all crummy and lousy plans. This clown, who looks more and more like he is so dumb he couldn’t pour water out of a rubber boot, thinks all he has to do to accomplish something is to pass a law requiring it. The sad part is the US Senate is controlled by idiots who go along with everything he suggests. Surely these people who are ripping us off for billions for staff, travel money, conferences and all kinds of other goodies are not as stupid as their supreme leader is. They must be afraid they are going to be called Racists for disagreeing this dumb azzed Oaf. I am insulted that this Oaf thinks that I and the American people are so dumb that he can continue to tell the same lies over and over and we will believe him.

  5. Neo says:

    I’m willing to believe that perhaps the President didn’t know the details, but to have the HHS Secretary go to Capitol Hill with the same “Sargent Schultz defense” is beyond reason. “Whatever” .. OMG
    There was no requirement by the text of the ACA to rollout healthcare.gov on Oct 1st, but somebody let it go online anyway, knowing it was “unstable.” They did the POTUS no favor.
    Then Obama goes into this defense blaming Mitt Romney and the Heritage Foundation for coming up with the idea. Mitt Romney did it a hell of a lot better than Team Obama. It looked like the boy who claimed his dog ate his homework, not the “Leader of the Free World.”
    The defense of the “if you like your insurance, you can keep it” is just an exercise in pure unadulterated prevarication. Blaming the insurance companies. Cue the boy who claimed his dog ate his homework, yet again.
    The childish reaction of this President is beyond partisan politics. The man is a failure, a lair, incompetent, and shows no ability to actually lead. Perhaps Democrats will feel the shame that Republicans felt with Richard Nixon back in 1973, but I doubt it.

  6. oneal lane says:

    The fact that it failed its roll out is ultimately meaningless. Obama care is the law of the land and unfortunately it’s not going away.

  7. […] than that, as I noted in my previous post, HHS/CMS decided that this huge, complex effort would best be served by skimping on the required […]

  8. […] to even come close. I have posted many times on the evidence of their ignorance (see here, here, here). Some of the more interesting bits of evidence that have come out since the failed launch of […]

  9. […] and enrollment data inside ObamaCare, and then maps to insurance plan data, etc. Worse than that, as I noted in my previous post, HHS/CMS decided that this huge, complex effort would best be served by skimping on the required […]