SA Health takes disaster recovery gamble to save money

South Australia's Health department has decided not to implement a secondary site for disaster recovery with its new state-wide pathology system in an effort to save costs from a project that is running late and over budget.

In late 2015, the state's audit office looked into SA Health's enterprise pathology laboratory information system (EPLIS) overhaul and warned it was likely to run out of money before it was complete.

The project was funded from a 2012, $30.4 million investment to replace two ageing and incompatible Cirdan Ultra instances and as many as 30 smaller pathology systems across in labs and hospitals across the state.

SA Health in 2014 signed a contract with Cerner to introduce its Millennium suite of pathology lab information modules, signalling the start of the transformation.

But just one year later the audit office found that the agency had bungled a couple of key components and was at serious risk of running out of money, with no contingency funding to dip into.

One of those key components included an assumption that the solution could run out of a single high availability data centre, according to the auditors.

After getting further advice, the department later decided it should look into how much a secondary data centre would cost - settling on $5 million - and undertook discussions with Cerner to identify a suitable disaster recovery option, the audit office said at the time.

But it today revealed that SA Health has since decided not to go with a secondary DR site in order to reduce pressure on its budget.

It was already forced last year to go to cabinet with hands outstretched for $2 million more so it could pay Cerner for certain services that weren't included in its original contract.

The agency told the audit office that it had considered the impacts of not having a secondary site for DR and "determined that the risks were acceptable given the unlikely event of a catastrophic failure".

SA Health said it was willing to wear any downtime to avoid having to pay an extra $2.5 million upfront to establish the site and then $596,000 annually to run it.

The audit office warned the agency that while the likelihood of an entire data centre going down was low, should any outages occur the damage would run into the millions of dollars, not to mention potential detriment to patient care and safety.

SA Health argued the Millenium solution had been designed as high-availability in a single data centre, with 15 minute recovery point objective and a maximum six hours of tolerable disruption.

A specially designed backup solution frequently backs up all database logs to disk, which are then replicated to tape daily - making a secondary DR site "not justifiable", SA Health said.

The agency promised the audit office it would review any potential impacts from a failure in its data centre and develop "appropriate business continuity plans" by September.

Problems remain

The disaster recovery issue is just one of many to be pointed out by the auditor looking into the project.

The audit office said SA Health still did not properly understand the complex system functionality and data requirements needed to effectively build and configure the solution, and had underestimated the effort required for certain activities in the project.

It also warned that support for the existing Ultra database may expire before it is decommissioned (it won't be supported beyond this coming December).

SA Health said it was in discussions with vendor Cirdan to continue using the platform into next year.  

So far the Millenium pathology system has only been implemented at the Women's and Children's Hospital.

SA Health had originally intended to roll out the platform across the state by May 2017, with all metro sites completed by October 2016.

But delays with planning, design, testing, documentation, data migration, problems with a related billing overhaul, and increases in scope as well as the delayed opening of the Royal Adelaide Hospital have meant multiple rescheduling of timelines.

SA Health told the audit office it won't introduce the new system at any other sites until "critical and high" defects are resolved or it has an acceptable workaround.

It currently has 10 active risks - four high and five medium - as well as 178 system defects, 16 of which are high and 98 considered medium, the agency told the audit office.

It said it expects to fully remediate most issues by December this year. The next implementation, should all things go well, will be at the Royal Adelaide Hospital.

The project is now expected to cost upwards of $40 million, but the department told the audit office it couldn't determine exactly how much more it would need to spend. How any increases will be funded is currently unclear, the audit office said.