Another GAO Look at Navy ERP

Back on Nov 7th, I posted a link to a GAO report on the Navy’s ERP implementation of SAP.

The biggest takeaways from that report were:

1. There was nothing either right or wrong with SAP. There’s no evidence in the report to suggest their experience would have been any better or any worse had they had used Oracle or some other ERP product. That means that the lessons are equally valuable to us regardless of the tool we use.
2. Process discipline, particularly requirements management, is critically important.
3. Using the same tool in four separate pilot implementations did not lead to common processes nor interoperability across the four pilots. All four pilot implementations used SAP, but they were configured very differently. Common tools won’t guarantee common processes for the Coast Guard either.

As soon as I wrote that post, someone alerted me a newer report.  Today I’ll share a link to the report and a few thoughts on its relevance to us.

You can find the report here.  It’s from September 2008 and entitled “DOD BUSINESS SYSTEMS MODERNIZATION: Important Management Controls Being Implemented on Major Navy Program, but Improvements Needed in Key Areas”

The Navy’s ERP Program is orders of magnitude larger than CG-LIMS.  Still, of the three weaknesses listed on the first “Highlights” page, the third is definitely relevant to us:

A defined process for proactively avoiding problems, referred to as risk management, has been established, but risk mitigation strategies have not been effectively implemented for all significant risks, such as those associated with data conversion and organizational change management, as well the risks associated with the above-cited weaknesses.

Data conversion from our myriad legacy systems will present the same challenges for us as it did for the Navy.  The many levels of organization change management must be managed.  We’re developing a system amidst the most profound organization change in the Coast Guard since “Streamlining” over a decade ago.  We’ll be deploying it into an environment where people are likely to be weary of change, since they will have just finished changing all their business processes to align to a new business model using existing tools.

There are a few more bits below I thought are relevant for us:

From page 4:

With respect to the cost estimate, our analysis showed that it was not derived using all key estimating practices contained in relevant guidance. For example, the estimate was not grounded in a historical record of comparable data from similar programs and was not based on a reliable schedule baseline, which are both necessary to having a cost estimate that can be considered credible and accurate. These practices were not employed for various reasons, including DOD’s lack of historical data from similar programs and the lack of an integrated master schedule for the program’s first increment that includes all three releases.

DoD is struggling to develop and accurate cost model based on historical cost data for ERP implementations.  CG-LIMS as currently envisioned is going to involve multiple COTS tools, which will make cost estimaton more challenging and more uncertain.

From Page 12, in explaining schedule slip of 22 months from original baseline and 4 months from the revised baseline:

According to program documentation, these delays were due, in part, to challenges experienced at NAVAIR in converting data from legacy systems to run on the new system and implementing new business procedures associated with the system.

The challenge for us will to ensure we build a reasonable schedule.  We can mitigate the risks by building small chunks and starting with the community that has the best data.  Even so, the business processes will change for everyone and must be managed.

From page 23:

To its credit, the program office has followed DOD’s BEA compliance guidance. However, this guidance does not adequately provide for addressing all relevant aspects of BEA compliance. Moreover, DON’s enterprise architecture, which is a major component of DOD’s federated BEA, as well as key aspects of DOD’s corporate BEA, have yet to be sufficiently defined to permit thorough compliance determinations. In addition, current policies and guidance do not require DON investments to comply with its enterprise architecture. This means that the department does not have a sufficient basis for knowing if Navy ERP has been defined to minimize overlap with and duplication of other programs’ functionality and maximize interoperability among related programs.

We have every intent to comply with the CG and DHS’s EA, but we’re going to have the same struggle to identify standards that don’t exist or are still eveolving.

There’s some good stuff there on cost estimation.  A footnote on page 30 includes a link to a draft GAO Cost Assessment Guide that I’ve shared with our BFM to use in putting together the program LCCE.

The Navy ERP Program got kudos from GAO for its requirements management.  From Page 48:

The program office is effectively implementing requirements traceability for its 1,733 Release 1.0 system requirements. To verify this traceability, we randomly selected and analyzed 60 of the 1,733 system requirements and confirmed that 58 of the 60 were traceable both backward to higher level requirements and forward to design specifications and test results. The remaining 2 had been allocated to the other releases, and thus we also confirmed the program’s ability to maintain traceability between product releases. In doing so, the program utilized a tool called DOORS, which if implemented properly, allows each requirement to be linked from its most conceptual definition to its most detailed definition, as well as to design specifications and test cases. In effect, the tool maintains the linkages among requirement documents, design documents, and test cases even if requirements change.

We’re laying the groundwork to do exactly the same thing using DOORS and SA.

Bottom line — we can strive to manage requirements as well as them, and must plan for the challenges of data conversion and business process change management.

For any of you who read the report, if you think there are other lessons in there for us, please add a comment by clicking on the link at the top of this post.