Putting People First

As part of the evaluation process that was done for three COTS products, we brought in our usability experts from TRACEN Yorktown’s Performance Technology Center.  CDR Ruckstuhl, the CG-LIMS rep from CG-1B3,  put together a study plan that made the most of the limited time we had hands-on with the products. That plan is in ePMO and Inteldocs.  We’ll share the results as soon as their available.

To do this “right,” one could argue that we would need dozens of trained people of all ranks using fully configured systems with real data.  That wouldn’t be feasible.  We would literally have to go through the whole configuration, implementation, and training process for all three products before we could do such a test.

I was absolutely delighted with the study plan my friend put together. It reflected the “Discount Usability Engineering” approach favored by Jakob Nielsen, whose work had a significant influence on me when I was leading a project that was largely a custom software development effort.  If you’re a software professional and don’t have his book Usability Engineering on your shelf, but it! If you’ve never seen his list of “Ten Usability Heuristics” go look at them now. I like the book so much the first time I read it, that it was the subject of four posts on my ALMIS project site.  You can find those four posts here.

The usability assessment is still in routing and should arrive this week, but I wanted to get this post out as a placeholder and to whet your appetite.  Assessing the usability of COTS tools up front as part of the selection process isn’t something we’ve done well in the past.  We did it the best we could for CG-LIMS, and learned some lessons that’ll lead us to do it even better way down the road when we go through the process again.

Continuing the Analysis

The last post shared the plan that OSC used to select COTS EAM tools for the first increment of CG-LIMS.  Now we’ll go to the next step.  You guessed it, this post will share the results of the second phase of the analysis.  The whole report is in ePMO or Inteldocs.

I read the report.  Here is what I told the team after I read it:

I just completed review of the initial report you passed.  With the clarification your provided earlier to Mr. Sylvester, it’s clear to me that your team and FGM’s are following the process that had been established and are working toward a defensible solution.

Concise, clear analysis.  Thanks for that.
Regards,
Dan

I stand by that assessment.  It provided a defensible process to choose a small number of products to evaluate further and wring out in the lab at OSC.

I’ve seen some describe the analysis as nothing more than what you get from Gartner.  I absolutely disagree with that assessment.

At tomorrow’s Quarterly Program Review, OSC’s will share some of the results of this report in PowerPoint slides that walk through the results.  For now, I just wanted everyone to know where to find the report.

Selecting COTS EAM Tools for CG-LIMS

In their role as the CG-LIMS System Development Agent, OSC conducted an analysis that was intended to determine what COTS tool or tools would be used for the first increment of CG-LIMS.  OSC’s System Engineering and Technical Services contractor partnered with another firm to conduct the analysis.

Throughout the process, the Sponsor’s Representative, Asset Manager, OSC Project Officer, and me as the project manager were involved.

The proces was conducted in three phases.  The report of the first phase is in ePMO and Inteldocs.

The Introduction spells out the work they did.

The purpose of the Phase 1 analysis effort is to capture the set of criteria against which candidate EAM products will be evaluated and to establish an objective methodology against which those products can be ranked and quantitatively compared. Appendix A lists the EAM products to be evaluated while Appendix B enumerates the evaluation criteria in a checklist format. Section 2 lists the resources utilized to derive the evaluation criteria, which draws heavily upon already established criteria developed by the USCG and its supporting contractors prior to this analysis effort. Due to the compressed schedule and limited resources available to carry out the evaluation, section 3 describes a staged analysis approach designed to eliminate unsuitable EAM candidates as quickly as possible so that resources are expended for a deep analysis only on the most promising EAM candidates. Section 4 groups the evaluation criteria into a collection of categories that capture different perspectives through which to view candidate EAM products, thus providing the USCG with the ability to establish priorities (cost, vendor maturity, functional coverage, etc.) that encompass all appropriate business, technical, functional, and lifecycle aspects of the product evaluation. The weighting factors for ranking the relative importance of the evaluation categories are presented in section 5. Section 6 combines the evaluation categories and their relative ranking of importance into a mathematical methodology that provides an objective way to rank and compare the candidate EAM products.

The Evaluation Approach represented what I believed to be a sound initial strategy to meet our time constraints:

The criteria listed in Appendix B and the methodology presented in Section 6 will be used to rank each of the candidate EAM products in Appendix A. However, to accelerate the evaluation process, not all of the EAM products will be evaluated against the full set of criteria presented in Appendix B. Instead, a multi-stage analysis approach will be used that rapidly “filters” the candidate EAM products into a smaller, more manageable subset. The analysis stages are designed to objectively eliminate unsuitable products as quickly as possible from consideration before resources are needlessly applied to perform deeper analysis activities, such as evaluating architectural conformance or performing a detailed usability analysis.

The analysis stages are:

•    Stage 1: Initial Evaluation – each of the candidate EAM products will be evaluated against critical pass/fail criteria. Those products that fail to meet the minimal criteria will be eliminated from further consideration.

•    Stage 2: Functional Coverage – candidate EAM products that meet the critical pass/fail criteria will be evaluated against the functional capabilities identified as required for a successful USCG Increment 1 release, which primarily means that the selected EAM product must be deployable with no (or acceptably minimal) loss of functionality from that which is currently available in the deployed legacy systems. The methodology described in section 6 will be used to rank the candidate EAM products, and then the top 50% products will proceed to the final analysis stage.

•    Stage 3: Detailed Evaluation – candidate EAM products that successfully passed the preceding stages will be evaluated against the full set of criteria in Appendix B and then ranked by the methods described in section 6.

As each of the analysis stages are executed, the criteria in Appendix B may be further refined to incorporate lessons learned, or to clarify and more accurately state individual criterion.

CG-LIMS in the C4&IT Strategic Plan

Back in August, the CIO announced the publication of the Command, Control, Communication, Computers and Information Technology (C4&IT) Strategic Plan for FY 2008 to 2012.  He made a blog post announcing it on August 26th that you can find here.

I read it and added it to my list of things I wanted to share with the team in this forum.  It’s only 60 pages, but I rather than just tell you to read the whole thing, I want to make sure you know where to find it, (here) and I want to let you know what it says about CG-LIMS.

Appendix A is the CG-6 Performance Plan, where it lists initiatives tied to specific goals.

Under Goal 2: Technology, within the section 2.2 of “Enterprise-wide” initiatives, we find CG-LIMS.  Here’s what it says:

Logistics System Modernization

Continue to develop the Coast Guard Logistics Information Management System (CG-LIMS) to transform the Coast Guard’s logistics systems in support of a Coast Guard-wide common logistics business model. (Primary POC: CG-63)

Major Milestones
FY08 Q4: Complete all Logistics Information Management System (LIMS)/CG-LIMS transition activities

Key Performance Indicators

• Knowledge transfer from Integrated Coast Guard Systems (invaluable in assessing LIMS/eMESA future usability)
• Cost effective solution supporting all Preliminary Operational Requirements Document (PORD) requirements and common business model
• Successful achievement of exit criteria for next acquisition phase
• Proven, stable, and secure enterprise architecture design
• Achievement of project plan objectives
• Timely and effective roll out
• Delivery of increments on or ahead of schedule for logistics community

We completed the transition on time in 4QFY08, and had been well on the way to developing a cost effective solution to meet PORD requirements.

There other goals that relate to CG-LIMS that make it worth skimming the entire document.

There’s a lot of good stuff in there, and I wanted to make sure you knew where to find it.  If you come across something you want to make sure the team reads, feel free to paste it in as a comment to this post, and it’ll show up at the top of the list of comments in the rightmost column.

Since this has been such a short read for you, I’ll past a few more excerpts I want to make sure you read.  The description of Goal 2 is:

Deliver mission-focused, interoperable, innovative, and net-centric C4&IT using enterprise-wide solutions, an optimized infrastructure, and electromagnetic spectrum efficiency.

And the Intent of Goal 2 is (emphasis added):

Coast Guard missions are increasingly dependent on the quality of our technology. Operators and support staff use C4&IT solutions throughout the Coast Guard to safeguard our oceans and waterways, enforce maritime laws, and serve our Nation. Interoperable, net-centric solutions provide our operators seamless communication with internal and external partners such as the DoD and its components; state, local, and tribal governments; and intelligence agencies. In addition, during times of war, our ability to transition from governmental responsibilities to defensive capabilities requires optimized and innovative C4&IT resources. To satisfy mission demands and operator needs, we must deliver mission-focused, interoperable, and net-centric C4&IT using enterprise-wide solutions, an optimized infrastructure, and electromagnetic spectrum efficiency.

An Amazingly Temporary TEMP

Last post I told you about a PMP that was in the approval cycle that needs to be re-written to reflect the new approach.  I’d planned to also share with you the Test and Evaluation Master Plan, or TEMP, which I thought by now would be approved, but now needs to be re-written.

This was another planning artifact that we felt good about.  It reflected the approach we were taking all along with the project: leverage OSC’s processes and use the HQ project staff to create the thin layer of the bare minimum additional stuff or translation needed to comply with the MSAM.  We didn’t want to bog the team down in duplicating effort.  We wanted to satisfy both the MSAM and SDLC with a minimum of duplication. We tried to keep the TEMP simple so OSC could focus on following or improving their processes.

I think we achieved that with the TEMP.  We showed how we were going to use OSC’s processes, or slight modifications to them, the answer the requirements of the MSAM.

Depending on the new Acquisition Strategy, much of this approach may change.  The timing will absolutely change.  Depending on how many more layers of integration and contractor program management we have, there may be many more levels of integration testing and acceptance testing.  Our approach only worked because the SDA was a CG entity, and had responsibility for CG-LIMS, for the SOA, for the ESB, and for the legacy systems.

At this point,  I don’t even want to think about it how complicated the TEMP may become with a different Acquisition Strategy.  I just want to share with everyone where we were when we stopped.

The draft TEMP is in ePMO or Inteldocs.