The Verification Cross-reference Matrix (VCRM)

Background

After defining and organizing requirements, managers need to track compliance.  This post describes a method used in engineering, but the method helps with any kind of project. The Verification Cross-reference Matrix (VCRM) addresses, or puts you on the path to addressing, many needs.

1. Requirements, as documented through surveys, interviews, analyses, contracts, Statements of Work, standards, or regulations often mix requirements with preferences and background information. Before you begin work, you need to isolate the essential information to gain approval of mutual understanding between customers and the planners or designers.
2. To determine or reflect the steps of the service or the components of a product (for example, in a Work Breakdown Structure), you need to organize requirements.
3. Before starting work, you need the team's agreement that the requirements make sense. For example, are they SMART -- Specific, Measurable, Achievable, Realistic, and Timely? (There are many variations on SMART, and it is only the beginning of the types of tests one can apply to requirements. For example, is the requirement written in active voice? Does each sentence have exactly one requirement? Is each requirement verifiable?)
4. The planners or designers need the requirements in a format that allows them to check off each requirement as they put it into effect in the plan, solution architecture, or detailed design.
5. Risk analysts need a list of requirements for early what-if analysis.
6. Test engineers need to identify parameters that will require measurement. They will need to add requirements to build measuring points and measuring devices into the system, or they will need to obtain test or monitoring equipment.
7. Verification requires planning the conduct of testing so that it occurs in a logical order and in coordination with other processes such as project phases, start-up, or burn-in.
8. Validation and customer acceptance require relating the results of verification back to the original requirements as evidence of compliance with the terms of the contract and fulfillment of the customer's needs.

This post provides a summary of traditional methods that support the above needs.  Each topic above could lead to another post, or even books. When exploring this area, one should consider another tangential topic, requirements traceability. Traceability allows creating threads showing linkage of the details to the general requirements. This allows management to

• identify every detail that may be affected when a more general requirement changes
• prevent creation of details that the general requirements do not authorize
• ensure that the details fulfill every general requirement
• re-use portions of the product's or service's organization in future projects for cost and schedule estimating and for planning or design.

The VCRM

A VCRM lists the requirements of a specification and identifies the method(s) for verifying them. The details included in a VCRM vary with the customer, the phase of the contract, the nature of the contract, and the relationships within the contract.

Simple VCRMs such as the one used by NASA in figure 1 list requirements and the appropriate quality control methods for each requirement. This would more properly be called a “Requirements Verification Matrix.” Detailed VCRMs may include considerably more, such as test ownership, verification requirements, and verification results.

Figure 1. Simple VCRM from a NASA specification. (Click for larger version.)

The QC methods normally include Inspection, Analysis, Demonstration, and Test (IADT).

• Inspection includes qualitative observation. "The fruit basket shall include three Granny Smith apples." (You don't need to run DNA tests to verify that.)
• Analysis includes computation or comparison to historical or experimental data.
• Computation: One might have to calculate the power of munitions, since using them destroys them.
• Comparison/Similarity: Since Project A used Widgets, Project B can re-use the historical data to show that the Widgets meet its requirements.
• Modeling and simulation: We have all seen photos and videos of smoke streaming over the wings of model airplanes or cars in wind tunnels. Did you know that scientists can run computerized simulations of galaxies colliding? They get the results a lot quicker than when they wait around billions of years to see what happens.
• Demonstration verifies performance of a function that does not require qualitative measurement.
• Test verifies that a function executes within specified parameters.

Variations

Figure 2 shows an example VCRM from the Department of Transportation that includes the complete text of each requirement, added information about the verifications, and the party responsible for each requirement's verification.

Figure 2. Example VCRM from the DOT. (Click for larger version.)

The DOT calls what NASA used (figure 1), a Requirements Verification Matrix. The simpler table does not really cross-reference the requirements to anything else. The DOT adds part of the Test Plan (columns 4-6) before calling it a VCRM.

The DOT adds Certification of Compliance as a test method. They don’t define it, but it sounds like a type of Inspection where you inspect a certificate rather than inspecting the product. Since the contract has multiple sellers, Certification of Compliance might refer to using certificates provided by an equipment provider. For example, if the Government furnishes its own equipment for the contractor to install, it may already have verified the equipment’s compliance. Other examples would include calibration certificates or certifications by independent testing labs.

Figure 3 shows an example of a procurement specification that allocates verifications to contractual phases and cross-references performance requirements to verification requirements. Figure 3A shows the table and figure 3B shows verification requirements.

Figure 3A. VCRM used by U.S. Army Corps of Engineers. (Click for larger version.)

Figure 3B. Verification requirements used by U.S. Army Corps of Engineers. (Click for larger version.)

Figure 4 includes a “VCRM” that documents not only the verification plan, but also the data that determines compliance. This goes beyond calling it a “VCRM.” It should, perhaps, be called a Verification Report.

Figure 4. Verification requirements used by U.S. Army Corps of Engineers. (Click for larger version.)

Forcing design and requirements engineers to identify verification methods protects both Buyer and Seller by ensuring that the requirements are verifiable. By definition, one cannot verify an unverifiable requirement. Without objective verification criteria, a Seller can falsely claim to have fulfilled the contract or a Buyer can claim the Seller did not fulfill the contract. Including VCRMs in specifications allows general agreement about requirements verification methods and prevents problems during product acceptance and at contract closure.

In the world of paper requirements documents, VCRMs occupy an appendix or the beginning of the Quality Assurance section. Each document contains its own VCRM. Simple tables contain the data. However, Requirements and Test engineering should control VCRMs centrally. The reason requires taking a tangent into requirements control.

Tools

Many projects now used a database such as DOORS to control requirements. Databases' storage tends to isolate a sentence from its context, forcing statement of each requirement explicitly and independently. The repetition is a pain, but it ensures design quality by forcing exhaustive identification of requirements and by reducing or identifying ambiguity in the scope.

At one company, a project manager employed a distributed requirements database. He allocated requirements using spreadsheets and distributed them to the various design teams. He reassembled the spreadsheets later.

During the project, multiple teams took ownership of some requirements, while nobody took ownership of others, especially when one team would try to transfer ownership of a requirement to another team. Project management had to play catch-up with ownership changes and the teams had to take corrective actions to address dropped requirements.

The lack of centralized control also resulted in the loss of rationales and lessons learned. A robust requirements database would have retained such information. Even a requirement labeled "not a requirement" would have been retained in previous baselines.

Instead, teams kept records at inconsistent levels of detail and often discarded records after deeming them irrelevant to their own efforts.

A requirements database allows you to trace up and down the layers requirements as they are decomposed. One can use Excel spreadsheets for a simple set of requirements. A larger operation needs tools such as DOORS, RTM, or a database developed in house.

Such tools allow one to verify product scope by ensuring that lower-level specifications satisfy each top-level requirement. The help prevent requirements creep by verifying the contractual authority of each bottom-level requirement. This is called traceability.

One can also take the requirements database to the next level by adding (or linking to it) a Verification layer. The Verification layer identifies the Test plan that verifies each requirement. A robust database could even contain the test documents and results.

Case Study

A friend, Jim Pannunzio, once worked on a contract for a first-of-its-kind product. Since many requirements had high risk of being unachievable, the contract provided incentive fees for accomplishing the high-risk requirements.

The company had not used the right tools, so they had not traced some successful tests to the requirements and had failed to bill the customer. This cost them money. When Jim cross-referenced the verifications to the requirements, he found a number of such requirements. Jim also found requirements that the company had failed to verify. This allowed him to suggest further verification.

By identifying untested requirements and untraced successful verifications, Jim’s attention to detail brought hundreds of thousands of dollars of added revenue to his company and turned the project from an economic failure into a success.

Conclusion

Tight integration between requirements and verification provided by a VCRM forms a vital link between design and thorough quality control, between contractual provisions and acceptance, and between the buyer’s and seller’s bank accounts.

For more information see:

Scukanec, Stephen. A Day in the Life of a Verification Requirement. Northrop Grumman. 28 January 2008. http://sstc-online.org/2009/pdfs/SJS2408.pdf
or
Scukanec, Stephen, and James van Gaasbeek. A Day in the Life of a Verification Requirement. Northrop Grumman. 24 October 2007. http://www.dtic.mil/ndia/2007systems/Wednesday/AM/Track1/5536_1004_1042.pdf

Sources:

Figure 1: Goddard Space Flight Center, Greenbelt, Maryland. National Polar-Orbiting Operational Environmental Satellite System (NPOESS) Preparatory Project (NPP), Science Data Segment Requirements Specification, GSFC 429-05-11-01. April 7, 2005. http://nppwww.gsfc.nasa.gov/PEATE/NPP_SDS_Lev3Req.doc. Downloaded 1 June 2011.

Figure 2: Office of Operations, Federal Highway Administration, Department of Transportation. Testing Programs for Transportation Management Systems: A Technical Handbook. Appendix A, Example Verification Cross Reference Matrix. June 20, 2007. http://ops.fhwa.dot.gov/publications/tptms/handbook/app_a.htm. Downloaded 1 June 2011.

Figure 3: Engineer Research And Development Center, Corps Of Engineers, U.S. Army Topographic Engineering Center, Alexandria, VA. Performance Specification: Improved
Position and Azimuth Determining System (IPADS), MIL-PRF-52955C. 8 November 2002. https://aais.ria.army.mil/AAIS/Award_web_03/DAAE2003D01500000/DAAE2002R0177/attach_exhib/spec_attachment_2.pdf. Downloaded 1 June 2011

Figure 4: Baun, Rich. GLAST Large Area Telescope: LAT Pre-Shipment Review: LAT Level Test Verification Process. Gamma-ray Large Area Space Telescope. 27 April 2006. http://www.slac.stanford.edu/exp/glast/flight/rdm/livepreship/04_Test_Verification_Process.pdf. Downloaded 1 June 2011.

Copyright 2011, 2013, Richard Wheeler -- Permission granted for non-profit or personal use with a link to this post.

IT Metrics and Productivity Institute (ITMPI) Premium membership gives members free access to 400 PDU-accredited webinar recordings and waives the PDU processing fees. The library is growing at about 100 webinars per year. Check it out: http://mbsy.co/dPHm?s=e