Do overlapping test strategies enhance reliability, or just add cost?

On the night of July 1, 2002, at 36,000 ft. over southern Germany, Bashkirian Airlines flight 2937 collided in mid-air with DHL flight 611. The result was total fatality. In the ensuing investigation, no single cause was identified. Instead, multiple factors emerged as contributory causes that led to the tragedy. One was that one aircraft followed the Resolution Advisory’s instruction of the in-flight Traffic Collision Avoidance System (TCAS) to descend, in order to avoid collision, while the other followed the conflicting air traffic controller’s instruction to similarly descend. This by itself would not have caused the accident, as there were backup systems. There were other factors such as human error, organizational deficiencies, ambiguities in safety protocols, ongoing maintenance work to the radar system, faulty phone lines and more. Even in safety critical industries like air-travel, with its many redundant systems, accidents happen.

In the aftermath, Prof. James T. Reason conducted an analysis¹ based on his model on cumulative act effect, which is sometimes called the Swiss Cheese Model.² The tool is highly valuable to industries such as automotive and healthcare, where mistakes can lead to death. It is also valuable in high-reliability industries like banking, e-commerce and government agencies that deal with the interest of the masses, conduct financial transactions and hold private information where failure could lead to financial catastrophe.

Prof. Reason put forward a model whereby any system or process would have successive barriers of defenses. But each barrier has its weaknesses (holes). Provided the subsequent barrier does not have similar holes, a potential failure can be averted. However, if the holes somehow align, this presents a trajectory for the failures to occur. This is visualized as layers of Swiss cheese, from which the name is derived (Figure 1). The holes are categorized into two sources: active failures and latent conditions. Active failures are the unsafe acts themselves, whereas latent conditions consist of organizational influences, unsafe supervision and preconditions for failure. Latent conditions foster the failure long before the actual failure occurs (e.g., insufficient pilot training due to cost reduction, improper pairing of two inexperienced pilots, fatigued air crew, etc).

We see a similar model in the electronics manufacturing test industry, where successive test systems (barriers) are put in place to screen PCBA defects on a production line to prevent defective products from reaching end-users. However, we have to note that this EMT model is focused only on test coverage, as the barriers put in place are meant to screen defects, whereas the Swiss Cheese Model is more encompassing, factoring in other conditions. Nevertheless, the principle is the same: If holes in each barrier are aligned, it creates a trajectory for failure to occur.

To gain an idea of how big these holes are in each barrier, we refer to a widely cited study from manufacturing test consultant Stig Oresjo, who presented a Venn diagram on the effectiveness of each test strategy from the results of nine studies (Figure 2).³

The numbers in the diagram represent the number of defects caught by a particular test strategy in the experiment. Whenever a number is shown in an overlapping area of two circles, it means that is the number of defects caught by both test strategies represented by those circles. For example, the number 67 is an overlapping area between AXI and ICT; this means 67 defects were caught by both AXI and ICT. In a similar fashion, the number 103 above means that 103 defects were caught by all test strategies, effectively creating multiple barriers for these defects. The same Venn diagram can be illustrated another way (Figure 3). Here we see each test strategy, with its inherent effectiveness in catching defects, providing multiple defensive barriers to prevent the escape of failures.

With this as a foundation, next month we will discuss the EMT trends regarding overlapping test coverage areas and the influence on warranty claims.

References

1. J. Reason, E. Hollnagel and J. Paries, “Revisiting the Swiss Cheese Model of Accidents,” EEC Technical/Scientific Report No. 2006-017, October 2006. 
2. Swiss Cheese Model, http://en.wikipedia.org/wiki/Swiss_cheese_model.
3. Stig Oresjo, Results from 2007 Industry Defect Level and Test Effectiveness Studies, Agilent, 2007. http://cp.literature.agilent.com/litweb/pdf/5989-8725EN.pdf.

Andrew Tek is product manager, In-circuit Test Group, Agilent Technologies (agilent.com); This e-mail address is being protected from spambots. You need JavaScript enabled to view it .