Comparison of Types of Biometric Testing

There are three commonly recognized categories of biometric tests: Technology Tests, Scenario Tests and Operational Tests.

Technology tests are performed in a laboratory environment and generally focus on product software, rather than the product itself. They are designed to validate the underlying theory of biometric identification methods, and to identify areas of improvement for future research and design. Technology testing is the most common form of biometric testing and is normally used to produce a repeatable and scalable assessment of an algorithm/sensor using offline data processing, producing FRR, FAR, & EER results that can be illustrated readily in graphics such as the DET curve.

Technology tests can consist of, but are not limited to:

Testing is conducted using offline processing of data. This data will be representative of the data the biometric device acquires. However, the data will be independent of the device and never seen before by the developers. Such data can be acquired from large databases. Some key attributes of technology tests are:


The Detection Error Trade-Off (DET) Curve

Receiver-Operator Curve

Scenario Testing is carried out in an environment that models a real-world target application of interest. Each tested system will have its own acquisition sensor and will receive slightly different data. Consequently, if multiple systems are being compared, care will be required that data collection across all tested systems is in the same environment with the same population using the same policies. Test results will be repeatable only to the extent that the modeled scenario and the variables associated with operator group selection can be carefully controlled.

BSI's enhanced scenario test procedures include a volunteer operator group of 500+ individuals available year round to evaluate biometric products. Demographic information can be modeled after the clients required user group to create a more accurate test scenario. In BSI's approach, the statistical method used by the test team to determine the number of volunteer operators required to achieve the most accurate results are those developed by Dr. Michael Schuckers of St. Lawrence University. BSI's enhanced scenario testing employs 200-400 testers who come in six to eight times during the course of a test cycle. The resulting 10,000+ activations shifts the burden of analysis from theoretical computer simulations to thousands of live operator activations.

Operational Testing is an evaluation of a complete biometric system in a real world environment. Problems often arise once a new technology is taken out of the lab and placed in a real world environment. During operational testing, BSI will monitor and document these problems and provide expert recommendations. This test will determine if the overall performance of the system meets the requirements of the client.

To better serve the client, BSI will act as the program manager and oversee all installation and configuration of the biometric equipment in the operational environment. In addition, the team will monitor and document every aspect of the biometric system, the environment, and generate a report that includes expert recommendations.

Practical Experience

In July 2004, BSI was the site for the U.S. Department of Homeland Security (DHS) e-passport interoperability testing session. Tests were conducted under the auspices of the DHS US-VISIT program to evaluate the interoperability of e-passports and passport readers. Over 50 companies and 18 countries were represented. NBSP's labs at BSI are equipped with the appropriate tools for testing biometric systems and products, and evaluating the results of such testing under published protocols that are fair, reasonable, and predictable with regard to the practices employed. The testing program is supported by a reference library housing all relevant technical and operational data related to the technologies and products under test.