Highly efficient evaluation design (HEED) for comparing algorithms used to detect nuclear materials

Radiological or nuclear materials are detected in vehicles by processing data from sensor systems. The key goals/objectives are to detect threats and control the number of "nuisance" alarms caused by non-threat materials. It is important to evaluate algorithms efficiently and fairly. Experiments can be done in silico or using real loads of cargo containing hidden radiation sources which is time-consuming and costly. In either mode, sensor data serve as inputs to the algorithms, whose outputs can be compared to ground truth. The goal is not to map the complete characteristics of an algorithm, but to compare algorithms. Therefore it is most efficient to concentrate on experimental configurations that reveal meaningful differences between the algorithms. The methods of Combinatorial Experimental Design are used to generate configurations that will find all situations in which one or two levels of key parameters reveal such a difference. In practice, an efficient set must be further reviewed by subject matter experts, who are tasked to specify configurations that would reveal meaningful differences among the algorithms. Experts may also assign importance values to the remaining configurations, based on other considerations related to the likelihood or consequences of the corresponding threat.