Small-sample inference for non-inferiority in binomial experiments

We compare the exact sizes and powers of various procedures for testing hypotheses concerning differences of sample proportions. We eliminate the effect of the remaining parameter following the method of Berger and Boos,1 who suggest constructing a confidence interval for the nuisance parameter, calcu- lating the supremum of the p-value as the nuisance parameter varies over this interval, and reporting as the p-value for the test this supremum, plus the com- plement of the coverage probability for the interval. Our method forces the size, i.e. the power at the null hypothesis, to be strictly controlled, compared to the standard z-test, which is anti-conservative. We also found that we can make modest improvements in power by optimizing over the coverage probability of the nuisance parameter confidence interval.