12,13 In addition to ROC analysis, Cass and colleagues 13 used two alternative measures of the predictive quality of the screening test: the Euclidean distance of the ROC curve to the upper left corner of the ROC graph, and the hit rate when the false-positive rate was held at a “reasonably low” level of 2.5%. 11 The same method was used to evaluate the effects of selecting screening test items on the basis of item-total correlations or item-total regression. Receiver operator characteristic (ROC) analysis was used to evaluate the effects of screening test length on the predictive qualities of the screening test. In recent studies, several aspects of sequential testing in SP-based tests have been investigated. Summarizing, the final outcome is “passed” when either the screening test or the complete test result is passed, and the final outcome is “failed” only when the complete test was failed. It seems reasonable that all students who did not pass the screening test are offered the opportunity to show their ability in the complete test. The latter group of examinees continues to take the second test to obtain the information needed for a reliable pass/fail decision. The screening test is used to take a first decision between a clear pass or a clear fail and a borderline result. 10,11,12,13 In a sequential approach, the complete test is divided into two parts: a screening test and a second test. 3 Then the use of a sequential testing procedure may help to reduce the costs. 3,9 If most examinees perform well relative to the pass/fail cutoff score, short tests can still yield reproducible pass/fail decisions, in particular for examinees at upper ability levels. 1,2,3,4,5,6,7,8 The reproducibility of the test scores is related to test length. As an alternative to receiver operator characteristic analysis, minimization of the loss function was found to be an appropriate method to determine the optimum cutoff value of the screening test.Įxaminations that use standardized patients (SP-based tests) are adequate in many respects, but the related costs can be a serious drawback, particularly since SP-based tests require a large sample of stations leading to a considerable test length. This can considerably reduce testing time (30% to 40%), while keeping the percentage of false positives at an acceptably low level of less than 0.2%. In a sequential SP-based test, the pass/fail cutoff score of the screening test should be stringent. Using those cutoff values, only 26% and 11% of the students would have had to take the complete test to get a “true” score, while only 0.2% and 0.0% of the students who passed the screening test went on to fail the complete test (false positives). Accordingly, stringent pass/fail cutoff scores in the screening test (75% for Group 1 and 80% for Group 2) produced optimum results. The negative predictive value (probability that a student would fail the complete test if he or she had failed the screening test) was low (96%). They developed and evaluated a new method to determine the optimum cutoff score of the screening test, a method based on minimization of the loss represented by the (weighted) numbers of false positives and negatives in the screening test. They investigated efficiency and validity as a function of the cutoff score of the screening test. In a simulation experiment based on the data from those tests, the authors considered the first day as the screening test and the second day as the second test. In 1994–95, first- through fourth-year (Group 1) and sixth-year (Group 2) medical students at the University of Maastricht took SP-based tests. This study investigated whether sequential testing increases efficiency with only a minor decrease of validity. In this approach, students take a short screening test only those who fail take a second test. Educators who use standardized-patient-based (SP-based) tests may save resources by using sequential testing.
0 kommentar(er)
