Automated Content Analysis of Multiple-Choice Test Item Banks

Authors: Ford, John M., Thomas A. Stetz, Marilyn M. Bott, and Brian S. O’Leary (US Office of Personel Management)

Description: Test item review is a specialized type of content analysis conducted to identify and correct test item flaws early in the test development process. Test item reviewers not only examine the targeted content of a test but also remove inappropriate content and balance various types of incidental content.Anautomated content analysis implementation of Hiller’s verbal ambiguity scales and Laffal’s General Concept Dictionary of English was used to examine 576 multiple-choice test items before and after test item reviewand revision by experienced item editors. Hiller’s scales detected some problems with item clarity. Laffal’s categories detected content imbalance between test forms but not inappropriate item
content.

Reference: Ford, J.M., Stetz, T.A., Bott, M.M. & B.S. O’Leary. Automated content analysis of multiple-choice test item banks. Social Science & Computer Review, 18, 258-271.