Knowledge assessment: A comparison between human experts and computerized procedures
We compared performance of human experts and a simple computer algorithm in a knowledge assessment task. Six experienced mathematics tutors were asked to predict the responses of students in a standard high school mathematics test. Experts first selected some of the items in the test one by one, for which they were shown the student's responses. Their predictions on the rest of the items were based on this information. The computer algorithm performed the same task on the same items and students.
Performance for both experts and the computer increased as they received information about additional items. Experts predicted between 59% and 77% of the items correctly. The computer algorithm predicted between 70% and 80% correctly, and was never worse than any of the experts. A detailed comparison showed that the computer algorithm's performance is higher both in the selection and the prediction task. Giving the experts more detailed information about the student's response, in the form of the student's answer sheet and scrap paper, did not change the computer algorithm's superiority.
The second issue we investigated was the possibility of obtaining the structure of the domain under study via systematic questioning of teachers-experts. The feasibility of a computerized querying technique was demonstrated through an application on fifty items taken from the standard high school mathematics curriculum. This procedure was employed to question five expert-teachers and, from their responses, construct the structure of the domain. The results show that this technique is applicable in a realistic setting; the reduction of the number of questions put to the experts on the relations between the fifty items was remarkable. However, the data indicate that, despite a good agreement across experts concerning item difficulty and other indirect measures, the resulting structures differ to a further extent than expected.
0288: Educational evaluation
0710: Educational software