Knowledge assessment: A comparison between human experts and computerized procedures

1991 1991

Other formats: Order a copy

Abstract (summary)

We compared performance of human experts and a simple computer algorithm in a knowledge assessment task. Six experienced mathematics tutors were asked to predict the responses of students in a standard high school mathematics test. Experts first selected some of the items in the test one by one, for which they were shown the student's responses. Their predictions on the rest of the items were based on this information. The computer algorithm performed the same task on the same items and students.

Performance for both experts and the computer increased as they received information about additional items. Experts predicted between 59% and 77% of the items correctly. The computer algorithm predicted between 70% and 80% correctly, and was never worse than any of the experts. A detailed comparison showed that the computer algorithm's performance is higher both in the selection and the prediction task. Giving the experts more detailed information about the student's response, in the form of the student's answer sheet and scrap paper, did not change the computer algorithm's superiority.

The second issue we investigated was the possibility of obtaining the structure of the domain under study via systematic questioning of teachers-experts. The feasibility of a computerized querying technique was demonstrated through an application on fifty items taken from the standard high school mathematics curriculum. This procedure was employed to question five expert-teachers and, from their responses, construct the structure of the domain. The results show that this technique is applicable in a realistic setting; the reduction of the number of questions put to the experts on the relations between the fifty items was remarkable. However, the data indicate that, despite a good agreement across experts concerning item difficulty and other indirect measures, the resulting structures differ to a further extent than expected.

Indexing (details)

Educational evaluation;
Educational software
0623: Psychology
0623: Experiments
0288: Educational evaluation
0710: Educational software
Identifier / keyword
Education; Psychology
Knowledge assessment: A comparison between human experts and computerized procedures
Kambouri, Maria
Number of pages
Publication year
Degree date
School code
DAI-B 52/12, Dissertation Abstracts International
Place of publication
Ann Arbor
Country of publication
United States
Falmagne, Jean-Claude; Landy, Michael S.
New York University
University location
United States -- New York
Source type
Dissertations & Theses
Document type
Dissertation/thesis number
ProQuest document ID
Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works.
Document URL
Access the complete full text

You can get the full text of this document if it is part of your institution's ProQuest subscription.

Try one of the following:

  • Connect to ProQuest through your library network and search for the document from there.
  • Request the document from your library.
  • Go to the ProQuest login page and enter a ProQuest or My Research username / password.