The effect of removing examinees with low motivation on item response data calibration

Abstract

Many item-response models (IRM) used to estimate student abilities and test item parameters in large-scale assessments (LSA) do not account for the effect of student low motivation. This effect may pose a threat to the validity of test data interpretations. The first purpose of this study was to evaluate the effect of removing examinees with low motivation on the estimates of examinee abilities and test-item parameters calibrations using an item response theory model. The second purpose was to examine the significance of the relationship between students’ motivation and their mathematics achievement, as measured by the LSA. Student motivation as defined by expectancy-value theory and self-efficacy theory was identified from self-report data using a principal component analysis. Two components scores, mathematics values and interest, were computed for each examinee to create two groups of examinees with high and low motivation. These groups were used to examine the effect of removing examinees with low motivation on the estimates of test item parameters and student abilities between a 3- parameter logistic (3PL) and modified 3PL IRM. The effect of student motivation on their academic achievement was examined using a hierarchical linear model (HLM). The results suggested that the modified 3PL IRM seemed to minimize the effect of low motivation on the estimates of student abilities and test item parameters when compared to the 3PL. The results from the HLM suggested that student mathematics values and interest are significant predictors of students’ academic achievement. The outcome of this study builds on the research work of Swerdzewski, Harmes, and Finneys (2011) and supports the research work of Wise and DeMars (2006); Wolf and Smith (1995) in the use of IRM and measures of student motivation to provide more accurate interpretations of LSA data.