| Description |
Student evaluations of teaching (SET) in higher education are ubiquitous. Many SETs utilize Likert-type scales and data from these evaluations are frequently used as data points in the retention, promotion, and tenure reviews of instructors. This makes them a high impact activity. Research has revealed a number of validity issues with Likert scales that cast doubt on the validity of the results produced by them. The current study proposed a new method for evaluating instruction in higher education that utilizes the forced choice method popularized by personality psychology. This method uses confirmatory factor analytic techniques to avoid the ipsative data problem of conventional forced choice scoring. The current study delivered both a Likert scale (SL) and a forced choice (SFC) version of an instructor evaluation instrument to students. Instructors' self-evaluation and department chairs' evaluation of their instructors served as dependent variables. Results showed that the forced choice method had some indication of better discriminant validity, evidenced by lower correlations between three forced choice factors (ranging from .12 to .63), compared to the Likert version (ranging from .95 to .96). Fewer responses than anticipated resulted in a simplified analysis of convergent validity. Neither of the trait estimates generated by the two student instructor evaluation methods correlated with those generated by the instructor self-evaluations. The Student Likert instrument had low to moderate correlations and the Student Forced Choice instrument had low correlations with the department chair evaluations of instructors. The slightly greater evidence of convergence for the Likert compared to the forced choice may be explained, in part, by a method effect, since the department chair instrument also used a Likert scale. Discussion focused on the surprising finding that students, instructors, and department chairs largely disagreed on the quality of instructors, and on how to improve the instrument and study method for more robust results. |
