Inference-Making and Reasoning: Refinement of an Assessment for Use in Gateway Biology Courses
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A160335 to the University of Illinois, Urbana-Champaign. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
Setting: Research sites include large state universities in Pennsylvania and in Illinois.
Dr. Jennifer Cromley, PI
For Study 1, we will use existing data collected over 7 years from approximately 1,800 students (primarily freshmen and sophomores) enrolled in introductory biology courses. For Studies 2 and 3, we will collect data from approximately 1,400 students enrolled in similar courses.
The assessment that will be refined and validated focuses on measuring the types of reasoning and inference-making that are critical for a deep understanding of science-related course material and that play a vital role in transfer of learning to new contexts. In particular, the assessment focuses on students' ability to demonstrate applied reasoning with recently presented information. The existing 20-minute-long, multiple-choice assessment presents students with brief passages and questions that require them to reason using biological principals with new material. For example, they read about the immune system, which is not typically covered in the introductory courses, and choose from the possible responses. The incorrect responses reflect common, specific reasoning errors students often make in the introductory biology courses. We have used the existing assessment for our research and have found strong Cronbach's alpha reliability but have not fully validated it using psychometric approaches. Based on initial IRT scaling and DIF analyses, the existing assessment has excellent psychometric properties, so we have collected additional think-aloud protocols on new passages, which we will code and use to develop new items that comprise a parallel form.
Research Design and Methods:
Using secondary and primary data, we will conduct three studies aimed to refine an existing assessment of inference-making and reasoning in biology and ensure its validity and reliability. In Study 1, we have used existing data collected from students in biology courses. We have calibrated items and tested them for differential functioning by students' race, sex, and socio-economic status. In Study 2, we have conducted cognitive interviews with a new sample of students to understand the relation between cognitive processes and item thetas, as well as to compare cognitive processes between students who get items correct versus incorrect. Study 3 has three parts. In part one, we will conduct tests of validity and reliability, item calibration using a multidimensional model, and tests for differential item functioning. In part two, we will use a measurement decision theory (Bayesian) approach to set a cut score that can identify students at high risk of course failure, while controlling for students' ACT Reading scores and High School GPA. In part three, we will determine whether the assessment is effective for documenting student growth over a semester.
In addition to student responses to the assessment, we will collect course grades, scores on a final exam or project (which is included in calculation of grades), students' intention to remain in a STEM major, and demographics (e.g., age, sex, race, parental education/ socio-economic status, major, self-reported ACT Reading score).
Data Analytic Strategy:
We will use multiple analytic strategies including confirmatory factor analysis, item response theory, and differential item functioning to ensure the validity and appropriate use of the full assessment.