Sample:

For Study 1, we will use existing data collected over 7 years from approximately 1,800 students (primarily freshmen and sophomores) enrolled in introductory biology courses. For Studies 2 and 3, we will collect data from approximately 1,400 students enrolled in similar courses.

​

Assessment:

The assessment that will be refined and validated focuses on measuring the types of reasoning and inference-making that are critical for a deep understanding of science-related course material and that play a vital role in transfer of learning to new contexts. In particular, the assessment focuses on students' ability to demonstrate applied reasoning with recently presented information. The existing 20-minute-long, multiple-choice assessment presents students with brief passages and questions that require them to reason using biological principals with new material. For example, they read about the immune system, which is not typically covered in the introductory courses, and choose from the possible responses. The incorrect responses reflect common, specific reasoning errors students often make in the introductory biology courses. We have used the existing assessment for our research and have found strong Cronbach's alpha reliability but have not fully validated it using psychometric approaches. Based on initial IRT scaling and DIF analyses, the existing assessment has excellent psychometric properties, so we have collected additional think-aloud protocols on new passages, which we will code and use to develop new items that comprise a parallel form.

​

Research Design and Methods:

Using secondary and primary data, we will conduct three studies aimed to refine an existing assessment of inference-making and reasoning in biology and ensure its validity and reliability. In Study 1, we have used existing data collected from students in biology courses. We have calibrated items and tested them for differential functioning by students' race, sex, and socio-economic status. In Study 2, we have conducted cognitive interviews with a new sample of students to understand the relation between cognitive processes and item thetas, as well as to compare cognitive processes between students who get items correct versus incorrect. Study 3 has three parts. In part one, we will conduct tests of validity and reliability, item calibration using a multidimensional model, and tests for differential item functioning. In part two, we will use a measurement decision theory (Bayesian) approach to set a cut score that can identify students at high risk of course failure, while controlling for students' ACT Reading scores and High School GPA. In part three, we will determine whether the assessment is effective for documenting student growth over a semester.