Poster Session 10 Program Schedule
02/17/2024
09:00 am - 10:15 am
Room: Shubert Complex (Posters 1-60)
Poster Session 10: Neurodevelopmental | Congenital Conditions
Final Abstract #12
Attention Check Questions in Self-Report Measurement of Math and its Non-Cognitive Factors in College Students.
Cristina Boada, University of Houston, Houston, United States
Cassidy Salentine, University of Houston, Houston, United States
Paul Cirino, University of Houston, Houston, United States
Category: Learning Disabilities/Academic Skills
Keyword 1: pediatric neuropsychology
Keyword 2: academic skills
Keyword 3: learning
Objective:
Online surveys are a common method of data collection. The use of “attention-check” questions are an effective method of identifying careless responding in surveys (Liu & Wronski, 2018; Meade & Craig 2012; Ward & Meade, 2023), which occurs in 10-12% of undergraduate samples (Meade & Craig, 2012). Instructed response type attention checks are straightforward and the most recommended (Meade & Craig, 2012; Ward & Meade, 2023). This study evaluated the effect of instructed response attention check questions on the measurement of math ability and non-cognitive factors commonly related to math (self-efficacy and math anxiety). We evaluated both level differences as well as whether check questions alter the relationship of non-cognitive factors to math. We expected that incorrect responding to check questions would lower math performance but were unable to make hypotheses about level of self-report non-cognitive factors. We predicted that incorrect responding to check questions would moderate the relationship between both math anxiety and self-efficacy to math performance.
Participants and Methods:
Participants were 424 undergraduates (age 20.4, SD=2.7) at a large southwestern university. The sample was majority female (74%) but diverse socioeconomically and in race/ethnicity. The non-cognitive measures were researcher developed Math Anxiety (MA) and Math Self-Efficacy (MSE; Betz & Hackett, 1993) scales, with items selected directly targeting the use/manipulation of math in everyday life; both showed good reliability (α=.95). The two math scales were also researcher developed; one was a pure symbolic computational measure (EM-A) and the other consisted of word problems in an everyday context (EM-B). These measures had good reliability (α=.80 and α=.73). The four check questions were embedded in the surveys and two groupings were formed – one consisting of those who provided the correct answer for all items versus those who did not, and a second consisting of those who got all correct or only one answer incorrect versus those with more items incorrect. Correlational, ANOVA, and ANCOVA models were utilized.
Results:
Descriptively, check questions were skewed – 75% participants answered all check questions correctly, and 8% missed only one. Relations of both MA and MSE with EM-A and EM-B were modest though significant (|r|=.22 to .37) and in the expected directions (all p<.001). Check questions were related to level of all tasks (p<.001), with incorrect responses resulting in lower math performance, lower MSE, and higher MA. Check questions did not moderate the relation of MA or MSE to either math performance, with some suggestion that MA was more strongly related to EM-B in those who missed check questions, though only when failing several.
Conclusions:
Check questions showed a clear relation to both self-report and math performance measures. However, check questions did not alter the relation of MA or MSE to math performance in general. These results affirm extant relations of key self-perceptions to math using novel measures and highlight the need to evaluate the validity of self-report measures, even outside of objective performance indicators. Future work could examine the effect of attention checks in domains other than math and investigate other types of attention checks.
| 
