Poster Session 10 Program Schedule
02/17/2024
09:00 am - 10:15 am
Room: Shubert Complex (Posters 1-60)
Poster Session 10: Neurodevelopmental | Congenital Conditions
Final Abstract #36
An Investigation of Factors that Increase the Rate of False-Positives on Performance Validity Testing in ADHD Evaluations
John-Christopher Finley, Northwestern University FSM, Chicago, United States
Jason Soble, University of Illinois College of Medicine, Chicago, United States
Category: Forensic Neuropsychology/Malingering/Noncredible Presentations
Keyword 1: performance validity
Keyword 2: attention deficit hyperactivity disorder
Keyword 3: psychometrics
Objective:
Embedded validity indicators (EVIs) are commonly used to establish the veracity of neurocognitive performance during attention-deficit/hyperactivity disorder (ADHD) evaluations. EVIs are gaining popularity because they are time-and cost-efficient; but they are also prone to higher false-positive rates (FPRs). EVIs with ≥90% specificity are recommended to maintain a FPR of ≤10%. However, it may be unrealistic to expect every EVI to maintain this low FPR in real-life settings. Although some research suggests aggregating EVIs can reduce the FPR, the evidence remains inconclusive. Furthermore, there is minimal discussion, particularly in the ADHD literature, about other factors that could influence FPRs. Along with the number of EVIs administered, it is possible that FPRs may depend on whether the validity indicators are embedded within tests of certain cognitive functions or within tests that require verbal, motor, or timed components. It is also possible that FPRs depend on the precision of the cutoff scale (e.g., T-scores versus scaled scores) or whether the cutoff is demographically adjusted. The current study sought to examine all of these factors in relation to the FPRs across 15 different EVIs in a sample of adults undergoing ADHD evaluation.
Participants and Methods:
Participants were 565 adult ADHD referrals with valid neurocognitive performance as determined by passing ≥2 performance validity tests (Dot Counting Test, Rey 15-Item Test/Recognition, or Reliable Digit Span). The EVIs included had cutoffs with >90% specificity based on ADHD cross-validation studies. FPRs were determined by the frequency of EVI failures within our sample.
Results:
Analyses revealed only 9 of the 15 empirical EVIs demonstrated an acceptable (≤10%) FPR. When all the EVI failures were combined into an aggregate index, the FPR reduced to 7.4%. Validity indicators embedded with tests of attention yielded significantly (ps <.001) higher FPRs than tests of any other cognitive ability, including processing speed, executive functioning, language, and memory. Similarly, FPRs were significantly higher (χ2 = 6.72, p = .009, V = .07) for validity indicators embedded within tests broadly associated with "frontal networks” dysfunction compared to others. FPRs also were significantly higher (χ2 = 4.63, p = .031, V = .06) for EVIs without a motoric component. FPRs did not significantly differ based on EVIs with timed or verbal components. FPRs were also not significantly influenced by EVIs with demographically adjusted cutoffs or cruder precision scaling metrics such as using scaled scores versus T-scores. Lastly, exploratory factor analysis revealed two primary factors: 'Complex Processing Speed' and 'Sustained Attention'. These factors explained 46% of the variance in the FPRs with the ‘Sustained Attention’ EVIs yielding significantly higher FPRs, χ2 = 9.79, p = .002, V = .09.
Conclusions:
Aggregating multiple EVIs may reduce the likelihood of misclassifying patients with genuine ADHD as invalid performers. It may be particularly helpful to consider which type of test the indicators are embedded within. FPRs were consistently higher among EVIs within tests of attention that do not require motor skills (e.g., Letter-Number Sequencing), potentially indicating that these EVIs are prone to conflating invalid with genuinely impaired performance.
| 
