INS NYC 2024 Program

Paper	Paper Session 09 Program Schedule 02/15/2024 04:00 pm - 05:25 pm Room: West Side Ballroom - Salon 4 Paper Session 09: Assessment and Psychometrics Final Abstract #1 Performance Validity Outcomes in the Multisite Clinics of the National Neuropsychology Network Emily Carter, University of Florida, GAINESVILLE, United States Laura Umfleet, Medical College of Wisconsin, Milwaukee, United States Daniel Drane, Emory University, Atlanta, United States David Loring, Emory University, Atlanta, United States Steven Riese, University of California, Lost Angeles, Los Angeles, United States Joseph Gullett, University of Florida, Gainesville, United States Jordan Schwartz, University of California, Los Angeles, Los Angeles, United States Kristen Enriquez, University of California, Los Angeles, Los Angeles, United States Lucia Cavanagh, University of California, Los Angeles, Los Angeles, United States Robert Bilder, University of California, Los Angeles, Los Angeles, United States Russell Bauer, University of Florida, Gainesville, United States Category: Assessment/Psychometrics/Methods (Adult) Keyword 1: performance validity Keyword 2: psychometrics Keyword 3: diversity Objective: Developed in forensic contexts and thought by some to measure general aspects of task engagement rather than specific NP functions, Performance Validity Tests (PVTs) have become standard practice in neuropsychological (NP) evaluations. In the context of validating entire NP evaluation batteries, clinicians need to understand base rates of PVT failures in the general clinical context, how PVT performance differs according to NP diagnosis, and how PVT performance relates to performance on clinical tests of NP function. This study provides necessary data to investigate and address some of these needs. Participants and Methods: Participants (N=12,554, 53% female, 78% Caucasian) underwent NP evaluations at one of four sites of the National Neuropsychology Network (NNN; UCLA, Emory, Medical College of Wisconsin, University of Florida), and were selected for this study if they completed one or more PVT’s (N=5,358). Using widely used cutoffs, we evaluated the TOMM (N=265), ToPF (N=1583), CVLT2 FC (N=361), CVLT3 FC (N=1535), Reliable Digit Span (N=4767), and RBANS Effort Index (EI, N=330). PVT pass-fail rates were calculated for key demographic variables, and relationships with several widely used memory tests (CVLT2, CVLT3, WMS-IV Logical Memory I and II, WMS-IV Visual Reproduction I and II) were examined. The PVT performance of 3 common diagnostic groups in the NNN dataset (Mild Cognitive Impairment [N=375], Dementia [N=104] and Concussion/Mild TBI [N=108]) was also evaluated. Results: In the overall sample, 57.3% of participants passed all PVT’s, while 21% failed one, 15.2% failed two, 5.3% failed 3, and 0.7% failed four. Failure rates differed substantially among the PVT’s: TOMM = 9.4%, ToPF = 4.3%, CVLT2 FC=14.7%, CVLT3 FC=6.8%, RDS=7.3%, RBANS-EI=56%. Age effects on PVT performance were quite weak, with the strongest relationship (RBANS-EI) explaining only 14.7% of variance in age. Sex differences in PVT performance were negligible. On all PVT’s except RBANS-EI, Black/African American participants had higher failure rates(+5.5% to +17.7%) than White participants; all p’s <.05), while on all except CVLT-2 and CVLT3 FC, Asian participants had lower (-7.3% to -14.1%) rates than White participants (all p’s<.05). Correlations among PVT and memory test performances were all below .35, and, while sometimes significant due to large N’s, reflected small effect sizes (.05-.19). In the diagnostic group analysis, 76% of this subset passed all PVT’s (MCI = 76%, Dementia = 83%, mTBI=69%. Most remaining participants failed one (15% MCI, 12% Dementia, 7% mTBI) or two (5% MCI, 3% Dementia, 16% mTBI) PVT’s. Conclusions: In a large multi-site clinic sample, common PVT’s differed in their overall failure rates, showed nonsignificant relationships with age and sex, and did not strongly differentiate three diagnostic groups commonly seen for NP evaluation. A concerning finding, supported by previous literature, is the apparent racial differences in failure rates. It will be important to further understand and track down sources of this effect, and to temper interpretation accordingly. Notably, fewer than half of the NNN patients in this database underwent performance validity testing, reflecting persisting variability among clinicians in implementing this important component of NP evaluations.

Paper

02/15/2024
04:00 pm - 05:25 pm
Room: West Side Ballroom - Salon 4

Paper Session 09: Assessment and Psychometrics

Final Abstract #1

Performance Validity Outcomes in the Multisite Clinics of the National Neuropsychology Network

Emily Carter, University of Florida, GAINESVILLE, United States
Laura Umfleet, Medical College of Wisconsin, Milwaukee, United States
Daniel Drane, Emory University, Atlanta, United States
David Loring, Emory University, Atlanta, United States
Steven Riese, University of California, Lost Angeles, Los Angeles, United States
Joseph Gullett, University of Florida, Gainesville, United States
Jordan Schwartz, University of California, Los Angeles, Los Angeles, United States
Kristen Enriquez, University of California, Los Angeles, Los Angeles, United States
Lucia Cavanagh, University of California, Los Angeles, Los Angeles, United States
Robert Bilder, University of California, Los Angeles, Los Angeles, United States
Russell Bauer, University of Florida, Gainesville, United States

Category: Assessment/Psychometrics/Methods (Adult)

Keyword 1: performance validity
Keyword 2: psychometrics
Keyword 3: diversity

Objective:

Developed in forensic contexts and thought by some to measure general aspects of task engagement rather than specific NP functions, Performance Validity Tests (PVTs) have become standard practice in neuropsychological (NP) evaluations. In the context of validating entire NP evaluation batteries, clinicians need to understand base rates of PVT failures in the general clinical context, how PVT performance differs according to NP diagnosis, and how PVT performance relates to performance on clinical tests of NP function. This study provides necessary data to investigate and address some of these needs.

Participants and Methods:

Participants (N=12,554, 53% female, 78% Caucasian) underwent NP evaluations at one of four sites of the National Neuropsychology Network (NNN; UCLA, Emory, Medical College of Wisconsin, University of Florida), and were selected for this study if they completed one or more PVT’s (N=5,358). Using widely used cutoffs, we evaluated the TOMM (N=265), ToPF (N=1583), CVLT2 FC (N=361), CVLT3 FC (N=1535), Reliable Digit Span (N=4767), and RBANS Effort Index (EI, N=330). PVT pass-fail rates were calculated for key demographic variables, and relationships with several widely used memory tests (CVLT2, CVLT3, WMS-IV Logical Memory I and II, WMS-IV Visual Reproduction I and II) were examined. The PVT performance of 3 common diagnostic groups in the NNN dataset (Mild Cognitive Impairment [N=375], Dementia [N=104] and Concussion/Mild TBI [N=108]) was also evaluated.

Results:

In the overall sample, 57.3% of participants passed all PVT’s, while 21% failed one, 15.2% failed two, 5.3% failed 3, and 0.7% failed four. Failure rates differed substantially among the PVT’s: TOMM = 9.4%, ToPF = 4.3%, CVLT2 FC=14.7%, CVLT3 FC=6.8%, RDS=7.3%, RBANS-EI=56%. Age effects on PVT performance were quite weak, with the strongest relationship (RBANS-EI) explaining only 14.7% of variance in age. Sex differences in PVT performance were negligible. On all PVT’s except RBANS-EI, Black/African American participants had higher failure rates(+5.5% to +17.7%) than White participants; all p’s <.05), while on all except CVLT-2 and CVLT3 FC, Asian participants had lower (-7.3% to -14.1%) rates than White participants (all p’s<.05). Correlations among PVT and memory test performances were all below .35, and, while sometimes significant due to large N’s, reflected small effect sizes (.05-.19). In the diagnostic group analysis, 76% of this subset passed all PVT’s (MCI = 76%, Dementia = 83%, mTBI=69%. Most remaining participants failed one (15% MCI, 12% Dementia, 7% mTBI) or two (5% MCI, 3% Dementia, 16% mTBI) PVT’s.

Conclusions:

In a large multi-site clinic sample, common PVT’s differed in their overall failure rates, showed nonsignificant relationships with age and sex, and did not strongly differentiate three diagnostic groups commonly seen for NP evaluation. A concerning finding, supported by previous literature, is the apparent racial differences in failure rates. It will be important to further understand and track down sources of this effect, and to temper interpretation accordingly. Notably, fewer than half of the NNN patients in this database underwent performance validity testing, reflecting persisting variability among clinicians in implementing this important component of NP evaluations.