Project Goal:

Develop an Item Response Theory (IRT) model to determine if the more statistical principled approach of IRT yields sufficient benefits as compared to the Safety Measurement Systems (SMS) to be integrated into FMCSA’s motor carrier prioritization process.

Background:

Section 5221 of the Fixing America’s Surface Transportation ACE (FAST Act) required the National Academy of Sciences (NAS) to conduct a study of the Compliance Safety Accountability (CSA) program and its SMS. On June 27, 2017, the NASEM panel issued a report titled “Improving Motor Carrier Safety Measurement,” which argues for FMCSA to adopt a more statistically principled approach. In response to the NASEM recommendation and the corresponding corrective action plan, FMCSA initiated a Correlation Study, a multi-year effort focused on developing and evaluating an IRT model to determine if the statistical rigor and complexity of IRT yields sufficient benefits compared to SMS to be integrated into the Agency’s prioritization process.

Summary:

The goal of this project will be met through completion of several task areas. The team began by carrying out an exploratory data analysis of FMCSA inspection data from MCMIS. During this phase, the team analyzed and evaluated carrier roadside inspection and violation data. While not a required model input, the team also evaluated vehicle miles traveled (VMT) as a potential measure of carrier exposure. Next, IRT model structures were formulated based on the knowledge gained during the Exploratory Data Analysis phase, as well as any models completed to date. The team then moved on to testing with synthetic data, a key step in testing different model structures due to complexities and anomalies seen in real data, which makes it difficult to see how different data elements impact the overall performance of the model. The team then used several different subsets of the carrier population to test small-scale IRT models before running them on the full carrier data set. The final test for each IRT model structure was to run it on the full set of carrier data. The team is now developing a communications and outreach plan to communicate study findings and next steps.

Contractor:

Volpe