Expert Support for Improving Motor Carrier Safety Measurement
Goal:
Establish a standing committee that will assist the Federal Motor Carrier Safety Administration (FMCSA) in developing and testing an Item Response Theory (IRT) model to measure motor carrier safety.
Background:
On June 27, 2017, the National Academy of Sciences (NAS) published the report entitled “Improving Motor Carrier Safety Measurement," which was required by Section 5221 of the Fixing America's Surface Transportation Act, 2015 (FAST Act). The NAS report concluded that the Safety Measurement System (SMS), in its current form, is structured in a reasonable way and its method of identifying motor carriers for alert status is defensible. In addition, NAS agreed that FMCSA’s overall approach, based on crash prevention rather than prediction, is sound. NAS did not recommend any immediate changes to FMCSA’s prioritization system but did provide FMCSA with six recommendations to improve the system:
-
Recommendation 1 - FMCSA should develop an Item Response Theory (IRT) model over the next 2 years. If it is then demonstrated to perform well in identifying motor carriers for alerts, FMCSA should use it to replace SMS.
-
Recommendation 2 - FMCSA should continue to collaborate with States and other agencies to improve the quality of Motor Carrier Management Information System (MCMIS) data in support of SMS.
-
Recommendation 3 - FMCSA should investigate ways of collecting data that will likely benefit the recommended methodology for safety assessment.
-
Recommendation 4 - FMCSA should structure a user-friendly version of the MCMIS data file used as input to SMS without any personally identifiable information to facilitate its use by external parties, such as researchers, and by carriers.
-
Recommendation 5 - FMCSA should undertake a study to better understand the statistical operating characteristics of the percentile ranks to support decisions regarding the usability of public scores.
-
Recommendation 6 - Given that there are good reasons for both an absolute and a relative metric on safety performance, FMCSA should decide on the carriers that receive SMS alerts using both the SMS percentile ranks and the SMS measures, and the percentile ranks should be computed both conditionally within safety event groups and over all motor carriers.
Summary:
Once established and funded on an ongoing basis, an independent committee appointed by the President of the NAS will assist FMCSA in developing and testing an IRT model to measure motor carrier safety. In addition, the standing committee will solicit input from stakeholders and provide advice to FMCSA in the implementation of the other five recommendations. Once established, the standing committee will meet quarterly, and on an ad-hoc basis if needed, to review FMCSA’s progress in implementing its action plan and to provide guidance and advice. This standing committee will a) assist FMCSA in refining its analysis methodologies; b) assist in identifying and utilizing current research in the transportation and related communities; c) review the effectiveness results of the IRT model and provide advice; and d) promote transparency in FMCSA’s analysis and prioritization activities.
Outcomes:
Formation of a committee of 10-12 experts to advise FMCSA on its improvements to the measurement of safety; committee meetings and summary notes/letter reports detailing recommendations.
Milestones: |
Completed: |
| October 2017: Formation of committee. | ☑ |
| January 2018: Establish committee Web page | ☑ |
| February 2018: First committee meeting | ☑ |
Funding:
FY17 Funding: $140,000
Current Status:
Project complete.
Contractor:
NAS
Last updated: Thursday, May 30, 2024