3. EVALUATION APPROACH 3.1 EVALUATION OVERVIEW The primary intent of the FOT was to determine the extent to which existing
vulnerabilities in the transportation of hazardous materials can be reduced,
thereby reducing the potential for a catastrophic event with resulting loss
of life and property. The benefit-cost analysis was designed to measure this
benefit and determine which component technologies or integrated systems offer
the best mix of improved security balanced against reasonable costs for deployment
and operations. Deploying these technologies and systems will require an investment by the
trucking industry, which is an industry with very low returns - a profit margin
of 2 percent is not uncommon. An important aspect of this evaluation was to
determine whether or not the deployment of these technologies and systems would
generate a positive return on investment for industry. The potential return
from deployment was quantified as a measurable improvement in operating efficiencies
or improvement in overall operations. The objective assessment of this "bottom
line" impact is critical for determining policy options for structuring
a deployment program: market-based, where operating efficiencies drive deployment;
or a mandate in the interests of national security; or a package of incentives
designed to encourage and facilitate deployment. 3.2 ASSESSMENT FRAMEWORKS The main evaluation impact categories examined by the Evaluation Team are
safety, security, and operational efficiency. As detailed in Figure 3-1, these
impact categories feed the benefit-cost analysis according to macroeconomic/societal
(macro) public sector benefit-cost results (stemming from security and safety
benefits) and microeconomic/industry (micro) private sector benefit-cost results
(derived from operational efficiency improvements and enhanced safety). The
macro/societal and micro/industry benefit-cost measurements analysis was conducted
to determine the following: - Are the industry operational efficiency benefits significant enough to drive
widespread industry deployment of test technology systems?
- If not, are the macro benefits large enough to warrant government action
to facilitate wide-scale national deployment?
The evaluation assessments were conducted within the scope of the FOT and
extended the FOT findings to the larger universe of truck-based HAZMAT shipments
(for the four primary load types) through rigorous analytical frameworks. These
frameworks utilized primary and secondary industry survey data; detailed motor
carrier census records; market analysis of technology products and services
that are commercially available; and the opinions provided by two august groups
of leading national experts in HAZMAT shipping, public safety, security and
risk assessment - an Expert Steering Committee and a 26-member Delphi Panel.
The assessments determined what measurable benefits exist, and established
and implemented analytical frameworks to monetize potential benefits and to
weigh these against any costs that would have to be incurred to realize the
benefits. Detailed discussion of methodology are presented in Volume III,
Section 3: HAZMAT FOT Security Benefits Assessment and Section 4: Benefit-Cost
Analysis and Industry Deployment Potential, and are summarized in the following
sections of this synthesis document. Figure 3-1. Evaluation Framework. 
It should be noted that the test technologies were designed to enable real-time
communications and information exchange among drivers, dispatchers and other
authorized parties; to track assets; to secure vehicles, loads, and shipping
documentation; and to enable driver or automated exception alerts in response
to crises or deviations in operational characteristics outside of set parameters.
The technologies themselves and their usage are not specifically designed to
provide explicit or traditional safety benefits (i.e., reducing the frequency
and severity of crashes). The test technologies are not designed to warn drivers of obstacles in proximity
to their vehicles, lane departure, imminent vehicle rollover conditions, or
conditions signaling driver fatigue. The exceptions include the beneficial impacts
of frequent driver/dispatcher communications that allow a dispatcher to assess
the driver's condition; position tracking to determine possible speeding, or
capabilities that provide responders to HAZMAT incidents timely notification
of the incident; and location and the type and quantity of HAZMAT involved to
enhance the rapidity and appropriateness of response. Potential safety
benefits in terms of crashes avoided and enhanced emergency response are proffered,
but are mostly qualitative in nature. Quantitatively, the evaluation focused
on the remaining two key assessment areas: Security and Operational Efficiency.
The estimation of benefits and costs, payback periods, and industry deployment
potential was based on a stepwise analytical framework. Benefits were derived
as operational improvements and reductions in potential impacts of terrorist
activities involving truck-based HAZMAT shipments. The framework for these assessments
(as illustrated in Figures 3-2 and 3-3) is based on the following inputs: - Technology performance and participant perceptions, as defined by the FOT,
established technology functionality, efficacy, user acceptance, and operational
improvements. These data also provided inputs to the Delphi process described
in Volume III, Section 3.
- HAZMAT carrier demographics (total number of trucks and fleet size distributions)
for each load type that was included in the FOT. These data, defined through
queries of the FMCSA's Motor Carrier Management Information System (MCMIS),
established the number of trucks that would represent potential full deployment
of the test technologies. The results of the Deployment Team's motor carrier
survey. Returned by 153 motor carriers, the respondent demographics
represented a broad diversity of fleet sizes, range of operations, routing
variability, general operational characteristics and levels of fleet management
technologies currently used and those to be employed in the near-term. These
results were validated using other industry technology deployment studies
and applied to the demographics of HAZMAT carriers reported in the FMCSA Motor
Carrier Management Information System (MCMIS) database to estimate levels
of current technology market penetration and total market potential.3
- A Technology Compendium, which defined current and near-future levels of
motor carrier technology adoption, and pricing and functionality of commercially
available technology products with similar capabilities as the technologies
testing during the FOT. The latter provided a range of potential industry
costs.
Other important
evaluation goals included: assessing technical performance of the test suites;
user acceptance and perceptions of the pros and cons of using the technologies;
issues of deployment costs; privacy and the potential for governmental intervention;
and defining the market potential for the deployment of the technologies. The data were collected
through direct observations; reviews of technology transaction records; interviews
and survey questionnaires of motor carrier and state agency participants and
non-participants; and consultation of secondary data sources. These assessments
are the basis for the overall evaluation findings. The processes involved
in the Efficiency and Security Assessments are detailed in Volume III, Sections
2 and 3, respectively, and are summarized in the sections that follow. Figure 3-2. Operational Efficiency Benefit-Cost Analysis Process Flows. 
Figure 3-3. Security Benefit-Cost Analysis Process Flows. 
Footnotes 3 ATA Foundation, Motor
Carrier Technologies - Fleet Operational Impacts and Implications for Intelligent
Transportation Systems/Commercial Vehicle Operations, October 1999; ATA
Foundation, NAFTA Priority Corridor Comprehensive ITS/CVO Plan Motor Carrier
Technology Survey, August 2000; American Transportation Research Institute
- GartnerG2, Trucking Technology Survey, 2003.
Continue to:Table of Contents >
Introduction >
Section 1 >
2 >
Section 3 >
4 >
5 >
6 >
7 >
8 >
9 >
10 >
11
|
