UL 4600 3rd Edition, 2023: UL Standard for Safety Evaluation of Autonomous Products

Description

This standard covers the safety principles, RISK mitigation, tools, techniques, and lifecycle processes for building and evaluating a SAFETY ARGUMENT for vehicles that can operate in an AUTONOMOUS mode, whether the ITEM is individual or part of a team such as a PLATOON.

Operation is assumed to occur without human supervision and without expectation of human intervention in performing and supervising the dynamic driving task and other normal system operations based upon the current ITEM state and ability to sense and otherwise interpret the operating environment. Human contributions to safety in other than normal operation are considered (e.g., maintenance), as are interactions with humans who are not operating the ITEM (e.g., pedestrians).

This standard generally uses the term “ITEM” rather than “system” or “product” when referring to the scope of the SAFETY CASE as well as the operation of the ITEM. This approach is in recognition of the possibility that the safety of the ITEM might rely upon infrastructure, services, support processes, and other factors that might not normally be considered part of a system such as a vehicle per se, but which materially affect its safety and therefore are all considered within the scope of the ITEM being assessed for CONFORMANCE. For a team of vehicles the ITEM might be scoped as an individual vehicle that is a member of a team or instead the ITEM might be designed to function as a team as a whole without reducing the safety of any one vehicle.

This standard assumes that the ITEM autonomously operates starting at some well-defined initial state to some other well-defined end state without human intervention. Human input might influence the selection of desirable states (e.g., via an occupant requesting a destination). However, the extent to which human operators MITIGATE or introduce RISK by performing or supervising a dynamic control task (e.g., by driving or taking responsibility for monitoring system operation) is outside the scope of the standard. Similarly, the extent to which human operator performance or non-performance is involved in RISKS related to transferring human driver control to or from the ITEM is also outside the scope of the standard. However, ensuring that the ITEM itself properly performs any change of control functions if and when it is supposed to is generally within the scope of the standard since it can adversely affect operation in fully AUTONOMOUS mode as well. Thus, while portions of this standard might be helpful for addressing less than fully AUTONOMOUS vehicles, issues involving human driver responsibilities, vigilance, and ability to properly accept responsibility for vehicle control are out of scope for this standard.

While information security is an essential topic, the details of that area are out of scope for this standard beyond a general requirement for a Security Plan and PROMPT ELEMENTS that are possibly unique to AUTONOMOUS vehicle operation in comparison to other vehicular security requirements. Reasonably foreseeable misuse and abuse as well as physical attacks (e.g., physical sensor damage) are in scope.

The requirements of this standard are considered to be at a necessary, but possibly not sufficient, level of completeness and rigor to create an acceptably well-formed and acceptably complete ITEM SAFETY CASE. In particular, PROMPT ELEMENT lists are considered non-exhaustive, with an expectation that design teams will include additional ITEMS as relevant to the ITEM and its OPERATIONAL DESIGN DOMAIN.

Elements in scope

Specific aspects of ITEM operation and SAFETY RELATED issues which are explicitly intended to be in scope for this standard include:

a) Operation of AUTONOMOUS ITEMS in potentially unstructured environments

EXAMPLE: A vehicle is the first vehicle directed into an open farm field containing a mixture of viable and non-viable areas for traversal and parking as part of an ad hoc overflow event parking process. There are no lane markings and no positioning beacons. Moreover, there are cows and hay bales randomly placed in the field. There are no humans assisting with organizing vehicle parking positions. This situation, if within the ITEM ODD, is in scope for the standard.

EXAMPLE: An AUTONOMOUS truck operating on a paved median to navigate around a road blockage. A truck in this situation may find itself in an opposing direction traffic lane or an unstructured off-road environment while passing the obstacle and attempting to return back to a normal travel lane.

EXAMPLE: A crowd has spilled into the street at a fire scene. Emergency response equipment, response personnel, victims, and casual observers are moving without regard to normal road use patterns. Fire hoses, falling pieces of burning debris, small explosions, traffic signal power outages, damaged pavement, and other disruptions to normal infrastructure expectations exist. Multiple injured people at building exits are calling for pickup by AUTONOMOUS vehicle ride hail services to be transported to urgent care medical facilities. This situation is in scope for the standard.

NOTE: A particular ITEM’s SAFETY CASE might require a structured environment for SAFE operation as specified by an OPERATIONAL DESIGN DOMAIN (ODD) description. However, structure is not assumed to be present by default. Therefore, operation in unstructured environments must be specifically disclaimed by the SAFETY CASE if applicable.

b) Operation with potentially inaccurate, incorrect, incomplete, or misleading data provided by sensors

c) The effects of potentially inaccurate, incorrect, incomplete, or biased data, including test data, field report data, other validation data and machine learning training data.

d) The effects of potentially imprecise, inaccurate, or incomplete simulation models.

e) Potential defects and failures of hardware and/or software in the ITEM, data collection functions, data processing functions, communications, engineering support systems, tools, and infrastructure support.

f) Human contributions to potential RISK, including occupants, pedestrians, other road users, nonroad users, cargo handlers, maintainers and inspectors. This includes acts of omission and commission; accidental and malicious physical acts; and human roles in creating as well as mitigating RISK.

g) Lifecycle considerations, including design data collection, engineering data management, tool qualification, design, implementation, testing, other validation, field data collection, operations, maintenance, updates, upgrades, and retirement. Lifecycle considerations also encompass potential changes to the environment which may affect ODDs, changes in object types, changes in behaviors, etc.

h) Inclusion of RISK mitigation and other aspects of contributions to the SAFETY CASE made by CONFORMANCE with other standards, and in particular both ISO 26262 and ISO 21448 standards for products within scope for those standards.

i) Ability to use a heterogeneous approach to arguments, including use of diverse standards to support safety (e.g., use of different but ACCEPTABLE functional safety standards for different ITEM subsystems).

None of the described in-scope topics is intended to require that the ITEM successfully delivers full service in all situations described. Rather, the requirement is to consider all PROMPT ELEMENTS and ARGUE that RISK is ACCEPTABLE despite these factors. In many cases that will involve crafting an ODD that excludes problematic PROMPT ELEMENTS. However, excluding a PROMPT ELEMENT from the ODD (or similar approach) creates an obligation to ARGUE that the exclusion does not itself result in unacceptable RISK.

EXAMPLE: Unpaved roads without lane markings are excluded from the ODD. The SAFETY CASE generally ARGUES that geo-fencing and map creation will exclude all unpaved roads. It is further ARGUED that this exclusion encompasses quickly identifying roads undergoing repaving projects that are temporarily unmarked but still carrying traffic.

EXAMPLE: Snow is excluded from the ODD. Snow is still part of the SAFETY CASE to cover un-forecast snow that occurs during an operational mission. The SAFETY CASE generally ARGUES that it can successfully terminate a mission via in-lane stop despite snow. It further ARGUES (with evidence) that snow will happen so infrequently in the deployment location that the elevated product RISK presented by occasional in-lane stops is ACCEPTABLE.

Scope limitations

A significant scope limitation of this standard is that it does not cover the detailed topics relevant to ensuring that humans are able to provide effective safety supervision for an AUTONOMOUS ITEM. Rather, coverage is limited to fully AUTONOMOUS operation with no human supervision as well as aspects of the ITEM during human supervised operation that do not relate to arguing human supervision effectiveness. Similarly, aspects of the ability of a human operator to safely control the ITEM are out of scope. More specifically, the following are explicitly intended to be out of scope for this standard:

a) Aspects of ITEMS as well as ITEM-level safety of operational modes for which the locus of control is outside the ITEM itself

EXAMPLES: End-to-end system-level safety (including the human supervisor or driver) of teleoperated modes of operation is excluded to the degree it relies upon a human teleoperator to control, supervise, or otherwise ensure system safety.

NOTE: A product-level “ITEM” is intended to include offboard functions that participate in control of a vehicle, such as a cloud-based route planning system.

NOTE: The proper response to teleoperated commands and proper transmission of teleoperation data out of the vehicle is in scope. However, whether teleoperation is actually SAFE at a system level is out of scope due to human involvement in the driving task.

b) Human factors related to safety during or after a handoff or mode switch that makes a human responsible for the dynamic control task safety.

EXAMPLE: The details of ensuring that a human supervisor is available and able to safely take over ITEM operation upon request and continue to operate the vehicle safely when some or all autonomy functions are disabled are out of scope.

c) RISK mitigation or other SAFETY ARGUMENT credit taken for the contribution of humans to the dynamic driving task (e.g., human driver, human safety supervisor, human teleoperator, issues of human alertness, issues of human situational awareness).

EXAMPLE: The details of how SAFE and effective human/machine interfaces should be provided to teleoperators are out of scope.

d) Road testing of prototype vehicles to the degree that SAFETY ARGUMENT takes credit for a human performing and/or supervising the dynamic driving task.

e) Specifics regarding how to ensure that humans acceptably meet expectations for non-driving roles in ITEM safety. To be clear, identifying the human contribution to the SAFETY CASE (e.g., via performing inspections, or a human’s ability to correctly perceive and interpret signals provided by the ITEM) is in scope. However, specifying the details of how to actually ensure that the contribution is being done in an ACCEPTABLE manner in terms of human behaviors, psychology, limitations, and so on is out of scope. While competency frameworks, staff skill lists, and experience requirements are potentially helpful topics to cover a SAFETY CASE, specifics regarding these topics are out of scope for this standard.

f) Specifics regarding how to evaluate the suitability and effectiveness of human interface devices. To be clear, the need to IDENTIFY such devices and the need to ensure suitability and effectiveness is in scope, but specifying requirements for how to meet that need is out of scope.

EXAMPLE: A vehicle intended to, among other things, transport occupants in AUTONOMOUS operation is not supposed to transfer vehicle control to an occupant under any circumstance during a mission. It does in fact attempt to transfer vehicle control to an occupant, providing three seconds of warning.

In Scope for UL 4600: Vehicle attempting a handoff to a driver when it was not supposed to.

In Scope for UL 4600: Incorrectly attempting to transfer control in violation of fully AUTONOMOUS operational mode.

Out of Scope for UL 4600: Whether three seconds is enough warning for effective handoff and whether the occupant is a qualified driver.

EXAMPLE: An AUTONOMOUS vehicle is designed to transfer control to a human driver under some circumstances with a 10 second warning when the driver is qualified, competent, and aware of this handoff mission parameter.

In Scope for UL 4600: Transferring control without the full 10 seconds of warning; A defectively designed brake control mechanism that under some circumstances prevents the human driver from actually regaining control at the designated time (e.g., human driver’s brake pedal disabled despite having attempted to perform a handoff); Whether brake pedal actuation by a human is intended to initiate a handoff under specified conditions.

Out of Scope for UL 4600: Whether ten seconds is enough warning for effective handoff in any particular handoff situation; Whether depression of a brake pedal by a human driver is a SAFE (from human factors point of view) handoff initiation mechanism; Checking whether occupant is a qualified driver (including having appropriate class of driver’s license); Checking whether the driver has sufficient cognitive ability for SAFE operation; Checking whether human driver is in correct seating position to assume operational control; Safety of vehicle once control has been transferred to the human driver; Effectiveness of Advanced Driver Assistance (ADAS) functions in mitigating RISK while under human driver control.

There are a number of additional topics out of scope. Reference to these topics should be made where relevant to the SAFETY CASE, but specifics such as PROMPT ELEMENTS to provide technical depth are not included in this standard:

a) The specific intended function (e.g. surface cleaning, fragile cargo delivery)

NOTE: This topic might be covered by an end product standard.

b) End product requirements

NOTE: It is intended that end product standards can reference or require this standard.

c) Legal and policy issues

EXAMPLES: Determining liability, what records retention policies are appropriate, what level or product RISK is actually ACCEPTABLE to society.

d) Ethical issues

EXAMPLES: Resolving questions of ACCEPTABLE RISK, evaluating comparative severity of different LOSS event scenarios

e) Electric Vehicle safety

EXAMPLES: SAFE battery design, SAFE battery management algorithms, battery thermal management

f) General vehicle safety

EXAMPLES: Crash mitigation, occupant restraints, refueling/recharging safety

g) Non-SAFETY RELATED quality aspects of performing the intended function

EXAMPLES: Ride quality, fuel economy

h) Effectiveness of crash and injury mitigation mechanisms

EXAMPLES: Seat belts, air bags, child seats

Also out of scope for the standard is any implied redefinition of existing standards and accepted practices to the degree that they are ACCEPTABLE to support the SAFETY CASE being made. Reference to these topics should be made where relevant to the SAFETY CASE, but specifics are not included in this standard. These topics include:

a) Government regulations

EXAMPLES: FMVSS, Federal Communications Commission radio frequency interference emission certification

b) Mechanical aspects of the ITEM

EXAMPLES: Sharp edges, pinch points, window lift motor closing pressure limits

c) Legacy operational procedures

EXAMPLES: Car door operation, child locks on car doors, cargo loading and loading

d) Specifics of vehicle support for other than normal operations (except to the degree that the autonomy is expected to provide such support and that such provision is SAFETY RELATED)

EXAMPLES: Autonomy support for non-SAFETY RELATED routine maintenance procedures in situations which do not present a hazard

e) Sufficiency and performance of controls and ITEM response when a human is operating or supervising the dynamic driving task

EXAMPLE: As required by FMVSS

f) Licensing, training, qualification and other aspects of ensuring human competence to operate or participate in the vehicle lifecycle

NOTE: Credit can be taken for these in the SAFETY CASE only to the degree that any standard or procedure conferring or documenting qualification provides objective evidence of abilities