Accordingly, assurance decomposes into two parts: a) ensuring that the model is an accurate representation of the world as it changes through time and b) ensuring that the actions are safe (and effective), given the model. Both perception and action may employ AI, including machine learning (ML), and these present challenges to assurance. However, it is usually feasible to guard the actions with traditionally engineered and assured monitors, and thereby ensure safety, given the model. Thus, the model becomes the central focus for assurance.
We propose an architecture and methods to ensure the accuracy of models derived from sensors whose interpretation uses AI and ML. Rather than derive the model from sensors bottom-up, we reverse the process and use the model to predict sensor interpretation. Small prediction errors indicate the world is evolving as expected and the model is updated accordingly. Large prediction errors indicate surprise, which may be due to errors in sensing or interpretation, or unexpected changes in the world (e.g., a pedestrian steps into the road). The former initiate error masking or recovery, while the latter requires revision to the model. Higher-level AI functions assist in diagnosis and execution of these tasks.
Although this two-level architecture where the lower level does "predictive processing" and the upper performs more reflective tasks, both focused on maintenance of a world model, is derived by engineering considerations, it also matches a widely accepted theory of human cognition.
PDF
PDF
or 25 minute video
Second generation slides and video for IFIP WG10.4 meeting on Intelligent Vehicle Dependability and Security (IVDS), January 2021
PDF
or 27 minute video
(unfortunately, this video has audio dropouts; I suggest you just ride
them out while reading the corresponding slides).
Alternatively, here's a recording from the actual workshop.
Third generation slides for a Newton Institute
workshop, 26 July 2022.
PDF
@inproceedings{Jha-etal:Safecomp20,
AUTHOR = {Susmit Jha and John Rushby and N. Shankar},
TITLE = {Model-Centered Assurance for Autonomous Systems},
PAGES = {228--243},
CROSSREF = {Safecomp20}
}
@proceedings{Safecomp20,
BOOKTITLE = {Computer Safety, Reliability, and Security ({SAFECOMP} 2020)},
TITLE = {Computer Safety, Reliability, and Security ({SAFECOMP} 2020)},
YEAR = 2020,
EDITOR = {Ant\'{o}nio Casimiro and others},
ADDRESS = {Lisbon, Portugal},
MONTH = sep,
SERIES = {Lecture Notes in Computer Science},
VOLUME = 12234,
PUBLISHER = {Springer}
}