Abstract

We present a framework for identifying, communicating, and addressing risk in shared-autonomy mobile manipulator applications. This framework is centered on the capacity of the mobile manipulator to sense its environment, interpret complex and cluttered scenes, and estimate the probability of actions and configuration states that may result in task failures, such as collision (i.e., identifying “risk”). If the threshold for acceptable risk is exceeded, a remote operator is notified and presented with timely, actionable information in which the person can quickly assess the situation and provide guidance for the robot. This framework is demonstrated with a use case in which a mobile manipulator performs machine tending and material handling tasks.

References

1.
Thakar
,
S.
,
Fang
,
L.
,
Shah
,
B.
, and
Gupta
,
S. K.
,
2018
, “
Towards Time-Optimal Trajectory Planning for Pick-and-Transport Operation With a Mobile Manipulator
,”
IEEE International Conference on Automation Science and Engineering (CASE)
,
Munich, Germany
,
Aug. 20–24
, pp.
981
987
.
2.
Annem
,
V.
,
Rajendran
,
P.
,
Thakar
,
S.
, and
Gupta
,
S. K.
,
2019
, “
Towards Remote Teleoperation of a Semi-Autonomous Mobile Manipulator System in Machine Tending Tasks
,”
ASME Manufacturing Science and Engineering Conference (MSEC)
,
Erie, PA
,
June
,
Vol. 58745, p. V001T02A027
.
3.
Thakar
,
S.
,
Rajendran
,
P.
,
Kabir
,
A. M.
, and
Gupta
,
S. K.
,
2020
, “
Manipulator Motion Planning for Part Pickup and Transport Operations From a Moving Base
,”
IEEE Trans. Autom. Sci. Eng.
,
19
(
1
), pp.
191
206
.
4.
Štibinger
,
P.
,
Broughton
,
G.
,
Majer
,
F.
,
Rozsypálek
,
Z.
,
Wang
,
A.
,
Jindal
,
K.
,
Zhou
,
A.
,
Thakur
,
D.
,
Loianno
,
G.
,
Krajník
,
T.
, and
Saska
,
M.
,
2021
, “
Mobile Manipulator for Autonomous Localization, Grasping and Precise Placement of Construction Material in a Semi-structured Environment
,”
IEEE Rob. Autom. Lett.
,
6
(
2
), pp.
2595
2602
.
5.
Thakar
,
S.
,
Malhan
,
R. K.
,
Bhatt
,
P. M.
, and
Gupta
,
S. K.
,
2022
, “
Area-Coverage Planning for Spray-Based Surface Disinfection With a Mobile Manipulator
,”
Rob. Auton. Syst.
,
147
, p.
103920
.
6.
Kot
,
T.
, and
Novák
,
P.
,
2018
, “
Application of Virtual Reality in Teleoperation of the Military Mobile Robotic System Taros
,”
Int. J. Adv. Rob. Syst.
,
15
(
1
), p.
1729881417751545
.
7.
Al-Hussaini
,
S.
,
Thakar
,
S.
,
Kim
,
H.
,
Rajendran
,
P.
,
Shah
,
B. C.
,
Marvel
,
J. A.
, and
Gupta
,
S. K.
,
2021
, “
Human-Supervised Semi-Autonomous Mobile Manipulators for Safely and Efficiently Executing Machine Tending Tasks
,”
Artificial Intelligence for Human–Robot Interaction Symposium, AAAI Fall Symposium Series
,
Washington, DC
,
Nov. 4–6
.
8.
Robotic Industries Association
,
2016
,
“RIA TR15.306: Task-Based Risk Assessment Methodology,” Technical Report Registered with American National Standards Institute (ANSI)
.
9.
Inam
,
R.
,
Raizer
,
K.
,
Hata
,
A.
,
Souza
,
R.
,
Forsman
,
E.
,
Cao
,
E.
, and
Wang
,
S.
,
2018
, “
Risk Assessment for Human-Robot Collaboration in an Automated Warehouse Scenario
,”
2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA)
,
Turin, Italy
,
Sept. 4–7
, Vol. 1, IEEE, pp.
743
751
.
10.
Wan
,
N.
,
Li
,
L.
,
Ye
,
C.
, and
Wang
,
B.
,
2019
, “
Risk Assessment in Intelligent Manufacturing Process: A Case Study of an Optical Cable Automatic Arranging Robot
,”
IEEE Access
,
7
, p.
105892
.
11.
Marvel
,
J. A.
,
Falco
,
J.
, and
Marstio
,
I.
,
2014
, “
Characterizing Task-Based Human–Robot Collaboration Safety in Manufacturing
,”
IEEE. Trans. Syst. Man. Cybernet.: Syst.
,
45
(
2
), pp.
260
275
.
12.
El Makrini
,
I.
,
Merckaert
,
K.
,
De Winter
,
J.
,
Lefeber
,
D.
, and
Vanderborght
,
B.
,
2019
, “
Task Allocation for Improved Ergonomics in Human-Robot Collaborative Assembly
,”
Interaction Stud.
,
20
(
1
), pp.
102
133
.
13.
Aven
,
T.
,
2013
, “
On the Meaning of a Black Swan in a Risk Context
,”
Safety science
,
57
, pp.
44
51
.
14.
Marvel
,
J. A.
, and
Bostelman
,
R.
,
2014
, “
A Cross-Domain Survey of Metrics for Modelling and Evaluating Collisions
,”
Int. J. Adv. Rob. Syst.
,
11
(
9
), p.
142
.
15.
Liu
,
H.
, and
Wang
,
L.
,
2021
, “
Collision-free Human-Robot Collaboration Based on Context Awareness
,”
Rob. Comput. Integr. Manuf.
,
67
, p.
101997
.
16.
Kozai
,
K.
, and
Hashimoto
,
M.
,
2018
, “
Determining Robot Grasping-Parameters by Estimating” “Picking Risk”
,”
2018 International Workshop on Advanced Image Technology (IWAIT)
,
Chiang Mai, Thailand
,
Jan. 7–10
, IEEE, pp.
1
4
.
17.
Azizi
,
V.
,
Kimmel
,
A.
,
Bekris
,
K.
, and
Kapadia
,
M.
,
2017
, “
Geometric Reachability Analysis for Grasp Planning in Cluttered Scenes for Varying End-Effectors
,”
2017 13th IEEE Conference on Automation Science and Engineering (CASE)
,
Xi'an, China
,
Aug. 20–23
, IEEE, pp.
764
769
.
18.
Akinola
,
I.
,
Varley
,
J.
,
Chen
,
B.
, and
Allen
,
P. K.
,
2018
, “
Workspace Aware Online Grasp Planning
,”
2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Madrid, Spain
,
Oct. 1–5
, IEEE, pp.
2917
2924
.
19.
Pardi
,
T.
, and
Stolkin
,
R.
,
2018
, “
Choosing Grasps to Enable Collision-Free Post-Grasp Manipulations
,”
2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids)
,
Beijing, China
,
Nov. 6–9
, IEEE, pp.
299
305
.
20.
Sundaralingam
,
B.
, and
Hermans
,
T.
,
2019
, “
Relaxed-Rigidity Constraints: Kinematic Trajectory Optimization and Collision Avoidance for In-Grasp Manipulation
,”
Autonom. Rob.
,
43
(
2
), pp.
469
483
.
21.
Marturi
,
N.
,
Kopicki
,
M.
,
Rastegarpanah
,
A.
,
Rajasekaran
,
V.
,
Adjigble
,
M.
,
Stolkin
,
R.
,
Leonardis
,
A.
, and
Bekiroglu
,
Y.
,
2019
, “
Dynamic Grasp and Trajectory Planning for Moving Objects
,”
Autonom. Rob.
,
43
(
5
), pp.
1241
1256
.
22.
Xiao
,
X.
,
Dufek
,
J.
, and
Murphy
,
R. R.
,
2020
, “
Robot Risk-Awareness by Formal Risk Reasoning and Planning
,”
IEEE Rob. Autom. Lett.
,
5
(
2
), pp.
2856
2863
.
23.
Vogl
,
G. W.
,
Weiss
,
B. A.
, and
Helu
,
M.
,
2019
, “
A Review of Diagnostic and Prognostic Capabilities and Best Practices for Manufacturing
,”
J. Intell. Manuf.
,
30
(
1
), pp.
79
95
.
24.
Sirkin
,
D.
,
Martelaro
,
N.
,
Johns
,
M.
, and
Ju
,
W.
,
2017
, “
Toward Measurement of Situation Awareness in Autonomous Vehicles
,”
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
, pp.
405
415
.
25.
Dini
,
A.
,
Murko
,
C.
,
Yahyanejad
,
S.
,
Augsdörfer
,
U.
,
Hofbaur
,
M.
, and
Paletta
,
L.
,
2017
, “
Measurement and Prediction of Situation Awareness in Human-Robot Interaction Based on a Framework of Probabilistic Attention
,”
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Vancouver, Canada
,
Sept. 24–28
, IEEE, pp.
4354
4361
.
26.
Gervits
,
F.
,
Thurston
,
D.
,
Thielstrom
,
R.
,
Fong
,
T.
,
Pham
,
Q.
, and
Scheutz
,
M.
,
2020
, “
Toward Genuine Robot Teammates: Improving Human-Robot Team Performance Using Robot Shared Mental Models
,”
Proceedings of the 19th International Conference on Autonomous Agents and Multi agent Systems
,
Auckland, New Zealand
,
May 9–13
, pp.
429
437
.
27.
International Organization for Standardization
,
2011
, “
Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models
,”
Standard, International Organization for Standardization/International Electrotechnical Commission
,
Geneva, CH
.
28.
Chatzimichailidou
,
M. M.
, and
Dokas
,
I. M.
,
2016
, “
Introducing Risksoap to Communicate the Distributed Situation Awareness of a System About Safety Issues: An Application to a Robotic System
,”
Ergonomics
,
59
(
3
), pp.
409
422
.
29.
Schaefer
,
K. E.
,
Straub
,
E. R.
,
Chen
,
J. Y.
,
Putney
,
J.
, and
Evans III
,
A. W.
,
2017
, “
Communicating Intent to Develop Shared Situation Awareness and Engender Trust in Human-Agent Teams
,”
Cogn. Syst. Res.
,
46
, pp.
26
39
.
30.
Chen
,
J. Y.
,
Procci
,
K.
,
Boyce
,
M.
,
Wright
,
J.
,
Garcia
,
A.
, and
Barnes
,
M.
,
2014
, “
Situation Awareness-Based Agent Transparency
,”
US Army Research Lab Aberdeen Proving Ground, MD, Human Research and Engineering
, Technical Report No. ARL-TR-6905.
31.
Yamamoto
,
T.
,
Terada
,
K.
,
Ochiai
,
A.
,
Saito
,
F.
,
Asahara
,
Y.
, and
Murase
,
K.
,
2018
, “
Development of the Research Platform of a Domestic Mobile Manipulator Utilized for International Competition and Field Test
,”
2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Madrid, Spain
,
Oct. 1–5
, IEEE, pp.
7675
7682
.
32.
Valner
,
R.
,
Vunder
,
V.
,
Zelenak
,
A.
,
Pryor
,
M.
,
Aabloo
,
A.
, and
Kruusamäe
,
K.
,
2018
, “
Intuitive “Human-on-the-Loop” Interface for Tele-Operating Remote Mobile Manipulator Robots
,”
International Symposium on Artificial Intelligence, Robotics, and Automation in Space (i-SAIRAS)
,
Madrid, Spain
,
June
, pp.
1
8
.
33.
Labbé
,
M.
, and
Michaud
,
F.
,
2019
, “
Rtab-map as an Open-Source Lidar and Visual Simultaneous Localization and Mapping Library for Large-Scale and Long-Term Online Operation
,”
J. Field Rob.
,
36
(
2
), pp.
416
446
.
34.
Thakar
,
S.
,
Rajendran
,
P.
,
Kim
,
H.
,
Kabir
,
A. M.
, and
Gupta
,
S. K.
,
2020
, “
Accelerating Bi-Directional Sampling-Based Search for Motion Planning of Non-Holonomic Mobile Manipulators
,”
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Paris, France
,
October
, pp.
6711
6717
.
35.
Kabir
,
A.
,
Kanyuck
,
A.
,
Malhan
,
R. K.
,
Shembekar
,
A. V.
,
Thakar
,
S.
,
Shah
,
B. C.
, and
Gupta
,
S. K.
,
2019
, “
Generation of Synchronized Configuration Space Trajectories of Multi-Robot Systems
,”
IEEE International Conference on Robotics and Automation (ICRA)
,
Montreal, Canada
,
May 20–24
, pp.
8683
8690
.
36.
Thakar
,
S.
,
Rajendran
,
P.
,
Annem
,
V.
,
Kabir
,
A. M.
, and
Gupta
,
S.
,
2019
, “
Accounting for Part Pose Estimation Uncertainties During Trajectory Generation for Part Pick-Up Using Mobile Manipulators
,”
2019 International Conference on Robotics and Automation (ICRA)
,
Montreal, Canada
,
May 20–24
, IEEE, pp.
1329
1336
.
37.
Kabir
,
A. M.
,
Thakar
,
S.
,
Bhatt
,
P. M.
,
Malhan
,
R. K.
,
Rajendran
,
P.
,
Shah
,
B. C.
, and
Gupta
,
S. K.
,
2020
, “
Incorporating Motion Planning Feasibility Considerations During Task-Agent Assignment to Perform Complex Tasks Using Mobile-Manipulators
,”
IEEE International Conference on Robotics and Automation
,
Paris, France
,
May
, pp.
5663
5670
.
38.
Kabir
,
A. M.
,
Thakar
,
S.
,
Malhan
,
R. K.
,
Shembekar
,
A. V.
,
Shah
,
B. C.
, and
Gupta
,
S. K.
,
2021
, “
Generation of Synchronized Configuration Space Trajectories With Workspace Path Constraints for an Ensemble of Robots
,”
Int. J. Rob. Res.
,
40
(
2–3
), pp.
651
678
.
39.
Fox
,
D.
,
Burgard
,
W.
, and
Thrun
,
S.
,
1997
, “
The Dynamic Window Approach to Collision Avoidance
,”
IEEE Rob. Autom. Mag.
,
4
(
1
), pp.
23
33
.
40.
Rajendran
,
P.
,
Thakar
,
S.
, and
Gupta
,
S. K.
,
2019
, “
User-Guided Path Planning for Redundant Manipulators in Highly Constrained Work Environments
,”
IEEE International Conference on Automation Science and Engineering (CASE)
,
Vancouver, Canada
,
Aug. 22–26
, pp.
1212
1217
.
41.
Rajendran
,
P.
,
Thakar
,
S.
,
Kabir
,
A. M.
,
Shah
,
B. C.
, and
Gupta
,
S. K.
,
2019
, “
Context-Dependent Search for Generating Paths for Redundant Manipulators in Cluttered Environments
,”
IEEE International Conference on Intelligent Robots and Systems (IROS)
,
Macao, China
,
Nov. 3–8
, pp.
5573
5579
.
42.
Rajendran
,
P.
,
Thakar
,
S.
,
Bhatt
,
P. M.
,
Kabir
,
A. M.
, and
Gupta
,
S. K.
,
2021
, “
Strategies for Speeding Up Manipulator Path Planning to Find High Quality Paths in Cluttered Environments
,”
ASME J. Comput. Inf. Sci. Eng.
,
21
(
1
), p.
011009
.
43.
Ratliff
,
N.
,
Zucker
,
M.
,
Bagnell
,
J. A.
, and
Srinivasa
,
S.
,
2009
, “
Chomp: Gradient Optimization Techniques for Efficient Motion Planning
,”
2009 IEEE International Conference on Robotics and Automation
,
Kobe, Japan
,
May 12–17
, IEEE, pp.
489
494
.
44.
Kalakrishnan
,
M.
,
Chitta
,
S.
,
Theodorou
,
E.
,
Pastor
,
P.
, and
Schaal
,
S.
,
2011
, “
Stomp: Stochastic Trajectory Optimization for Motion Planning
,”
2011 IEEE International Conference on Robotics and Automation
,
Shanghai, China
,
May 9–13
, IEEE, pp.
4569
4574
.
45.
Kuffner
,
J. J.
, and
LaValle
,
S. M.
,
2000
, “
Rrt-Connect: An Efficient Approach to Single-Query Path Planning
,”
ICRA’00, IEEE International Conference on Robotics and Automation
,
San Francisco, CA
,
Apr. 24–28
, Vol. 2, IEEE, pp.
995
1001
.
46.
Sucan
,
I. A.
,
Moll
,
M.
, and
Kavraki
,
L. E.
,
2012
, “
The Open Motion Planning Library
,”
IEEE Rob. Autom. Mag.
,
19
(
4
), pp.
72
82
.
47.
Pan
,
J.
,
Chitta
,
S.
, and
Manocha
,
D.
,
2012
, “
Fcl: A General Purpose Library for Collision and Proximity Queries
,”
2012 IEEE International Conference on Robotics and Automation
,
Saint Paul, MN
,
May 14–18
, IEEE, pp.
3859
3866
.
48.
Konur
,
S.
,
2013
, “
A Survey on Temporal Logics for Specifying and Verifying Real-Time Systems
,”
Front. Comput. Sci.
,
7
(
3
), pp.
370
403
.
49.
Van Gansbeke
,
W.
,
Neven
,
D.
,
De Brabandere
,
B.
, and
Van Gool
,
L.
,
2019
, “
Sparse and Noisy Lidar Completion With RGB Guidance and Uncertainty
,”
2019 16th International Conference on Machine Vision Applications (MVA)
,
Tokyo, Japan
,
May 27–31
, IEEE, pp.
1
6
.
50.
Park
,
J.-H.
,
Shin
,
Y.-D.
,
Bae
,
J.-H.
, and
Baeg
,
M.-H.
,
2012
, “
Spatial Uncertainty Model for Visual Features Using a Kinect™
,”
Sensors
,
12
(
7
), pp.
8640
8662
.
51.
Choi
,
J.
,
Hwang
,
E.
,
So
,
H.-Y.
, and
Kim
,
B.
,
2003
, “
An Uncertainty Evaluation for Multiple Measurements by Gum
,”
Accreditation Quality Assurance
,
8
(
1
), pp.
13
15
.
52.
Curless
,
B.
, and
Levoy
,
M.
,
1996
, “
A Volumetric Method for Building Complex Models from Range Images
,” SIGGRAPH, J. Fujii, ed., ACM, pp.
303
312
.
You do not currently have access to this content.