Ch5 Teleoperation PDF

5 Teleoperation What is a telesystem? Teleoperation– it is a “What is hnecessary/temporary evil” or a different style of AI? uman supervisory control? What is semi-autonomy and h...

5 Teleoperation What is a telesystem? Teleoperation– it is a “What is hnecessary/temporary evil” or a different style of AI? uman supervisory control? What is semi-autonomy and how it is different? What types of domains is teleoperation good for? © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 1  5 Specific Learning Objectives Be able to label the 7 components of a telesystem Define telesystem, supervisory control, teleoperation, shared Motivation control, traded control, remote control, telepresence, Components Control proprioception, exteroception, exproprioception -RC -Manual -Shared If given a description of a supervisory control scheme, classify -Autonomous Case Study it as manual, traded, shared, or autonomy Semi-autonomy -OOTL Summary Give at least 3 characteristics of a domain that is suitable for telesystems Describe the out of the loop control (OOTL) problem and why it is a concern for semi-autonomy © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 2  5 Outline Theory – What is Teleoperation? – Motivation Components Control 7 Components of a telesystem -RC -Manual – Types of supervisory control -Shared -Autonomous Case Study – Summary Semi-autonomy -OOTL Summary Practice – Case Studies – Human Out Of The Loop Control problem – Assessment of where telesystems make sense © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 3  5 Recap: AI AI is good at lower level behaviors and skills (reactive layer), and at symbolic reasoning Motivation Components (deliberative layer) Control -RC -Manual But not so good at converting sensor data into -Shared -Autonomous symbols Case Study Semi-autonomy -OOTL Summary © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 4  5 Solution? AI is good at lower level behaviors and skills (reactive layer), and at symbolic reasoning Motivation Components (deliberative layer) Control -RC -Manual But not so good at converting sensor data into -Shared -Autonomous symbols Case Study Semi-autonomy -OOTL Summary …Telesystems or teleoperation: – Where human and robot are physically separated and must interact to accomplish the task – Connotation is human must interact THRU the robot © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 5  5 Where We’re Going… Teleoperation is the state of the practice in most DoD and Public Safety mobile robotics Motivation Components It is important to understand it both as a legitimate, Control -RC -Manual stand-alone approach to robotics for remote -Shared -Autonomous presence applications but also as a transitional path Case Study Semi-autonomy -OOTL to autonomous taskable agents Summary Next podcast, it is important to understand why teleoperation is unlikely to be the fail-safe mode for most robotics, even line-of-sight Later in the course, discuss Human-Robot Interaction © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 6 5 TASKABLE AGENT VERSUS REMOTE PRESENCE © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 7  5 Definitions Taskable agent – Given a task Motivation Components – Executes without supervision – Returns (with information) Control -RC -Manual -Shared -Autonomous Case Study Semi-autonomy -OOTL Examples Summary – Deep Space One probe © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 8  5 Definitions Remote presence Motivation Components – Human and robot share the task and the role(s) Control -RC -Manual – Task execution is blended (joint cognitive system) -Shared -Autonomous Case Study Semi-autonomy -OOTL Examples Summary © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 9  5 Difference Taskable agent – Given a task Motivation Components – Executes without supervision – Returns (with information) Control -RC -Manual -Shared -Autonomous Case Study Semi-autonomy -OOTL Remote presence Summary – Human and robot share the task and the role(s) – Task execution is blended (joint cognitive system) The two are not mutually exclusive– (Not mutually exclusive means that two instances or outcomes can occur simultaneously.) © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 10  5 Relevance for Teleoperation Taskable agent Teleoperation is a Motivation Components – Given a task “workaround,” Control -RC – Executes without supervision especially for -Manual -Shared -Autonomous – Returns (with information) perceptual Case Study Semi-autonomy deficiencies -OOTL Summary Remote presence Teleoperation is the – Human and robot share the desired end-state task and the role(s) – Task execution is blended (joint cognitive system) © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 11 5 A COMPONENTS OF A TELESYSTEM © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 12  Components of a Telesystem 5 (after Uttal 89) Remote Motivation Components Control Local -RC Communi- Sensor -Manual -Shared -Autonomous Display cation Case Study Semi-autonomy -OOTL Control Summary Control Effector Power © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 13  7 Components of a Telesystem 5 At Local (2) – display Remote – local control device Local Communication (1) Communi- Sensor At Remote (4) Display cation – remote control Control device – sensor Control – effector Effector – power Power © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 14  5 Hot New Trend! Motivation Components Control -RC -Manual -Shared -Autonomous Case Study Semi-autonomy -OOTL Summary © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 15  A Bit Harder With 5 Non-Anthropomorphic Systems Motivation Components Control -RC -Manual -Shared -Autonomous Case Study Semi-autonomy -OOTL Summary Remote Local © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 16  5 How Do You Control A Telesystem? Referred to as: Human supervisory control Motivation Components – Term is not limited to unmanned vehicles Control -RC Manned systems: Auto-pilot; fly-by-wire -Manual -Shared -Autonomous Factory automation Case Study Semi-autonomy -OOTL Summary © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 17  5 Supervisory Control Definition (after Sheridan 92) – One or more human operators are intermittently giving directives Motivation and continually receiving information from a computer that itself Components closes an autonomous control loop through artificial effectors and Control -RC sensors to the controlled process or task environment -Manual -Shared -Autonomous Ramifications Case Study Semi-autonomy – Human is always involved, if only to set objectives -OOTL Summary Ex. NASA MER – Information may be the lack of information (e.g., nothing interesting has happened) – Computer is always involved. Ways include: Project effect of actions Compensate for time delays Inner-loop control Safety/self-protection reflexes © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 18  5 Types of Human Supervisory Control Can you see the robot in its environment? Where is the major part of the intelligence: the local (operator) Motivation or the remote (robot)? Components Control -RC -Manual -Shared -Autonomous Case Study Robot Semi-autonomy Primary -OOTL Summary controller Operator Primary controller Operator Operator Can see robot Can’t see robot © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 19  Robot Social Semi-Autonomous Primary controller Interactions Control Human Primary Remote Control Manual control controller Human Human Can see robot Can’t see robot 20  5 Types of Human Supervisory Control This is what “human supervisory control” generally refers to Supervisory Control Motivation Components Control Autonomy -RC Robot -Manual -Shared Primary Social -Autonomous controller Interactions Shared control Case Study Semi-autonomy Traded control Teleoperation -OOTL Summary Operator Primary Remote Controlled Manual control controller Operator Operator Can see robot Can’t see robot © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 21  5 Types of Human Supervisory Control Supervisory Control Motivation Components Control Autonomy -RC Robot -Manual -Shared Primary Social -Autonomous controller Interactions Shared control Case Study Semi-autonomy Traded control Teleoperation -OOTL Summary Operator Primary Remote Controlled Manual control controller Operator Operator Can see robot Can’t see robot © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 22  Supervisory Control 5Robot Primary controller Autonomy Shared control Manual Control Teleoperation Traded control Operator Primary Manual control controller Operator Can’t see robot Motivation Components Control -RC -Manual Sometimes -Shared called a -Autonomous Case Study “telefactor” Semi-autonomy -OOTL Remote Summary Local Sensor Communi- Display cation Only inner Control loop Control Effector Power © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 23  5 Downside of Manual Control You can’t see the robot but the interfaces are insufficient to be aware of where the robot is Motivation Components You need: Control -RC -Manual – Proprioception: measurements of the movements relative -Shared -Autonomous to an internal frame of reference Case Study Semi-autonomy – Exteroception: measurements of the layout of the -OOTL Summary environment and object relative to the robot’s frame of reference – Exproprioception: measurement of the position of the robot body or parts relative to the layout of the environment. © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 24  Remote Sensor Local Only inner Communication loop Display Control Control Effector Power 25 Teleoperator and Telefactor can trade places controlling a task Remote Sensor Local Communication Display Control Control Effector Power 26 Teleoperator and Telefactor shared control for a task Remote Sensor Local Communication Display Control Control Effector Power 27  5 Example: Manual Control Motivation Components Control -RC -Manual -Shared -Autonomous Case Study Semi-autonomy -OOTL Summary © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 28  Supervisory Control 5Robot Primary controller Autonomy Shared control Traded Control Teleoperation Traded control Operator Primary controller Manual control But here when needed Mostly here Operator Can’t see robot Motivation Components Control -RC -Manual -Shared -Autonomous Case Study Semi-autonomy -OOTL Remote Summary Local Sensor Communi- Display cation Control Control Effector Power © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 29 5 Teleoperator and Telefactor can trade places controlling a task Remote Sensor Local Communication Display Control Control Effector Power © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 30  5 Mars Sojourner: Traded Control Motivation Components Control -RC -Manual -Shared -Autonomous Case Study Semi-autonomy -OOTL Summary © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 31 5 Shared Control In shared control, both the teleoperator and the telefactor contribute simultaneously to the control. Typically, the teleoperator provides deliberative inputs and the telefactor provides reactive control, essentially splitting responsibilities along the layers of the canonical software organizational architecture in chapter 4. Antilock brakes can be considered a form of shared control where the car can react to a skid faster than a human can and can execute the correct response of pumping the brakes more reliably © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 32 5 Guarded Motion An important form of control that has been used since 1980 is guarded motion. Guarded motion is defined as a type of human- in-the-loop control, where the robot guards itself from unintended consequences of human directives. © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 33 5 Guarded Motion guarded motion has five components: 1. Autonomy Intervention Criteria 2. Command Integration Method 3. Monitored Condition 4. Interface Modality 5. Display Preprocessing © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 34 5 Human Factors Cognitive Fatigue Latency Human: Robot Ratio human out-of-the-loop control problem © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 35 5 Cognitive Fatigue Keyhole effect Simulator sickness © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 36 5 Latency One problem with teleoperation for space applications, telemedicine, or applications over long distances is that it can be inefficient or dangerous because of large time delays. A large time delay can result in the teleoperator giving a remote command, unaware that it will place the remote in jeopardy or that an unanticipated event, such as a rockfall, might occur and destroy the robot before the teleoperator can see the event and command the robot to flee. A rule of thumb, or heuristic, is that the time it takes to do a task with traditional teleoperation grows linearly with the transmission delay. A teleoperation task that took 1 minute for a teleoperator to guide a remote to do on the Earth might take 2.5 minutes to do on the Moon, and 140 minutes on Mars. © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 37 5 Human: Robot Ratio Safe human-robot ratio is as follows: Nh= Nv+ Np+1 where Nh is the number of humans, Nv is the number of vehicles, and Np is the number of payloads. © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 38 5 Human Out-of-the-Loop Control Problem Human out-of-the-loop (OOTL) control studies since 70’s show that people may not seamlessly not take over from automation failures © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 39 5 Human Out-of-the-Loop Control Problem DarkStar unmanned aerial vehicle. (Photograph courtesy of DefenseLink Office of the Assistant Secretary of Defense-Public Affairs.) © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 40 5 Guidelines for Determining if a Telesystem Is Suitable for an Application According to The Concise Encyclopedia of Robotics, teleoperation is best suited for applications where: 1. The tasks are unstructured and not repetitive. 2. The task workspace cannot be engineered to permit the use of industrial manipulators. 3. Key portions of the task intermittently require dexterous manipulation, especially hand-eye coordination. © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 41 5 Guidelines for Determining if a Telesystem Is Suitable for an Application 4. Key portions of the task require object recognition, situational awareness, or other advanced perception. 5. The needs of the display technology do not exceed the limitations of the communication link (bandwidth, time delays). 6. The availability of trained personnel is not an issue. © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 42  Mini-Summary: 5 Telesystem Theory What is teleoperation or telesystems? Motivation Components Is it a “temporary evil” or a different style of AI? Control -RC – a popular, immediate alternative to full autonomy in -Manual -Shared taskable agents -Autonomous Case Study – Telesystems are the “end state” for remote presence Semi-autonomy -OOTL applications; joint cognitive system Summary What is Human supervisory control? – how you control a telesystem. The distinctions are not so important because often shift dynamically during a mission. Manual control Traded, Shared Full autonomy © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 43  5 Summary Telesystems are a popular, immediate alternative to full autonomy Motivation Components Control Human supervisory control is how you control a -RC -Manual -Shared telesystem -Autonomous Case Study Semi-autonomy -OOTL Supervisory control usually refers to the spectrum Summary from manual control to autonomy Problems include: cognitive fatigue, communications and many:one human to robot ratios Teleoperation or shared control is generally not a reliable “fail safe” or backup plan because of the OOTL control problem © 2019 Robin Murphy Introduction to AI Robotics 2nd Edition (MIT Press 2019) 44