04-VR-Tracking.pdf

Virtual Reality Prof. Dr. Oliver Staadt Chair in Visual Computing 4. Tracking Overview Tracking, Calibration, and Registration Coordinate Systems Characteristics Stationary Tracking Systems Mobile Sensors Optical Tracking Sensor Fusion www.augmentedrealitybook.org Tracking...

Virtual Reality Prof. Dr. Oliver Staadt Chair in Visual Computing 4. Tracking Overview Tracking, Calibration, and Registration Coordinate Systems Characteristics Stationary Tracking Systems Mobile Sensors Optical Tracking Sensor Fusion www.augmentedrealitybook.org Tracking 3 Tracking, Calibration, and Registration Registration = alignment of spatial properties Calibration = offline adjustment of measurements Calibration Spatial calibration yields static registration no calib Offline: once in lifetime or once at startup n-s ra pa pati tion t S ra ic at tion t tra ram al of gis Alternative: autocalibration ck e s re ing ters ati p de of al vic es Tracking = dynamic sensing and Tracking Dynamic Registration measuring of spatial properties registration Tracking yields dynamic registration Tracking in AR/VR always means “in 3D”! www.augmentedrealitybook.org Tracking 4 Coordinate Systems Eye Local object coordinates coordinates Perspective transformation Calibrate offline For both camera and display Model transformation View transformation Track for moving objects, Track for moving objects, if there are static objects as well if there are no static objects Track for moving observer Global world coordinates www.augmentedrealitybook.org Tracking 5  Frames of Reference World-stabilized E.g., billboard or signpost Body-stabilized E.g., virtual tool-belt Screen-stabilized Heads-up display www.augmentedrealitybook.org Tracking 6 Measurement Coordinates Global vs. local measurements Global  larger (or unlimited) workspace Local  better accuracy Absolute vs. relative measurements Absolute  coordinate system defined in advance Relative  incremental sensing www.augmentedrealitybook.org Tracking 7 Physical Phenomena Electromagnetic radiation Visible light Infrared light Laser light Radio signals Magnetic flux Sound Physical linkage Gravity Inertia www.augmentedrealitybook.org Tracking 8 Measurement Principle Signal strength Signal direction Time of flight Absolute time Signal phase Requires synchronized clocks www.augmentedrealitybook.org Tracking 9 Degrees of Freedom (DOF) DOF = independent dimension of measurement Full tracking requires 6DOF 3DOF position (x, y, z) 3DOF orientation (roll, pitch, yaw) Some sensor deliver only a subset E.g., gyroscope  3DOF orientation only E.g., tracked LED  3DOF position only E.g., mouse  2DOF position only www.augmentedrealitybook.org Tracking 10 Measured Geometric Property Trilateration: 3 distances Triangulation: 2 angles, 1 distance P3 d3 M M d1 d2 α1 α2 P1 P2 P1 d12 P2 www.augmentedrealitybook.org Tracking 11 Sensor arrangement Multiple sensors in rigid geometric configuration E.g., stereo camera rig Sparse or dense sensors E.g., digital camera is dense 2D array of intensity sensor with know angles Advanced technical issues Sensor synchronization Sensor fusion www.augmentedrealitybook.org Tracking 12 Sensor Group Arrangement Outside-in Inside-out Stationary mounted sensors Mobile sensor(s) Good position, poor orientation Good orientation, poor position www.augmentedrealitybook.org Tracking 13 Signal Sources Passive sources Natural signals E.g., natural light, earth magnetic field Active sources Electronic components producing physical signal Can be direct or indirect (reflected) E.g., acoustic, optical, radio waves Most forms require open line of sight No sources Most important example: inertia www.augmentedrealitybook.org Tracking 14 Measurement error Accuracy How close is measurement to true value Affected by systematic errors Can be improved with better calibration Precision How closely do multiple measurements agree (random error, noise) Varies per type of sensor Varies per degree of freedom Can be improved with filtering (more computation, more latency) Resolution Minimum difference that can be discriminated between two measurements Cannot be reached in practice because of noise www.augmentedrealitybook.org Tracking 15 Temporal Characteristics Update rate: Number of measurements per time interval Measurement latency Time it takes from occurrence of physical event to data becoming available End-to-end latency Time it takes from occurrence of physical event to presentation of a stimulus www.augmentedrealitybook.org Tracking 16  102 Possible Tracking ErrorsP. Grimm et al. orange: drift blue: latency green: noise measurement time (From: Virtual und Augmented Reality) Stationary Tracking Systems Mechanical Tracking Electromagnetic Tracking Ultrasonic Tracking www.augmentedrealitybook.org Tracking 18 Mechanical Tracking Track end-effector of articulated arm Joints with 1, 2, or 3 DOF CyberGrasp Fakespace BOOM Rotary encoders or potentiometers High precision Fast Freedom of operation limited www.augmentedrealitybook.org Tracking 19 Electromagnetic Tracking Stationary source produces three orthogonal magnetic fields Current induced in sensor coils Measurement of strength and phase of signal Signal strength falls off quadratically with distance Working range: half-sphere with 1-3m radius Problems with electromagnetic interference www.augmentedrealitybook.org Tracking 20 Razer Hydra Ultrasonic Tracking Measures time of flight of sound pulse Trilateration of 3 measurements Requires synchronized time (cables) or more than 3 measurements Low update rate (10-50Hz) due to slow speed of sound Possible fusion with fast inertial sensors (e.g., InterSense IS-600) Requires open line of sight Suffers from noise or change of temperature Wide-area configuration, e.g., AT&T BAT system Microphones mounted in ceiling www.augmentedrealitybook.org Tracking 21 Image: Joseph Newman Mobile Sensors Global Positioning System Wireless Networks Magnetometer Gyroscope Linear Accelerometer Odometer www.augmentedrealitybook.org Tracking 22 Global Positioning System Planet-scale outside-in radio wave time-of-flight Requires clock synchronization Must receive signals from at least 4 satellites satellites GPS receiver www.augmentedrealitybook.org Tracking 23 Differential GPS Compensate for atmospheric distortion Receive correction signal from base station via network Real-Time Kinematics (RTK) Differential GPS also uses signal phase satellites GPS receiver base station correction signal www.augmentedrealitybook.org Tracking 24 Wireless Networks Measure signal strength from WiFi, Bluetooth, mobile phone towers Potential trilateration/triangulation Mostly only good for coarse location (e.g., based on WiFi SSID) Fingerprinting: carefully map the signal reception in a given area Recent use: Bluetooth iBeacon in department stores Assisted GPS: accelerate GPS initialization using WiFi or GSM id Skyhood, Google, Broadcomm etc. www.augmentedrealitybook.org Tracking 25  Magnetometer Electronic compass Measure direction of Earth magnetic field in 3D Principle: magnetoresistance (Hall effect) watch Often very distorted measurements www.augmentedrealitybook.org Tracking 26  Radial Gyroscopes movement Determines rotational velocity Coriolis movement Electronic gyro Image: Hideyuki Tamura Measures Coriolis force of small vibrating object Micro-electromechanical system (MEMS) High update rate (1KHz) Only relative measurements Must integrate once to determine orientation  drift Laser gyro (fiber-optic gyro) Measures angular acceleration based on light interference rotation Large, expensive, used in aviation www.augmentedrealitybook.org Tracking 27 Linear accelerometer MEMS device spring spring Displacement of small mass mass Measures Change of electric capacity, or Piezoresistive effect of bending Subtract gravity (the difficult part!) Integrate twice numerically to get position mass Drift problems Combine lin.acc., gyro + compass into inertial measurement unit (IMU) www.augmentedrealitybook.org Tracking 28 Odometer Mechanical or opto-electrical wheel encoder E.g., traditional ball mouse www.augmentedrealitybook.org Tracking 29 Optical Tracking Optical sensors Model-Based versus Model-Free Tracking Illumination Markers versus Natural Features Target Identification www.augmentedrealitybook.org Tracking 30 Optical Sensors Digital cameras are cheap and powerful CCD (charge coupled devices) – professional photography CMOS (complementary metal oxide semiconductor) – fast and cheap Computer vision techniques improve with Moore’s law Lenses are becoming the most limiting part Bayer pattern www.augmentedrealitybook.org Tracking 31 Model-Based versus Model-Free Tracking Model-based Model-free A tracking model representing At start-up, no tracking model is the 3D world is available available Compare the model to Most build a temporary tracking observations in the images model while tracking Measurements only relative to starting point www.augmentedrealitybook.org Tracking 32  ARTTrack Illumination Passive illumination Natural (or existing) light sources Visible spectrum 380-780nm Cannot track when it is too dark (mostly indoors) Active illumination Often infrared spectrum LED beacons Camera with infrared filter delivers high contrast Not suitable with sunlight Microsoft Kinect V1 Structured light Project a known pattern into the scene Projector with regular light or laser Laser ranging Measure time of flight taken by laser pulse Steerable MEMS mirror for scanning laser LIDAR (light radar): long range laser used in surveying www.augmentedrealitybook.org Tracking 33 Leap Motion 2 cameras, 3 infrared LEDs Short-distance reflection of the hands www.augmentedrealitybook.org Tracking 34 Valve/HTC Vive “Lighthouses” = two scanning infrared lasers Photodiodes on head pick up lasers www.augmentedrealitybook.org Tracking 35 Markers vs Natural Features Fiducials markers Artificial tracking targets Square shapes yield 4 points (enough for pose ) Circular shapes yield only 1 point Image: Daniel Wagner Digital marker model exists first, marker manufactured second (e.g., printing) Natural feature tracking Existing visual features in the environment Physical features exist first, tracking model reconstructed second www.augmentedrealitybook.org Tracking 36 Image: Andrei State, UNC Chapel Hill Flat Marker Designs Image: Daniel Wagner www.augmentedrealitybook.org Tracking 37 Retro-Reflective Ball Markers Light reflected towards light-source Illuminate with infrared LED flash Infrared camera observes bright blobs 4 or more spheres in known configuration to recover 6DOF pose Multiple targets distinguished by their geometric configuration www.augmentedrealitybook.org Tracking 38 Advanced Realtime Tracking GmbH  Image: Martin Hirzer Natural Features Detect salient interest points in image Must be easily found Location in image should remain stable when viewpoint changes Requires textured surfaces Alternative: can use edge features (less discriminative) Match interest points to tracking model database Database filled with results of 3D reconstruction Matching entire (sub-)images is too costly Typically interest points are compiled into “descriptors” www.augmentedrealitybook.org Tracking Image: Gerhard Reitmayr 39 Marker Target Identification More targets or features  more easily confused Must be as unique as possible Image: Mark Fiala Square markers 2D barcodes with error correction E.g., 6x6=36 bits (2 orientation, 6-12 payload, rest for error correction) Marker tapestries Spherical targets 5 spheres in different geometric configurations Can distinguish 10-20 targets Pulsed LEDs Image: Greg Welch, UNC Chapel Hill www.augmentedrealitybook.org Tracking 40 Natural Feature Target Identification Individual natural interest points too easily confused Rely on co-occurency of interest points for detection Probabilistic search methods used to deal with errors Vocabulary trees Random sampling consensus Image: Martin Hirzer www.augmentedrealitybook.org Tracking 41 Complementary Sensor Fusion Combining sensors with different degrees of freedom Sensors must be synchronized (or requires inter-/extrapolation) E.g., combine position-only and orientation-only sensor E.g., orthogonal 1D sensors in gyro or magnetometer are complementary www.augmentedrealitybook.org Tracking 42 Competitive Sensor Fusion Different sensor types measure the same degree of freedom Redundant sensor fusion Use worse sensor only if better sensor is unavailable E.g., GPS + pedometer Statistical sensor fusion (see next slide) www.augmentedrealitybook.org Tracking 43 Statistical Sensor Fusion Important form of competitive fusion for higher quality Combine measurement to improve quality Establish statistical estimate of the true system state Predict future system state  Correct from observation (measurement) Extended Kalman filter for Gaussian error distribution Unscented Kalman filter for highly non-linear systems Particle filter for systems with multiple state hypothesis E.g., maintain estimate with fast IMU + update when slow computer vision results come in www.augmentedrealitybook.org Tracking 44  Cooperative Sensor Fusion Primary sensor relies on information from secondary sensor to obtain its measurements E.g., A-GPS combines celltower + GPS Combination of inside-out + outside-in Stereo cameras with known epipolar geometry PointGrey LadyBug Non-overlapping cameras (e.g., 360°) Indirect sensing (cont’d) www.augmentedrealitybook.org Tracking 45 Image: Georg Klein Next time: Kinect