Deep Learning in Augmented Reality

Deep Learning-Based Object Detection in Augmented Reality

• Recent advancements in augmented reality (AR) and artificial intelligence (AI) are driving innovation across various fields and industries.
• Computer vision (CV) and AR technologies enable analysis and understanding of environments.
• This study systematically reviews research integrating AR/mixed reality (MR) and deep learning for object detection.
• Sixty-nine relevant papers were analyzed, focusing on: application analysis of deep learning-based object detection; and analyzing object detection computations on servers or local AR devices to understand the relationships between algorithms and AR technology.
• The advantages of deep learning-based object detection in solving AR problems and limitations hindering its widespread adoption are critically assessed.
• The integration of AR and CV shows promise for future innovation.

Augmented Reality Technologies and Devices

• AR technologies, first introduced in the 1960s, have advanced significantly in recent decades.
• AR systems comprise an image generating optical unit and a projection surface for displaying virtual content.
• AR devices are categorized into wearable devices, handheld devices, projection-based displays (spatial AR), and holographic displays.
• Wearable AR devices (headsets, goggles, glasses) are prominent, allowing hands-free interaction with digital content overlaid on the real world.
• Handheld devices (smartphones, tablets) offer simpler AR experiences, suitable for everyday use in navigation and gaming.
• Projector-based displays allow augmenting any surface without a mediating device.
• Holographic displays generate 3D content using diffractions of light, also capable of creating interactive experiences without intermediary devices.

Deep Learning-Based Object Detection

• Deep learning, particularly convolutional neural networks (CNNs), is frequently used in object detection.
• CNN algorithms break down images into segments, processing each portion to identify and classify objects.
• Region-based CNNs (R-CNN) use region-of-interest (ROI) to target object detection, improving speed.
• Fast R-CNN and Faster R-CNN algorithms enhance efficiency by reducing the number of regions to process.
• Mask R-CNN extends Faster R-CNN by adding a mask branch for object segmentation, enabling pixel-level accuracy.
• You Only Look Once (YOLO) processes entire images, offering fast real-time detection.
• Single Shot Multibox Detector (SSD) provides higher processing speed by eliminating the need for region proposal networks.

Review Methodology

• The review adhered to PRISMA guidelines to select relevant literature (Scopus, Web of Science, IEEE Xplore, ACM, and ScienceDirect).
• Research articles focused on both server/cloud-based and local device-based computation.
• 4835 initial records were reduced to 69 after rigorous screening based on relevance.

Object Detection in Augmented Reality

• Deep learning finds application in several industries.
• Manufacturing: AR-based systems using deep learning for assembly processes, tool/part detection, and providing instructions can enhance efficiency and reduce human errors
• Driving and Autonomous Vehicles: obstacle detection in augmented reality HUDs can enhance awareness and possibly reduce accidents. Algorithms detect signs, obstacles, and lanes to enhance driving safety.
• Assistance: AR used for navigation (especially for those with visual impairments); elderly assistance; or interactive product browsing in retail settings.

Computation Platforms

• Studies use either local AR devices or remote servers for object detection computations, each with advantages and disadvantages.
• Remote server processing typically offers faster speeds and greater computational power.
• Local device processing provides faster response times suitable for demanding applications/environments with low latency.

Study Results and Evaluation Metrics

• Most studies for object recognition involve images/video.
• Some studies focused on point clouds for 3D object detection.
• A combination of factors - computation efficiency, performance (accuracy/precision), and subjective user feedback (e.g., usability, error rates) - used for evaluating AR applications.
• Publication rates for this topic increased after 2016 and peaked in 2019-2020

Discussion and Conclusion

• Key challenges include: computational resource requirements, limited training datasets/models, and challenges related to latency and mobile/wearable device power limitations for real-time performance.
• User studies are scarce.
• The need for subjective measure is emphasized. Further consideration of user experience should enhance deep learning-based object detection in augmented/mixed reality.