Presentation
Cross-Layer Innovations for Enabling Real-Time and Efficient Eye Tracking in VR/AR
DescriptionEye tracking has become an essential human-machine interaction modality for providing an immersive experience in numerous virtual and augmented reality (VR/AR) applications requiring high throughput (e.g., 240 FPS), a small form factor, and enhanced visual privacy. Despite the growing demand for such eye tracking systems, prior arts are still limited in their achievable throughput (e.g., still <30 FPS) and thus cannot fully satisfy the desired real time performance requirements, e.g., >240 FPS for supporting frequent and essential human-machine interaction on mobile AR/VR devices with limited computing resources. In this talk, I will present a first-of-its-kind intelligent eye-tracking system, dubbed i-FlatCam, integrating a lensless camera of a much smaller form factor and an eye-tracking acceleration chip. In particular, i-FlatCam features (1) a predict-then-focus eye tracking pipeline to boost efficiency without compromising accuracy, (2) a unified compression scheme to enable fully on-chip processing and improved FPS, and (3) a dedicated intra-channel reuse design for depth-wise convolutional layers to increase hardware utilization. As a highlight, i-FlatCam demonstrates the first eye-tracking pipeline with a lensless camera and achieves 3.16 degrees of award-winning accuracy, 253 FPS, 91.5 µJ/Frame, and a 6.7mm×8.9mm×1.2mm camera form factor, paving the way for next-generation AR/VR devices. Our i-FlatCam system won first place in the University Demo Best Demonstration at DAC 2022. Finally, I will conclude my talk by briefly introducing a more general eye-tracking system for AR/VR, which was recently selected as an IEEE Micro Top Pick of 2023.
Event Type
Special Session (Research)
TimeWednesday, July 12th2:30pm - 3:00pm PDT
Location3001, 3rd Floor
AI
