A report appeared in 2018 that Apple was working on mixed reality glasses. It was also reported that it was working on a headset with an 8K display for each eye and expected to release in 2020. Now, US Patent & Trademark Office (USPTO) has published the Apple patent application for Mixed Reality headset which appears to have futuristic features such as head-mounted display, an array of sensors to track the eyes, gestures and facial expressions and a controller to merge Virtual Reality with Augmented Reality. The application, titled “Display System Having Sensors” which was first reported by Variety, was filed on 21st March, 2019 and the patent published last week.
Details of Apple Patent Application for Mixed Reality Headset
The patent also explains the plans to utilize an array of sensors data to gather information from the wearer of Mixed Reality headset. As per the report, the headset might include right and left displays, light sensors to gather info like intensity, color and directions, eyebrow sensors to track facial expressions, world mapping sensors to track movement and location, eye-tracking sensors, lower jaw sensing units to track expression, head posture sensing units to track a user’s orientation and motion, hand sensors to track position, movement and gestures, an inertial measurement system unit to augment the info, and left and right cameras.
Such sensing units would make it possible for Apple to more reasonably recreate a user’s facial expression in Mixed Reality. Apple has recently developed an animated AR emoji application for tracking facial expression. The iPhone’s selfie cam tracks the facial expressions, and after that translate those expressions to animated emoji. This animated feature is known as Animoji.
The only obstacle with the approach is that users are unable to capture their face when they are wearing the headset. That’s the reason why Apple aiming to combine the data from different sensing units. The eye-tracking cameras would also be used for biometric authentication. As per the application it may use cameras for gesture tracking also.
The info collected by the sensor on the head-mounted display is then sent as input to a controller which then renders the frames including the virtual content based upon the info collected.
It also reported that the controller may have single or multi-processor. As per the patent application “The controller is configured to render an avatar of the user’s face for display in the 3D virtual view based at least in part on information collected by the one or more eye tracking sensors, the one or more eyebrow sensors, and the one or more lower jaw sensors.”
The VR system may use LCD, digital light processing or liquid crystal on silicon technology — or even “a direct retinal projector system that scans left and right images, pixel by pixel, to the subject’s eyes.”
There’s still a lot of variables, and there’s no guarantee that Apple is certainly developing a headset matching these specs. The Apple Patent Application for Mixed Reality Headset tosses affordable light on the tech giant’s strategies for its AR/VR item line-up.