Apple wins patent for a scene camera system for a mixed reality headset that includes a 2D array of cameras

Today, the US Patent and Trademark Office has officially granted Apple a patent for a new HMD camera system that includes a two-dimensional array of cameras that take pictures of specific parts of a scene.

Today, the US Patent and Trademark Office has officially granted Apple a patent for a new HMD camera system that includes a two-dimensional array of cameras that take pictures of specific parts of a scene.

The cameras are positioned along a spherical surface so that the cameras have adjacent fields of view. The cameras’ entrance pupils are positioned at or near the user’s eye while the cameras also form enhanced images on the sensor. Methods for reducing the number of cameras in the matrix are also described, as well as methods for reducing the number of pixels read from the matrix and processed by the pipeline.

While today’s patent report can be appreciated by all Apple fans, the patent will be especially appreciated by the engineers, optics engineers, and technologists who go after all things VR headsets.

Simulated environments for virtual reality systems and/or mixed environments for mixed reality systems can be used to provide an interactive user experience for multiple applications, such as applications that add virtual content to a real-time view of a viewer’s environment, applications that create 3D virtual worlds (Metaverse), and interact with environments Virtual training, games, remote control of drones or other mechanical systems, display of digital media content, interaction with the Internet, exploration of landscapes or virtual environments, or the like.

Apple’s patent covers scene cameras for video head-mounted displays (HMDs) that can be used in mixed reality (MR) or virtual reality (VR) systems.

On traditional HMDs, one or more sight cameras can be mounted to the front of the HMD. However, the point of view (POV) of scene cameras is greatly compensated for and differs from the POV of the user’s eyes. Apple’s invention corrects the point of view (POV) of cameras to match the user’s POV by shifting the cameras’ entrance pupils toward the user’s eyes.

In some models, the HMD includes two-dimensional arrays of small form factor cameras (for example, one matrix for the left eye, a second matrix for the right eye) that capture images of specific parts of a scene from the real world in front of the user. The cameras are positioned along a curve or spherical surface so that the cameras have non-overlapping, contiguous fields of view (FOVs).

To achieve a more accurate representation of the user’s perspective, the cameras’ optics are configured so that the cameras’ entrance pupils are positioned in the matrix behind the cameras’ image planes formed at the image sensors, and at or near the user. the eye while the cameras also compose enhanced images on the sensor. In this way, each set of cameras capture views of the scene from the same perspective as the user’s eye.

In some models, one sensor can be used to take pictures of several cameras (for example, four) in the matrix. In these models, the optics of cameras used with a sensor can be shifted or modified to align image planes with the sensor surface.

In some embodiments, a curved sensor and simpler lens system can be used to provide a wider field of view for each camera and thus reduce the number of cameras in the array.

Apple FIG patent. Figure 1a below shows a head-mounted monitor (HMD) that includes an array of cameras with entrance pupils at or near the user’s eye; Figs. 2 shows part of the camera matrix.

2 patents for HMD 1A 2 . camera

Apple FIG patent. Figure 3 below shows an example of a camera with an entrance pupil at or near the user’s eye that can be used in a matrix as shown in the figures. 1a and 2.

(Click on the image below to enlarge)

3 fig.  3 patents for hmd camera

Fig apple. 9A and 9B below demonstrate reducing the number of cameras by decreasing resolution and shifting the input pupil of the cameras into the peripheral regions.

4 Apple HMD Patent Camera 9A B 16 17

Apple FIG patent. Figure 16 above graphically shows the addition of a passive meniscus lens in front of the camera assembly to increase the field of view (FOV) for perimeter cameras; Figs. Figure 17 graphically illustrates the addition of a negative meniscus lens in front of the camera array to gradually compensate the cameras’ view towards the periphery.

Fig Apple patented. 20A to 20C below graphically illustrates an example of a scene camera that includes a negative meniscus lens in front of the cameras in the camera assembly.

5 Apple Tin Patent 20A BC HMD Camera System

For more details, see Apple patent 11,448,886.

Few of the inventors of Apple

Dan Henigan: Optical Mechanical Engineering Manager

Saito Kenichi: Optical Engineer (previously at Canon and Fuji Photo Film)

Brett Miller: Camera Incubator Engineering Manager

Noah Bedard: Prototyping Engineer

Kathryn Berkner: Senior Engineering Manager – Camera Incubator

10.52FX - Stripe Patent Granted

Leave a Comment