How does the ZED 3D perception work?

ZED stereo cameras are passive stereo vision cameras that reproduce the way human binocular vision works. They embed two synchronized image sensors separated by a fixed, factory-calibrated distance called the baseline. By matching the same visual features in the left and right images and measuring their horizontal shift (the disparity), the ZED SDK computes the distance of every pixel through triangulation, building a dense three-dimensional model of the observed scene in real time.

Because ZED cameras rely on ambient light and natural scene texture instead of projecting infrared patterns or laser pulses, they are fully passive sensors; multiple cameras can operate side by side without interfering with each other, indoors and outdoors, even in direct sunlight.

How is the depth map generated?

The camera provides two hardware-synchronized color video streams (left and right views): the USB cameras (ZED 2i, ZED Mini) send them as a side-by-side stream, while the GMSL2 cameras (ZED X, ZED X Mini, ZED X Nano) are processed by the ISP of the NVIDIA® Jetson™ host. The ZED SDK rectifies the two images using the factory calibration, performs stereo matching on the GPU to compute a disparity map, and converts it into a metric depth map. The AI-based NEURAL depth modes significantly improve robustness on low-texture and reflective surfaces; see Depth Modes.

Developers have simultaneous access to:

• The synchronized left and right color images

• The dense depth map and confidence map

• A 3D point cloud with color and surface normals

All outputs are spatially and temporally aligned; there is no need to register a separate RGB sensor to the depth data, as required by most active depth cameras.

How does stereo vision compare to other depth sensing technologies?

Active depth sensors based on structured light or Time-of-Flight are typically limited to short range and indoor use, since sunlight overwhelms their infrared signal. LiDAR performs well outdoors but provides sparse geometric data without color and at a higher cost.

Passive stereo combines the strengths of both worlds: depth perception from a few centimeters (ZED X Nano) up to 35 m (ZED X and ZED 2i with narrow lenses), full-resolution color and depth from the same sensor pair, and reliable operation both indoors and outdoors. This makes ZED cameras suitable for robotics, AR/VR, video analytics, security, VFX, logistics, agriculture, and 3D scanning applications.

Can I use the ZED for people counting and tracking?

Yes. The long depth range of stereo vision makes ZED cameras a good fit for people counting and tracking in large areas. In addition, the ZED SDK includes dedicated Object Detection and Body Tracking modules that combine AI detection with 3D depth information to localize and track people in metric space, with unique IDs and 3D skeleton data. Keep in mind that fine segmentation of small body parts, such as fingers, is limited at long range because disparity resolution decreases with distance.

Best practices for stereo depth sensing

• Working range: each model has an optimal range defined by its baseline and focal length; see For the depth sensing, what is the range for the ZED cameras? As a rule of thumb, depth accuracy is highest in the near range; place the camera as close to the region of interest as the application allows.

• Scene texture: stereo matching needs visual features. Homogeneous, textureless, or specular surfaces (white walls, green screens, glass, mirrors) can generate unstable or missing depth. Adding texture or using the NEURAL depth modes mitigates this.

• Lighting: ZED cameras use color images, not IR, for depth perception; avoid very low-light environments, or add scene illumination. Outdoors, direct sunlight is not a problem.

• Fast motion: to capture fast movements, use the high frame rate modes (HD720 @ 60 FPS or WVGA @ 100 FPS on the USB cameras, SVGA @ 120 FPS on the GMSL2 cameras) to reduce motion blur, which degrades stereo matching. The global shutter sensors of the ZED X family are free from rolling shutter distortion.

• Cable length: the ZED 2i supports USB 3.0 cables up to 10 m without external power supply; see Can I use the ZED camera with a USB 3 extension cable? For long cabling and multi-camera setups, the ZED X family uses a GMSL2 interface that transmits video up to 15 m with low latency.

What is the depth accuracy?

Stereo depth is estimated by triangulation from the disparity image. The depth resolution follows this relation:

Dr = Z² * alpha

where Z is the distance to the target and alpha is a constant that depends on the baseline, the focal length, and the matching sub-pixel precision.

In practice, depth error grows quadratically with distance. The accuracy specified for each camera model is reported in the depth range article and in the datasheets. Accuracy can also be affected by outlier measurements on homogeneous and textureless surfaces, which usually produce temporal instability in the depth values; the SDK confidence map can be used to filter these unreliable pixels.