Can I get the 3D position of a Custom Object, a Ball or a Small Object?

The ZED SDK provides the left and right rectified images of the scene and computes the 3D position of each pixel. When an object is too far or too small, the SDK will detect that the estimation isn't accurate and will leave its pixels as not-estimated.

As everyone has custom needings for the detection of their specific object, the ZED SDK isn't providing features to locate it in the image. This part is up to the user, and we're recommending to use OpenCV with color segmentation or shape detection.

 

My object appears in the depth map

To know if the depth of your object is estimated in every situation, you can open the Depth Viewer tool, and take a look at the bottom-left depth map.

  • Windows: C:\Program Files (x86)\ZED SDK\tools
  • Linux: /usr/local/zed/tools

You should also make sure to select the higher resolution possible that fit your framerate needings.

Resolution Framerates
HD2K 15 fps
HD1080 30/15 fps
HD720 60/30/15 fps
WVGA 100/60/30/15 fps

 The top-right gear icon will allow you to use various depth settings, with ULTRA being the most accurate mode.

Once you know that your object is always visible in the depth map, you can start writing your program that will detect it in the images. You can start with our OpenCV sample.

When you have detected the position of your object in the images, you can query the ZED SDK for its 3D position. To do so, the easiest way it to retrieve the point cloud from the SDK and read the values in the point matching the pixel position. A sample displaying the position of the center pixel is available in out Depth Sensing tutorial.

 

My object is too small to appear in the depth map

We already tried some small object detection and using the depth map isn't always the best solution.
We figured out that detecting the position of the object in the right and left image is more robust. Once the position of the object is known in the two images, thanks to the ZED calibration, you just have to compute the disparity of the two centers to obtain the depth.

disparity = abs(centerLeft.x - centerRight.x)
depth = (zed.getCameraInformation().calibration_parameters.left_cam.fx * zed.getCameraInformation().calibration_parameters.T.x) / (-disparity);

You can then convert this pixel + depth information to a 3D coordinate:

float z=depth;
float x = (( centerLeft.x - cx) * z) / (fx);
float y = (( centerLeft.y - cy) * z) / (fy);


fx,fy,cx,cy are corresponding to the left camera parameters, you can access them from zed.getCameraInformation().calibration_parameters.left_cam
Check the API documentation for more.

You can also compute the distance of the camera to this point using:

float distToPoint = sqrtf(x*x + y*y + z*z);

 

When using the disparity method to estimate the 3D position of an object, the higher the image resolution is, the more accurate the results will be.