Tracking jumps in Unity when using VR headset in AR Passthrough mode

In AR pass-through mode, the ZED Plugin for Unity continuously compares data from the ZED Mini IMU against the position of the VR headset. This is used to compensate visual latency and better align virtual VR controllers on their real-world counterpart.

However, there are cases where the IMU's orientation and the headset can get permanently out of sync. This causes the tracking to shake as it tries to correct the error. This can happen if:

  • The mount for the ZED Mini was not properly attached to the headset.
  • You are using a headset for which we don't offer a mount, and did not center it properly - particularly if the front of the headset is curved.
  • The mount on the headset is loose and shifts at runtime.
  • Rarely, the ZED Mini's IMU may have a calibration issue.

If this is the source of the problem, then you will see sharp movements constantly, even if the headset is still and there is nothing moving in your environment. Sometimes the issue can start after a minute or two into running your scene.

 

 

Correct Alignment Issues

If one of the above points is true, then the best solution is to reposition the ZED Mini as necessary to fix its alignment. This should not only solve the tracking issues, but improve the pass-through experience as well.

If using an Oculus Rift or HTC Vive, make sure the mount aligns well with the included mounting template sticker. If you need replacement mounts or templates, you can purchase them as part of our VR Kit. If using a Vive Pro, you can download a 3D-printable mount for it here.

If using another headset, make sure that the ZED Mini is facing straight forward, and is adhered strongly enough that it won't move at runtime. If you make either of the above adjustments, run your scene for a few minutes while the headset is on a flat surface.

 

Disable Offset Check

The Unity plugin sets the HMD's rotation as a "prior" for the ZED Mini's IMU. Commenting out the line where this happens will remove the tracking issues.

In ZEDMixedRealityPlugin.cs, go to the AdjustTrackingAR() method and comment out the following line (line 603 as of writing):

zedCamera.SetIMUOrientationPrior (ref latencyPose.rotation);

This should solve the problem immediately. If not, your ZED Mini is likely rotated significantly off-center. It's better to adjust the camera's physical position to solve this, but if you can't, then go to the InitTrackingAR() function and comment out this line (line 554 as of writing):

dllz_drift_corrector_set_calibration_const_offset_transform(ref const_offset);

If neither Option 1 or Option 2 helps you, send an email to support@stereolabs.com with details.