Working with colour and depth

Questions & Answers
1

Hi!
I have a C57 and I’m trying to integrate the nxLib SDK into our software using C++.
I have a few questions:

  1. We need both colour and depth frames. When initialising the camera, do we have to find the serial number for the stereo device and run cmdOpen with that as a parameter, then run another cmdOpen with [serial]-Color? Then when streaming does cmdCapture work with both depth and colour or do we need to do something for each? Do we have to get the depth frame from the depth device and colour frame from the colour device? Sorry if I missed a sample or guide that explains this, I have tried looking but not found anything.

  2. For various reasons (our software is used with multiple 3D camera SDKs) we handle frame alignment and projection/deprojection ourselves so we need the colour and depth intrinsics, and the extrinsics (rotation & translation) between them. I wanted to check which intrinsics are correct for the depth image (ie, what we should use to deproject depth to points) as there are several potential options in the tree: Monocular/Left or Right and Stereo/Left or Right, all of which have slightly different values.

  3. I have the colour->depth extrinsics from [colour device]/ResolvedLink (hopefully that is correct). Is [stereo device]/ResolvedLink the depth->colour extrinsics?

Thanks!
James

2

Hi,

  1. Yes, the stereo and color devices are separate and you can use them independently in the API. If you want to use both, you can specify both serials to the commands (most commands also take a list of cameras, so you can handle both with a single execution). The images and 3D data are inside of the respective camera nodes.

    You can find some general information on the two devices for a C-series camera in this guide.

  2. The reprojection matrix for the disparity map is in Calibration/Dynamic/Stereo/Reprojection and the camera matrix for the left rectified images is in Calibration/Dynamic/Stereo/Left/Camera. Also see this guide for the pixel layout of the different stereo camera data.

    Unfortunately the description of the calibration nodes in the manual is not very clear at the moment. To summarize the difference between the different nodes of the stereo calibration:

    • Monocular nodes are for raw images and Stereo nodes are for rectified images / 3D data.
    • Dynamic includes the dynamic recalibration and nodes outside of it are the original calibration.

  3. I am not sure if I understand the question correctly here.

    <Serial>-Color/Link contains the relative position of the color sensor to the stereo camera (assuming a factory calibration where the color sensor link targets the stereo camera).

    The ResolvedLink nodes contain the workspace (world) position of each device. For the color sensor this is the same as above by default, since the stereo camera does not have a link by default. If you change its link you can move both the stereo camera and its color sensor together, which would affect both their ResolvedLink. Also see this guide on the general concept of the link tree in the NxLib.