it is not possible to convert the color image to the perspective of the stereo camera because it does not contain any depth information.
When using texture data, processing usually starts with the color image (where some features are detected) and then fetches the 3D data corresponding to those 2D features. For such an approach you need the 3D data from the perspective of the color camera. This can be computed using the RenderPointMap command as described in this guide.