Producing 3D point clouds with a stereo camera in OpenCV

In my last posts, I showed you how to build a stereo camera, calibrate it and tune a block matching algorithm to produce disparity maps. The code is written in Python in order to make it easy to understand and extend, and the total costs for the project are under 50€. In this post, I’ll show you how to use your calibrated stereo camera and block matcher to produce 3D point clouds from stereo image pairs, like this:

Two images taken with a calibrated stereo camera pair, with two perspectives of the resultant point cloud.

Short and sweet: How to make a point cloud without worrying too much about the details

The entire workflow for producing 3D point clouds from stereo images is doable with my StereoVision package, which youcan install from PyPI with:

pip install StereoVision

or fork directly on GitHub.

So let’s get straight to the good stuff: How to produce and visualize point clouds. Afterwards I’ll explain the code behind it and how it’s changed since the last posts, so that you’ll know whether you need to change anything on your end before you can start producing your point clouds.

One utility that comes with the package, images_to_pointcloud, that takes two images captured with a calibrated stereo camera pair and uses them to produce colored point clouds, exporting them as a PLY file that can be viewed in MeshLab. You can see an example at the top of the page.

If you’ve built your stereo camera and calibrated it, you’re ready to go. You can call it like this:

me@localhost:~> images_to_pointcloud --help
usage: images_to_pointcloud [-h] [--use_stereobm] [--bm_settings BM_SETTINGS]
                            calibration left right output

Read images taken with stereo pair and use them to produce 3D point clouds
that can be viewed with MeshLab.

positional arguments:
  calibration           Path to calibration folder.
  left                  Path to left image
  right                 Path to right image
  output                Path to output file.

optional arguments:
  -h, --help            show this help message and exit
  --use_stereobm        Use StereoBM rather than StereoSGBM block matcher.
  --bm_settings BM_SETTINGS
                        Path to block matcher's settings.

On my machine this takes about 2.3 seconds to complete on two images with dimensions 640×480. 1.89 seconds is for writing the output file – disk access is just about the most expensive thing you can do. That means that I need about 0.4 seconds to read two pictures, create a 3D model of what they saw, do some very rudimentary filtering to eliminate the most obviously irrelevant points. That’s pretty good, if you ask me.

Here are some example results:

Two point clouds produced with images_to_pointcloud

If you’re looking for an easy way to take pictures, you can use show_webcams to view the stereo webcam views and, if desired, save pictures in specified intervals.

As with any passive stereo camera setup, you’ll have the best results with a texture-rich background. If your results are bad, try tuning your block matching algorithm and make sure that you’ve told the programs the correct device number for the left and right camera – otherwise you might be trying to work crosseyed, and that won’t work well at all.

So what’s changed in the sources?

The bottom line: If you’ve already calibrated your camera, you’ll need to modify your calibration or do it again. Also, you won’t be able to work with the StereoBMTuner for at least a few days to tweak your block matching algorithm. This is only a problem if you have a special setup – otherwise the default settings should work fine now.

If you’ve followed the previous posts, you’ll have built and calibrated your stereo camera and perhaps started producing disparity maps. If you were working with the old repository, you might have been producing bad calibrations. Since then there have been a few changes to core parts of the code in the calibration program, calibrate_stereo.py, which means that you’ll need to update your old calibration. The reason for this is that I was getting bad point clouds using the disparity-to-depth map produced by OpenCV’s stereoRectify function. I’ve replaced the matrix with one based off of the stereo_match.py sample in the OpenCV sources that examines the images you provide and makes some assumptions about the camera’s focal length. It should work fine for cameras with focal lengths similar to the human eye, which most webcams are. You can either use calibrate_cameras from the new package to recalibrate your camera or just correct your own calibration by loading the StereoCalibration, looking at the blame for the old calibrate_stereo.py, replacing your calibration’s disparity-to-depth map with the new one that’s produced in StereoCalibrator.calibrate_cameras and then using the calibration’s export function to save it back to disk.

As much as I dislike inferring camera properties from the image’s metadata, this should work fine for most examples and is accurate enough for my purposes. The matrix returned by OpenCV just wasn’t doing what it needed to and the fix was easy and fairly robust for the use cases that I foresee for the library.

Another change is that the StereoBMTuner, which you might have read about in the last post, has changed quite a bit, as I explain in my next post. I had nice experience with it and was able to produce some okay-looking disparity maps, but in general, I was so unsatisfied with the resultant point clouds that I started working with OpenCV’s StereoSGBM. It’s another block matching algorithm that also works quite well, and I discovered that my camera pair was able to work just fine with standard settings that I harvested from one of the OpenCV sample scripts. The new tuner class now allows tuning both types of block matchers by analyzing the block matcher and creating methods on-the-fly to pair with sliders for the correct parameters – a lot of fun to program.

So how does it work?

We’ve been working thus far with a CalibratedPair class, which is an interface to the two cameras in your stereo rig. It inherits from the StereoPair class, with the major difference that it takes a StereoCalibration object that it uses to deliver stereo rectified images rather than the raw images taken with your cameras. This class used to directly hold a block matching object from OpenCV, restricting access to its parameters and taking care of all of its details. The complexity of dealing with the block matcher has now been externalized into a group of three classes.

These classes are an abstract class, BlockMatcher, which implements all the functionality of both BlockMatchers I’ve worked with in OpenCV so far through a unified API and defines the remaining interface that each class has to implement itself. That means that a CalibratedPair doesn’t actually need to know what kind of BlockMatcher you’re using – it delegates work to its BlockMatcher and lets the BlockMatcher take care of the details itself.

There are two BlockMatcher subclasses: StereoBM and StereoSGBM. Each BlockMatcher does certain things – produce disparity maps, for example, or produce point clouds. It uses a block matching object from the OpenCV library and protects the parameters that are relevant for that block matcher, preventing the user from setting ones that will result in errors. It’s up to the user to make sure that the settings he uses also make sense.

StereoPair now interacts with the BlockMatcher of your choice and gives you a PointCloud back. PointCloud is a new class that holds the coordinates of the points produced by the block matching algorithm paired with the colors stored in the image at that location. The class implements one very basic filter – removing the coordinates detected at infinity, where the block matcher wasn’t able to find a match. It can also export itself to a PLY file so you can view the point cloud in full color in MeshLab.

So now you know how to use the code yourself. I hope that the design is understandable enough that it’s easy for you to implement your own programs using it.

I will probably implement some more robust filtering at a later point in time, but for the moment I’m happy with these results. Hope you are too! If you’re using these programs to make point clouds yourself, I’d love to know what you’re doing with them and what your experience is.

About erget

My name’s Daniel Lee. I've founded a company for planning solar power. I've worked on analog space suit interfaces, drones and a bunch of other things in my free time. I'm also involved in standards work for meteorological data. I worked for a while German Weather Service on improving forecasts for weather and renewable power production. I led the team for data ingest there before moving to EUMETSAT.

‹ Extracting files from git into their own repository

Optimizing point cloud production from stereo photos by tuning the block matcher ›

Tagged with: computer vision, OpenCV, point cloud, python
Posted in Uncategorized