Filling in the Blanks: Dealing with Missing Data in 3D Point Clouds

Zivid SDK 2.9 introduces hole-filling in point clouds.

SDK 2.9 has a whole variety of features already available in the preview, but we decided we could add one more thing we have been working on for a while. Ladies, gents, and other non-binary entities, may I introduce to you: the hole-filling filter from Zivid!

Getting good, true-to-life 3D data can be tricky. Often, we find that we need to disregard points because of too high noise, wrong data, and artifacts that create features that are not part of the scene. Removing incorrect points is a critical step in providing the most accurate data we can from the Zivid cameras, but it does come with a new problem and a picking system’s worst enemy: holes.

When there are holes in a point cloud, it creates an unknown that the system must deal with. This can show itself in multiple ways depending on the application: more processing time, poor model matching, fewer pick locations, and less robust picking locations.

With Zivid, one of our strong suits is the direct correspondence between the 2D image and 3D data that the Zivid cameras return. Problems can arise when you find that the 2D points that the detection software has determined at the proper pick location do not have the corresponding 3D data.

Holes in the point cloud can lead to a decrease in the number of reliable, usable pick points available for an object. Patchy surfaces can be troublesome, for instance, for systems employing suction grippers. As the grippers require continuous areas, at least the size of the gripper, the number of valid picking positions will be reduced. In the worst case, the object will not be pickable at all. Let us have a look at a common warehouse material infamous for its challenging surface: Bubble wrap.

Bubble wrap before (left) hole-filling and after (right) hole-filling

Before

After

These images are directly from a Zivid point cloud, showing a bubble-wrapped object. On the left, we see the object after applying point-removing filters, such as noise, cluster, and reflection filters. On the right, we see the object after applying the new hole-filling filter. The hole-filling filter interpolates missing values from neighboring data in the point cloud, leaving us with a smooth, continuous surface.

Let’s check out a more encompassing example from a full-piece picking scene. In the image below,  the cyan coloring shows where 3D  data has been reintroduced to the point cloud using the hole-filling filter. 

Another place where this filter can prove beneficial is when dealing with sparse data on extremely dark and absorbent objects. Below is one of the Zivid calibration targets set up angled away from the camera. This angle, along with the matte black material, requires a bit more effort to get quality data. When having high contrast scenes, it requires a 3D HDR capture to get good data across all of the surfaces. The white of the board requires smaller apertures with lower exposure times, and the dark areas require another acquisition with a long exposure time.

With the new hole-filling filter, it is possible to interpolate the data, resulting in a faster return time for the point cloud and one less acquisition in your 3D capture. The example below shows that instead of adding in an extra acquisition, we used hole-filling to fill in the gaps in the darkest sparse data sections. 

Now, we do understand that this filter might not be for everyone. Regarding inspection solutions, the risk associated with introducing incorrect data is higher, as it could result in objects passing quality control that shouldn’t. On the other hand, when you have sparse data on an object with minimum data thresholds for CAD matching in a picking solution, hole-filling in the sparse data can prove very useful. 

At the moment, hole filling is in experimental mode and still subject to changes in future releases. We will continue to innovate on this topic to provide a more optimized option in the future. In the meantime, please test out this new feature and let us know what you think! Customer feedback is our favorite way to help innovate. 😊