Zivid SDK 2.7: Preserve the Details in your Point Clouds

Our latest SDK 2.7 introduces some new and improved functionality to take your Zivid 3D camera to even better levels of performance. We always have a relentless emphasis on enabling your Zivid 3D color camera to deliver even more stunning results, even when faced with the toughest scenes to capture. Keeping the great Zivid ease-of-use is also at the front of our minds too. This SDK release holds true to that commitment as you will see. 

On reflection 

No matter what industry you are working in, shiny and highly reflective parts can be the ultimate challenge for machine vision and detection.  The ability to see truly can be make or break when it comes to vision-guided robotics. It isn’t any surprise really when we consider how light behaves when hitting a very shiny or mirror-like surface, as humans we even see it with our own eyes in the form of glare and specular reflections. Luckily, we are equipped with the best AI out there and are able to adapt and understand what and where these reflections and distortions are. Sometimes we use other tools, like polarized sunglasses to help us along the way. But what about machine vision?  

AI and machine learning have come a long way, but it’s always valuable to give them the best quality data as a starting point. We have designed tools to help get closer to real-world data and our new reflection filter is a prime example, of such an instance. Think of it as the software version of a pair of Ray-Bans for your 3D vision.  

Fig 1. Ghost planes produced by reflections with shiny objects. 

Exorcising ghosts 

In 3D vision, we generate three-dimensional point clouds using two-dimensional sensors. To do this involves a combination of different techniques from special structured light patterns, through high-resolution capture by the sensor, to complex calculations in a GPU. The unpredictable nature of reflections and interreflections can create what are called ghost planes. Ghost planes are easily identifiable when looking at a point cloud as in fig.1. They are perpendicular to the camera’s vertical axis and seemingly just hovering in the air.  

Now, as humans, we automatically know that these planes do not exist and that the points are incorrect data. However, current software and detection algorithms can see these as a problem. To combat this problem, we need to find ways to filter out these artifacts so we can offer high-quality accurate data to detection systems and other processing methods. 

In a nutshell, what we have achieved with this new reflection filter is better overall performance at reducing ghost planes, and importantly, it processes even faster. So better results, in less time. What’s not to like?  

Filtering options 

Without going too much into our secret sauce, let’s just say we have made filtering algorithms that can identify when these ghost planes occur and remove them.  This works rather well, but as with all filters, there is a limit you have to decide on. If you filter too aggressively, then you can crop out other data in the scene, like thin edges or wires that can appear to be hovering.  With analog filters, it can be possible to fine-tune this point for each application, we found the two best options. For the small thin objects in your scenes, like thin pipes, wires, and container edges, we have the local filter. For the high reflections and extremely shiny areas, we have the global filter.  

Settings::Processing::Filters::Reflection::Removal::Experimental::Mode  is where the reflection filter mode can be set to ‘global’ or ‘local’. 

The global filter setting is the more aggressive of the two and removes ghost planes very effectively, but if thin wires are present, it will probably remove them also. 

The local filter setting is gentler and whilst still removing most ghost planes in each situation, it is designed to preserve any thin wires that are present in the scene. 

Fig 2. Before and after use of new reflection filter. 

Aside from new reflection filters, have plenty of other new and exciting things in the latest v2.7 SDK. 

Consistent color for HDR captures 

If you are needing a bit more control over the color in the scene? Check out our newest feature for choosing the capture mode for your 2D data with Settings::Processing::Experimental::Color::Mode.  

Normally, for an HDR capture, all the acquisitions are combined together to make one color image. However, this can sometimes give inconsistent 2D data with changing scenes. This new Color::Mode setting will give you consistent colors across repeated captures. Having this consistent 2D can be crucial to certain detection and deep learning algorithms. 

Thermal stabilization

In this SDK release we have introduced a new thermal stabilization mode which will warm up the camera to a stable state and attempt to keep it there. This result in more stable and predictable trueness performance for your point clouds.

Pose Detection of Zivid calibration boards

If you capture a point cloud of an official Zivid calibration board it is now possible to detect and return the pose of the board. This can be helpful for validating hand-eye calibration or transform the origin of the point cloud to the board.

We hope that you all are as excited about this release as we are! The changes don’t stop here though, we have added many other improvements and bug fixes along the way as well. Some nice ones are: 

• Better support for newer AMD GPU
• GPUs with the RDNA architecture can run Zivid SDK. 
• We have decreased the time for HDR captures. 
• We have decreased the time for Stripe Engine captures. 
• And finally, we can now return to you the pose of the checkerboard through the SDK. 

If you want to see all the improvements we have made, check out our changelog. 

Knowledge Base resources 

As always, we have supported this SDK release with articles, updates, and code snippets in the Zivid Knowledge Base

Some highlights are:

• Thermal stabilization (New)
• Cloud Logging (New)
• Reflection Filter (Updated)
• Tonemapping (Updated)
  

   

To view all the new resources we have put here to help you understand more and develop faster here’s the Zivid Knowledge Base 2.7 version history.

To see all the new improvements in SDK 2.7, check out the changelog.

So, some good stuff in this release, we hope you like and use these new improvements. See you next time for SDK 2.8 and as always happy developing with Zivid!