Normals: What are they and how they help us pick better

Surface normals are the vectors we find on surfaces that help us understand the nature of that surface.  

Often abbreviated to just ‘normals,’ they are perpendicular to the tangent plane line at a particular point on the surface. They yield especially useful information about a curved surface’s characteristics but conversely can tell us much about the planarity of a flat or angular surface also. So normals can tell us the rate of change of curvature when a surface has an angular departure and tell us just how perfectly flat something is. This is already looking interesting, isn’t it?  

 Fig. 1 Surface normals on a curved surface. 

You have probably already gathered that normals are a theoretical construct and as such are not limited in number, theoretically, an infinite number of normals can exist on surfaces. However, we need to identify them by some point of reference and capture them, and that method of capture will dictate how many normals we can work with. In 3D vision, the sensor of the camera is what dictates this number. So, Zivid cameras, with their 2.3-million-pixel sensors can deliver normals data for every pixel which means a normal value for every point in the point cloud.  

 Fig. 2 angular deviation of normal vector on a sphere. 

A theoretical construct with very real-world uses

Normals give us a powerful complementary tool when assessing scenes and making decisions about that scene. Even with excellent point cloud detail, some objects can present a challenge. The addition of normals to our object identification toolkit can be an invaluable aid. We will take a look at a couple of instances where normals provide assistance. 

Spotting the gap 

A stand-out example is finding the gaps between closely aligned boxes. This is possible with particularly good point clouds but can be problematic in certain instances. Take for example a thin strip of black tape running across the top of a box. When a point cloud is used alone that strip of black tape might look remarkably similar to the black shadow of a gap between boxes. With high-quality point clouds such as the Zivid color 3D cameras can deliver, you could probably work out the difference with the right processing. But when you introduce the normals data you will clearly see the normals transition through 90 degrees at the box edge if it is seeing a gap between the boxes. When the image is just a black strip of tape the normals will remain pointing in the same direction as no gap exists.  

 Fig.3 Normals on the surface of a box 

Using normals in CAD matching 

3D CAD matching is a quite common method for locating objects in a range of robotic applications. These can range from bin pick-and-place to machine tending and assembly. This technique of using normals data is a powerful complement to having an already (hopefully) detailed point cloud. It helps in finding the correct object, but especially the correct pose of an object, and whether the object is flipped on one side or another. It is not unusual to find that objects with simpler, more uniform shapes pose more of a problem here due to a lack of very defining features to act as an anchor point for the detection and subsequent calculations. 

When CAD matching is used it relies on a CAD reference of the object to be detected. This CAD reference will have normals as a part of its data. As the normals have a reference back to a singular reference plane the industrial PC or controller for the robot cell can compare the scene to make a match on the shape and pose by lining the two up together. 

The methods used to match the CAD image with the 3D point cloud of the scene have many different and subtle approaches dependent on numerous factors such as the complexity of the object and scene. Often a kernel is used to identify a small, but easily identifiable section of an object. The kernel can be thought of as a small ‘window’ on the scene and this is stepped across the scene with the task of recognizing a particular object feature. When that feature is located, because we have the CAD file as a reference, we can calculate the pick point even if that is not a part of the kernel’s scene. 

Use of normals to reject artifacts 

When combined with some point cloud post-processing, normals have the potential to reduce unwanted spurious effects and distortions from our point cloud. Shiny objects especially can produce point cloud artifacts that are unrecognizable as part of the object they represent. This is due to specular light interreflections from the object, polished metal and glass surfaces are particularly prone to this.  

When this happens not only is the object producing them sometimes unrecognizable, but the artifacts produced can obscure other scene detail. But what these artifacts contain are normals data, and that comes to our aid. When we analyze the points on an object, we expect to see some sort of uniformity, either inconsistent alignment of normals on flat surfaces or on changing surfaces we still expect to see some reasonably steady rate-of-change. What reflective artifacts will reveal is a very chaotic arrangement of normals that generally are impossible to exist in the real world. We can filter for the appearance of this behavior and then remove the offending artifacts, thus cleaning up our point cloud and making the scene more ‘true-to-reality’ and useful. 

 Fig.4 Noisy image can be interpreted as curved or flat surface 

 Fig.5 Addition of normals data reveals objects true shape. 

 Fig. 6 uniformly arranged normals on a Zivid calibration board. 

Zivid normals addition to SDK 2.5 

We introduced normals support in our latest Zivid SDK 2.5. It was the major feature of this release, and we are happy to be able to offer this especially useful feature to our customers. The new CopyNormalsXYZ function allows developers to obtain normals and use them in your application.   

Calculation of normals can be a computationally intensive task, and there are various approaches. At Zivid we have implemented a novel solution for obtaining normals data that is highly accurate and executes very quickly and efficiently on a GPU whilst the data is still in the GPU’s memory. This negates the need for transferring the data to the host CPU for calculating the normals calculated with a third-party application. This makes development much more straightforward with everything in one SDK, and it is having an execution time advantage. We have published an article on normals in our knowledge base which is accompanied by a range of examples. 

Use of normals in applications  

So, we have seen normals data is particularly useful to determine an object’s surface characteristics, and indeed also to reject spurious characteristics. It is an essential part of the 3D vision toolkit, and we look forward to hearing from our customers how they have used this new functionality to overcome challenges and achieve even better performance from their 3D vision system. 

Good luck from Team Zivid!