The close relation between spatial kinematics and line geometry has been proven to be fruitful in surface detection and reconstruction. However, methods based on this approach are limited to simple geometric shapes that can be formulated as a linear subspace of line or line element space. The core of this approach is a principal component formulation to find a bestfit approximant to a possibly noisy or impartial surface given as an unordered set of points or point cloud. We expand on this by introducing the Gaussian process latent variable model, a probabilistic nonlinear nonparametric dimensionality reduction approach following the Bayesian paradigm. This allows us to find structure in a lower dimensional latent space for the surfaces of interest. We show how this can be applied in surface approximation and unsupervised segmentation to the surfaces mentioned above and demonstrate its benefits on surfaces that deviate from these. Experiments are conducted on synthetic and realworld objects.
Extracting structural information as shapes or surfaces from an unordered set of 3D coordinates (point cloud) has been an important topic in computer vision [
However, in this work, we mainly focus on detecting simple geometrical surfaces such as planes, spheres, cylinders, cones, spiral and helical surfaces, surfaces of revolution, etc. as described in [
Although a mathematically very elegant way to describe 3D surfaces, this approach does have several drawbacks. First, the surface classification is strict. This means that only very primitive shapes can be treated. Realworld objects do not always fall neatly into one of these categories. For example, imperfect shapes like a slight bend plane, a sphere with a dent or shapes found in nature or human anatomy. Blindly following this method, results in a misrepresentation of the data. Second, because PCA minimises an L2norm, it is very sensitive to outliers. This can be mitigated by iterative RANSAC or by downweighting the outliers. However, this comes at the cost of increased computation time. Third, realworld point clouds show various other imperfections like nonuniform sampling, noise, missing regions, etc. This highly affects the quality of the representation. Fourth, the authors of [
Most of these drawbacks can be attributed to the eigenvalue problem (or its PCA formulation) used to find an appropriate linear subspace in
In our approach, we can handle shapes (or surfaces) that fall in the categories described in [
For completeness, we mention that in recent years various deep learning techniques have been successfully introduced to discover objects in point clouds. A thorough overview can be found in [
To summarise, for every point on a given point cloud, we can formulate a socalled line element. Dimensionality reduction on the set of line elements reveals the characteristics of the surface captured by that point cloud. Existing methods rely on PCA, which is a linear mapping. In contrast, our model is built on the Gaussian process latent variable model, which allows for nonlinear mapping. This results in a more nuanced way of representing the surface. The main contributions of this work are the following:
We expand existing methods based on kinematic surfaces and line element geometry by introducing GPLVM to describe surfaces in a nonlinear way.
We apply our method to surface approximation.
We test our method to perform unsupervised surface segmentation.
We demonstrate our method to perform surface denoising.
All of our 3D models, sets of line elements, trained GPLVM, notebooks with code and many more experiments and plots can be found on our GitHub repository (
The rest of this paper is structured as follows. In the next section, we give some theoretical background on line element geometry, kinematic surfaces, approximating the data, Gaussian processes and Gaussian process latent variable models in particular. The
In projective 3space
For example, a line
Not every combination of six numbers yields a straight line. To do so, the following condition is necessary and sufficient:
This is called the
This quadric is called the
Plücker coordinates of a line in
Each point on a smooth surface
Rigid body motions can be seen as a superposition of rotations and translations. These can be extended by adding a scaling, making them the family of
As defined in [
Here, we will give an overview of such motions
Examples of these surfaces can be found in
This alternative way of describing surfaces as linear complexes of line elements opens up a new way of studying them, as explained below.
Suppose a scanning process results in a set of points
First, we calculate the unit normal vectors from the point cloud at every point. This topic has been very well documented in the literature. We refer the reader to [
Second, according to Equation (
Some surfaces are invariant under more than one oneparameter transformation [
Further examination of the coordinate
Although this is a very elegant and powerful approach, some issues are discussed in the works listed above. First, this method is very sensitive to outliers. A solution proposed by the authors is to apply a RANSAC variant or to downweigh the outliers by iteratively
This obviously results in longer computation times. Second, numerical issues can arise calculating the eigenvalues (especially for planes and spheres). Third, some shapes do not fall into the classification of these simple geometric forms. This is certainly the case for organic surfaces that can be found in nature or when modelling the human body. Reconstructing a surface based on a simple geometric shape is obviously only valid if the surface resembles the shape well. Fourth, some shapes are either a combination or a composition of the elementary simple shapes (e.g., pipe constructions). In this case, the question arises of what constitutes as a small eigenvalue and where to draw the line between steadily increasing values. Even though some guidance is given in the literature, these thresholds are often applicationspecific parameters.
Our approach provides a solution for these issues, by finding a representative lower dimensional latent space for the line elements in a more flexible nonlinear way. This is no longer a linear subspace in
By definition, a Gaussian process (GP) is a stochastic process (a collection of random variables), with the property that any finite subset of its variables is distributed as a multivariate Gaussian distribution. It is a generalization of the multivariate Gaussian distribution to infinitely many variables. Here, we only give an overview of the main aspects. We refer the reader to the book [
Let a dataset
The conditional predictive posterior distribution of the GP can be written as:
The hyperparameters
In the literature, many kernel functions have been extensively studied and reviewed. An overview can be found in [
In this work, we do implement a different lengthscale parameter for every input dimension. This technique is called automatic relevance determination (ARD) and allows for functions that vary differently in each input dimension [
Principal component analysis (PCA) transforms a set of data points to a new coordinate system, in which the greatest variance is explained by the first coordinate (called the first principal component), the second greatest variance by the second coordinate, etc. This reprojection of the data can be exploited in dimensionality reduction by dropping the components with the smallest variance associated. The result will still contain most of the information of the original data. This method can also be explained as a statistical model known as probabilistic PCA (PPCA) [
In dimensionalityreduction, the representation of the original data by its retained principal components can be interpreted as a latent variable model, in which the
In [
However, by choosing a nonlinear kernel, we can establish a nonlinear relationship between the latent variables
In the original GPLVM, the unobserved inputs are treated as latent variables which are optimised. Another approach is to variationally integrate them and compute a lower bound on the exact marginal likelihood of the nonlinear variable model. This is known as the Bayesian Gaussian process latent variable model (BGPLVM) [
The GPLVM is a map from latent to data space. As such, it preserves local distances between points in the latent space. However, this does not imply that points close by in the data space are also close by in the latent space. To incorporate this extra feature, one can equip the GPLVM with a socalled back constraint [
In this section, we explain how all of the abovementioned concepts come together in our approach. To recap, we can represent a straight line in 3D space by Plücker coordinates, which are sixtuples. By adding a seventh component, we can specify a point on that line. We can do this for every
The theory of kinematic surfaces links the line elements that are contained in a linear line element complex to an equiform kinematic surface. Finding this complex comes down to solving an ordinary eigenvalue problem. The dimensionality of the linear subspace, in which the line elements live, and the resulting eigenvalues determine the type of surface. In essence, this is dimensionality reduction on the sevendimensional line elements via PCA.
However, PCA is a linear mapping. In contrast, our model is built on the Gaussian process latent variable model (GPLVM), which allows for nonlinear mapping. This results in a more nuanced way of representing the surface. Our model is only given the sevendimensional line elements and finds the mapping from a latent (unobserved) space to these line elements. Each of the seven dimensions of the line elements is assigned to a Gaussian process (GP). The outputs (predictions) of those GPs are the components of the line elements. The inputs (training data) are the latent points, which are treated as variables of the overall model and are optimised (or integrated out in the Bayesian formulation).
In the next section, we will describe in more detail how we compose the datasets and elaborate on our experiments.
All 3D models, datasets, plots and trained GP models described below can be found in our GitHub repository. These include supplementary experiments and plots we omitted here, so we do not overload the text.
To assess the latent representation of various 3D shapes, we composed a collection of both synthetically generated point clouds and realworld scanned objects. An overview can be found in
The line elements are calculated in Matlab R2020b and exported as commaseparated values (.csv). These serve as the data space for the GPLVM models, which are implemented using the python GPy library (
We trained a Bayesian Gaussian process latent variable model for all examples of the equiform kinematic surfaces listed in
All surfaces show a clear structure in their latent space. Note that the number of small eigenvalues does not necessarily correspond with the number of relevant dimensions in latent space. The latter is the result of an optimization algorithm in which both the latent points and the kernel hyperparameters are found. This can be seen in the ARD contribution plot for the helical surface. The plot for the cylinder of revolution even has a significant value for all seven dimensions. This effect can be thought of as overfitting [
Another important remark is that the mapping from latent to data space is nonlinear. Care must be taken when interpreting the 2D latent space plots. For instance, the spiral surface clearly has a onedimensional subspace. However, the 2D plot shows a scattered cloud of points. This is an artefact of the visualization. The ARD plot indicates that only dimension 1 has a significant contribution. Another example is given by the cylinder without revolution. Its subspace in
So far, nothing has been gained by this new Bayesian GPLVM way of representing surfaces. The difference with the approach described in
First, we apply our method to a bend torus. This is a surface of revolution, which we altered using the Simple Deform modifier in Blender to bend it 45° around an axis perpendicular to the axis of rotational symmetry of the original torus. This removes the rotational symmetry altogether. The results can be seen in
Second, we look at the surface of the point cloud obtained by scanning a pear as described above. This is an organic shape, so it possesses the same challenges as working with shapes that can be found in other items from nature or when modelling the anatomy of humans and animals. We are only interested in the shape of the body, so we removed the stalk and the bottom part when cleaning up the 3D model. In this case, the 3D shape resembles a surface of revolution, but the axis is bent irregularly and the rotational symmetry is broken (not all normals intersect the axis of rotation). The results can be seen in
Both the bend torus and the pear can not be described by an equiform kinematic surface. Applying the methods of
A major challenge in point cloud classification is the segmentation of subregions within that cloud. Once points are grouped together in simpler shapes, the underlying structure can be found via either our method or the methods described in [
As we want to separate coherent groups of points in latent space, we care about their local distances. Points close by in the latent space should be close by in the data space as well. Therefore, we expand our GPLVM with a back constraint as described in
To demonstrate our approach, we first designed three objects composed of different simpler geometric shapes. They can be found in
First, we created a 3D model called Mixture 1, which consists of a cylinder and cone, neither without rotational symmetry. Both of those shapes individually show one small eigenvalue and a clearly distinguishable curve in their 2D latent space, as can be seen in
Second, the 3D model named Mixture 2 consists of a noisy cylinder of revolution where one end is closed by a demisphere. The former is characterised by two small eigenvalues and the latter by three. Again, this behaviour can be clearly observed in the latent space. Notice how the BCGPLVM formulates latent shapes for each part that are consistent with the kinematic surface described in
Finally, in the 3D model Mixture 3, we grouped together the upper half of a sphere, a cone of revolution, a cone without revolution and a cylinder without revolution. These parts have three, two, one and one small eigenvalues, respectively. As this model consists of four different parts, the segmentation is more complex. Nonetheless, the BCGLVM is able to find distinct substructures in the latent space, even in just two dimensions.
For a realworld and more challenging example, we scanned a metal hinge, as described above. It can be found in
Once a latent space is found, the segmentation can be done via either a manual selection of the latent points or a form of unsupervised learning. In the case of separable clusters, we can perform the wellstudied kmeans clustering algorithm or draw (hyper)planes determined by support vector machines (SVM). The details of these are outside the scope of this work. The reader is referred to [
In general, a Gaussian process can handle noise very well, even in low data regimes [
From a predicted line element
To demonstrate this, we again work on the bend torus model. We introduce random noise with the Blender Randomize tool and select a hundred vertices at random, which we translate to simulate shot noise. The results can be seen in
This work presented the first findings for this new GPLVM approach to describe 3D surfaces. In this manuscript, we wanted to focus on the theoretical principles themselves and not overload the paper with additional research questions that determine the limits of this idea. Even though these are both interesting and important in realworld applications, we leave them for future work.
We have shown surface segmentation for surfaces that are the combination of a few different simpler geometrical shapes. The question remains how many subregions can be detected and what the complexity of those regions can be?
We presented the Bayesian GPLVM and the GPLVM with back constraints. There are more variations on this topic investigated in the literature. A recent paper describes a generalised GPLVM with stochastic variational inference [
A line element is formed by a line and a point on that line. By working with normal lines for points on a surface, we effectively introduced a second socalled
The prediction as seventuples made by the model does not automatically follow the Grassmann–Plücker relationship in Equation (
This only holds for
This approach is taken in [
We provided a theoretical introduction to kinematic surfaces and showed how they could be used to perform surface detection. Many simple geometric shapes manifest themselves as linear subspaces of line or line element space. This approach is limited by the linearity of the underlying eigenvalue problem. We expanded on this by reformulating this as a probabilistic nonlinear nonparametric dimensionality reduction technique known as the Gaussian process latent variable model. We showed how this could be applied to many simple geometric surfaces, as well as surfaces that do not fall into any of these categories. Moreover, we showed the benefits of unsupervised surface segmentation and surface denoising. We presented findings on synthetically generated surfaces and scanned realworld objects.
The main goal of the current study was to determine the feasibility of applying the Gaussian process latent variable model to line element geometry. Even though several experiments are explained, and several more are included in the
Another natural progression of this work is to exploit further the found latent space in the case of missing data. Point clouds sometimes have missing regions, caused by bad lighting conditions, occluded areas or areas that simply can not be reached by the scanning device. Finding the 3D coordinates for the missing points is a classic example of the missing data problem. In our case, it manifests itself as a region in the latent space that is missing values. If the found structure in the latent space is enough to reconstruct those missing latent points, then according to data space points can also be inferred by the Gaussian process latent variable model.
More broadly, we plan to study the benefits of working on latent spaces not just for line elements, but for the lines themselves (in which case, we drop the point on the line and only keep the description of the line). This technique could be used in the calibration of various devices that are built on the usage of straight lines. Cameras, for instance, produce images in which a pixel is the result of a single incoming ray of light. Another example is galvanometric laser scanners, which guide a laser beam by means of two rotating mirrors. Calibrating such a device means finding the relationship between the two angles of the mirrors and the outgoing beam. So, in this case, a 2D latent space must exist. This would be a fruitful area for further work.
All of our 3D models, sets of line elements, trained GPLVM, notebooks with code and many more experiments and plots can be found on our GitHub repository
Conceptualisation, I.D.B., C.H.E. and R.P.; methodology, I.D.B., C.H.E. and R.P.; software, I.D.B.; validation, I.D.B.; formal analysis, I.D.B. and R.P.; investigation, I.D.B.; resources, I.D.B.; data curation, I.D.B.; writing—original draft preparation, I.D.B.; writing—review and editing, I.D.B., C.H.E. and R.P.; visualisation, I.D.B.; supervision, R.P.; project administration, I.D.B.; funding acquisition, R.P. All authors have read and agreed to the published version of the manuscript.
Not applicable.
Not applicable.
All of our 3D models, sets of line elements, trained GPLVM, notebooks with code and many more experiments and plots can be found on
We thank the anonymous reviewers whose comments helped improve the quality of this paper.
The authors declare no conflict of interest.
Examples of equiform kinematic surfaces.
ARD contributions for the dimensions of the latent space for the examples of equiform kinematic surfaces.
A 2D representation of the points of the kinematic surfaces in their latent space. The amount of black in the background indicates the posterior uncertainty of the BGPLVM.
Results for a bend torus. One latent dimension is found to be dominant.
Results for a real world pear scanned with a mobile phone app.
Three synthetically generated 3D models by combining primitive surfaces. The BCGPLVM is able to show distinguishable structures for points in latent space.
The results for a realworld scanned metal hinge. Again, the BGPLVM is able to separate the points in latent space.
A bend torus. Noise is added to the entire surface. Moreover, A hundred vertices were translated. (
An overview of the surfaces and their corresponding GPLVM and properties. Larger point clouds are subsampled uniformly to a smaller set. The number of points retained for training is given in # Train. Noise is the Gaussian noise variance hyperparameter for the GPLVM model, which is either a fixed value or a value that has to be learned along with the other hyperparameters. The number of inducing points for the Bayesian GPLVM is shown in # Ind. To make sure we do not end up in local minima, we restarted the training of the model a number of times given in # Restarts.
Model  # Vertices  # Train  Noise  # Ind  # Restarts  

Cylinder of revolution  BGPLVM  2176  2176 

15  10 
Cone of revolution  BGPLVM  2176  2176  free  50  10 
Spiral cylinder  BGPLVM  2210  2210  free  25  10 
Cylinder w/o revolution  BGPLVM  1717  1717  free  50  10 
Cone w/o revolution  BGPLVM  2210  2210  free  50  10 
Surface of revolution  BGPLVM  2816  2500  free  50  10 
Helical surface  BGPLVM  2582  2500  free  25  10 
Spiral surface  BGPLVM  1842  1842  free  50  10 
Torus  BGPLVM  2048  2048  free  50  10 
Torus bend  BGPLVM  2048  2048  free  25  10 
Pear  BGPLVM  6356  5000  free  50  5 
Mixture 1  BCGPLVM  10,653  1000 

NA  3 
Mixture 2  BCGPLVM  6555  1000 

NA  3 
Mixture 3  BCGPLVM  15,681  1000 

NA  3 
Hinge  BGPLVM  22,282  5000  free  50  3 
Torus bend noisy  BGPLVM  8192  8192  free  50  3 