Advanced usage

Go to:

• Notebook configuration

• Clustering initialisation

  • Short initialisation

  • Manual custom initialisation

  • Initialisation via a builder

Notebook configuration

import sys

import numpy as np

import cnnclustering
from cnnclustering import cluster, hooks
from cnnclustering import _types, _fit

Print Python and package version information:

# Version information
print("Python: ", *sys.version.split("\n"))

print("Packages:")
for package in [np, cnnclustering]:
    print(f"    {package.__name__}: {package.__version__}")

Python:  3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27)  [GCC 9.3.0]
Packages:
    numpy: 1.20.2
    cnnclustering: 0.4.3

Clustering initialisation

Short initialisation for point coordinates

In the Basic usage tutorial, we saw how to create a Clustering object from a list of point coordinates.

# Three dummy points in three dimensions
points = [
    [0, 0, 0],
    [1, 1, 1],
    [2, 2, 2]
]
clustering = cluster.Clustering(points)

The created clustering object is now ready to execute a clustering on the provided input data. In fact, this default initialisation works in the same way with any Python sequence of sequences.

# Ten random points in four dimensions
points = np.random.random((10, 4))
clustering = cluster.Clustering(points)

Please note, that this does only yield meaningful results if the input data does indeed contain point coordinates. When a Clustering is initialised like this, quite a few steps are carried out in the background to ensure the correct assembly of the object. To be specific, the following things are taken care of:

• The raw input data (here points) is wrapped into a generic input data object (a concrete implementation of the abstract class _types.InputData)

• A generic fitter object (a concrete implementation of the abstract class _fit.Fitter) is selected and associated with the clustering

  • The fitter is equipped with other necessary building blocks

In consequence, the created clustering object carries a set of other objects that control how a clustering of the input data is executed.

print(clustering)

Clustering(input_data=InputDataExtComponentsMemoryview, fitter=FitterExtBFS(ngetter=NeighboursGetterExtBruteForce(dgetter=DistanceGetterExtMetric(metric=MetricExtEuclideanReduced), sorted=False, selfcounting=True), na=NeighboursExtVectorCPPUnorderedSet, nb=NeighboursExtVectorCPPUnorderedSet, checker=SimilarityCheckerExtSwitchContains, queue=QueueExtFIFOQueue), predictor=None)

To understand the setup steps and the different kinds of partaking objects better, lets have a closer look at the default constructor for the Clustering class in the next section.

Manual custom initialisation

The init method of the Clustering class has the following signature:

print(cluster.Clustering.__init__.__doc__, end="\n\n")

Clustering.__init__(self, input_data=None, fitter=None, predictor=None, labels=None, unicode alias: str = u'root', parent=None, **kwargs)

A Clustering does optionally accept a value for the input_data and the fitter keyword argument (let’s ignore the others for now). A plain instance of the class can be created just like this:

plain_clustering = cluster.Clustering()
print(plain_clustering)

Clustering(input_data=None, fitter=None, predictor=None)

Naturally, this object is not set up for an actual clustering.

plain_clustering.fit(0.1, 2)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-33a427bb7428> in <module>
----> 1 plain_clustering.fit(0.1, 2)

src/cnnclustering/cluster.pyx in cnnclustering.cluster.Clustering.fit()

AttributeError: 'NoneType' object has no attribute 'n_points'

Starting from scratch, we need to provide some input data and associate it with the clustering. Trying to use just raw input data for this, however, will result in an error:

plain_clustering.input_data = points

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-74ffe4d30bb9> in <module>
----> 1 plain_clustering.input_data = points

src/cnnclustering/cluster.pyx in cnnclustering.cluster.Clustering.input_data()

TypeError: Can't use object of type ndarray as input data. Expected type InputData.

Info: If you know what you are doing, you can still associate arbitrary input data to a clustering by assigning to Clustering._input_data directly.

We need to provide a valid input data object instead. The recommended type for point coordinates that can be constructed from a 2D NumPy array is _types.InputDataExtComponentsMemoryview.

plain_clustering.input_data = _types.InputDataExtComponentsMemoryview(points)

This input data type is used to wrap the raw data and allows generic access to it which is needed for the clustering. For more information on what exactly has to be implemented by a valid input data type, see the Demonstration of (generic) interfaces tutorial. We could have chosen to pass a valid input data type to a Clustering directly on initialisation:

print(
    cluster.Clustering(
        _types.InputDataExtComponentsMemoryview(points)
    )
)

Clustering(input_data=InputDataExtComponentsMemoryview, fitter=None, predictor=None)

As you see, this initialisation creates a Clustering that carries the input data wrapped in a suitable type, but nothing else. This is different from the starting example where we passed raw data on initialisation which triggered the assembly of a bunch of other objects.

So we are not done yet and clustering is not possible because we are still missing a fitter that controls how the clustering should be actually done.

plain_clustering.fit(0.1, 2)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-33a427bb7428> in <module>
----> 1 plain_clustering.fit(0.1, 2)

src/cnnclustering/cluster.pyx in cnnclustering.cluster.Clustering.fit()

AttributeError: 'NoneType' object has no attribute 'fit'

The default fitter for any common-nearest-neighbours clustering is _fit.FitterExtBFS. If we want to initialise this fitter, we additionally need to provide the following building blocks that we need to pass as the following arguments:

• neighbours_getter: A generic object that defines how neighbourhood information can be retrieved from the input data object. Needs to be a concrete implementation of the abstract class _types.NeighboursGetter.

• neighbours: A generic object to hold the retrieved neighbourhood of one point. Filled by the neighbours_getter. Needs to be a concrete implementation of the abstract class _types.Neighbours.

• neighbour_neighbours: As neighbours. This fitter uses exactly two containers to store the neighbourhoods of two points.

• similarity_checker: A generic object that controls how the common-nearest-neighbours similarity criterion (at least c common neighbours) is checked. Needs to be a concrete implementation of the abstract class _types.SimilarityChecker.

• queue: A generic queuing structure needed for the breadth-first-search approach implemented by the fitter. Needs to be a concrete implementation of the abstract class _types.Queue.

So let’s create these building blocks to prepare a fitter for the clustering. Note, that the by default recommended neighbours getter (_types.NeighboursGetterExtBruteForce) does in turn require a distance getter (that controls how pairwise distances for points in the input data are retrieved) which again expects us to define a metric. For the neighbours containers we choose a type that wraps a C++ vector. The similarity check will be done by a set of containment checks and the queuing structure will be a C++ queue.

# Choose Euclidean metric
metric = _types.MetricExtEuclidean()
distance_getter = _types.DistanceGetterExtMetric(metric)

# Make neighbours getter
neighbours_getter = _types.NeighboursGetterExtBruteForce(
    distance_getter
)

# Make fitter
fitter = _fit.FitterExtBFS(
    neighbours_getter,
    _types.NeighboursExtVector(),
    _types.NeighboursExtVector(),
    _types.SimilarityCheckerExtContains(),
    _types.QueueExtFIFOQueue()
)

This fitter can now be associated with our clustering. With everything in place, a clustering can be finally executed.

plain_clustering.fitter = fitter

plain_clustering.fit(0.1, 2)

-----------------------------------------------------------------------------------------------
#points   r         c         min       max       #clusters %largest  %noise    time
10        0.100     2         None      None      0         0.000     1.000     00:00:0.000
-----------------------------------------------------------------------------------------------

The described manual way to initialise a Clustering instance is very flexible as the user can cherry pick exactly the desired types to modify the different contributing pieces. On the other hand, this approach can be fairly tedious and error prone. In the next section we will see how we solved this problem by facilitating the aggregation of a clustering according to pre-defined schemes.

Initialisation via a builder

We did see so far how to assemble a Clustering instance from scratch by selecting the individual clustering components manually. In the beginning we did also see that we could create a Clustering seemingly automatically if we just pass raw data to the constructor. To fill the gap, let’s now have a look at how a Clustering can be created via a builder. A builder is a helper object that serves the purpose of correctly creating a Clustering based on some preset requirements, a so called recipe. When we try to initialise a Clustering with raw input data (that is not wrapped in a valid generic input data type), a ClusteringBuilder instance actually tries to take over behind the scenes.

The ClusteringBuilder class has the following initialisation signature:

print(cluster.ClusteringBuilder.__init__.__doc__)

ClusteringBuilder.__init__(self, data, preparation_hook=None, registered_recipe_key=None, clustering_type=None, alias=None, parent=None, **recipe)

It requires some raw input data as the first argument. Apart from that one can use these optional keyword arguments to modify its behaviour:

• preparation_hook: A function that accepts raw input data, does some optional preprocessing and returns data suitable for the initialisation of a generic input data type plus a dictionary containing meta information. If None, the current default for this is hooks.prepare_points_from_parts which prepares point coordinates for any input data type accepting a 2D NumPy array. If no processing of the raw input data is desired, use hooks.prepare_pass. The default preparation hook can be set via the class attribute _default_preparation_hook (must be a staticmethod).

• registered_recipe_key: A string identifying a pre-defined clustering building block recipe. If this is None, the current default is "coordinates" which can be overridden by the class attribute _default_recipe_key. The recipe key is passed to hooks.get_registered_recipe to retrieve the actual recipe. The key "none" provides an empty recipe.

• clustering_type: The type of clustering to create. The current default (and the only available option out of the box) is Clustering and can be overridden via the class attribute _default_clustering. This allows the use of the builder for the creation of other clusterings, e.g. for subclasses of Clustering.

• alias/parent: Are directly passed to the created clustering.

• **recipe: Other keyword arguments are interpreted as modifications to the default recipe (retrieved by registered_recipe_key).

To start with the examination of these options, we should look into what is actually meant by a clustering recipe. A recipe is basically a nested mapping of clustering component strings (matching the corresponding keyword arguments used on clustering/component initialisation, e.g. "input_data" or "neighbours") to the generic types (classes not instances) that should be use in the corresponding place. A recipe could for example look like this:

recipe = {
    "input_data": _types.InputDataExtComponentsMemoryview,
    "fitter": "bfs",
    "fitter.getter": "brute_force",
    "fitter.getter.dgetter": "metric",
    "fitter.getter.dgetter.metric": "euclidean",
    "fitter.na": ("vector", (1000,), {}),
    "fitter.checker": "contains",
    "fitter.queue": "fifo"
}

In this recipe, the generic type supposed to wrap the input data is specified explicitly as the class object. Alternatively, strings can be used to specify a type in shorthand notation. Which abbreviations are understood is defined in the hooks.COMPONENT_NAME_TYPE_MAP. In the fitter case, bfs is translated into _fit.FitterExtBFS. Dot notation is used to indicate nested dependencies, e.g. to define components needed to create other components. Similarly, shorthand notation is supported for the component key, as shown with fitter.getter which stands in for the neighbours getter required by the fitter. Abbreviations on the key side are defined in hooks.COMPONENT_ALT_KW_MAP. For the "fitter.na" component (one of the neighbours container type needed that the fitter needs), we have a tuple as the value in the mapping. This is interpreted as a component string identifier, followed by an arguments tuple, and a keyword arguments dictionary used in the initialisation of the corresponding component. Note also, that the recipe defines only "fitter.na" (neighbours) and not "fitter.nb" (neighbour_neighbours) in which case the same type will be used for both components. Those fallback relation ships are defined in hooks.COMPONENT_KW_TYPE_ALIAS_MAP.

This recipe can be now passed to a builder. Calling the build method of a builder will create and return a Clustering:

print(
    cluster.ClusteringBuilder(
        points,
        preparation_hook=hooks.prepare_pass,
        registered_recipe_key="none",
        **recipe
    ).build()
)

Clustering(input_data=InputDataExtComponentsMemoryview, fitter=FitterExtBFS(ngetter=NeighboursGetterExtBruteForce(dgetter=DistanceGetterExtMetric(metric=MetricExtEuclidean), sorted=False, selfcounting=True), na=NeighboursExtVector, nb=NeighboursExtVector, checker=SimilarityCheckerExtContains, queue=QueueExtFIFOQueue), predictor=None)

For the initial example of using point coordinates in a sequence of sequences, the builder part is equivalent to:

# The recipe registered as "coordinates":
# {
#     "input_data": "components_mview",
#     "fitter": "bfs",
#     "fitter.ngetter": "brute_force",
#     "fitter.na": "vuset",
#     "fitter.checker": "switch",
#     "fitter.queue": "fifo",
#     "fitter.ngetter.dgetter": "metric",
#     "fitter.ngetter.dgetter.metric": "euclidean_r",
# }

print(
    cluster.ClusteringBuilder(
        points,
        registered_recipe_key="coordinates",
    ).build()
)

Clustering(input_data=InputDataExtComponentsMemoryview, fitter=FitterExtBFS(ngetter=NeighboursGetterExtBruteForce(dgetter=DistanceGetterExtMetric(metric=MetricExtEuclideanReduced), sorted=False, selfcounting=True), na=NeighboursExtVectorCPPUnorderedSet, nb=NeighboursExtVectorCPPUnorderedSet, checker=SimilarityCheckerExtSwitchContains, queue=QueueExtFIFOQueue), predictor=None)

It is possible to modify a given recipe with the explicit use of keyword arguments. Note, that in this case dots should be replaced by a double underscore:

print(
    cluster.ClusteringBuilder(
        points,
        registered_recipe_key="coordinates",
        fitter__ngetter__dgetter__metric="precomputed"
    ).build()
)

Clustering(input_data=InputDataExtComponentsMemoryview, fitter=FitterExtBFS(ngetter=NeighboursGetterExtBruteForce(dgetter=DistanceGetterExtMetric(metric=MetricExtPrecomputed), sorted=False, selfcounting=True), na=NeighboursExtVectorCPPUnorderedSet, nb=NeighboursExtVectorCPPUnorderedSet, checker=SimilarityCheckerExtSwitchContains, queue=QueueExtFIFOQueue), predictor=None)

The above modification makes the recipe match the "distances" recipe. Other readily available recipes are "neighbourhoods" and "sorted_neighbourhoods". The users are encouraged to modify those to their liking or to define their own custom recipes.

Newly defined types that should be usable in a builder controlled aggregation need to implement a classmethod get_builder_kwargs() -> list that provides a list of component identifiers necessary to initialise an object of itself.