Find the natural groups hidden in your data

Clustering is the search for groups of objects that are more similar to each other than to the rest. Nobody tells the algorithm which groups exist or how many there are: that is why it is called unsupervised. Compare with classification (supervised), where the groups are known in advance and the goal is to predict them.

In biology and agronomy the "objects" are typically genotypes, accessions, sites, plots, soil profiles, samples, patients or genes, and the "variables" are the traits, environmental measurements, species counts, markers or expression values recorded on them.

A good clustering has two properties at once: compactness (members of a cluster are close) and separation (clusters are far apart). Almost every method, index and figure in this app is a way of measuring one or both.

Every clustering algorithm returns clusters, even when there are none. Cutting a dendrogram of uniformly scattered points still yields groups; k-means will happily split a single cloud into three. So the first step is to ask whether the data contain structure worth finding.

• Hopkins statistic compares the distances from random locations to their nearest data point with the distances from real points to their nearest neighbour. Around 0.5 the data look uniform; values approaching 1 indicate strong clustering. A common rule: below 0.75 be cautious, above 0.9 be confident.
• VAT image (visual assessment of tendency): the distance matrix is reordered so that similar objects sit together; dark diagonal blocks reveal clusters, a smooth gradient reveals none.
• PCA or MDS preview: a scatter of the first two dimensions often shows groups, gradients or a single blob at a glance.
• Correlations and redundancy: highly correlated variables count several times in a Euclidean distance; consider dropping one, using Mahalanobis distance or clustering on principal components.
• Outliers and missing values: a single extreme object becomes its own cluster in hierarchical methods and drags k-means centres. Decide how to treat them before, not after.

Algorithms never see your variables; they see a matrix of dissimilarities between pairs of objects. Change the coefficient and you change the clusters, so this choice deserves more attention than the choice of algorithm. A similarity s (1 = identical) becomes a dissimilarity through d = 1 − s or d = √(1 − s).

Quantitative variables

• Euclidean: the straight line. The default for k-means and Ward, sensitive to scale and to outliers.
• Manhattan (city block): sum of absolute differences; more robust to extreme values.
• Minkowski generalises both; Chebyshev keeps only the largest difference.
• Canberra: differences relative to magnitude, useful for skewed positive data.
• Correlation-based (Pearson, Spearman, Kendall, cosine): two objects are similar when their profiles rise and fall together, regardless of level. The standard choice for gene expression and for shape rather than size.
• Mahalanobis: Euclidean after removing correlations among variables.

Binary variables (presence/absence, 0/1)

• Jaccard and Dice–Sørensen ignore double zeros: two sites that both lack a species are not made similar by that absence. Preferred in ecology and for markers.
• Simple matching, Rogers–Tanimoto, Sokal–Sneath count double zeros as agreement: right when 0 and 1 are symmetric states (male/female, allele A/B).
• Russell–Rao, Ochiai, Kulczynski, Hamming: further variants offered in Block 3 with their formulas.

Nominal, ordinal and mixed variables

• Nominal: simple matching across categories (equivalently Hamming on dummy columns).
• Ordinal: replace levels by ranks and use Manhattan, or treat them inside Gower.
• Gower distance handles any mixture: each variable contributes a 0–1 dissimilarity of the appropriate kind and the average is taken. It is the natural partner of PAM.

Ecological counts and abundances

• Bray–Curtis: the workhorse for species × site tables; ignores double zeros, bounded 0–1.
• Hellinger and chord transformations followed by Euclidean distance make abundance data suitable for k-means and Ward.
• Chi-square distance, Morisita–Horn, Canberra, Kulczynski (quantitative) and Jaccard on abundances (Ružička) complete the set.

Euclidean and Manhattan distances add differences across variables. A variable measured in kilograms per hectare (thousands) swamps one measured in pH units (5–8). Unless the units are already comparable, standardise:

• z-score: subtract the mean, divide by the standard deviation. Each variable gets mean 0 and variance 1. The default.
• Range (0–1): divides by max − min; keeps the shape of the distribution, sensitive to outliers.
• Robust: median and interquartile range, when outliers are present.
• Log or square-root: for skewed counts before any distance.
• Hellinger, chord, proportions: transformations specific to abundance data (see lesson 3).

Agglomerative methods start with every object in its own cluster and repeatedly merge the two closest clusters until one remains. Divisive methods (DIANA) start from one cluster and split. Both produce a hierarchy: a tree with a merge height at every node, drawn as a dendrogram.

"Closest clusters" must be defined, and that definition is the linkage rule:

Two diagnostics tell you how well a tree represents the original distances:

• Cophenetic correlation: correlation between the original distances and the heights at which pairs are joined. Above 0.75 the tree is a faithful summary; UPGMA usually scores highest here.
• Agglomerative coefficient: how strongly clustered the objects are (close to 1 = clear structure). It grows with sample size, so compare it only among trees of the same data.

The height of a node is the dissimilarity at which its two branches were merged. Long vertical branches mean well-separated groups; a cluster is convincing when the branch below the cut is short and the branch above it is long.

• The left–right order of the leaves is largely arbitrary: any node can be flipped without changing the tree (2ⁿ⁻¹ equivalent drawings). Do not read "neighbours" horizontally; read the height at which two objects join.
• Cutting the tree at a height, or asking for k groups, produces a partition. The cut can be at a single height or at different heights in different branches (dynamic cut).
• For large trees, zoom into a branch, collapse sub-clusters into triangles, or show only the top part.

What ClusteringPro draws. Rectangular, triangular, horizontal and radial (circular) layouts; branches coloured by cluster or by a height gradient; leaf labels coloured, rotated, resized or replaced by symbols; a movable cut line; cluster boxes; a legend you can position; editable titles, axes, fonts, palettes (categorical, viridis-type, colour-blind safe) and line widths; export to PNG, TIFF, SVG, JPG and WEBP at up to 900 dpi. Every figure remembers your edits for the report.

Different linkage rules, distances or variable sets give different trees for the same objects. Instead of trusting one, compare them:

• Tanglegram: two trees face to face with the leaves connected. Leaves are re-ordered to minimise crossings; the entanglement (0 = perfect match, 1 = total disagreement) summarises the result.
• Cophenetic correlation between trees: correlation of the two sets of merge heights.
• Baker's gamma: rank correlation of the level at which each pair of objects first joins in each tree.
• Fowlkes–Mallows index B_k: agreement of the partitions obtained by cutting each tree into k groups, for every k.
• Correlation matrix of methods: run all eight linkages and see which ones agree; a stable structure appears in most of them.

k-means chooses k centres and alternates two steps until nothing changes: assign every object to its nearest centre, then move each centre to the mean of its objects. It minimises the total within-cluster sum of squares, is fast for large data and has no tree. Variants: Hartigan–Wong (the usual default, usually best), Lloyd/Forgy, MacQueen; smarter starts with k-means++; many random starts (nstart) to avoid poor local optima.

• k-medoids (PAM): centres are real observations (medoids), so any distance works, including Gower and Bray–Curtis, and outliers weigh less.
• CLARA: PAM on repeated samples, for thousands of objects.
• Fuzzy c-means (FANNY): every object receives a membership degree in each cluster instead of a hard label. Useful for gradients and for objects sitting between groups; the Dunn partition coefficient measures how crisp the result is.
• Hierarchical k-means: a Ward tree supplies the initial centres, k-means refines them; removes the dependence on random starts.

Limits. k-means assumes round clusters of similar size and variance, in Euclidean space, and needs k in advance. Curved, elongated or nested shapes (moons, rings) are split incorrectly; single linkage, DBSCAN or spectral clustering recover them. The live demo above lets you see this happen.

DBSCAN grows clusters from core points, those with at least minPts neighbours within radius ε. Clusters can have any shape, the number of clusters is not fixed in advance, and isolated points are labelled noise rather than forced into a group. The k-nearest-neighbour distance plot suggests ε: sort the distance to the minPts-th neighbour and look for the knee. HDBSCAN* removes ε altogether and extracts clusters of varying density from a hierarchy.

Model-based clustering assumes the data come from a mixture of Gaussian distributions, one per cluster, with its own mean and covariance. The EM algorithm estimates the parameters and gives each object a probability of belonging to each cluster; the BIC chooses both the number of clusters and the covariance shape (spherical, diagonal, ellipsoidal; equal or varying volume, shape and orientation). It is the principled way to get uncertainty with your labels, and it handles elliptical clusters that k-means cannot.

There is no single correct answer; there are criteria, and the domain has the last word. ClusteringPro computes them for every k in a range and for any method, then reports a consensus:

• Elbow: total within-cluster sum of squares drops fast until the true k, then flattens. Subjective when the bend is gentle.
• Average silhouette: how well each object sits in its cluster compared with the nearest alternative (lesson 11); pick the k that maximises the mean.
• Gap statistic: compares the within-cluster dispersion with what is expected under no structure (bootstrap of uniform data); choose the smallest k whose gap is at least the gap of k + 1 minus one standard error. Unlike the other criteria it can also answer k = 1: no structure.
• Calinski–Harabasz (ratio of between to within dispersion), Davies–Bouldin (lower is better), Dunn (higher is better), plus the C-index, McClain–Rao, PBM, Ratkowsky–Lance, Hartigan, Krzanowski–Lai and Ball–Hall indices: thirteen criteria in all, and the app tallies their votes.
• For hierarchies: the largest jumps in merge heights, and the support of each node by multiscale bootstrap.
• For model-based clustering: BIC. For DBSCAN: the knee of the kNN-distance plot.

Internal validation uses the data alone:

• Silhouette width s(i) for every object: near 1 = well placed, near 0 = between two clusters, negative = probably misassigned. The silhouette plot shows them sorted within each cluster; the mean is the overall quality (0.7+ strong, 0.5 reasonable, below 0.25 no substantial structure).
• Dunn index (smallest between-cluster distance / largest cluster diameter) and connectivity (whether neighbours share a cluster).

Stability asks whether the clusters survive perturbation:

• Bootstrap Jaccard stability: resample the objects, recluster, and measure how often each cluster reappears (above 0.75 = stable, below 0.5 = doubtful).
• Multiscale bootstrap p-values for hierarchical clusters (AU / BP): which branches of the tree are supported by the data.
• Variable removal: clusters that vanish when one variable is dropped depend on that variable alone.

External validation compares the clusters with a known grouping (species, origin, treatment): Rand and adjusted Rand index, normalised mutual information, purity, variation of information, plus a cross-table with a chi-square test. The same indices compare two clusterings with each other.

Comparing algorithms. Run hierarchical, k-means, PAM and model-based side by side over a range of k and rank them by connectivity, Dunn and silhouette before you commit.