Clustering API

The clustering API is under active development. The objects documented here are the intended public entry points exported by compresso.clustering.

Types

class compresso.clustering.SparseVector(indices, values, size)[source]

Small sparse vector used as a reusable cluster handle.

Parameters:

  • indices (ndarray)

  • values (ndarray)

  • size (int)

class compresso.clustering.ScoredTag(tag_id, name, score, count=0.0, metadata=<factory>)[source]
Parameters:

  • tag_id (int)

  • name (str)

  • score (float)

  • count (float)

  • metadata (Mapping[str, Any])

class compresso.clustering.SparseCluster(cluster_id, centroid, entity_indices, source_cluster_ids=(), parent_cluster_ids=(), child_cluster_ids=(), tags=(), label=None, description=None, stats=<factory>, metadata=<factory>)[source]
Parameters:

  • cluster_id (str)

  • centroid (SparseVector)

  • entity_indices (ndarray)

  • source_cluster_ids (tuple[str, ...])

  • parent_cluster_ids (tuple[str, ...])

  • child_cluster_ids (tuple[str, ...])

  • tags (tuple[ScoredTag, ...])

  • label (str | None)

  • description (str | None)

  • stats (Mapping[str, Any])

  • metadata (Mapping[str, Any])

compresso.clustering.SparseClusterGraph

alias of SparseClusterSet

class compresso.clustering.SparseClusterSet(clusters, n_entities, n_features, active_cluster_ids=None, entity_ids=None, feature_ids=None, assignment_mode='dominant_signed', history=(), metadata=<factory>)[source]
Parameters:

  • clusters (tuple[SparseCluster, ...])

  • n_entities (int)

  • n_features (int)

  • active_cluster_ids (tuple[str, ...] | None)

  • entity_ids (ndarray | None)

  • feature_ids (ndarray | None)

  • assignment_mode (str)

  • history (tuple[Mapping[str, Any], ...])

  • metadata (Mapping[str, Any])

property entity_to_cluster_ids: dict[int, list[str]]

Map entity index to active cluster ids.

fill_missing_cluster_labels(label_fn=None, text_fn=None, *, roots=None, active_roots_only=True, overwrite=False, verbose=False)[source]

Count or fill missing labels below selected graph roots.

When label_fn is not provided this is a count-only traversal. When label_fn is provided, the method returns a new graph with filled labels and leaves the original graph unchanged.

Return type:

tuple[SparseClusterSet, int, int]

Parameters:

  • label_fn (Callable[[object], object] | None)

  • text_fn (Callable[[SparseCluster], object] | None)

  • roots (Iterable[str | SparseCluster] | None)

  • active_roots_only (bool)

  • overwrite (bool)

  • verbose (bool)

Pipeline

class compresso.clustering.ClusteringPipeline(steps, verbose=False)[source]

Composable class-based sparse clustering pipeline.

A pipeline starts with one clustering step that consumes an SRPTensor. Every later step consumes and returns a SparseClusterSet. This separates discovery from optional graph linking, merging, labeling, tagging, and filtering.

Parameters:

Examples

>>> graph = ClusteringPipeline([
...     TopMSignedClustering(top_m=3, min_cluster_size=5),
...     EntityContainmentLink(threshold=1.0),
...     MaterializeLinkMerges(),
...     SizeFilter(min_cluster_size=10),
... ]).fit(srp)
class compresso.clustering.AbstractClusterTransform[source]

Base class for pipeline steps that transform an existing cluster graph.

Transform steps consume and return a SparseClusterSet. They are used after a clustering step for linking, merging, labeling, tagging, or filtering clusters.

class compresso.clustering.AbstractClustering[source]

Base class for pipeline steps that build clusters from an SRPTensor.

Clustering steps are the first step in a ClusteringPipeline. They consume sparse entity representations and return a SparseClusterSet.

class compresso.clustering.AbstractMerging[source]

Base class for transforms that create merge-parent clusters.

Merge steps are non-destructive by convention: original clusters remain in the graph as children and new parent nodes become the active frontier.

Clustering Steps

class compresso.clustering.DominantSignedClustering(min_cluster_size=1, show_progress=False)[source]

Cluster each entity by its single strongest signed feature.

This is the simplest activation clustering mode. For every SRP row, the feature with largest absolute activation is selected, including its sign. Entities with the same (feature, sign) key form one cluster.

Parameters:

  • min_cluster_size (int) – Drop clusters with fewer entities than this value during construction.

  • show_progress (bool) – Show a tqdm progress bar while assigning rows.

class compresso.clustering.TopMSignedClustering(top_m=1, min_cluster_size=1, show_progress=False)[source]

Cluster each entity by each of its top-m signed features.

Every entity may belong to multiple clusters: one for each of its strongest top_m signed feature activations. With top_m=1 this is equivalent to DominantSignedClustering.

Parameters:

  • top_m (int) – Number of strongest features per entity to use.

  • min_cluster_size (int) – Drop clusters with fewer entities than this value.

  • show_progress (bool) – Show tqdm progress while building clusters.

class compresso.clustering.ComboSignedClustering(top_m=1, combo_size=1, min_cluster_size=1, show_progress=False)[source]

Cluster entities by exact combinations of signed features.

For each entity, take its top top_m signed features and create exact AND-combination keys of length combo_size. Two entities share a cluster only when they contain the same signed feature combination.

Parameters:

  • top_m (int) – Candidate feature pool size for each entity.

  • combo_size (int) – Size of exact signed-feature combinations.

  • min_cluster_size (int) – Drop clusters with fewer entities than this value.

  • show_progress (bool) – Show tqdm progress while building clusters.

class compresso.clustering.FeaturePathClustering(top_m=1, max_depth=None, min_cluster_size=None, min_activation=None, show_progress=False)[source]

Build a hierarchy by recursively splitting on next strongest features.

At the root level each entity starts from one of its top top_m signed features. Inside each cluster, entities are split again by the next strongest feature not already used in the path. This creates an explicit feature-path hierarchy such as feature A -> feature B -> feature C.

Parameters:

  • top_m (int) – Number of starting signed features per entity.

  • max_depth (int | None) – Maximum feature-path depth. None allows paths up to available sparse features.

  • min_cluster_size (int | None) – Stop or drop nodes smaller than this size. None disables the size stop.

  • min_activation (float | None) – Ignore candidate features whose absolute activation is below this value. None disables the threshold.

  • show_progress (bool) – Show tqdm progress while building paths.

class compresso.clustering.SRPSimilarityClustering(threshold, top_k=100, min_cluster_size=2, normalize_rows=True, min_local_density=None, centroid_top_k=None, batch_size=1024, show_progress=False)[source]

Build clusters as connected components in SRP similarity space.

The SRP rows are compared by dense similarity, edges with similarity above threshold are kept, and connected components become clusters. This is useful when the cluster should be defined by representation similarity rather than by a shared feature id.

Parameters:

  • threshold (float) – Minimum pairwise similarity for an edge.

  • top_k (int | None) – Limit each row to its top-k nearest neighbors before thresholding. None compares against all rows.

  • min_cluster_size (int) – Drop connected components smaller than this size.

  • normalize_rows (bool) – If True, L2-normalize rows before similarity computation.

  • min_local_density (float | None) – Optional fraction of within-component neighbors an entity must keep to remain in a component.

  • centroid_top_k (int | None) – If provided, keep only this many largest centroid features.

  • batch_size (int) – Number of rows scored per matrix multiplication batch.

  • show_progress (bool) – Show tqdm progress while scoring batches.

Linking and Merging

Add parent-child links based on entity containment without merging nodes.

This is a graph-linking step, not a merge step. It preserves the active frontier and only adds DAG edges from contained child clusters to broader parent clusters. A child may receive multiple parents.

Parameters:

  • threshold (float) – Minimum entity containment score in [0, 1].

  • child_scope (Literal['active', 'all', 'leaves', 'roots']) – Which clusters may become children: "active", "all", "leaves", or "roots".

  • parent_scope (Literal['active', 'all', 'leaves', 'roots']) – Which clusters may become parents.

  • require_parent_larger (bool) – Require the parent candidate to contain more entities than the child.

  • skip_existing_ancestors (bool) – Avoid adding links that are already implied by existing ancestry.

  • verbose (bool) – Print linking progress.

  • show_progress (bool) – Show tqdm progress while scanning child clusters.

Add parent-child links based on feature-support containment.

This is the feature analogue of EntityContainmentLink. It creates graph edges without creating merge nodes or changing the active frontier.

Parameters:

  • threshold (float) – Minimum feature-containment score in [0, 1].

  • signed (bool) – Compare signed (feature, sign) pairs instead of feature ids only.

  • child_scope (Literal['active', 'all', 'leaves', 'roots']) – Which clusters may become children.

  • parent_scope (Literal['active', 'all', 'leaves', 'roots']) – Which clusters may become parents.

  • require_parent_larger (bool) – Require the parent candidate to have a larger feature support.

  • skip_existing_ancestors (bool) – Avoid adding links already implied by existing ancestry.

  • verbose (bool) – Print linking progress.

  • show_progress (bool) – Show tqdm progress while scanning child clusters.

class compresso.clustering.MaterializeLinkMerges(parent_scope='active', include_descendants=False, min_children=1, normalize_centroids=True, activate=True, verbose=False)[source]

Convert existing links into explicit merge-parent nodes.

Link steps create edges only. This transform adds a new non-destructive parent node for linked structures so that downstream renderers and recommendation code can treat the linked group as a concrete cluster.

Parameters:

  • parent_scope (Literal['active', 'all', 'leaves', 'roots']) – Which linked parent clusters should be materialized.

  • include_descendants (bool) – If True, include all descendants of a linked parent, not just its direct children.

  • min_children (int) – Minimum number of linked children required to create a materialized node.

  • normalize_centroids (bool) – L2-normalize materialized parent centroids.

  • activate (bool) – If True, add materialized nodes to the active frontier.

  • verbose (bool) – Print materialization details.

class compresso.clustering.EntityIoUMerge(threshold, max_rounds=10, normalize_centroids=True, verbose=False, show_progress=False)[source]

Merge active clusters whose entity sets have high intersection-over-union.

Active clusters are treated as nodes in a similarity graph. Pairs with entity IoU greater than or equal to threshold are connected, connected components are materialized as new parent clusters, and the process repeats until convergence or max_rounds.

Parameters:

  • threshold (float) – Minimum entity IoU in [0, 1].

  • max_rounds (int) – Maximum iterative merge rounds.

  • normalize_centroids (bool) – L2-normalize newly created parent centroids.

  • verbose (bool) – Print merge progress.

  • show_progress (bool) – Show tqdm progress while comparing clusters.

class compresso.clustering.EntityContainmentMerge(threshold=1.0, max_rounds=10, normalize_centroids=True, verbose=False, show_progress=False)[source]

Merge active clusters when one entity set is mostly contained in another.

Containment is |A intersection B| / min(|A|, |B|). A strict subset has score 1.0 even when IoU is small. This is useful for collapsing small highly-specific clusters into broader parents.

Parameters:

  • threshold (float) – Minimum containment score in [0, 1].

  • max_rounds (int) – Maximum iterative merge rounds.

  • normalize_centroids (bool) – L2-normalize newly created parent centroids.

  • verbose (bool) – Print merge progress.

  • show_progress (bool) – Show tqdm progress while comparing clusters.

class compresso.clustering.FeatureContainmentMerge(threshold=1.0, signed=True, max_rounds=10, normalize_centroids=True, verbose=False, show_progress=False)[source]

Merge active clusters when feature supports are mostly contained.

Feature containment is |F_A intersection F_B| / min(|F_A|, |F_B|). With signed=True, positive and negative use of the same feature are treated as different support elements.

Parameters:

  • threshold (float) – Minimum feature-containment score in [0, 1].

  • signed (bool) – Compare signed (feature, sign) pairs instead of feature ids only.

  • max_rounds (int) – Maximum iterative merge rounds.

  • normalize_centroids (bool) – L2-normalize newly created parent centroids.

  • verbose (bool) – Print merge progress.

  • show_progress (bool) – Show tqdm progress while comparing clusters.

class compresso.clustering.CentroidSimilarityMerge(threshold, metric='cosine', top_k=None, max_rounds=10, min_group_size=2, normalize_centroids=True, verbose=False, show_progress=False)[source]

Merge active clusters whose sparse centroids are similar.

This creates non-destructive parent nodes from connected components in a centroid-similarity graph. It is useful after initial clustering when different feature handles point to nearby regions in sparse space.

Parameters:

  • threshold (float) – Minimum centroid similarity. For cosine this must be in [-1, 1].

  • metric (Literal['cosine', 'dot']) – Similarity function: "cosine" or raw "dot" product.

  • top_k (int | None) – Optional nearest-neighbor restriction. A pair is considered only when one cluster is in the other’s top-k neighbors.

  • max_rounds (int) – Maximum iterative merge rounds.

  • min_group_size (int) – Minimum connected component size to materialize.

  • normalize_centroids (bool) – L2-normalize newly created parent centroids.

  • verbose (bool) – Print merge progress.

  • show_progress (bool) – Show tqdm progress while comparing centroids.

class compresso.clustering.LabelDuplicateMerge(cluster_scope='active', case_sensitive=False, mark_children_hidden=True, min_group_size=2, normalize_centroids=True, verbose=False)[source]

Merge clusters that have exactly the same label string.

This is useful after LLM or rule-based labeling, where duplicate labels can indicate duplicate semantic segments. When mark_children_hidden=True, merged children are preserved but marked as hidden for later compaction or UI rendering.

Parameters:

  • cluster_scope (Literal['active', 'all', 'leaves', 'roots']) – Which clusters are considered for duplicate-label grouping.

  • case_sensitive (bool) – If False, normalize labels by case before grouping.

  • mark_children_hidden (bool) – Mark duplicate children with metadata key render_hidden.

  • min_group_size (int) – Minimum number of clusters with the same label required to merge.

  • normalize_centroids (bool) – L2-normalize newly created parent centroids.

  • verbose (bool) – Print merge summary.

class compresso.clustering.SemanticSimilarityMerge(embed_fn, threshold=0.9, text_fn=None, label_fn=None, label_text_fn=None, cluster_scope='active', max_rounds=10, min_group_size=2, normalize_embeddings=True, normalize_centroids=True, verbose=False, show_progress=False)[source]

Merge clusters whose labels/descriptions are semantically similar.

The user supplies embed_fn so Compresso does not depend on a specific embedding model or API key. Candidate cluster texts are embedded, similar clusters are grouped with a maximal-clique strategy, and new non-destructive semantic parent nodes are created. Optional callbacks can label those new parent nodes.

Parameters:

  • embed_fn (Callable[[list[str]], ndarray]) – Function mapping a list of cluster texts to a 2D NumPy embedding array.

  • threshold (float) – Minimum cosine similarity between cluster text embeddings.

  • text_fn (Callable[[SparseCluster], str] | None) – Function mapping a cluster to text. Defaults to cluster description, then label, then empty text.

  • label_fn (Callable[[object], object] | None) – Optional user function that names a newly created semantic parent.

  • label_text_fn (Callable[[SparseCluster, list[SparseCluster]], object] | None) – Optional function that builds the input object passed to label_fn from the parent cluster and its children.

  • cluster_scope (Literal['active', 'all', 'leaves', 'roots']) – Which clusters should be considered for semantic merging.

  • max_rounds (int) – Maximum iterative semantic merge rounds.

  • min_group_size (int) – Minimum similar group size to materialize.

  • normalize_embeddings (bool) – L2-normalize text embeddings before similarity.

  • normalize_centroids (bool) – L2-normalize newly created parent centroids.

  • verbose (bool) – Print merge progress.

  • show_progress (bool) – Show tqdm progress while processing rounds.

class compresso.clustering.TagSimilarityMerge(threshold, metric='weighted_jaccard', max_rounds=10, normalize_centroids=True, verbose=False, show_progress=False)[source]

Merge active clusters with similar assigned tag profiles.

Run AssignTags before this step. Tags are compared either by weighted Jaccard over tag scores or by cosine similarity of tag-score vectors. Matching groups are materialized as non-destructive parents.

Parameters:

  • threshold (float) – Minimum tag-profile similarity in [0, 1].

  • metric (Literal['weighted_jaccard', 'cosine']) – Tag similarity metric: "weighted_jaccard" or "cosine".

  • max_rounds (int) – Maximum iterative merge rounds.

  • normalize_centroids (bool) – L2-normalize newly created parent centroids.

  • verbose (bool) – Print merge progress.

  • show_progress (bool) – Show tqdm progress while comparing clusters.

Post-processing

class compresso.clustering.CompactHiddenClusters(hidden_key='render_hidden', verbose=False)[source]

Remove hidden clusters and rewire visible graph edges.

Hidden nodes are usually produced by LabelDuplicateMerge with mark_children_hidden=True. This transform physically removes them from the graph while preserving visible ancestor/descendant connectivity.

Parameters:

  • hidden_key (str) – Metadata key used to decide whether a cluster should be removed.

  • verbose (bool) – Print compaction summary.

class compresso.clustering.PruneRedundantRoots(verbose=False)[source]

Remove active clusters that are descendants of another active cluster.

This is usually used after link/materialization steps to make the active frontier easier to render. It does not delete nodes; it only updates active_cluster_ids.

Parameters:

verbose (bool) – Print pruning summary.

class compresso.clustering.AssignTags(entity_tag_matrix, tag_names, method='tfidf', top_k=5, min_score=0.0)[source]

Assign top tags to clusters from an entity-tag matrix.

The tag matrix is expected to be shaped (n_entities, n_tags) and can be a dense or sparse matrix-like object accepted by the implementation. Tags are aggregated over each cluster’s entity indices and stored as ScoredTag objects.

Parameters:

  • entity_tag_matrix (object) – Matrix containing entity-tag counts or weights.

  • tag_names (Sequence[str]) – Names corresponding to columns of entity_tag_matrix.

  • method (Literal['tfidf', 'counts']) – Scoring method: "tfidf" downweights globally common tags, while "counts" uses raw aggregate counts.

  • top_k (int) – Maximum number of tags stored per cluster.

  • min_score (float) – Drop tags whose score is not greater than this threshold.

class compresso.clustering.LabelClusters(entity_metadata, text_extractor, label_fn, cluster_scope='active', overwrite=False, verbose=False, show_progress=False)[source]

Populate cluster labels/descriptions with user-provided callbacks.

Compresso coordinates the loop but does not own prompting, API keys, or model initialization. text_extractor converts a cluster plus metadata into a domain-specific input object, and label_fn converts that object into a label result. The result may be a string, (label, description), or a mapping understood by the implementation.

Parameters:

  • entity_metadata (object) – Metadata table or object used by text_extractor.

  • text_extractor (Callable[[SparseCluster, object], object]) – Callable (cluster, entity_metadata) -> object.

  • label_fn (Callable[[object], object]) – Callable that returns a label/description for one extracted text object.

  • cluster_scope (Literal['active', 'all', 'leaves', 'roots']) – Which clusters should be labeled.

  • overwrite (bool) – If False, skip clusters that already have a label.

  • verbose (bool) – Print labeling progress.

  • show_progress (bool) – Show tqdm progress while labeling clusters.

class compresso.clustering.AssignUnclusteredToNearestCluster(srp, metric='cosine', min_similarity=None, top_k_clusters=1, cluster_scope='active', coverage_scope='active', assigned_weight=1.0, centroid_top_m=None, centroid_top_k=None, normalize_centroids=True, verbose=False)[source]

Expand coverage by assigning uncovered entities to nearest clusters.

This step does not create singleton clusters. Instead, for each cluster that receives newly assigned entities, it creates an expanded parent node with the original cluster as a child. This preserves the discovered core cluster while adding coverage for entities outside the current frontier.

Parameters:

  • srp (SRPTensor) – Original entity representations used for nearest-cluster assignment.

  • metric (Literal['cosine', 'dot']) – Similarity between entity rows and cluster centroids: "cosine" or "dot".

  • min_similarity (float | None) – Optional minimum similarity required for assignment.

  • top_k_clusters (int) – Number of nearest clusters to assign each uncovered entity to.

  • cluster_scope (Literal['active', 'all', 'leaves', 'roots']) – Which clusters can receive assignments.

  • coverage_scope (Literal['active', 'all']) – Which clusters count as already covering an entity: "active" or "all".

  • assigned_weight (float) – Weight of newly assigned entity vectors when recomputing expanded centroids.

  • centroid_top_m (int | None) – Keep only this many largest centroid features in expanded parents.

  • centroid_top_k (int | None) – Deprecated alias for centroid_top_m.

  • normalize_centroids (bool) – L2-normalize expanded parent centroids.

  • verbose (bool) – Print assignment summary.

class compresso.clustering.SizeFilter(min_cluster_size)[source]

Keep only active clusters with at least min_cluster_size entities.

This changes the active frontier only; filtered-out clusters remain in the graph and can still be inspected through clusters.clusters.

Parameters:

min_cluster_size (int) – Minimum entity count required for an active cluster to stay active.

Persistence and Helpers

compresso.clustering.save_cluster_graph(graph, path)[source]
Return type:

Path

Parameters:
compresso.clustering.load_cluster_graph(path)[source]
Return type:

SparseClusterSet

Parameters:

path (str | Path)

compresso.clustering.graph_to_dict(graph)[source]
Return type:

dict[str, Any]

Parameters:

graph (SparseClusterSet)

compresso.clustering.graph_from_dict(data)[source]
Return type:

SparseClusterSet

Parameters:

data (Mapping[str, Any])

compresso.clustering.cluster_srp(srp, *, mode='dominant_signed', top_m=1, combo_size=1, min_cluster_size=1, post_merge_min_cluster_size=None, entity_ids=None, activation_iou_threshold=None, entity_tag_matrix=None, tag_names=None, tag_method='tfidf', top_k_tags=5, tag_similarity_threshold=None, tag_similarity_metric='weighted_jaccard', max_merge_rounds=10, verbose=False, show_progress=False)[source]

Convenience pipeline for sparse clustering.

Semantic labeling/merging is intentionally not part of v1; callers can add it later as callbacks operating on the returned SparseClusterSet.

Return type:

SparseClusterSet

Parameters:

  • srp (SRPTensor)

  • mode (Literal['dominant_signed', 'top_m_signed', 'combo_signed'])

  • top_m (int)

  • combo_size (int)

  • min_cluster_size (int)

  • post_merge_min_cluster_size (int | None)

  • entity_ids (ndarray | None)

  • activation_iou_threshold (float | None)

  • tag_names (Sequence[str] | None)

  • tag_method (Literal['tfidf', 'counts'])

  • top_k_tags (int)

  • tag_similarity_threshold (float | None)

  • tag_similarity_metric (Literal['weighted_jaccard', 'cosine'])

  • max_merge_rounds (int)

  • verbose (bool)

  • show_progress (bool)

compresso.clustering.run_clustering_pipeline(srp, steps, *, verbose=False)[source]

Run an explicit sparse clustering pipeline.

The first step consumes an SRPTensor and must return a SparseClusterSet. Every following step consumes and returns a SparseClusterSet. This keeps base cluster construction separate from optional merge/tag/filter phases while staying easy to compose with functools.partial or small lambdas.

Return type:

SparseClusterSet

Parameters: