{ "cells": [ { "cell_type": "markdown", "id": "e5fb7eea-a8ac-46c6-9e33-27a647b5f414", "metadata": {}, "source": [ "# Clustering Example\n", "\n", "This example assumes that the `compresso-pytorch` and `compresso-recsys` packages are installed:\n", "\n", "```bash\n", "pip install \"compresso-pytorch@git+https://github.com/zombak79/compresso.git\"\n", "pip install \"compresso-recsys@git+https://github.com/zombak79/compresso-recsys.git\"\n", "```\n", "\n", "This notebook demonstrates a basic clustering workflow. We first import Compresso and its recommender-system extension, then download Amazon metadata and build a checkpoint for the experiment.\n" ] }, { "cell_type": "markdown", "id": "a6577858-9606-4507-9da4-0fe17410aa45", "metadata": {}, "source": [ "## Build checkpoint & encode item descriptions" ] }, { "cell_type": "code", "execution_count": 1, "id": "169f0da5-1b87-4266-9302-65415fd9ea8e", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "24f594fa87f946a3b48813417c18c719", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Building checkpoint: 0%| | 0/6 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
item_idtitleentity_textimage_url
1448B01BYKUI9CHP 65 Black Ink Cartridge | Works with HP AMP ...Title: HP 65 Black Ink Cartridge | Works with ...https://m.media-amazon.com/images/I/61GAiSy74W...
2115B07F9SVWXWQuartet Dry Erase Markers, Whiteboard Markers,...Title: Quartet Dry Erase Markers, Whiteboard M...https://m.media-amazon.com/images/I/61b1qTIOLB...
1921B078Z65VQYCanon PIXMA G4210 Wireless All-In-One Supertan...Title: Canon PIXMA G4210 Wireless All-In-One S...https://m.media-amazon.com/images/I/81sbe681tY...
4175B0BXR5F7LSWestcott B-70 8ths Graph Beveled Ruler, 12 inTitle: Westcott B-70 8ths Graph Beveled Ruler,...https://m.media-amazon.com/images/I/81e+DtAFL5...
134B0003WN0DOSharpie Fine Electro Pop Marker, Fine Point, A...Title: Sharpie Fine Electro Pop Marker, Fine P...https://m.media-amazon.com/images/I/81W3-nzwxi...
\n", "" ], "text/plain": [ " item_id title \\\n", "1448 B01BYKUI9C HP 65 Black Ink Cartridge | Works with HP AMP ... \n", "2115 B07F9SVWXW Quartet Dry Erase Markers, Whiteboard Markers,... \n", "1921 B078Z65VQY Canon PIXMA G4210 Wireless All-In-One Supertan... \n", "4175 B0BXR5F7LS Westcott B-70 8ths Graph Beveled Ruler, 12 in \n", "134 B0003WN0DO Sharpie Fine Electro Pop Marker, Fine Point, A... \n", "\n", " entity_text \\\n", "1448 Title: HP 65 Black Ink Cartridge | Works with ... \n", "2115 Title: Quartet Dry Erase Markers, Whiteboard M... \n", "1921 Title: Canon PIXMA G4210 Wireless All-In-One S... \n", "4175 Title: Westcott B-70 8ths Graph Beveled Ruler,... \n", "134 Title: Sharpie Fine Electro Pop Marker, Fine P... \n", "\n", " image_url \n", "1448 https://m.media-amazon.com/images/I/61GAiSy74W... \n", "2115 https://m.media-amazon.com/images/I/61b1qTIOLB... \n", "1921 https://m.media-amazon.com/images/I/81sbe681tY... \n", "4175 https://m.media-amazon.com/images/I/81e+DtAFL5... \n", "134 https://m.media-amazon.com/images/I/81W3-nzwxi... " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "with cr.read_checkpoint(checkpoint_path) as root:\n", " split = cr.load_recsys_split(root)\n", " meta = split[\"entity_metadata\"]\n", "\n", "meta[[\"item_id\", \"title\", \"entity_text\", \"image_url\"]].sample(5)" ] }, { "cell_type": "markdown", "id": "f5b058d3-475d-4b8c-9ebf-e4818dffa8bf", "metadata": {}, "source": [ "Next, we create semantic embeddings with the `sentence_transformers` library.\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "1297191e-98c8-45c3-bf0d-7742b66a2458", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "044a7771f90e4f8e998f29d54de1254d", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Loading weights: 0%| | 0/103 [00:00 str:\n", " \n", " prompt = f\"\"\"You are given a list of items, each with metadata.\n", "\n", " 1. Identify the most common concept that links all the items\n", " 2. Convert that concept into a segment name suitable for a recommendation system.\n", " \n", " The segment name must be title-case, concise (2-5 words), and sound natural — like a content \n", " category. Prefer distinctive labels over generic ones.\n", " \n", " Output: Only the final segment name, nothing else.\n", " \n", " Items:\n", " {items}\n", " \n", " Segment name:\"\"\"\n", "\n", " response = ollama.chat(\n", " model=MODEL,\n", " messages=[\n", " {\n", " \"role\": \"system\",\n", " \"content\": (\n", " \"You are a strict classification model. \"\n", " \"Return only one segment name. Never explain.\"\n", " ),\n", " },\n", " {\n", " \"role\": \"user\",\n", " \"content\": prompt,\n", " },\n", " ],\n", " options={\n", " \"temperature\": 0.0,\n", " \"num_predict\": 1000,\n", " \"num_ctx\": 8192,\n", " },\n", " think=\"low\",\n", " stream=False,\n", " )\n", "\n", " content = response[\"message\"][\"content\"] if response[\"message\"][\"content\"] is not None else None\n", " \n", " return content\n", "\n", "def item_texts(cluster: compresso.clustering.types.SparseCluster, metadata: pd.DataFrame, limit: int = 50) -> str:\n", " rows = metadata.iloc[cluster.entity_indices] # select rows with items that are members of the cluster\n", " if len(rows) > limit: # optionally sample maximum allowed number of items to save tokens\n", " rows = rows.sample(limit)\n", " return \"\\n\".join(rows[\"title\"].fillna(\"\").tolist()) # just return item titles, one per row\n", "\n", "\n", "print(\"Items texts for the first cluster in the graph:\\n\")\n", "texts = item_texts(cluster_graph.clusters[0], meta, limit=10)\n", "print(texts)\n", "\n", "print(\"\\nLLM label for the first cluster:\", label_cluster(texts))" ] }, { "cell_type": "markdown", "id": "36ef2d7d-2414-4ac2-bfdf-eab3ad0aa58a", "metadata": {}, "source": [ "Now we can apply the labeling function to the full cluster graph.\n" ] }, { "cell_type": "code", "execution_count": 9, "id": "38f4d084-a492-4db7-b88d-13284f410493", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "711f0b9b34b04666a7dc4cc847d72ada", "version_major": 2, "version_minor": 0 }, "text/plain": [ "label_clusters: 0%| | 0/28 [00:00\n", " \n", " \n", " Cluster\n", " Items (sample)\n", " \n", " \n", " \n", " \n", " Desktop & Handheld Calculators\n", "
\n", " \n", " \n", " Office Organization Essentials\n", "
\n", " \n", " \n", " Office & Packaging Supplies\n", "
\n", " \n", " \n", " Photo Printing Paper\n", "
\n", " \n", " \n", " HP Printer Ink\n", "
\n", " \n", " \n", " Paper Trimmers\n", "
\n", " \n", " \n", " Dry Erase & Marker Supplies\n", "
\n", " \n", " \n", " Binder Tab Dividers\n", "
\n", " \n", " \n", " Notebook & Paper\n", "
\n", " \n", " \n", " Pens & Pencils\n", "
\n", " \n", " \n", " Pencil Sharpeners\n", "
\n", " \n", " \n", " Premium Colored Pencils\n", "
\n", " \n", " \n", " Label & File Organization\n", "
\n", " \n", " \n", " Index Card Sets\n", "
\n", " \n", " \n", " Staplers & Staples\n", "
\n", " \n", " \n", " Envelopes\\n\\n\n", "
\n", " \n", " \n", " Wireless All‑In‑One Printers\n", "
\n", " \n", " \n", " Thermal Laminators & Pouches\n", "
\n", " \n", " \n", " Office Printer Paper\n", "
\n", " \n", " \n", " Ergonomic Gaming Desk Mat\n", "
\n", " \n", " \n", " Cordless Home Phone Systems\n", "
\n", " \n", " \n", " Epson All‑In‑One Printing Solutions\n", "
\n", " \n", " \n", " Tombow Mono Stationery Essentials\n", "
\n", " \n", " \n", " Multi‑Compartment Desk Caddy\n", "
\n", " \n", " \n", " Electric Pencil Sharpeners\n", "
\n", " \n", " \n", " Paper Shredders\n", "
\n", " \n", " \n", " Planner & Calendar Collection\n", "
\n", " \n", " \n", " Office Chair Mats\n", "
\n", " \n", " \n", "" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "from IPython.display import HTML\n", "\n", "# construct a dataframe with cluster label and 5 random sampled item images belonging to the cluster\n", "def visualize_clusters(cluster_graph):\n", " data = [\n", " (\n", " cluster_graph.clusters[i].label,\n", " meta.iloc[cluster_graph.clusters[i].entity_indices].image_url.sample(5).to_list()\n", " ) for i in range(len(cluster_graph.clusters))\n", " ]\n", " \n", " df = pd.DataFrame(data, columns=[\"Cluster\", \"image_urls\"])\n", "\n", " def image_gallery(urls):\n", " imgs = [\n", " f''\n", " for url in urls\n", " ]\n", " return \"
\" + \"\".join(imgs) + \"
\"\n", "\n", " df_render = df.copy()\n", " df_render[\"Items (sample)\"] = df_render[\"image_urls\"].map(image_gallery)\n", " \n", " html = df_render[[\"Cluster\", \"Items (sample)\"]].to_html(\n", " escape=False,\n", " index=False\n", " )\n", "\n", " return HTML(html)\n", "\n", "visualize_clusters(cluster_graph)" ] }, { "cell_type": "markdown", "id": "50be2702-7915-4aa4-b38f-9002a96c44a8", "metadata": {}, "source": [ "## Using clustering pipelines\n", "\n", "The same steps can be combined with `ClusteringPipeline`. Results should be similar, although labels may vary because they are generated by an LLM.\n" ] }, { "cell_type": "code", "execution_count": 12, "id": "3cb2c206-e44a-4b9b-a27e-12d34d86260c", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1c02e6daa51046cd9689773be30386d0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "build_srp_similarity_clusters: batches: 0%| | 0/139 [00:00\n", " \n", " \n", " Cluster\n", " Items (sample)\n", " \n", " \n", " \n", " \n", " Office & Home Standard Calculators\n", "
\n", " \n", " \n", " Office Organization Essentials\n", "
\n", " \n", " \n", " Office Stationery & Packing\n", "
\n", " \n", " \n", " Photo Printing Paper\n", "
\n", " \n", " \n", " HP Printer Ink\n", "
\n", " \n", " \n", " Paper Trimmers\n", "
\n", " \n", " \n", " Dry Erase & Whiteboard Supplies\n", "
\n", " \n", " \n", " Binder Tab Organizers\n", "
\n", " \n", " \n", " Notebook & Paper\n", "
\n", " \n", " \n", " Pens & Pencils\n", "
\n", " \n", " \n", " School & Office Pencil Sharpeners\n", "
\n", " \n", " \n", " Premium Colored Pencils\n", "
\n", " \n", " \n", " Labeling & Filing Essentials\n", "
\n", " \n", " \n", " Index Card Essentials\n", "
\n", " \n", " \n", " Staplers & Accessories\n", "
\n", " \n", " \n", " Secure Mailing Envelopes\n", "
\n", " \n", " \n", " Ink & Toner Supplies\n", "
\n", " \n", " \n", " Thermal Laminators & Pouches\n", "
\n", " \n", " \n", " Printing Paper\n", "
\n", " \n", " \n", " Ergonomic Gaming Desk Mat\n", "
\n", " \n", " \n", " Cordless Home Phone Systems\n", "
\n", " \n", " \n", " Office All‑In‑One Printers\n", "
\n", " \n", " \n", " Mono Stationery Essentials\n", "
\n", " \n", " \n", " Desk Organizers\n", "
\n", " \n", " \n", " Electric Pencil Sharpeners\n", "
\n", " \n", " \n", " Office Paper Shredder\n", "
\n", " \n", " \n", " Planner & Calendar Collection\n", "
\n", " \n", " \n", " Office Chair Mats\n", "
\n", " \n", " \n", "" ], "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pipeline_cluster_graph = cc.ClusteringPipeline(\n", " [\n", " cc.SRPSimilarityClustering(\n", " threshold=0.5, # minimum semantic similarity between items inside a cluster\n", " top_k=None, # None = all pairs above threshold\n", " min_cluster_size=20, # smaller clusters are discarded \n", " normalize_rows=True, # centroids of the cluster will be normalized\n", " min_local_density=None, # optional cleanup\n", " centroid_top_k=8, # how many top features are included in centroid definition\n", " batch_size=32,\n", " show_progress=True,\n", " ),\n", " cc.LabelClusters(\n", " entity_metadata=meta,\n", " text_extractor=item_texts,\n", " label_fn=label_cluster,\n", " cluster_scope=\"all\",\n", " show_progress=True,\n", " ),\n", " ]\n", ")(sparse_embeddings)\n", " \n", "visualize_clusters(pipeline_cluster_graph)" ] }, { "cell_type": "code", "execution_count": null, "id": "f4ff0d86-4c42-40d2-968d-b6aef33e6d9a", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.7" } }, "nbformat": 4, "nbformat_minor": 5 }