RAILS: Retrieval-Augmented Incremental LLM Clustering at Scale
RAILS reports better clustering scores by pairing LLM labeling with retrieval and incremental batching.
The paper says full-label prompts and serial per-document LLM clustering do not scale cleanly. Its method keeps a growing label pool, retrieves against it, and uses bounded concurrent batches. Across six public benchmarks, it reports gains over the strongest prior LLM-clustering method: 59.3% accuracy, 74.8% NMI, and 54.7% ARI. The authors also say it has replaced an HDBSCAN stage in a SaaS ticket-topic-discovery pipeline. ArXiv · AI/CL/LG's note
The paper says full-label prompts and serial per-document LLM clustering do not scale cleanly. Its method keeps a growing label pool, retrieves against it, and uses bounded concurrent batches. Across six public benchmarks, it reports gains over the strongest prior LLM-clustering method: 59.3% accuracy, 74.8% NMI, and 54.7% ARI. The authors also say it has replaced an HDBSCAN stage in a SaaS ticket-topic-discovery pipeline. ArXiv · AI/CL/LG's note
score 5