Long online discussions, such as a ChangeMyView thread with hundreds of replies, are often condensed by a short unsupervised pipeline. Each comment is embedded as a vector, the vectors are compressed to a few dimensions, similar comments are grouped into clusters, and a language model labels each cluster so readers see a table of contents instead of a wall of text. Every stage has settings a person picks by hand: how many dimensions to keep, how many neighbours count as "local", and how small a group can be before it is treated as noise. These settings behave differently on a 40-comment thread and a 2,000-comment thread. Yet published systems pick one configuration heuristically and apply it everywhere. A deployed system also cannot tune per thread, because a new thread arrives with no reference summary to score against.
This note proposes to treat each discussion thread as a separate tuning task and to learn, from threads that do have reference key points, a model that predicts good pipeline settings for a new thread from label-free features of that thread, such as its size, embedding density and reply depth. It then asks which settings actually matter. The deliverable is either a learned per-thread configuration predictor or, if the analysis shows one or two settings dominate, a simple scaling rule that practitioners can adopt. The claim is falsifiable: per-thread predicted settings should give higher key-point coverage on held-out threads than the best single global configuration tuned with the same budget.
Core Claim
The quality of an unsupervised discussion summary depends mostly on a small number of pipeline settings, and their best values vary systematically with measurable properties of the thread. If that is true, hyperparameter transfer learning can close most of the gap between one global configuration and an oracle configuration tuned per thread with labels. Hyperparameter transfer learning here means reusing tuning results from past tasks to choose settings for a new task. The novelty is in treating the thread, rather than the dataset, as the unit of tuning, and in doing so zero-shot on the target thread, with no target evaluations at all. No paper in the file does either.
Prior Work and Where This Sits
The application regime comes from indicative summarization. Indicative Summarization of Long Discussions builds a two-level table of contents for ChangeMyView threads. It uses Sentence-BERT embeddings, UMAP projection (a nonlinear method that squeezes vectors into a few dimensions while keeping neighbours close) and HDBSCAN clustering (a density-based method that finds clusters of varying size and discards sparse points as noise), followed by LLM labelling. It chooses these settings heuristically, and it reports about 67% frame-assignment accuracy for GPT-4 without separating labelling errors from clustering errors. Enhancing Argument Summarization fine-tunes Sentence-BERT for clustering arguments and adds a selection step with a length penalty. It also supplies what this proposal needs most: a key-point matching model, reported at 90.01% micro precision on the ArgKP test set, and evidence that ROUGE-style metrics track argumentative coverage poorly. Neither paper tunes its clustering or penalty settings systematically. These two papers motivate the work and supply the pipeline and the scoring model.
Evaluation realism is constrained by ArgCMV. It shows that the older ArgKP benchmark consists of short isolated sentences, with over 95% of arguments mapped to zero or one key point. In ArgCMV, over 75% of arguments carry more than one key point. Hard clustering assigns each comment to exactly one group, so it is structurally mismatched to ArgCMV. We therefore score with soft, multi-label matching, and we report ArgKP and ArgCMV separately rather than averaging them. From Arguments to Key Points defines the matching task and its BERT-large baseline.
The tuning methods come from the second direction. Hyperparameter Importance Across Datasets uses functional ANOVA, a method that splits the variance in performance into the share due to each hyperparameter and to their interactions. On tabular learners it finds that a few "heavy hitter" settings dominate, and priors fitted to past runs speed up search. We apply the same decomposition across threads, and we also measure how the importance of each setting varies with thread size. Scalable Hyperparameter Transfer Learning shares a neural feature map across tasks with per-task Bayesian linear heads, which lets transfer scale linearly in the number of evaluations. We reuse that surrogate, but we condition its input on thread features so it can predict for a thread with zero evaluations. Perrone et al. warm-start with some target evaluations and note that the linear heads extrapolate poorly, so that weakness is a named risk here. Quantifying the Hyperparameter Sensitivity provides sensitivity measures such as relative performance at $k$ and the number of local maxima, which we use to describe how rugged each thread's landscape is.
Research Program
The tuning space covers the embedding model choice, UMAP dimensionality and neighbourhood size, HDBSCAN minimum cluster size and minimum samples, the cluster-selection method, and a soft-assignment threshold that lets one comment join several clusters. This last setting addresses the ArgCMV multi-key-point finding.
The objective for a thread $t$ and configuration $\lambda$ is key-point coverage. Coverage is the fraction of reference key points matched by at least one generated cluster label, where matching is decided by the Khosravani et al. matching model. We subtract a redundancy penalty. To keep the LLM from confounding the clustering study, the main experiments label clusters extractively, using the most central comment. A second stage repeats the key comparisons with LLM labels, so we can tell how much of the gain survives labelling. That separation is exactly what Syed et al. left open.
The work proceeds in three steps. First, on training threads we run a dense random search and fit functional ANOVA pooled across threads and within thread-size strata. Second, we train the thread-conditioned transfer surrogate on those runs and use it to predict a configuration for each held-out thread from label-free features only. Third, we fit the simplest rule the importance analysis suggests, for example minimum cluster size as a power of thread length, and compare it with the learned predictor.
Baselines
The naive reference point is the published configuration of Syed et al. applied unchanged to every thread. It shows what practitioners get today.
The strongest honest competitor is a single global configuration found by Bayesian optimization of mean coverage over all training threads. It gets the same number of total pipeline evaluations as the transfer method's offline phase and the same embedding models. If a global configuration is nearly as good as per-thread prediction, the core claim fails. A second strong competitor is the size-scaled heuristic rule described above, fitted on the training threads. It isolates whether a full learned surrogate adds anything over one hand-crafted feature.
An ablation keeps the transfer surrogate but removes all thread features, which reduces it to learning a prior over configurations. This isolates the per-thread conditioning, the key ingredient. An oracle tuned per test thread with its labels gives the ceiling. We report the fraction of the gap between global and oracle that each method closes.
For downstream context, the file reports 0.684 F1 for fine-tuned BERT-large on ArgKP matching (Bar-Haim et al.), 90.01% micro precision for the matching model (Khosravani et al.) and roughly 67% GPT-4 frame accuracy (Syed et al.). None of these is a coverage number for a clustering pipeline. No reported result was found on Lacuna for key-point coverage on ArgCMV, and Khosravani et al.'s coverage scores are not given numerically in the file, so we reproduce their pipeline as a baseline. Beating it means a statistically significant coverage gain on held-out threads under paired bootstrap, at matched redundancy.
Why not just…?
Why not just tune one global configuration carefully? This is the real threat. The experiments settle it directly through the gap-closed measure. The prior evidence is suggestive but indirect: van Rijn and Hutter find that the best values of heavy-hitter settings shift across datasets, and density-based clustering has an obvious scale dependence on the number of points. If the global configuration closes most of the gap, we report that as the finding, and the importance analysis still tells practitioners which knobs to set.
Why not just prompt a long-context LLM with the whole thread? Syed et al. show even GPT-4 at about 67% on frame assignment. Falk et al. report Gemma-2-9b as the best few-shot model on ArgCMV, but the file gives no number. We include a direct-prompting baseline with a comparable inference budget. We expect it to struggle on threads with hundreds of comments, where the pipeline remains useful. If prompting wins on small threads, the practical recommendation becomes a size-based switch between the two approaches.
Why not just scale minimum cluster size with thread length? This may well suffice, and we treat it as an outcome rather than a rival. The learned predictor has to beat the fitted rule to justify its complexity. The functional ANOVA interaction terms show whether any second setting has a thread-dependent best value that the rule misses.
Proposed Experiments
The labelled tasks are ArgKP topics (28 topics, about 24,000 pairs) and ArgCMV topics (about 3,000 topics, about 12,000 arguments). We hold out whole topics, and we test transfer from ArgKP to ArgCMV separately, since ArgCMV's longer and multi-key-point arguments make that shift the hardest one.
The Winning Arguments corpus (25,043 threads) has no key-point labels. We use it to check whether the thread features seen at test time fall within the training range, because extrapolation is where linear surrogate heads are known to fail.
To guard against the pipeline overfitting the matching model, we swap in a second matching scorer for evaluation only. We also run a small human coverage judgment on 50 held-out threads, checking that gains measured by the scorer hold up under human judgment.
The decisive check has two parts. The per-thread predictor must close a substantial part of the global-to-oracle gap on ArgCMV. The gain must also persist when clusters are labelled by an LLM, rather than being absorbed by labelling noise. If both hold, per-thread configuration is a real lever against information overload. If neither holds, the result is still useful: it shows that one tuned configuration is enough and tells the field to stop hand-tuning per corpus.
References
- Indicative Summarization of Long Discussions
- Enhancing Argument Summarization: Prioritizing Exhaustiveness in Key Point Generation and Introducing an Automatic Coverage Evaluation Metric
- ArgCMV: An Argument Summarization Benchmark for the LLM-era
- From Arguments to Key Points: Towards Automatic Argument Summarization
- Hyperparameter Importance Across Datasets
- Scalable Hyperparameter Transfer Learning
- Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks