Training Cross-Thread Answer Retrieval Through Its Own Sub-Decisions

On community question answering forums such as Qatar Living, a new question is often already answered somewhere in an older thread. The answer is buried among greetings, "same problem here," price complaints and side arguments. Systems are good at ranking comments inside the thread they belong to. They fail when they must pull a useful comment out of a different thread. In the SemEval-2017 shared task, the best in-thread ranker reached a mean average precision (MAP) of 88.43, while the best cross-thread system reached only 15.46 (SemEval-2017 Task 3: Community Question Answering). The cross-thread decision is really two decisions combined. First, is the older thread about the same thing as the new question? Second, is this comment substantive rather than chatter? The clearest evidence that these sub-decisions carry the problem comes from Joint Multitask Learning for Community Question Answering Using Task-Specific Embeddings. There, cross-thread MAP on SemEval-2016 rises from 56.00 to 63.49 when the system receives the true labels for the two sub-decisions instead of its own predictions.

We propose to build cross-thread retrieval as an explicit combination of those two sub-decisions inside one shared neural encoder, trained end to end. The cross-thread labels then send gradient back into the thread-similarity scorer and the comment-quality scorer. This is multi-task learning: one network trained on several related objectives at once. Its role here is specific. The point is not that extra tasks regularize the encoder in general. The point is that the main task's loss should reshape what the auxiliary scorers treat as "noise." A comment judged "potentially useful" for its own asker may be exactly the clue a different asker needs. A frozen comment filter cannot learn that.

Core Claim

Cross-thread answer retrieval in noisy forums fails mainly because the two sub-decisions it depends on are learned against the wrong targets. The comment-quality scorer is trained only on in-thread goodness. The thread-similarity scorer never sees whether a matched thread actually held a usable answer. Our model scores a comment $c$ from older thread $t$ for new question $q$ as

$$s(q, t, c) = \alpha\, \ell_B(q, t) + \beta\, \ell_A(t, c) + r(q, c).$$

Here $\ell_B$ is the logit of a question–question similarity head, $\ell_A$ is the logit of an in-thread comment-quality head, and $r$ is a small residual head that compares $q$ and $c$ directly. All three heads read one shared cross-attention Transformer encoder. The weights $\alpha$ and $\beta$ are learned scalars. Training sums the losses for in-thread goodness, question similarity and cross-thread relevance. The cross-thread loss backpropagates through $\ell_A$ and $\ell_B$. Our claim is that this end-to-end composition closes a substantial part of the 7.49-point gap between predicted and true auxiliary labels reported by Joty et al. We also claim the improvement is concentrated on comments annotated "Potentially Useful." Simpler designs do not achieve either result: a frozen pipeline that multiplies separately trained scores, a single cross-encoder trained only on cross-thread data, and an unstructured multi-task model with tuned loss weights.

Relation to Prior Work

Joty et al. supply the closest design. They train separate feed-forward networks per subtask over hand-built features, freeze their hidden layers, and add a conditional random field that encodes the constraint that a comment cannot answer the new question unless its thread is relevant. We keep that constraint but make it differentiable and place it inside a shared pretrained encoder. Their two-step training is exactly the limitation they name as future work. Their gold-label ablation motivates this proposal.

Knowledge-enhanced Hierarchical Attention for Community Question Answering with Multi-task and Adaptive Learning (KHAMA) also uses multi-task learning against forum noise. Its auxiliary task is whole-question topic categorization, and it reports a MAP of 90.76 on in-thread ranking. That auxiliary signal is coarse and says nothing about which comments are chatter. We use comment-level and thread-level auxiliaries that sit inside the target decision. Multi-Relational Graph based Heterogeneous Multi-Task Learning in Community Question Answering shares a graph network across five Stack Overflow tasks with hand-written cross-task rules. It treats text as fixed node features, which is the reverse of our focus.

Work on forum noise constrains the design. Selecting Sentences versus Selecting Tree Constituents for Automatic Question Ranking shows that pruning text helps ranking: tree pruning reached a MAP of 78.56 against 76.02 for full text. The same paper found that heuristic sentence selection run as a separate stage was fragile on test data because it discarded needed context. That is a direct argument against hard pre-filtering and for a soft, trainable quality score. PMI-cool at SemEval-2016 Task 3 shows that politeness words like "thanks" mark bad answers, so generic notions of sentiment or salience mislead. A learned quality head must be supervised on forum-specific goodness, which is what the in-thread labels provide. ICRC-HIT and the SemEval-2015 overview (SemEval-2015 Task 3) both locate most errors on the "Potentially Useful" class. Our analysis targets that class.

The multi-task optimization literature supplies diagnostic tools rather than the core method. Gradient Deconfliction via Orthogonal Projections onto Subspaces For Multi-task Learning (GradOPS) and RotoGrad address cases where task gradients point in opposing directions or differ wildly in size. Understanding and Improving Information Transfer in Multi-Task Learning argues that negative transfer can arise from mismatched input distributions even when tasks agree. The in-thread and cross-thread data plausibly have that property. We will measure gradient conflict before reaching for these tools. We expect conflict to be concentrated rather than pervasive, and we do not assume projection is needed.

Research Program

The first component is the model above. A single pretrained cross-encoder of BERT-base size reads a pair of texts and has three heads. One scores (question, own-thread comment) pairs for goodness. One scores (new question, older question) pairs for relevance. One scores (new question, foreign comment) pairs directly. Following the SemEval-2016 data setup used by Joty et al., the cross-thread candidates carry true labels for all three relations, so all heads can be supervised on the same items. Only the cross-thread score uses the composition.

The second component is a gradient-level diagnostic. During training we log the cosine between the shared-encoder gradients of each auxiliary loss and the cross-thread loss. We break this down by the comment's in-thread label (Good, Potentially Useful, Bad) and by whether its thread is relevant. The hypothesis is that conflict appears mainly on Potentially Useful comments in relevant threads. If so, a targeted fix is warranted. That fix would be GradOPS-style projection applied only to the in-thread loss on those examples, or a learned per-class down-weighting of that loss. We will not apply blanket gradient surgery.

The third component is an error analysis of what end-to-end training changes in the quality head. We will compare its scores before and after cross-thread gradients are allowed to flow. We will check which comments move: chatter that is lexically close to the new question and was wrongly matched, versus conditional or partial answers.

Baselines

The naive reference point is BM25 over the full comment text, plus the official SemEval organizer baselines. These show how much the noise hurts lexical matching. The strongest reported numbers on SemEval-2016 cross-thread retrieval are SUper_team's MAP of 55.41 and Joty et al.'s joint CRF at 56.00 (SemEval-2016 Task 3; Joint Multitask Learning). The ceiling of that design with true auxiliary labels is 63.49. On SemEval-2017 the best reported cross-thread MAP is IIT-UHH's 15.46. These older systems use weaker encoders, so beating them alone proves little.

The strongest real competitor is therefore a single-task pretrained cross-encoder fine-tuned only on cross-thread pairs, using the same backbone as our model. I found no reported result on Lacuna for a pretrained Transformer cross-encoder on SemEval cross-thread retrieval, so we must establish this number ourselves. It is the most likely way our claim fails. Its main constraint is the small amount of cross-thread training data.

The second competitor is a frozen product pipeline. It trains the in-thread and thread-similarity scorers as separate cross-encoders on their own labels, then ranks foreign comments by a tuned combination of their logits. This is Joty's dependency structure with a modern encoder and without joint training. It is identical to our model with the gradient through $\ell_A$ and $\ell_B$ stopped, so it is also the ablation that isolates our key ingredient.

The third competitor is a shared encoder with three independent heads. It is trained on the summed losses with loss weights tuned by grid search, and it has no composition. This tests whether structure matters beyond plain multi-task sharing.

We will also run two ablations of our own model. One removes the residual head $r$. The other replaces true auxiliary supervision with no auxiliary loss, so that only the cross-thread loss trains the composed heads.

Every system uses the same backbone, the same training data (SemEval-2016 English training and development sets, with 2017 as a second test), and the same budget of 30 hyperparameter trials selected on development MAP. Each runs with 5 seeds. We report mean and spread, with a paired permutation test over test queries.

Beating the field means three things. Our model must exceed the single-task cross-encoder and the frozen pipeline by a statistically significant MAP margin on both test years. It must close a measurable fraction of the gap between Joty's predicted-label and true-label results, which we will reproduce with our own encoder by feeding true labels to the pipeline. And it must not lose more than about one MAP point on in-thread ranking relative to a single-task in-thread model, compared against KeLP's 79.19 and 88.43 as reported reference points.

Why not just…?

Why not just fine-tune one cross-encoder on cross-thread pairs? This is a serious threat. Large pretrained encoders may already learn to ignore greetings and to judge thread relevance implicitly. The evidence against it is indirect. Cross-thread labels are scarce. The 2016-to-2017 collapse suggests the task is brittle. A Discriminative Semantic Ranker for Question Retrieval shows deep encoders lose fine-grained cues on community questions: removing its cross-layer connections cost about 4% in Recall@100 on Quora. This single-task model is our primary baseline. If it matches our model, we will report that the decomposition is unnecessary at this scale.

Why not just multiply the outputs of separately trained quality and similarity scorers? This is exactly the stop-gradient ablation. If it matches the end-to-end model, our central hypothesis is wrong. The hypothesis says cross-thread labels should change what the quality scorer treats as noise, and that would be refuted. Barrón-Cedeño et al.'s finding that a separate selection stage is fragile on test data, and Joty et al.'s 7.49-point dependence on auxiliary accuracy, suggest frozen auxiliaries leave error on the table. The ablation settles it directly. The per-class analysis then shows whether any difference comes from Potentially Useful comments.

Why not just share an encoder with tuned loss weights, or add gradient surgery? Tuned linear weighting is often competitive with conflict-resolution methods. We treat GradOPS and RotoGrad as conditional tools, applied only if the logged gradient cosines show concentrated conflict. If the unstructured multi-task model equals the composed one, the benefit comes from shared training rather than from the combination rule. The comparison isolates exactly that.

Why not just strip chatter with rules or a lexicon before ranking? The PMI-cool lexicons show that forum-specific signals beat generic ones. Hard filtering still discards context, as the sentence-selection results showed. We include a BM25 run and a frozen-pipeline run over rule-filtered text so the cost of hard removal is visible.

Decisive Checks

Three results decide the proposal. First, the composed end-to-end model must beat both the single-task cross-encoder and its own stop-gradient twin on SemEval-2016 and SemEval-2017 cross-thread MAP across seeds. Second, its advantage over the frozen pipeline must be larger on Potentially Useful comments than on Good or Bad comments, matching the mechanism we propose. Third, we will feed true auxiliary labels to both the frozen pipeline and the end-to-end model. If joint training is doing what we claim, the end-to-end model should sit measurably closer to its own true-label ceiling than the pipeline does.

If the first check fails while the stop-gradient twin matches the end-to-end model, the honest conclusion is that a modern encoder plus Joty's factorization is sufficient. The gradient diagnostic would still be useful: it would show whether in-thread and cross-thread supervision actually conflict on noisy comments.