Abstract
This paper investigates whether modern Information Retrieval (IR) models adhere to the "Term Discrimination Constraint" (TDC)—a formal rule stating that rarer terms should carry more weight than common ones. Through mathematical analysis and empirical validation, we demonstrate that several state-of-the-art models, including the log-logistic and Dirichlet language models, frequently violate this constraint in practice. Our findings suggest that the TDC rule itself may need to be redefined to better align with high-performing retrieval systems.
Introduction
In information retrieval, researchers utilize "axiomatic methods" to determine if a retrieval formula is mathematically sound. These methods consist of a set of "common sense" constraints that any effective IR model should follow. For example, a document containing a search term more frequently should generally rank higher than one containing it less.
One of the most critical rules is the Term Discrimination Constraint (TDC). It captures the "IDF effect": if two terms appear in a query, the rarer one (the one with a higher Inverse Document Frequency) should be more influential in determining the document's score. While most IR models pass basic tests, it has remained unclear whether modern, high-performing models actually satisfy the TDC under standard configurations.
Methodology
We performed a formal mathematical analysis of three prominent IR models:
- The Log-Logistic Model: A model based on information theory.
- Jelinek-Mercer (JM) Language Model: A standard probabilistic model using linear smoothing.
- Dirichlet Language Model: A standard probabilistic model using Bayesian smoothing.
Using Lagrangian optimization, we identified scenarios where these models might fail the TDC. We also distinguished between the general TDC and a stricter version called speTDC (special TDC), where query terms are isolated to specific documents to see if the "rare term preference" holds in a simplified context.
Empirical Analysis of TDC Violations
While the study is primarily theoretical, we utilized corpus statistics from two classic IR benchmarks—Robust (TREC) and TREC 1-2—to validate our mathematical findings. These datasets allowed us to calculate average term probabilities and verify how often the conditions for TDC violation occur in real-world data.
Our analysis revealed that high-performing models like the log-logistic and Dirichlet models do not naturally satisfy the general TDC constraint. For the Dirichlet model specifically, satisfying the TDC would require a smoothing parameter ($\mu$) to be extremely large (often > 2,000). In practice, researchers typically set $\mu$ between 500 and 1,000 to achieve optimal performance. Consequently, to obtain the best search results, the TDC rule must technically be violated.
As shown in the table above, using real data from the Robust and TREC 1-2 collections, the Dirichlet model violates the TDC for approximately 62.2% of query term pairs under standard settings. This high frequency of violation suggests that the constraint, as currently formulated, is at odds with empirical effectiveness.
Redefining the Constraint: speTDC
Despite failing the general TDC, all three models studied (Log-logistic, JM, and Dirichlet) were shown to be universally compliant with the speTDC constraint. This suggests that while the general TDC might be too broad or incorrectly formulated, the core intuition behind it remains valid in a more restricted context.
Conclusion
The fact that top-tier models work effectively in the real world despite failing the TDC indicates that the IR community should consider replacing the old TDC with the more precise speTDC version. Future work should focus on experimentally assessing constraint violations to determine how often these mathematical failures lead to poor search results for users. Moving toward the speTDC formulation will better align formal theory with the models that demonstrate the best practical performance.
Summary updated