Minimização do Impacto do Problema de Desvio de Conceito por Meio de Acoplamento em Ambiente de Aprendizado Sem Fim (Minimizing the Impact of the Concept Drift Problem by Using a Framework of Endless Learning) [in Portuguese]

Maísa Cristina Duarte, Estevam R. Hruschka Jr., Maria do Carmo NicolettiStil 2011

The quest for artificial intelligence that learns indefinitely, much like a human, faces a significant technical hurdle: Concept Drift. This paper, originally presented at the 8th Brazilian Symposium in Information and Human Language Technology, introduces RTWP (Read The Web in Portuguese). RTWP is a semi-supervised system designed to autonomously crawl the Portuguese web and expand its knowledge base while actively mitigating the feedback loops that lead to system degradation.

The Challenge of Concept Drift

Most machine learning models are trained on static datasets. In contrast, a Never-Ending Learning System (NELS) aims to operate 24/7, discovering new information based on its existing knowledge. However, in a semi-supervised "bootstrapping" environment, a single small error can be catastrophic. For example, if the system misidentifies a company as a city, it may then extract text patterns associated with that "city" that are actually characteristic of companies. This leads to a recursive cycle of errors known as concept drift, eventually rendering the knowledge base useless.

Methodology: The RTWP System

RTWP utilizes a bootstrapping approach across four integrated tasks: extracting Named Entities (ENs) from text patterns, discovering new patterns from known entities, and identifying semantic relations (e.g., "Capital of") between pairs of entities. To maintain system integrity, the researchers implemented three types of Coupling (Acoplamento):

  1. Self-Correction (Type 1): Penalizes conflicting scores between entities and patterns within the same category.
  2. Mutual Exclusion (Type 2): Leverages negative examples. If an entity is highly likely to be a "City," its probability of being a "Country" is automatically reduced.
  3. Semantic Constraints (Type 3): Validates relations. For instance, a "Capital of" relation is only accepted if the first entity is a "City" and the second is a "Country."

Key Results: Quality Over Quantity

The researchers found that enforcing these constraints is vital for long-term stability. While an uncoupled system might learn a higher volume of instances, its precision drops precipitously as errors accumulate.

Precision vs. Coverage for coupled and uncoupled systems.

As shown in the figure above, the coupled system maintains a precision level between 80% and 90% even as coverage increases. In contrast, the uncoupled system's precision hovers around 20-30%, indicating that the vast majority of its "learned" knowledge is incorrect.

The Impact of Promotion Limits

The study also examined the "Maximum Number of Promoted Patterns" (NMPT). Promoting too many patterns at once (NMPT=10) significantly increases the risk of error propagation.

Comparison of correct hits (Acerto) vs. errors (Erro) with and without coupling at NMPT=10.

The bar chart illustrates that without coupling at NMPT=10, the number of errors (red) far exceeds the number of correct hits (blue). By applying coupling, the system drastically reduces these errors, albeit at the cost of a lower total volume of learned entities. This "Quality over Quantity" trade-off is essential for a system intended to run indefinitely.

Conclusion

The RTWP project demonstrates that "learning what something isn't" is as crucial as learning what it is. By using an ontology-backed coupling framework, the researchers successfully minimized concept drift in a Portuguese-language context. Future work will focus on refining semantic relation constraints and implementing aggregated coupling to further harden the system against the complexities of the live web.

Summary updated

BibTeX
@inproceedings{duarte2011minimizacao,
  author = {Maísa Cristina Duarte and Estevam R. Hruschka Jr. and Maria do Carmo Nicoletti},
  title = {{Minimização do Impacto do Problema de Desvio de Conceito por Meio de Acoplamento em Ambiente de Aprendizado Sem Fim (Minimizing the Impact of the Concept Drift Problem by Using a Framework of Endless Learning) [in Portuguese]}},
  booktitle = {Stil},
  year = {2011},
}
Figure