TL;DR
This paper introduces a novel test-time reinforcement learning agent that tackles the challenging problem of mapping diverse third-party log schemas to a common format, even when vendor documentation is scarce or unavailable. By iteratively gathering external evidence through web searches and using a confidence-based reward system, the agent significantly improves mapping accuracy without requiring model updates or labeled data, making it highly practical for real-world enterprise deployments.
What Problem Does This Paper Address?
In modern enterprise IT environments, a critical challenge is integrating logs from a vast array of third-party sources. These logs contain vital information for security operations, threat detection, and overall system understanding. However, a major hurdle is schema mapping: aligning the often poorly documented or inconsistent schemas of incoming data with a standardized, common schema. This is particularly problematic during inference (test time) when vendor documentation might be missing, outdated, or difficult to access. Traditional methods like fine-tuning LLMs are costly, require labeled data, and can be brittle. This paper addresses the need for an efficient, adaptable, and transparent schema mapping solution that works effectively even with limited or no ground truth labels and documentation.
Methods
The core innovation is a test-time reinforcement learning (RL) agent that improves its schema mapping capabilities during inference without updating model weights or relying on pre-collected labeled data. The process unfolds iteratively:
- Initial Mapping: The agent uses a large language model (LLM) to generate an initial mapping hypothesis based on the source and target schemas, enriched with any available context.
- Conflict Detection: It identifies ambiguous or conflicting mapping attempts by running multiple inference passes with slight variations.
- Evidence Gathering: For detected conflicts, the agent formulates targeted web search queries to gather external evidence (e.g., definitions, examples) from the internet.
- Confidence-Based Reward: The collected evidence is used to refine the mapping. A confidence score, calculated based on the consistency of predictions across multiple inference attempts, serves as a proxy for accuracy. This confidence score acts as the reward signal: evidence that increases confidence is retained, while unhelpful evidence is discarded.
- Iterative Refinement: This cycle repeats, allowing the agent to dynamically build a contextually rich understanding and improve its mappings over time.
This approach is inspired by "verbal RL" where the agent learns through accumulating context and feedback rather than parameter updates.
Datasets
The evaluation was performed using two schemas:
- Source Schema: Microsoft Defender for Endpoint (195 fields).
- Target Schema: Trend Micro Common Schema (137 fields).
The ground truth consisted of 66 manually verified field mappings, curated by domain experts. No new benchmarks were introduced; the focus was on applying the method to a realistic enterprise scenario.
Key Insights and Ideas
- Confidence as a Proxy for Accuracy: The paper demonstrates that a confidence score, derived from prediction consistency, can effectively guide schema mapping improvement in the absence of ground truth labels. This is a crucial insight for practical, real-world applications.
- Test-Time Adaptation is Powerful: The agent's ability to self-improve during inference without retraining is a significant advantage, making it highly adaptable to new, unseen log formats and schemas.
- External Evidence is Key: Dynamically gathering evidence from external sources (like web searches) is more effective than relying solely on static knowledge bases or limited internal documentation.
- Transparency in Reasoning: The agent's process of evidence collection and confidence scoring provides a transparent reasoning trace, allowing human experts to focus their review on the most ambiguous cases, thereby reducing manual effort.
- Overconfidence is a Persistent Challenge: While confidence scores guide improvement, the system can become overconfident, where confidence values exceed actual accuracy. This highlights an area for future research in better confidence calibration.
Key Results
- The proposed method significantly improved mapping accuracy from 56.36% (single-shot LLM) and 72.73% (LLM with internal KB context) to 93.94% after 100 iterations.
- The number of low-confidence mappings requiring expert review was reduced by 85% (from 26 to 4 fields).
- The agent demonstrated selective evidence rejection, identifying when additional information was unlikely to improve results, preserving a high-quality evidence set.
- Experiments on overconfidence mitigation showed that increasing inference attempts from 3 to 10 improved calibration (confidence closer to accuracy) but slightly decreased overall accuracy.
Limitations and Future Work
- Evidence Quality Dependency: The system's performance is contingent on the quality and availability of external evidence sources. In domains with sparse documentation, improvements might be limited.
- Computational Overhead: The iterative process involving LLM calls and external searches can increase computational costs compared to single-shot approaches.
- Domain Specificity: The evaluation was focused on cybersecurity schemas. Further validation in other domains (e.g., healthcare, finance) is needed to confirm generalizability.
- Scope Limitations: The current work focuses on 1-to-1 mappings. Extending to more complex mapping cardinalities (1-to-N, N-to-M) is a direction for future work.
- Overconfidence Calibration: The paper identifies overconfidence as a limitation and suggests future work on improving confidence calibration, potentially by using multiple models for ensemble confidence or directly prompting LLMs for self-assessed confidence.
Summary updated