LoCt-Instruct: An Automatic Pipeline for Constructing Datasets of Logical Continuous Instructions

Hongyu Sun, Yusuke Sakai, Haruki Sakajo, Shintaro Ozaki, Kazuki Hayashi, Hidetaka Kamigaito, Taro WatanabeEMNLP 2025 · OpenReview

TL;DR

This paper introduces LoCt-Pipeline, an automated method for creating datasets of "logical continuous instructions" (LoCt-Instruct) that mimic real-world multi-step tasks. These datasets are crucial for evaluating how well Large Language Models (LLMs) can follow complex, coherent instructions, revealing a significant gap between current LLM capabilities and human performance.

What Problem Does This Paper Address?

Modern LLMs excel at many tasks, but they often struggle with "continuous instruction following" – the ability to execute a sequence of related, interdependent steps that build upon each other logically. Existing datasets for this task suffer from a lack of logical coherence, limited topic diversity, and often rely on rigid templates or extensive manual annotation, making them insufficient for robustly evaluating LLMs. The authors aim to bridge this gap by creating a more realistic and challenging benchmark.

Methods

The core of the paper is the LoCt-Pipeline, an automated framework for generating instruction chains. It works in four stages:

  1. Context and Instruction Preparation: It starts with a seed database of instruction-following data (MUFFIN dataset) and selects "pivot inputs" to serve as central contexts. Nearest neighbors based on embedding similarity are retrieved to ensure thematic relevance.
  2. Dialogue Generation: A set of 12-16 instructions is sampled for each pivot input. LLMs are then used to enumerate plausible instruction sequences, prioritizing logical coherence and continuity. The top-ranked sequence becomes the "optimal logical plan." A lighter-weight LLM then generates a multi-turn dialogue based on this plan, simulating a realistic interaction.
  3. Self-Evaluation & Feedback: An automated evaluation filters dialogues for factual errors, logical breaks, and completeness issues. Dialogues are rated as "Good," "Needs Post-processing," or "Poor."
  4. MSQA Conversion: Each user turn in the dialogue is treated as an instruction, and the assistant's reply becomes the gold response. A large language model then generates three plausible distractors for each instruction, creating a multi-selection question answering (MSQA) format. This transformation allows for precise evaluation and tests positional bias.

Datasets

  • MUFFIN: Used as the seed database for context and instruction preparation. It contains pairs of input topics with multiple coherent instructions and gold responses.
  • LoCt-Instruct: The newly introduced benchmark dataset constructed using the LoCt-Pipeline. It consists of logically coherent, multi-turn instruction chains presented in a multi-selection QA format.

Key Insights and Ideas

  • The Need for Logical Coherence: Real-world tasks are rarely single, isolated instructions. LLMs need to handle sequences where each step depends on the previous one, a capability that current datasets don't adequately test.
  • Automated Dataset Generation: Leveraging LLMs to generate complex, multi-turn dialogues and QA pairs significantly reduces the cost and effort of creating high-quality instruction-following benchmarks.
  • MSQA as a Robust Evaluation Metric: The multi-selection QA format, with carefully crafted distractors, provides a more stringent test than simple answer generation, revealing subtle issues like positional bias and error propagation.
  • LLMs Lag Behind Humans: The results clearly show a performance gap between even the most advanced LLMs and human evaluators on these complex, chained instructions, highlighting areas for improvement.

Key Results

  • Performance Gap: LLMs, including state-of-the-art models like GPT-4o, consistently underperform human evaluators on the LoCt-Instruct benchmark. For instance, human evaluation achieved an EM of 0.74 and F1 of 0.866, while GPT-4o achieved EM of 0.653 and F1 of 0.882.
  • Chain-of-Thought (CoT) Benefits: Incorporating CoT reasoning significantly improves LLM performance on LoCt-Instruct, demonstrating its effectiveness for complex, sequential tasks.
  • Positional Bias: The dataset effectively reveals positional biases in LLMs, where models tend to favor earlier options, especially in open-source models.
  • Dialogue Quality Trade-offs: The number of instructions sampled impacts dialogue quality. Too few instructions lead to insufficient material, while too many create a combinatorial burden. An optimal range of 12-16 instructions per instance was found to balance quality and cost.

Limitations and Future Work

  • Scale and Topical Diversity: The current evaluation is based on a limited number of instances (271) derived from the MUFFIN dataset, which may not fully represent rarer reasoning phenomena or cover the full spectrum of topical diversity.
  • Scalability and Hyperparameter Optimization: Scaling the LoCt-Pipeline to much larger datasets will require careful redesign of sampling strategies and hyperparameter tuning to maintain logical coherence and computational efficiency.
  • Reasoning Evaluation: The MSQA format is limited for evaluating reasoning. Future work might involve sentence-by-sentence evaluation methods.
  • Future Work: The authors plan to scale the method to construct massive, logically continuous MSQA datasets and use them to train models, aiming to further enhance multi-step reasoning capabilities. They also aim to explore more nuanced evaluation methods beyond MSQA.

Summary updated

Abstract

Continuous instruction following closely mirrors real-world tasks by requiring models to solve sequences of interdependent steps, yet existing multi-step instruction datasets suffer from three key limitations: (1) lack of logical coherence across turns, (2) narrow topical breadth and depth, and (3) reliance on rigid templates or heavy manual effort. We introduce LoCt-Pipeline, a novel pipeline that leverages modern LLMs’ reasoning capabilities to assemble rich, topic-related single-instruction data into multi-turn dialogues, producing chains that are logically coherent, progressively deepen in content, and span diverse domains without fixed templates or extensive human annotation. We employed this pipeline to construct LoCt-Instruct for assessing models’ problem-solving abilities. The generated chains serve as a testbed for benchmarking a variety of models, including reasoning-oriented architectures, instruction-tuned variants, and state-of-the-art closed-source LLMs on their capacity to follow and correctly respond to each step. Our results reveal a substantial performance gap between current LLMs and human solvers. These findings highlight the need for more robust continuous instruction following. We publicly release the dataset and end-to-end pipeline.
BibTeX
@inproceedings{sun2025loct,
  author = {Hongyu Sun and Yusuke Sakai and Haruki Sakajo and Shintaro Ozaki and Kazuki Hayashi and Hidetaka Kamigaito and Taro Watanabe},
  title = {{LoCt-Instruct: An Automatic Pipeline for Constructing Datasets of Logical Continuous Instructions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=9E0YswcUht},
}
Figure