Penquiry: A Pen-based Interactive In-situ Q&A System Leveraging LLMs

Jeongmin Rhee, Changhee Lee, Hyunwoo Kim, Kiroong Choe, Bohyoung Kim, Sungahn Ko, Jinwook SeoarXiv 2026 · DOI · arXiv

Here is a blog-style summary of the research paper introducing Penquiry, a system designed to bridge the gap between pen-based digital note-taking and Large Language Models (LLMs).

TL;DR

Penquiry is an in-situ Q&A system that allows students to query digital study materials directly using a stylus, eliminating the need to switch to a separate keyboard interface. By introducing intelligent features like "Content Snapping" and "Question Autocompletion," Penquiry translates rough sketches and minimal handwritten keywords into rich, context-aware LLM queries, significantly reducing the physical and cognitive burden of learning.

What Problem Does This Paper Address?

When students study complex subjects (like math, physics, or engineering), they often prefer the fluid, spatial nature of a tablet and stylus for active note-taking. However, when they need to clarify a concept using an LLM (like ChatGPT), they face a frustrating modality mismatch. They have to abandon their digital canvas, open a separate chat window, and type out their questions using a keyboard.

The authors identify two primary friction points in this standard workflow:

  1. The Referential Barrier: It is incredibly tedious to textually describe which specific part of a complex diagram, formula, or graph you are asking about (e.g., writing out: "Let theta denote the angle between segments d_1 and d_2 at point (x,y)").
  2. The Expressive Barrier: It is highly inefficient to translate 2D spatial or symbolic information—like mathematical equations or structural layouts—into rigid, 1D typed text strings.
Figure 1: Penquiry is an in-situ Q&A system that enables learners to query directly on digital study materials using a pen. (Left) Penquiry V1 leverages in-situ pen input to mitigate the referential and expressive barriers inherent in a separate keyboard interface for LLM queries. However, it introduces new challenges, such as uncertainty in referential grounding and the physical burden of handwriting lengthy questions. (Right) Penquiry V2 resolves these emergent barriers by introducing Content Snapping, which aligns user sketches with nearby document text elements, and Question Autocompletion, which generates full queries from minimal ink keywords. Together, these mechanisms mediate free-form pen input into LLM-interpretable questions while sustaining inquiry within the learner’s pen-centered workflow.

Methods

To solve these barriers, the authors iteratively designed two versions of Penquiry:

Penquiry V1: Direct In-Situ Pen Input

V1 allows users to enter a dedicated "Question Mode" and sketch directly on top of a PDF document using underlines, circles, or arrows to point to targets. When the user taps "AI Question," a multimodal pipeline powered by GPT-4o takes three inputs: a cropped image with the sketch, a cropped image without the sketch, and the underlying parsed PDF text. The LLM synthesizes these inputs and displays the answer in an "Answer Shape" (a sticky-note-like UI element) right in the document's margin.

Penquiry V2: Intelligent Mediation Layer

While V1 eliminated the keyboard, user testing revealed new "pen-specific" challenges: raw ink input can be ambiguous, and handwriting long, complex questions is physically exhausting. V2 introduces two core interaction techniques to solve this:

  • Content Snapping: When a user draws a rough circle or underline, the system automatically aligns ("snaps") the sketch to the nearest character-level PDF text elements. This gives immediate visual feedback, confirming the system correctly understood the user's spatial intent.
  • Question Autocompletion: Instead of forcing the user to write out a full sentence, the user can simply provide a minimal sketch or a quick keyword (e.g., writing "VS" between two terms). The system analyzes the local context and dynamically generates four candidate questions categorized by cognitive depth (Definition, Description, Interpretation, and Knowledge-based).

Datasets

Because this is a Human-Computer Interaction (HCI) systems paper rather than a traditional machine learning training paper, no new ML datasets were created. Instead, the authors evaluated the system using standard digital textbook chapters from open repositories:

  • Formative Study: Sections from OpenStax textbooks covering history, data science, algebra, and biology.
  • Study 1 (V1 vs. Baseline): Two chapters from a geotechnical engineering textbook on LibreTexts ("Standard Penetration Test" and "Cone Penetration Test").
  • Study 2 (V2 vs. V1): Two chapters from a climate science textbook ("Theory" and "Carbon").

Key Insights and Ideas

  • Raw Ink is Not Enough: Simply replacing a keyboard with a pen interface introduces secondary challenges. Without explicit visual feedback (like Content Snapping), users experience anxiety over whether the AI accurately captured their intended selection.
  • Autocompletion as a Cognitive Scaffold: Crucially, Question Autocompletion did more than just save physical handwriting effort; it acted as an intellectual springboard. Users frequently selected higher-level, "knowledge-based" questions suggested by the system that they hadn't initially considered, deepening their engagement with the material.
  • Shifting Proportions of Inquiry: As learning sessions progressed, the proportion of complex, deep-thinking questions naturally increased. The autocompletion mechanism successfully adapted to and supported this cognitive progression over time.

Key Results

The authors evaluated Penquiry through two user studies ($N=16$ each):

  • High Usability: Penquiry V2 achieved an "Excellent" System Usability Scale (SUS) score of 84.9 (out of 100), surpassing V1's score of 80.78.
  • Massive Autocompletion Adoption: In Study 2, 74.9% of all queries submitted by users utilized the Question Autocompletion feature, proving it effectively mitigated the physical burden of handwriting.
  • Cognitive Progression: Quantitative analysis showed a significant progressive increase in deep, knowledge-based questions across a study session—growing from 18.85% in the early phase to 40.43% in the late phase.
  • Performance: The system maintained low error rates (only 3.8% in V2) with an average end-to-end latency of 9.04 seconds, which includes both the autocompletion generation and final answer delivery phases.

Limitations and Future Work

  • Short-Term Evaluation: The user studies were conducted in controlled lab settings with 25-minute sessions. Longitudinal studies are required to see how Penquiry impacts long-term learning habits and retention.
  • The Risk of Over-Steering: Proactive autocompletion can occasionally "interfere" if a user already has a highly specific question in mind. It can also anchor learners to system-framed perspectives.
  • Adaptive Fading: The authors propose that future iterations should feature adaptive fading—providing aggressive query suggestions when a student encounters a brand-new topic, but gradually fading into the background as the student's proficiency and familiarity with the material increase.

Summary updated

Abstract

Pen-based digital devices remain a preferred medium for active, cognitively engaging study. Concurrently, Large Language Models (LLMs) have become indispensable for self-directed learning, enabling students to clarify concepts. However, a fundamental interaction gap exists between the fluid, spatial nature of pen-based workflows and the discrete, keyboard-heavy requirements of LLMs. We present Penquiry, an in-situ question-and-answer system that bridges this gap by enabling learners to pose questions directly on digital study materials via a pen. We characterize two primary interaction challenges in this multimodal transition: a Referential Barrier, which hinders grounding fine-grained visual elements into the query context, and an Expressive Barrier, which forces learners to translate diverse, non-textual intents--such as equations and diagrams--into rigid, typed sentences. To resolve these, Penquiry introduces a mediation layer featuring Content Snapping for unambiguous referencing and Question Autocompletion to expand sparse ink keywords into rich semantic queries. Through two iterative user studies (N = 16 per study), we demonstrate that Penquiry significantly reduces the cognitive and physical overhead of inquiry compared to traditional interfaces, providing a new blueprint for pen-based, in-situ AI interaction
BibTeX
@misc{rhee2026penquiry,
  author = {Jeongmin Rhee and Changhee Lee and Hyunwoo Kim and Kiroong Choe and Bohyoung Kim and Sungahn Ko and Jinwook Seo},
  title = {{Penquiry: A Pen-based Interactive In-situ Q\&A System Leveraging LLMs}},
  year = {2026},
  eprint = {2609.19870},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.19870},
}
Figure