Abstract: Humans increasingly interact with Artificial intelligence (AI) systems. AI systems are optimized for objectives such as minimum computation or minimum error rate in recognizing and interpreting inputs from humans. In contrast, inputs created by humans are often treated as a given. We investigate how inputs of humans can be altered to reduce misinterpretation by the AI system and to improve efficiency of input generation for the human while altered inputs should remain as similar as possible to the original inputs. These objectives result in trade-offs that are analyzed for a deep learning system classifying handwritten digits. To create examples that serve as demonstrations for humans to improve, we develop a model based on a conditional convolutional autoencoder (CCAE). Our quantitative and qualitative evaluation shows that in many occasions the generated proposals lead to lower error rates, require less effort to create and differ only modestly from the original samples.
Introduction
Human-to-AI communication is often bottlenecked by ambiguous, sloppy, or rushed human inputs, such as mumbling to a voice assistant or scribbling a digit on a tablet. Traditionally, engineers solve this by making the AI more robust. However, in safety-critical systems like autonomous driving or medical technology, it is highly beneficial for the human to know how to adjust their behavior to ensure the AI understands them perfectly.
Because deep learning models are "black boxes" that process information differently than humans do, humans don't intuitively know what specific changes will help the AI. This research frames the problem as a multi-objective optimization: how do we alter a human's input so that the AI recognizes it with higher certainty, while ensuring the new input requires minimal human effort to create and looks similar to their original style?
The H2AI Coach Framework
The authors develop the Human-to-AI (H2AI) Coach, which acts as an intermediary between the human and a frozen, pre-trained target AI classifier. The core architecture is a Conditional Convolutional Autoencoder (CCAE). An autoencoder is chosen over a standard GAN because the goal is to modify an existing input rather than generate a new one from scratch.
The model is trained using a linear weighted sum of four distinct loss functions:
- Reconstruction Loss (Minimal effort to change): Ensures the suggested image remains visually similar to the original human input.
- Classification Loss (Reduce misunderstanding): Minimizes the loss of the target AI classifier, forcing the generated image to be highly recognizable to the AI.
- Discriminator Loss (Realistic samples): A GAN-style discriminator ensures the generated images look like real digits rather than blurry mathematical artifacts.
- Efficiency Loss (Minimal effort to create): Encourages the removal of unnecessary strokes by minimizing total pixel intensity ("ink").
Results and Qualitative Analysis
Quantitatively, the baseline target CNN classifier had an accuracy of 95.97% on the standard MNIST dataset. By passing the test set through the H2AI Coach optimized with classification loss, the target classifier's accuracy on the modified digits jumped to near-perfect levels (99.98% – 100%).
Qualitatively, the autoencoder successfully smoothed out messy digits. When the efficiency loss was applied, the model actively deleted unnecessary flourishes, such as the top loop of a '5' or the tail of a '2'. The research highlights the "alien" nature of AI preferences: optimizing purely for classification confidence often results in blurry images that humans find hard to read, as the AI prioritizes specific pixel activations over crisp lines. The discriminator is crucial to bridge this gap, forcing AI-optimized suggestions back into the manifold of realistic human handwriting.
Conclusion and Future Work
The H2AI coach demonstrates that humans can expend less effort (use less ink) while simultaneously improving AI accuracy. However, balancing the loss weights ($\alpha$) is highly sensitive; excessive efficiency loss can cause digits to disappear, while excessive classification loss leads to "mode collapse" where the network outputs a prototypical digit regardless of the user's style.
Future work will explore UI/UX strategies for effectively showing these suggestions to users and expanding the framework to more complex modalities like speech recognition.
Summary updated