Steps Towards Best Practices For Robot Videos

Eric Rosen, Stefanie Tellex, Geroge KonidarisarXiv 2020 · arXiv

TL;DR

This paper proposes a formal set of "maxims" and a practical checklist to standardize how robotics researchers create and share videos. By advocating for transparency in film editing, trial reporting, and the use of supporting systems, the authors aim to move the field away from "movie magic" and toward a more honest characterization of a robot’s true capabilities and reliability.

What Problem Does This Paper Address?

In robotics, videos are a primary research artifact—they tell the story of a robot’s success in the real world in a way that code or papers cannot. However, the community currently relies on "unwritten guidelines" passed down from advisors to students.

This lack of formal standards creates a transparency gap. It is often difficult for a viewer to tell if a robot is performing a task autonomously using onboard sensors or if it is being "helped" by expensive motion-capture systems, hidden human intervention (Wizard-of-Oz), or aggressive video editing that hides dozens of failed attempts. Without best practices, "movie magic" (artistic editing) is frequently confused with "robot magic" (the actual research contribution), making it nearly impossible to judge the robustness and reliability of a system.

Methods

The authors propose a normative framework consisting of four core Maxims for Robot Videos. Rather than introducing a new algorithm, they provide a set of heuristic principles:

  1. Clear Film Edits: Edits should be used to improve comprehension (e.g., showing multiple camera angles simultaneously) rather than to hide the robot's limitations. If footage is spliced from different trials, it must be explicitly stated.
  2. Record All Trials: To demonstrate robustness, researchers should ideally provide access to all trial footage. At a minimum, videos should report how many attempts were required to achieve the successful run shown.
  3. Display Film Rate: If a video is sped up (to save time) or slowed down (to show fast motion), the playback speed (e.g., 2x, 0.5x) should be clearly displayed on the screen.
  4. Highlight Supporting Systems: Researchers must explicitly identify external aids like AR tags, motion capture rigs, or off-the-shelf motion planners. Viewers shouldn't have to guess whether a robot is "seeing" with its own eyes or relying on a structured environment.

The paper concludes with a practical Checklist that researchers can use during the post-production of their research videos to ensure they meet these transparency standards.

Datasets

Not discussed in the paper. This is a position paper proposing best practices rather than a study involving a specific training or evaluation dataset. However, the authors reference several existing robot videos from the community as qualitative case studies for "positive" and "negative" examples of their maxims.

Key Insights and Ideas

  • Failure as a Research Artifact: The authors argue that "blooper reels" or failure cases are not just for entertainment; they are vital scientific data. Transparency about where a system fails is exactly where new research directions are born.
  • The "One-Take" Fallacy: A single continuous shot of a success is often seen as the gold standard, but the authors point out this can be misleading if it took 50 tries to get that one shot. Showing multiple trials—even if edited for time—is a better indicator of reliability.
  • Movie Magic vs. Robot Magic: The paper draws a sharp line between artistic storytelling and scientific reporting. While a compelling narrative is good for engagement, it should never come at the cost of misrepresenting the robot's autonomy or speed.
  • Self-Critique: In a refreshing move, the authors use their own past videos as "negative examples," pointing out where they previously failed to be transparent about supporting systems or trial counts.

Key Results

The primary result of this paper is the Robot Video Checklist, which provides a concrete set of "Yes/No" questions for researchers:

  • Did you indicate if spliced shots are from different trials?
  • Did you display the film rate if the speed was manipulated?
  • Did you explicitly show or mention supporting off-the-shelf systems (like QR tags or speech-to-text)?

While the paper does not provide quantitative metrics (like accuracy), it provides a qualitative framework intended to improve the reproducibility and integrity of robotics research.

Limitations and Future Work

The authors acknowledge that this is a "first draft" and that their maxims are not exhaustive. They also note that for certain types of research—such as purely theoretical or systems-heavy papers—a video may not be necessary or practical.

Future work involves engaging the broader robotics community to refine these maxims and potentially integrating them into the submission requirements for major conferences (like ICRA or IROS), similar to how many AI conferences now require a "reproducibility checklist" for code and data.

Summary updated

Abstract

There are unwritten guidelines for how to make robot videos that researchers learn from their advisors and pass onto their students. We believe that it is important for the community to collaboratively discuss and develop a standard set of best practices when making robot. We suggest a starting set of maxims for best robot video practices, and highlight positive examples from the community and negative examples only from videos made by the authors of this article. In addition, we offer a checklist that we hope can act as an document that can be given to robotic researchers to inform them of how to make robot videos that truthfully characterize what a robot can and can not do. We consider this a first draft, and are looking for feedback from the community as we refine and grow our maxims and checklist.
BibTeX
@misc{rosen2020steps,
  author = {Eric Rosen and Stefanie Tellex and Geroge Konidaris},
  title = {{Steps Towards Best Practices For Robot Videos}},
  year = {2020},
  eprint = {2007.06218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.06218},
}
Figure