Message from the ISMAR 2018 Science and Technology Program Chairs

Joseph L. Gabbard, Jens Grubert, Shi‐Min Hu, Stefanie Zollmann2019 · DOI

This document is not a traditional research paper but rather the Message from the Program Chairs for the ISMAR 2019 conference. It outlines the organizational "behind-the-scenes" of one of the most prestigious venues in the field of Mixed and Augmented Reality.

TL;DR

The ISMAR 2019 Program Chairs detail the rigorous, multi-stage review process used to select the highest quality research for the 18th IEEE International Symposium on Mixed and Augmented Reality. By implementing a new double-blind review system and a synchronized global committee meeting, the chairs achieved a highly selective 22.1% acceptance rate for conference proceedings and an elite 8.6% for the IEEE TVCG journal.

What Problem Does This Paper Address?

The primary "problem" addressed here is the challenge of maintaining academic rigor, fairness, and transparency in a top-tier international conference. As the premier venue for Mixed and Augmented Reality (MAR), ISMAR must filter a high volume of submissions to identify work that is truly "cutting edge."

The chairs specifically highlight the need to evolve the review process to eliminate bias and ensure quality. This year’s process aimed to improve upon the changes made in 2017, tackling the logistical difficulty of coordinating a global Program Committee (PC) across three continents (Asia, Americas, and Europe) while transitioning to a more equitable reviewing standard.

Methods

The "methods" described are the procedural steps of the peer-review pipeline. Key components included:

  • Double-Blind Process: For the first time, ISMAR moved to a double-blind system where external reviewers were unaware of the authors' identities to ensure a fairer evaluation.
  • Three-Tier Reviewing: Each paper underwent three reviewing cycles. Every submission received at least four reviews: three from external experts and one from a PC member.
  • Primary/Secondary PC Roles: A "Primary" PC member coordinated the handling of a submission and assigned two external reviewers, while a "Secondary" PC member assigned one external reviewer and provided an additional expert review themselves.
  • Quality Control Window: A 10-day window was established for Primaries to check review quality and request improvements or additional reviews if the initial feedback was insufficient.
  • Synchronized Global PC Meeting: The final decisions were made during a two-day meeting held simultaneously across three locations (Beijing, Blacksburg, and Coburg) via a three-way video link.
  • Shepherding: For papers that were "conditionally accepted," the chairs assigned a "shepherd" from the PC to oversee the refinement process and ensure all requested modifications were met before final acceptance.

Datasets

Not discussed in the paper. (As an editorial message, this document does not utilize training or evaluation datasets.)

Key Insights and Ideas

  • The "Bulk" Decision Strategy: To maximize the efficiency of the two-day PC meeting, papers were categorized into "bulk accept," "bulk reject," and "to be discussed." This allowed the committee to focus their collective energy on the 68 papers (roughly 41% of total submissions) that were in the "gray area," rather than spending time on clear-cut cases.
  • Conflict Management: The chairs implemented strict protocols where reviewer assignments and names were hidden from any PC member or chair who had a conflict of interest with a specific paper.
  • Journal Integration: The process was designed to align with the high standards of the IEEE Transactions on Visualization and Computer Graphics (TVCG), allowing the very best papers to be published directly into a special issue of the journal rather than just the conference proceedings.

Key Results

The selection process resulted in a highly competitive program:

  • Total Submissions: 163 papers.
  • Conference Proceedings Acceptance: 36 papers (22.1% acceptance rate).
  • IEEE TVCG Special Issue Acceptance: 14 papers (8.6% acceptance rate).
  • Reviewer Effort: 265 reviewers participated, contributing an average of 1.8 reviews each.
  • PC Composition: 25 internationally renowned experts, plus 4 program chairs, totaling 29 voting members.

Limitations and Future Work

While the chairs do not explicitly list "limitations" in the scientific sense, they acknowledge the immense difficulty of the selection task ("this was not an easy task!").

Future Work in this context refers to the ongoing evolution of the conference. The chairs noted that they "continued with the successful changes made in 2017 and additionally modified the procedure," suggesting that the ISMAR review process is an iterative "living" system that will continue to be refined in subsequent years to meet the growing demands of the MAR research community.

Summary updated

Abstract

Presents the introductory welcome message from the conference proceedings. May include the conference officers' congratulations to all involved with the conference event and publication of the proceedings record.
BibTeX
@misc{gabbard2019message,
  author = {Joseph L. Gabbard and Jens Grubert and Shi‐Min Hu and Stefanie Zollmann},
  title = {{Message from the ISMAR 2018 Science and Technology Program Chairs}},
  year = {2019},
  doi = {10.1109/ismar-adjunct.2019.00007},
}

Loading related content...

Figure