Report of 2017 NSF Workshop on Multimedia Challenges, Opportunities and Research Roadmaps

Shih-Fu Chang, Alexander G. Hauptmann, Louis-Philippe Morency, Sameer K. Antani, Dick C. A. Bulterman, Carlos Busso, Joyce Yue Chai, Julia Hirschberg, Ramesh C. Jain, Ketan Mayer-Patel, Reuven Meth, Raymond Mooney, Klara Nahrstedt, Shrikanth S. Narayanan, Prem Natarajan, Sharon L. Oviatt, Balakrishnan Prabhakaran, Arnold W. M. Smeulders, Hari Sundaram, Zhengyou Zhang, Michelle X. ZhouarXiv 2019 · OpenReview

TL;DR

This report from a 2017 NSF workshop outlines the key challenges, opportunities, and a 5-15 year research roadmap for the multimedia and multimodal (MM) field. It highlights the need for more integrated, person-centered, and interpretable MM systems to unlock their full potential across various domains like healthcare, education, and smart infrastructure.

What Problem Does This Paper Address?

The paper addresses the rapidly evolving landscape of multimedia and multimodal (MM) research, driven by transformative technologies and pervasive computing. The MM community faces both immense opportunities and significant uncertainties. While new research is emerging at an unprecedented pace, there's a need to consolidate efforts, identify critical research areas, and chart a course for the next 10-15 years. The core problem is to move beyond siloed research and develop comprehensive MM systems that can tackle complex, real-world challenges, leading to impactful applications that benefit society.

Methods

The paper is not a traditional research paper with novel algorithms or experiments. Instead, it's a synthesis of expert opinions and discussions from a 2017 NSF workshop. The "methods" involved:

  • Consensus-based identification of research areas: 23 invited participants from academia and industry identified key MM challenges and opportunities.
  • Breakout group discussions: Participants delved into specific topics, reporting initial findings to the larger group.
  • Extensive deliberation: The whole group engaged in further discussion to refine the identified areas.
  • Roadmap development: For each identified topic, a summary was created outlining the state-of-the-art, key challenges, and a 5, 10, and 15-year research roadmap.

The paper synthesizes these discussions into a structured report, covering cross-cutting research areas and specific application domains.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • The "dominant interface paradigm" is multimodal: Mobile smartphones and other devices have made multimodal interaction the norm, driving the need for more sophisticated MM research.
  • MM research is crucial for "new knowledge": The synergy between different modalities is a prime source for generating novel insights and understanding complex systems.
  • Person-centered MM is the future: Moving beyond general applications, the focus is shifting towards understanding individuals and tailoring MM experiences to their unique characteristics and needs.
  • Interdisciplinary collaboration is essential: Tackling complex MM challenges requires bridging gaps between computer vision, machine learning, linguistics, psychology, and domain-specific fields (e.g., healthcare, education).
  • Interpretability and transparency are key for trust: As MM systems become more integrated into our lives, users need to understand how they work and trust their decisions.
  • Data is a double-edged sword: While abundant data fuels progress, challenges remain in data quality, annotation, privacy, and accessibility, especially for longitudinal and person-specific data.
  • A 5-15 year roadmap is critical: The field needs a long-term vision to guide research and funding, moving from basic multimodal analysis to complex systems-level theories and applications.

Key Results

This report doesn't present quantitative results in the traditional sense. Instead, its key "results" are the identified research areas, challenges, and a proposed roadmap for the MM field. These include:

  • Major Cross-Cutting Research Areas: Multimodal Foundational Methods, Multimodal Knowledge Discovery, Joint Knowledge Representation and Processing - Grounding, Person-Centered Multimodal Interaction, Multimedia Content Generation, Multimedia and Multimodal Systems, Data, Challenges and Evaluation.
  • Major Application Areas: Education, Healthcare, Smart Infrastructure, Social Good.
  • Key Challenges: Data sparsity, lack of longitudinal data, privacy concerns, need for robust and extensible multimodal behavior analysis, cross-cultural understanding, automated synthesis of multimodal persuasions, and limited methodological frameworks.
  • Roadmap: A phased approach (5, 10, 15 years) for advancements in fundamental methods, applications, and infrastructure, emphasizing areas like interpretable models, learning from limited data, and developing systems-level theories.

Limitations and Future Work

Limitations:

  • Limited time for discussion: The report acknowledges that due to the workshop's limited time, some identified areas were not discussed in depth by breakout groups.
  • Focus on specific areas: The report focuses on identified research areas and may not cover the entire breadth of MM research.

Future Work:

The paper extensively outlines future work through its 5, 10, and 15-year roadmaps across various research areas and applications. Key future directions include:

  • Developing advanced multimodal foundational methods: Focusing on interpretable models, learning from limited data, and better integration of diverse modalities.
  • Enhancing person-centered MM: Creating systems that understand individuals deeply and tailor interactions accordingly.
  • Advancing MM applications: Driving progress in healthcare, education, smart infrastructure, and social good through MM technologies.
  • Addressing data challenges: Developing better data collection, annotation, sharing, and privacy-preserving techniques.
  • Building robust infrastructure: Investing in computing resources (especially GPUs), data hubs, and standardized frameworks.
  • Fostering interdisciplinary and international collaboration: Bridging research communities and promoting global partnerships.
  • Developing systems-level theories: Moving beyond individual components to understand complex MM systems holistically.

Summary updated

Abstract

With the transformative technologies and the rapidly changing global R&D landscape, the multimedia and multimodal community is now faced with many new opportunities and uncertainties. With the open source dissemination platform and pervasive computing resources, new research results are being discovered at an unprecedented pace. In addition, the rapid exchange and influence of ideas across traditional discipline boundaries have made the emphasis on multimedia multimodal research even more important than before. To seize these opportunities and respond to the challenges, we have organized a workshop to specifically address and brainstorm the challenges, opportunities, and research roadmaps for MM research. The two-day workshop, held on March 30 and 31, 2017 in Washington DC, was sponsored by the Information and Intelligent Systems Division of the National Science Foundation of the United States. Twenty-three (23) invited participants were asked to review and identify research areas in the MM field that are most important over the next 10-15 year timeframe. Important topics were selected through discussion and consensus, and then discussed in depth in breakout groups. Breakout groups reported initial discussion results to the whole group, who continued with further extensive deliberation. For each identified topic, a summary was produced after the workshop to describe the main findings, including the state of the art, challenges, and research roadmaps planned for the next 5, 10, and 15 years in the identified area.
BibTeX
@misc{chang2019report,
  author = {Shih-Fu Chang and Alexander G. Hauptmann and Louis-Philippe Morency and Sameer K. Antani and Dick C. A. Bulterman and Carlos Busso and Joyce Yue Chai and Julia Hirschberg and Ramesh C. Jain and Ketan Mayer-Patel and Reuven Meth and Raymond Mooney and Klara Nahrstedt and Shrikanth S. Narayanan and Prem Natarajan and Sharon L. Oviatt and Balakrishnan Prabhakaran and Arnold W. M. Smeulders and Hari Sundaram and Zhengyou Zhang and Michelle X. Zhou},
  title = {{Report of 2017 NSF Workshop on Multimedia Challenges, Opportunities and Research Roadmaps}},
  year = {2019},
  eprint = {1908.02308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.02308},
}
Figure