Exploring Coherence in Visual Explanations

Malihe Alikhani, Matthew Stone

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

A wide range of communicative artifacts - perhaps the majority - involve the coordinated presentation of visual and linguistic information. We envisage computer systems that support access to information by using rich representations of the interpretation of such multimodal presentations. This paper advocates organizing such representations in terms of coherence relations [2, 19], a fundamental construct from the theory of natural language discourse that is often invoked to explain the integrated interpretation of the diverse communicative actions in face-to-face conversation [9, 25, 35]. Coherence relations come in constrained classes, such as the Explanation, Narration and Parallel relations, each of which establishes specific kinds of structural, logical, and intentional relationships among communicative actions. Representing these relationships can therefore provide a scaffold for organizing, disambiguating and integrating the interpretation of communication across modalities. This paper uses a case study of instructions presented using text and pictures to motivate and describe an analysis of multimodal discourse interpretation in terms of coherence relations and to sketch a roadmap for operationalizing the approach in computer systems.

Original languageEnglish (US)
Title of host publicationProceedings - IEEE 1st Conference on Multimedia Information Processing and Retrieval, MIPR 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages272-277
Number of pages6
ISBN (Electronic)9781538618578
DOIs
StatePublished - Jun 26 2018
Event1st IEEE Conference on Multimedia Information Processing and Retrieval, MIPR 2018 - Miami, United States
Duration: Apr 10 2018Apr 12 2018

Publication series

NameProceedings - IEEE 1st Conference on Multimedia Information Processing and Retrieval, MIPR 2018

Other

Other1st IEEE Conference on Multimedia Information Processing and Retrieval, MIPR 2018
Country/TerritoryUnited States
CityMiami
Period4/10/184/12/18

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Signal Processing
  • Media Technology

Keywords

  • discourse coherence
  • discourse interpretation
  • multimodal communication

Fingerprint

Dive into the research topics of 'Exploring Coherence in Visual Explanations'. Together they form a unique fingerprint.

Cite this