Concept based hybrid fusion of multimodal event signals

Yuhui Wang, Christian Von Der Weth, Yehong Zhang, Kian Hsiang Low, Vivek K. Singh, Mohan Kankanhalli

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Scopus citations


Recent years have seen a significant increase in the number of sensors and resulting event related sensor data, allowing for a better monitoring and understanding of realworld events and situations. Event-related data come from not only physical sensors (e.g., CCTV cameras, webcams) but also from social or microblogging platforms (e.g., Twitter). Given the wide-spread availability of sensors, we observe that sensors of different modalities often independently observe the same events. We argue that fusing multimodal data about an event can be helpful for more accurate detection, localization and detailed description of events of interest. However, multimodal data often include noisy observations, varying information densities and heterogeneous representations, which makes the fusion a challenging task. In this paper, we propose a hybrid fusion approach that takes the spatial and semantic characteristics of sensor signals about events into account. For this, we first adopt the concept of an image-based representation that expresses the situation of particular visual concepts (e.g. "crowdedness", "people marching") called Cmage for both physical and social sensor data. Based on this Cmage representation, we model sparse sensor information using a Gaussian process, fuse multimodal event signals with a Bayesian approach, and incorporate spatial relations between the sensor and social observations. We demonstrate the effectiveness of our approach as a proof-of-concept over real-world data. Our early results show that the proposed approach can reliably reduce the sensor-related noise, locate the event place, improve event detection reliability, and add semantic context so that the fused data provides a better picture of the observed events.

Original languageEnglish (US)
Title of host publicationProceedings - 2016 IEEE International Symposium on Multimedia, ISM 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages6
ISBN (Electronic)9781509045709
StatePublished - Jan 18 2017
Event18th IEEE International Symposium on Multimedia, ISM 2016 - San Jose, United States
Duration: Dec 11 2016Dec 13 2016

Publication series

NameProceedings - 2016 IEEE International Symposium on Multimedia, ISM 2016


Other18th IEEE International Symposium on Multimedia, ISM 2016
Country/TerritoryUnited States
CitySan Jose

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Computer Networks and Communications
  • Media Technology
  • Computer Science Applications


  • Events
  • Multimodal fusion
  • Multisensor data analysis
  • Situation understanding


Dive into the research topics of 'Concept based hybrid fusion of multimodal event signals'. Together they form a unique fingerprint.

Cite this