Situation recognition from multimodal data

Situation recognition is the problem of deriving actionable insights from heterogeneous, real-time, big multimedia data to benefit human lives and resources in difierent applications. This tutorial will discuss the recent developments towards converting multitudes of data streams including weather patterns, stock prices, social media, trafic information, and disease incidents into actionable insights. For multiple decades, multimedia researchers have been building approaches like entity resolution, object detection, and scene recognition, to understand difierent aspects of the observed world. Unlike the past though, now we do not need to undertake sense-making based on data coming from a single media element, modality, time-frame, or location of media capture. Real world phenomena are now being observed by multiple media streams, each complementing the other in terms of data characteristics, observed features, perspectives, and vantage points. Each of these multimedia streams can now be assumed to be available in real-time and increasingly larger portion of these come inscribed with space and time semantics. The number of such media elements available (e.g. tweets, Flickr posts, sensor updates) is already in the order of trillions, and computing resources required for analyzing them are becoming increasingly available. We expect these trends to continue and one of the biggest challenges in multimedia computing in the near term to be that of concept recognition from such multimodal data. As shown in Figure 1, the challenges in situation recognition are fundamentally difierent from those in object or event recognition. They involve dealing with multiple media, each capturing real world phenomena from multiple vantage locations spread over time. Detecting situations in time to take appropriate actions for saving lives and resources can transform multiple aspects of human life including health, natural disaster, trafic, economy, social reforms, business decisions and so on. Examples of such relevant situations include beautiful-days/ hurricanes/ wildfires, trafic (jams/ smooth/ normal), economicrecessions/ booms, block-busters, droughts/ great-monsoons, seasons (early-fall/ fall/ late-fall), demonstrations/ celebrations, social uprisings/ happiness-index, ash-mobs, ocking and so on. This tutorial will provide the audience with a thorough theoretical will bring together the work by multiple scholars working in the area of situation recognition both within and outside the multimedia research community. The attendees would be introduced to the difierent interpretations of situations across multiple fields, and how it builds upon and extends the efiorts on object detection, event detection, scene recognition and so on. The tutorial will provide a review of recent efiorts within the multimedia community towards detecting real-time situations, and the attendees will be introduced to multiple practical situation recognition approaches and applications. Specific attention will be paid to discussing the relevant open research challenges for the community to extensively advance the state of the art in situation recognition. Learning objectives: At the end of the tutorial the attendees should be able to 1. Describe the problem of situation recognition and how it is difierent from object detection, event recognition, scene understanding etc. 2. Outline the difierent interpretations of situations across difierent fields e.g. multimedia, ubiquitous computing, robotics, aviation etc. 3. Articulate a computational definition for the concept of"situation" and the problem of situation recognition. 4. Identify the important categories of operators needed for the task of situation recognition. 5. Relate to the practical experience of creating at least one practical situation recognition application using an open-source situation recognition toolkit. 6. Articulate the emerging trends in situation-based computing and identify the open challenges in the field of situation recognition.