Tracking urban heartbeat and policy compliance through vision and language-based sensing

Tahiya Chowdhury, Qizhen Ding, Ilan Mandel, Wendy Ju, Jorge Ortiz

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Sensing activities at the city scale using big data can enable applications to improve the quality of citizen life. While there are approaches to sense the urban heartbeat using sound, vision, radio frequency (RF), and other sensors, capturing changes at urban scale using such sensing modalities is challenging. Due to the enormous amount of data they produce and the associated annotation and processing requirement, such data can be of limited use. In this paper, we present a vision-to-language modeling approach to capture patterns and transitions that occur in New York City from March 2020 to August 2020. We use the model on ∼1 million street images captured by dashcams over 6 months. We then use the captions to train a language model based on Latent Dirichlet Allocation [4] and compare models from different periods using probabilistic distance measures. We observe distribution shifts in the model that correlate well with social distancing policies and are corroborated by different data sources, such as mobility traces. This language-based sensing introduces a new sensing modality to capture dynamics in the city with lower storage requirements and privacy concerns.

Original languageEnglish (US)
Title of host publicationBuildSys 2021 - Proceedings of the 2021 ACM International Conference on Systems for Energy-Efficient Built Environments
PublisherAssociation for Computing Machinery, Inc
Pages302-306
Number of pages5
ISBN (Electronic)9781450391146
DOIs
StatePublished - Nov 17 2021
Event8th ACM International Conference on Systems for Energy-Efficient Built Environments, BuildSys 2021 - Virtual, Online, Portugal
Duration: Nov 17 2021Nov 18 2021

Publication series

NameBuildSys 2021 - Proceedings of the 2021 ACM International Conference on Systems for Energy-Efficient Built Environments

Conference

Conference8th ACM International Conference on Systems for Energy-Efficient Built Environments, BuildSys 2021
Country/TerritoryPortugal
CityVirtual, Online
Period11/17/2111/18/21

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Renewable Energy, Sustainability and the Environment
  • Electrical and Electronic Engineering
  • Building and Construction
  • Computer Networks and Communications
  • Architecture

Keywords

  • COVID-19
  • computer vision and language
  • urban sensing

Fingerprint

Dive into the research topics of 'Tracking urban heartbeat and policy compliance through vision and language-based sensing'. Together they form a unique fingerprint.

Cite this