RI: SMALL: COLLABORATIVE RESEARCH: SEEING SURFACES: ACTIONABLE SURFACE PROPERTIES FROM VISION

This project is to enable computers and robots the capability of estimating actionable, physical properties of surfaces (the feel) from their appearance (the looks). The key idea is to leverage the deeply interwoven relation between radiometric and physical surface characteristics. By learning models that make explicit the physical surface properties encoded in full and partial measurements of radiometric appearance properties, computers can estimate crucial physical properties of real-world surfaces from passive observations with novel camera systems. This project paves the path for integrating these models and estimation algorithms into scene understanding, robotic action planning, and efficient visual sensing. The research results provide a currently missing but fundamental capability to computer vision that benefits a number of applications in areas of computer vision, robotics, and computer graphics. The project provides hands-on research opportunities for both undergraduate and graduate students and are integrated in the PIs' undergraduate and graduate courses taught at Drexel and Rutgers. They are also used as a backdrop for K-12 outreach activities including high school and middle school mentorship programs. The data collection activities provide an ideal platform to expose K-12 students to physics and computer science.This research investigates the methods to infer actionable surface properties from images and detailed surface reflectance measurements. The research activities are centered on four specific aims: 1) controlled and uncontrolled large-scale data collection of actionable physical properties and appearance measurements of everyday surfaces, 2) derivation of prediction models for deducing physical properties from local surface appearance, 3) integration of global semantic context including object and scene information, and 4) development of efficient appearance capture and its use for novel physics-from-appearance sensing. These research thrusts collectively answer the fundamental question of how computer vision can anticipate the physical properties of a surface without touching it and knowing what it is, laying the foundation for computational vision-for-action.