Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials
Name
1704.03822.pdf
Description
Submitted version
Size
9.19 MB
Format
Adobe PDF
Checksum (MD5)
e72bf2a3210bce7aebb31c4e3f35fa52
Author(s)
Yuan, Wenzhen
Wang, Shaoxiong
Dong, Siyuan
Adelson, Edward
Date Issued
July 2017
Publisher
IEEE
Citation
Yuan, Wenzhen, Wang, Shaoxiong, Dong, Siyuan and Adelson, Edward. 2017. "Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials."
Version
Original manuscript
Abstract
© 2017 IEEE. For machines to interact with the physical world, they must understand the physical properties of objects and materials they encounter. We use fabrics as an example of a deformable material with a rich set of mechanical properties. A thin flexible fabric, when draped, tends to look different from a heavy stiff fabric. It also feels different when touched. Using a collection of 118 fabric samples, we captured color and depth images of draped fabrics along with tactile data from a high-resolution touch sensor. We then sought to associate the information from vision and touch by jointly training CNNs across the three modalities. Through the CNN, each input, regardless of the modality, generates an embedding vector that records the fabric's physical property. By comparing the embedding vectors, our system is able to look at a fabric image and predict how it will feel, and vice versa. We also show that a system jointly trained on vision and touch data can outperform a similar system trained only on visual data when tested purely with visual inputs.
MIT Department
Massachusetts Institute of Technology. Department of Mechanical Engineering
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/cvpr.2017.478
