Repository logo
Log in(current)
Repository logoMIT Open ScholarshipDSpace@MIT
  1. Home
  2. MIT Open Access Articles
  3. MIT Open Access Articles
  4. What Do Different Evaluation Metrics Tell Us About Saliency Models?

What Do Different Evaluation Metrics Tell Us About Saliency Models?

Thumbnail Image
Download
Name

1604.03605.pdf

Description
Submitted version
Size

6.08 MB

Format

Adobe PDF

Checksum (MD5)

9924ea4c1f4a34b0e763bbd9378a2185

sword-2019-05-29T13:50:21.original.xml (130 B)
Original SWORD entry document
Author(s)
Bylinskii, Zoya
•
Judd, Tilke
•
Oliva, Aude
•
Torralba, Antonio
•
Durand, Frederic
Date Issued
March 2019
Journal
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Bylinskii, Zoya et al. "What Do Different Evaluation Metrics Tell Us About Saliency Models?" IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 3 (March 2019): 740 - 757 © 2019 IEEE
Version
Original manuscript
Abstract
How best to evaluate a saliency model's ability to predict where humans look in images is an open research question. The choice of evaluation metric depends on how saliency is defined and how the ground truth is represented. Metrics differ in how they rank saliency models, and this results from how false positives and false negatives are treated, whether viewing biases are accounted for, whether spatial deviations are factored in, and how the saliency maps are pre-processed. In this paper, we provide an analysis of 8 different evaluation metrics and their properties. With the help of systematic experiments and visualizations of metric computations, we add interpretability to saliency scores and more transparency to the evaluation of saliency models. Building off the differences in metric properties and behaviors, we make recommendations for metric selections under specific assumptions and for specific applications.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
http://creativecommons.org/licenses/by-nc-sa/4.0/
Persistent DSpace Link
https://hdl.handle.net/1721.1/129414
DOI of Published Version
https://doi.org/10.1109/tpami.2018.2815601
Repository logo
PrivacyPermissionsAccessibilityContact us
Repository logo
Notify us about copyright concerns.