VLEO-Bench: A Framework to Evaluate Vision-Language Models for Earth Observation Applications
Name
zhang-chenhui5-sm-idss-2025-thesis.pdf
Description
Thesis PDF
Size
15.24 MB
Format
Adobe PDF
Checksum (MD5)
f395b21ab53f7f793f45ef75b307e31a
Author(s)
Zhang, Chenhui
Advisor(s)
Wang, Sherrie
Date Issued
May 2025
Publisher
Massachusetts Institute of Technology
Abstract
Large Vision-Language Models (VLMs) have demonstrated impressive performance on complex tasks involving visual input with natural language instructions. However, it remains unclear to what extent capabilities on natural images transfer to Earth observation (EO) data, which are predominantly satellite and aerial images less common in VLM training data. In this work, we propose VLEO-Bench, a comprehensive evaluation framework to quantify the progress of VLMs toward being useful tools for EO data by assessing their abilities on scene understanding, localization and counting, and change detection tasks. Motivated by real-world applications, our framework includes scenarios like urban monitoring, disaster relief, land use, and conservation. We discover that, although state-of-the-art VLMs like GPT-4V possess extensive world knowledge that leads to strong performance on open-ended tasks like location understanding and image captioning, their poor spatial reasoning limits usefulness on object localization and counting tasks.
MIT Department
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
Terms of Use
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
Copyright retained by author(s)
Persistent DSpace Link
