Towards Automated Assessment of Crowdsourced Crisis Reporting for Enhanced Crisis Awareness and Response

Abstract

The availability of information during a climate crisis event is critical for crisis managers to assess and respond to crisis impact. During crisis events, affected residents post real-time crisis updates on platforms such as RiskMap and Twitter. These updates provide localized information, which has the potential to enhance crisis awareness and response. However, with limited resources, crisis managers may endure information overload from the inundation of these updates. Prior work has demonstrated the potential of machine learning (ML) methodologies to mitigate this problem. We have identified limitations in the prior work including the lack of involvement of crisis managers in the development and evaluation of a ML methodology.

To address these limitations, we propose a novel framework and ML methodology which investigate the efficacy of various ML methods in enhancing crisis awareness and response beyond model performance metrics. This framework aims to iteratively embed the information needs and priorities of crisis managers during crisis into the design of the ML methodology. We cooperated with crisis managers in Fukuchiyama City (FC), a city in Japan which is susceptible to flood events, and analyzed crowdsourced crisis image and text data from past FC flood events. We devised the Flood Presence image classification task, constructed Train/Dev/Test splits, and annotated images from FC. We report a weighted F1 score of 92.1% on the test split and 82.5% on the FC images. Using the results of our image analysis ML methodology and the insights we gained from crisis managers, we iterated on the design of our text analysis ML methodology. This led to the creation of the Human Risk text classification task which is tailored to a subset of the identified information needs of the crisis managers. To align with the priorities of crisis managers for this task, we determined the model evaluation metric to be the F2 score. We report an F2 score of 92.8% on an FC crisis text test dataset, which is a significant improvement over the baseline score of 43.4%.