Visual intelligence for online communities : commonsense image retrieval by query expansion

Abstract

This thesis explores three weaknesses of keyword-based image retrieval through the design and implementation of an actual image retrieval system. The first weakness is the requirement of heavy manual annotation of keywords for images. We investigate this weakness by aggregating the annotations of an entire community of users to alleviate the annotation requirements on the individual user. The second weakness is the hit-or-miss nature of exact keyword matching used in many existing image retrieval systems. We explore this weakness by using linguistics tools (WordNet and the OpenMind Commonsense database) to locate image keywords in a semantic network of interrelated concepts so that retrieval by keywords is automatically expanded semantically to avoid the hit-or-miss problem. Such semantic query expansion further alleviates the requirement for exhaustive manual annotation. The third weakness of keyword-based image retrieval systems is the lack of support for retrieval by subjective content. We investigate this weakness by creating a mechanism to allow users to annotate images by their subjective emotional content and subsequently to retrieve images by these emotions. This thesis is primarily an exploration of different keyword-based image retrieval techniques in a real image retrieval system. The design of the system is grounded in past research that sheds light onto how people actually encounter the task of describing images with words for future retrieval. The image retrieval system's front-end and back- end are fully integrated with the Treehouse Global Studio online community - an online environment with a suite of media design tools and database storage of media files and metadata.

(cont.) The focus of the thesis is on exploring new user scenarios for keyword-based image retrieval rather than quantitative assessment of retrieval effectiveness. Traditional information retrieval evaluation metrics are discussed but not pursued. The user scenarios for our image retrieval system are analyzed qualitatively in terms of system design and how they facilitate the overall retrieval experience.