Improving an image annotation and retrieval agent using commonsense and personal knowledge

Abstract

As digital photography continues to grow in popularity and usefulness, consumers can accumulate large quantities of images that may rarely be looked at over the course of a lifetime. Since computer vision has not yet become sophisticated enough to automatically recognize the content of images, the use of a software agent for integrated annotation and retrieval of images is proposed. This agent, ARIA, will combine commonsense knowledge, specific personal background information about the user, and the context in which they will use the images to assist him/her in the process of locating and using his/her images. The act of composing an email or webpage provides the context needed for the agent to present relevant images. ARIA looks at the words closest to the cursor and uses those as a query to the photo database. Not only does ARIA look for direct keyword matches, but it also expands the query with related words to increase the likelihood of locating pictures. General relationships are garnered from Open Mind, a commonsense knowledge repository. Because consumer photography is very personal, a repository of information specific to the user is also included to bridge the gap.