Prioritized text spotting using SLAM

Abstract

We show how to exploit temporal and spatial coherence of image observations to achieve efficient and effective text detection and decoding for a sensor suite moving through an environment rich in text at a variety of scales and orientations with respect to the observer. We use simultaneous localization and mapping (SLAM) to isolate planar "tiles" representing scene surfaces and prioritize each tile according to its distance and obliquity with respect to the sensor, and how recently (if ever) and at what scale the tile has been inspected for text. We can also incorporate prior expectations about the spatial locus and scale at which text occurs in the world, for example more often on vertical surfaces than non-vertical surfaces, and more often at shoulder height than at knee height. Finally, we can use SLAM-produced information about scene surfaces (e.g. standoff, orientation) and egomotion (e.g. yaw rate) to focus the system's text extraction efforts where they are likely to produce usable text rather than garbage. The technique enables text detection and decoding to run effectively at frame rate on the sensor's full surround, even though the CPU resources typically available on a mobile platform (robot, wearable or handheld device) are not sufficient to such methods on full images at sensor rates. Moreover, organizing detected text in a locally stable 3D frame enables combination of multiple noisy text observations into a single higher-confidence estimate of environmental text.