Multi-Modal Robotic Intelligence

Recently, substantial progress has been made in the machine perception, particularly computer vision, largely due to the advancements in deep learning techniques. However, robots often operate in unstructured environments, which differ greatly from the well-defined problems typically addressed in computer vision. Consequently, many existing computer vision solutions are not directly applicable to robotics. Additionally, modern intelligent robots have access to multi-modal sensory information, rather than relying on a single modality. Therefore, it is essential to explore the specific challenges of multi-modal robotic intelligence. A key requirement for all intelligent robots is the capability of active perception. Active perception is an effective approach to bridge the gap between robotics and machine perception. Intelligence emerges when the robot actively interacts with the environment, embodying a close coupling of perception and action within a continuous feedback loop. Perception guided action, and each movement generate information that informs subsequent movements. With a variety of available multi-modal sensory information, it is crucial for robots to leverage active perception techniques to achieve multi-modal robotic intelligence. This book introduces multi-modal robotic intelligence from the perspective of active perception. Extensive robotic multi-modal active perception problems are formulated and corresponding case studies are described. Specifically, this book is organized in three parts. Part I covers core concepts and approaches for multi-modal robotic intelligence. In Part II, active perception for robotic intelligence is described, which presents several typical active perception tasks. Part III further describes the multi-modal active perception for robotic intelligence. The book is primarily intended for researchers and graduates with a foundational knowledge of machine learning, spanning in a wide range of disciplines, particularly those involved in robotic intelligence and sensor fusion.