PhD Proposal: Manipulation Action Understanding: From Alpha towards Omega

Talk
Yezhou Yang
Time: 
04.23.2014 09:00 to 10:30
Location: 

AVW 4424

Modern intelligent agents will need to learn the actions that humans perform. They will need to recognize these actions when they see them and they will need to perform these actions themselves. We want to propose a cognitive system that interprets human manipulation actions from perceptual information (image and depth data) and consists of perceptual modules and reasoning modules that are in interaction with each other. The contributions of this work are given along two core problems at the heart of action understanding: a.) the grounding of relevant information about actions in perception (the perception - action integration problem), and b.) the organization of perceptual and high-level symbolic information for interpreting the actions (the sequencing problem). At the high level, actions are represented with the Manipulation Action Grammar, a context-free grammar and associated parsing algorithms, which organizes actions as a sequence of sub-events. Each sub-event is described by the hand, movements and the objects and tools involved, and the relevant information about these quantities is obtained from biological-inspired perception modules. These modules track the hands and objects and recognize the hand grasp, objects and actions using attention, segmentation, and feature description. Additionally, the lesson from the findings on mirror neurons is that the two processes of interpreting visually observed action and generating actions, should share the same underlying cognitive process. Recent studies on humans have shown that grammatical structures underlying our representation of manipulation actions, which are used both to understand and to execute these actions. Understanding manipulation actions is like understanding language, while executing them is like generating language. Experiments on two tasks, i.e. a robot observing people performing manipulation actions, and a robot executing manipulation actions accordingly, are proposed to validate the formalism.
Examining Committee:
Committee Chair: - Dr. Yannis Aloimonos
Dept. Representative - Dr. Donald Perlis
Committee Member (s) - Dr. Hal Daume III
- Dr. Cornelia Fermuller