UMD     This Site

Four authors with University of Maryland ties have written the chapter “Vision During Action: Extracting Contact and Motion from Manipulation Videos—Toward Parsing Human Activity,” in the book Modelling Human Motion: From Human Perception to Robot Design, published by Springer in July 2020.

ISR-affiliated Professor Yiannis Aloimonos (CS/UMIACS) and Associate Research Scientist Cornelia Fermüller (UMIACS), along with alumnus Konstantinos Zampogiannis (CS Ph.D. 2019) and current Ph.D. student Kanishka Ganguly, present an active, bottom-up method for the detection of actor-object contacts and the extraction of moved objects and their motions in RGBD videos of manipulation actions.

When we physically interact with our environment using our hands, we touch objects and force them to move: contact and motion are defining properties of manipulation. The core of the approach described in the new book lies in non-rigid registration: continuously warping a point cloud model of the observed scene to the current video frame, generating a set of dense 3D point trajectories.

Under loose assumptions, the authors employ simple point cloud segmentation techniques to extract the actor and subsequently detect actor–environment contacts based on the estimated trajectories. For each such interaction, using the detected contact as an attention mechanism, they obtain an initial motion segment for the manipulated object by clustering trajectories in the contact area vicinity and then jointly refine the object segment and estimate its 6DOF pose in all observed frames.

Because of its generality and the fundamental, yet highly informative, nature of its outputs, this approach is applicable to a wide range of perception and planning tasks. The authors qualitatively evaluate the method on a number of input sequences and present a comprehensive robot imitation learning example that demonstrates the crucial role of outputs in developing action representations/plans from observation.

Related Articles:
Cornelia Fermüller is PI for 'NeuroPacNet,' a $1.75M NSF funding award
Microrobots soon could be seeing better, say UMD faculty in Science Robotics
New model predictive control framework improves reactive navigation for autonomous robots
Student autonomous drone racing team takes 2nd place at IROS
'MorphEyes' stereo camera system improves quadrotor UAV navigation
MRC Faculty and Researchers to Present 16 Papers at International Robotics Conference - IROS 2020
$100,000 investment from Amazon to power Clark School initiatives in diversity, robotics research and education
Bera, Manocha and Shim awarded a BBI Seed Grant to examine the role of age and gender on gait-based emotion recognition
Alumnus Zhenyu Lin is Google software development engineer
Computational framework automatically optimizes the shape of tissue engineered vascular grafts

July 17, 2020

«Previous Story  



Current Headlines

Proloy Das Receives Honorable Mention in Charles Caramello Distinguished Dissertation Competition

Alum Victor De Oliveira elected Fellow of American Statistical Association

Alum Himanshu Tyagi promoted to Associate Professor at  Indian Institute of Science

Firebird Tour Makes Stop in St. Mary’s County

Chapin, Fiaz and Mavridis win Wylie Dissertation Fellowships

Nikolas Francis on the Neurophysiology of Listening

UMD Researchers to Design Greener Aircraft Engine

Voice-activated telehealth technology could strengthen patient-physician connection

Maryland Joins NSF-funded Effort to Help Set Nation's Direction for Engineering

Biofilm-fighting system for urinary catheters proves effective in simulated environment

Back to top  
Home Clark School Home UMD Home