ADSC researchers win international action recognition competition

Advanced Digital Sciences Center researchers Bingbing Ni and pei [dot] yong

The HARL competition invited participants to submit algorithms to automatically detect, classify and localize activities taking place in Kinect video. This emerging video modality outputs both color and depth image sequences, the latter showing the distance of each point in a scene to the camera. HARL is the world’s first competition that focuses on recognizing and localizing complex human activities in footage from a Kinect camera.

ADSC Researchers Bingbing Ni, left, and Yong Pei, right, won the 2012 Human Activities Recognition and Localization competition in September. They will present their winning method during the November 2012 International Conference on Pattern Recognition (ICPR).

HARL participants were given benchmark datasets of video clips showing complex activities taking place in a typical office setting. Participants created algorithms that could analyze a long video clip and precisely determine when and where the activity took place in the video.

“Previous activity recognition research focused primarily on classification, which means determining what the human is doing,” Ni said. “But in HARL’s more realistic and challenging setting, we had to determine when, where and what the humans were doing. Knowing when and where is more important and difficult than knowing what.”

According to Ni, by winning this award, the team has demonstrated that their algorithm and method for detecting activities is state-of-the-art and is the best option for recognizing actions in Kinect footage.

“Our team’s method is based on fusing the depth image with the grayscale image in every step of the processing pipeline, including human key pose extraction, object detection, interaction contextual modeling and a Bayesian network based action detector,” Ni said.

Harl Discussion photo

Ni and Pei developed algorithms to detect, classify and localize activities in Kinect video. They used their algorithms to identify actions such as a discussion between two people, see left, handshaking or talking on the phone.

Of the teams submitting results, Ni and Pei ranked first in both classification and localization of activities. Visit HARL for more information about the competition results.

Ni and Pei work with University of Illinois at Urbana-Champaign Electrical and Computer Engineering Professor Pierre Moulin on the Multi-modal Visual Analytics research project at ADSC. The project aims to develop methods for detection, recognition and localization of events and actions in depth video. This research has the potential to significantly impact video-based rehabilitation and sports training, video surveillance and many other areas.

TheAdvanced Digital Sciences Centeris aUniversity of Illinois at Urbana-Champaignresearch center in Singapore. It is led by Electrical and Computer Engineering and Computer Science faculty at the University of Illinois. ADSC focuses on breakthrough innovations in information technology.