App software edits photos, videos in real time

Posting selfies to Instagram and Facebook has been popular among teens and young adults for many years now, but ADSC researchers are now upping the selfie-game with their development of an app that allows people to take a short video clip and alter the background using pre-designed animated templates to create a Hollywood style mini-movie in seconds.

The app, SnapClip, was developed by an external partner Basilo Labs, incorporates algorithms developed at ADSC into its core modules. This allows users to do this photo and video editing quickly and in real time.

ADSC researchers Jiangbo Lu and Peter Niu developed the Intelligent Scribbles algorithm after previous research resulted in an efficient interactive image segmentation algorithm. Lu and Niu tweaked the already-developed algorithm into a mobile-friendly version that could achieve real-time performance with satisfying results. This allows users with Intelligent Scribbles software to edit images or videos more quickly within the app, rather than doing it in the cloud.

Intelligent Scribbles brings “object selection,” a desktop experience typically using a mouse with a software such as Adobe Photoshop or Microsoft's Grabcut, to mobile devices using fingertips. When a user scribbles carelessly over the object intended for editing or applying various effects, the algorithm intelligently and instantly determines the user's intention and does the selection job correctly.

"Such a functionality in achieving instant and accurate response in particular, is very uncommon for mobile platforms,” Lu said. “Turning an image of many individual pixels into semantically meaningful parts or objects enables a range of interesting editing possibilities, such as depth-of-field rendering, colorization/recolorization, cut and composition.”

According to Niu, the project developed from the idea that with the rising usage of smartphones, people are tending to use cell phones as the one-stop tool of taking, editing and sharing photos and videos.

“There are many mobile apps with the capability of photo and video editing, but they could only handle easy tasks such as color filtering, exposure and blurring, because of the limitations of computational power,” Niu said.

According to Niu, Intelligent Scribbles is unique to other photo and video editing apps, as it is more accurate and efficient. In addition, in order to make the app real-time on a mobile platform, they implemented the algorithm with mobile graphic processing unit (GPU). They’re finding that it’s possible to implement some complex tasks such as image layering and image completion on the mobile platform based on their unique algorithm and GPU implementation, which are areas where other apps don’t measure up.

“Before Intelligent Scribbles, generally layering a photo into different objects is only be done on PC with softwares like Adobe Photoshop,” Niu said. “Intelligent Scribbles enabled the mobile developer to achieve the image segmentation in their app in a very light and intuitive manner.”

SnapClip, the first app to use the Intelligent Scribbles software, comes with pre-designed animated templates that range from snowy nature scenes to horror scenes with poltergeists to middle-earth or a candy mountain. Users can take video clips up to 12 seconds, add freeze frames, include music from their own library and add text, titles and credits when creating their videos.

“The effect is that people can create a short video that often feels like a movie trailer with users themselves inside,” Niu said.

In addition to catering to consumers, the app is also designed as a marketing tool for brands. For example, with the World Cup currently airing on television, SnapClip has created World Cup effects, so users can upload videos with their team’s flag and colors.

The Intelligent Scribbles technology was licensed this past April by Exploit Technologies Pte Ltd (ETPL), the technology transfer arm of the Agency for Science, Technology and Research (A*STAR) in Singapore. The app is available on iTunes and is designed for both the iPhone and iPad.

Lu and Niu are a part of a larger research and technology transfer project at ADSC, Magic Touch, which has a portfolio of advanced image/video manipulation technologies that provide easy ways to add effects to photos and videos via smart phones. The ADSC team has created novel ways to convert still photos into moving images, create a condensed image from a video clip where part of the photo remains still while other selected parts of the image continue to move, as well as selecting and cutting parts of images from photos on mobile phones. Moving forward, the team will be working on fast and reliable video object selection and editing on mobile devices with as least as possible user's intervention.

“We feel this technology has pushed the boundary of what people could do using the smartphones, and we’ll further explore along the direction to broaden the spectrum of human life,” Niu said.