share

ENGINEERING

4min read

Android - Decoding and Encoding video simultaneously

While recently working on a project from ThinkApps, we did a lot of research about rapid video encoding and decoding on Android and I want to share my findings in this post. The first obvious question is, why do we need hardware decoding and encoding for videos? The answer is simple: processing done by hardware will be a lot faster than what software could do.

As you probably know, the Android ecosystem is very fragmented with a lot of different software and hardware vendors on the market. There are a variety device manufacturers such as LG, Samsung, and HTC and different hardware manufacturers like Nvidia and Qualcomm which provide components like CPU and GPU chipsets for the devices.

When talking about hardware accelerated processing, it must be at a low level i.e. “close to the hardware”. The code responsible for processing needs to be run in-parallel on the hardware components. This is why most hardware manufacturers provide their own components (encoders and decoders) that can be used within the Android ecosystem.

This hardware-close approach does have a problem though, all of these companies are competitors. Each has their own R&D; departments with superstar engineers and the majority of the results they produce are proprietary in nature. That means one can’t expect a single decoded video frame to be in the same desired format on all Android phones. One also can’t assume that all encoders will consume the input frames in the same format either.

Take Qualcomm for example. Qualcomm hardware decoders produce decoded frames in a format which is not described anywhere. There is no description available on the Qualcomm official site. Even in the official Android documentation, one can find every official color format except for the Qualcomm format which isn’t even mentioned!

We tried several different approaches to troubleshoot this problem. We checked the FFmpeg solution, which can be compiled with Android’s hardware decoding. But, we wanted to get to a point where we have a fully hardware accelerated decoding-postprocess-encoding chain. We also tried Qualcomm’s solution with their samples without any success. This seems to be because Qualcomm’s solution is only for phones which are using Qualcomm Snapdragon chipsets. Even with their sample application, the final result included broken frames on our Jelly Bean devices with Snapdragon chipsets.

What’s more, Qualcomm doesn’t provide ready-to-use binaries for the libraries. Qualcomm’s code is mostly native C and uses deep system integration. So in order to use it, we have to compile the native code for every major Android release separately using Android sources. Basically, this means that we have to compile five different shared object libraries, each of them including a full download and compilation of Android sources. Later, they can be packed into a single .apk file but we need to compile it again every time we change something or a major Android API update is released.

Apparently Google has noticed the problem of fragmentation related to the hardware accelerated video processing and has started to fix it. The fix is a MediaCodec class which has been available since Android 4.1. Unfortunately, the first release had a serious implementation flaw: it didn’t have support for a portable encoding/decoding chain. It worked fine for encoding (as long as you use OpenGL to draw movie frames) or decoding (as long as you want to display the decoded movie on Android OpenGL surface view) but when you tried to do a combined decoding/encoding cycle, you run into the problem of color format conversions again.

As of Android API level 18, (Jelly Bean 4.3+) there has been a way to better use hardware acceleration and the problem from the first release has been solved. There are new API calls (createInputSurface particularly) that are using OpenGL surface view as an intermediary (the decoder writes data to the surface view using hardware acceleration and the encoder can then read the data directly from the OpenGL surface view buffer) but current device coverage is extremely low for Jelly Bean 4.3. Official statistics show only 1.5% of all devices have it currently installed which means for now it’s not really a feasible solution for wider audiences. Most of the people who are using the latest API agree that the pre-Jelly Bean 4.3 APIs are not really portable across the board and it’s not feasible to try and perform the encoder/decoder loop on your own. This means that from now on, (Jelly Bean 4.3 and later) all hardware manufacturers need to support dual side conversion for their hardware decoders and encoders (decoder color format to RGB to encoder color format).

By using the MediaCodec solution, we were able to gain 6-7x more speed in terms of processing when compared to software processing. We tested this approach using 720p videos which had 1280x720 resolution recordings at 30fps.

Below you can find more information about playing with hardware accelerated decoding/encoding:

share


KonradLead Software Engineer

LEARN MORE

Contact us if you have any questions regarding the article or just want to chat about technology, our services, job offers and more!

POLIDEA NEWSLETTER

Sign in and expect sharp insights, recommendations, ebooks and fascinating project stories delivered to your inbox

The controller of the personal data that you are about to provide in the above form will be Polidea sp. z o.o. with its registered office in Warsaw at ul. Przeskok 2, 00-032 Warsaw, KRS number: 0000330954, tel.: 0048 795 536 436, email: hello@polidea.com (“Polidea”). We will process your personal data based on our legitimate interest and/or your consent. Providing your personal data is not obligatory, but necessary for Polidea to respond to you in relation to your question and/or request. If you gave us consent to call you on the telephone, you may revoke the consent at any time by contacting Polidea via telephone or email. You can find detailed information about the processing of your personal data in relation to the above contact form, including your rights relating to the processing, HERE.

Data controller:

The controller of your personal data is Polidea sp. z o.o. with its registered office in Warsaw at ul. Przeskok 2, 00-032 Warsaw, KRS number: 0000330954, tel.: [0048795536436], email: [hello@polidea.com] (“Polidea”)

Purpose and legal bases for processing:

 

Used abbreviations:

GDPR – Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016
on the protection of natural persons with regard to the processing of personal data and on the free movement
of such data, and repealing Directive 95/46/EC (General Data Protection Regulation)

ARES – Polish Act on Rendering Electronic Services dated 18 July 2002

TL – Polish Telecommunications Law dated 16 July 2004

1)        sending to the given email address a newsletter including information on Polidea’s new projects, products, services, organised events and/or general insights from the mobile app business world |art. 6.1 a) GDPR, art. 10.2 ARES and art. 172.1 TL (upon your consent)

Personal data:name, email address

2)       statistical, analytical and reporting purposes |art. 6. 1 f) GDPR (based on legitimate interests pursued by Polidea, consisting in analysing the way our services are used and adjusting them to our clients’ needs, as well as developing new services)

Personal data:name, email address

Withdrawal of consent:

You may withdraw your consent to process your personal data at any time.

Withdrawal of the consent is possible solely in the scope of processing performed based on the consent. Polidea is authorised to process your personal data after you withdraw your consent if it has another legal basis for the processing, for the purposes covered by that legal basis.

Categories of recipients:

Your personal data may be shared with:

1)       authorised employees and/or contractors of Polidea

2)       persons or entities providing particular services to Polidea (accounting, legal, IT, marketing and advertising services) – in the scope required for those persons or entities to provide those services to Polidea

 

Retention period:

1)       For the purpose of sending newsletter to the given email address – for as long as the relevant consent is not withdrawn

2)       For statistical, analytical and reporting purposes – for as long as the relevant consent is not withdrawn

Your rights:

 

Used abbreviation:

GDPR – Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016
on the protection of natural persons with regard to the processing of personal data and on the free movement
of such data, and repealing Directive 95/46/EC (General Data Protection Regulation)

According to GDPR, you have the following rights relating to the processing of your personal data, exercised by contacting Polidea via [e-mail, phone].

1)       to access to your personal data (art. 15 GDPR) by requesting sharing and/or sending a copy of all your personal data processed by Polidea

2)       to request rectification of inaccurate personal data
(art. 16 GDPR) by indicating the data requiring rectification

3)       to request erasure of your persona data (art. 17 GDPR); Polidea has the rights to refuse erasing the personal data in specific circumstances provided by law

4)       to request restriction of processing of your personal data (art. 18 GDPR) by indicating the data which should be restricted

5)       to move your personal data (art. 20 GDPR) by requesting preparation and transfer by Polidea of the personal data that you provided to Polidea to you or another controller in a structured, commonly used machine-readable format

6)       to object to processing your personal data conducted based on art. 6.1 e) or f) GDPR, on grounds relating to your particular situation (art. 21 GDPR)

7)       to lodge a complaint with a supervisory authority,
in particular in the EU member state of your habitual residence, place of work or place of the alleged infringement if you consider that the processing
of personal data relating to you infringes the GDPR
(art. 77.1 GDPR)

No obligation to provide data:

Providing your personal data is not obligatory, but necessary for Polidea to provide you the newsletter service

Refusal to provide the above data will result in inability to receive the newsletter service.

Profiling

In the process of providing the newsletter service, we make decisions in an automated way, including profiling, based on the data you provide.

 

“Profiling” means automated processing of personal data consisting of the use of your personal data to evaluate certain personal aspects relating to you, in particular to analyze or predict aspects concerning your personal preferences and interests.

 

The automated decisions are taken based on the analysis of clicked and viewed content. They affect the targeting of specific newsletter content to selected users registered to receive the newsletter service, based on the anticipated interests of the recipient.