Seeking a Developer-Friendly Video Encoding Solution

Adam Hasler builds digital products. He’s the lead developer at The Big Studio, a design-focused consultancy based in Boston. Not only does Adam engage in a lot of design, but he also does all the coding. Like other digital product leaders, Adam Hasler is first and foremost a developer and a designer. When it comes to other projects like video encoding, it’s usually out of scope for a typical day’s work.

Adam’s been working on an app that’s used by psychologists as an assessment mechanism. It’s the project of a psychologist, who’d been applying this assessment framework on paper, and administering it to people that way.

Adam’s task was to build a video quiz where subjects could click on a video and give their feedback. The video quiz component would then record where they clicked, and allow them to give feedback on why that moment resonated for them. Each subject’s feedback would then be compared to that of experts to assess whether or not they could read a situation as well.

“I needed to build a tool where subjects watching a video could say, ‘There, right there, that thing that happened is what I think is important,’” explains Adam Hasler. “In my first test build, I used a solution that involved uploading a video and running it through a script. It didn’t work. It was a disaster.”

Panda Is The Best Solution

To complete the project, Adam needed to build both a testing and an authoring component. Psychologists needed to be able to write the tests, so there were two user personas in that sense: a tester and a test taker. The tester would always be a psychologist, who wasn’t technologically savvy, so Adam had to make a really good test editing interface. Because of the nature of the project, he needed to:

1. Upload videos
2. Have them appear in different formats depending on the browser being used

“I discovered Panda through Heroku, and it ended up being the best solution,” says Adam. “With Panda, psychologists can author an assessment video by dragging and dropping it into a container I built. Panda uploads it, and collects the feedback. We don’t have to worry about uploading 4 different video file types, because Panda encodes the videos to work on different browsers.”

Panda Delivers Encoded Video To The Department Of Defense

 Thanks to Panda, the project has been highly successful. One of the key user personas is the Department of Defense, which is testing subjects for their response to conflicts.

“Because of the interactivity, I needed more than a video on a page,” explains Adam. “With Panda, I get that beautiful little jSON object back with all the information I would need to make all the difference for this very little, key component. I love Panda! It made my life so much easier. I think it’s so cool.”

7 minute read

On the surface, Panda is a pretty simple piece of software – upload a video, encode it into various formats, add a watermark or change frame rate, and deliver it to a data store.

Once you spend some time with it, it begins to show how complex each component can be – and how important it is to continuously improve each one.

When Panda was first built, it worked beautifully, and it was quick! But as time went on, and the volume of videos encoded per day increased, it became obvious that to keep pace with increasing speed requirements from customers, and maintain growth – core parts of the platform were going to need to be rethought.

We started looking at each component piece by piece, to find bottlenecks, optimize throughput and keep a fair operating expense so we could retain our price leadership. Panda might be a software platform – but having read the ‘The Goal‘ by Eli Goldratt about a manufacturing plant really reminded us of the process. (It’s a great read btw).

In July we updated to the most current versions of Ruby and Go – and added a memory cache to tasks that were maxing out our instances. Then we tackled the big scale bottleneck – the job manager.

Our biggest bottleneck: the Job Manager

The Job Manager is built to ensure that our customers video queues get processed as close to real-time as possible, and distributes transcoding jobs to the encoder clusters. Whether it’s 2000 encoders on 8 CPU cores each, or 1 encoder on 1 CPU core it’s important it’s allocated correctly.

It monitors all encoding servers running within an environment, receives new jobs, and assigns them to instance pools.

The Panda Job Manager was a single thread Ruby process, which worked well for quite some time. We noticed it would start struggling during peaks, and we had to do something about it. We started looking at where we could optimize it, by identifying each bottleneck one by one.

It was obvious that events processing was too slow in general, but before we even fired up a profiler, we managed to find a huge one just by looking at logs and comparing timestamps.

Redis Queue Architecture

Short digression: We use Redis queues for internal communication, and there was one such queue where all messages for the manager were being sent. The manager was constantly pooling this queue and most of its work was based on messages it received. Each encoding server had a queue in Redis too, and all these queues were used for communication between the manager and encoders.

Because a single Redis queue was used for new jobs as well as manager/encoders communication, huge numbers of the former were causing delays in the latter. And a slow down in internal communication meant that some servers were waiting unnecessarily long for jobs to be assigned.

Is Ruby and Redis the Answer?

The obvious solution was to split the communication into two separate queues: one for new jobs and another one for internal messaging. Unfortunately, Redis doesn’t allow blocking reads from more than one queue on a single connection.

We were forced either to implement Redis client that would use non-blocking IO to handle more that one connection in a single thread, or resort to multiple threads or processes. Writing our own client seemed like a lot of work, and Ruby isn’t especially friendly if you’d like to write multithreaded code (well, unless you use Rubinius).

Before trying to solve that, we launched manager within a profiler to get a clearer picture. It turned out that roughly 30% of time was spent at querying the database (jobs were saved, updated and deleted from the DB), and the remaining 70% was just running the Ruby code. Because we were a few orders of magnitude slower that we wished, optimizing neither just the database nor the Ruby code would be enough (and we still had to solve the queues issue). We needed something more thorough that a simple fix. 

Go baby, GO!

We started by rewriting the manager in Go. We didn’t want to waste time on premature optimization, so it roughly was a 1:1 rewrite, just a few things were coded differently to be more Go-idiomatic – but the mechanics stayed the same.

The result? Those 70% that were previously spent on Ruby code dropped to about 1%! That was great, we got almost 70% speed-up, but we were still nowhere near where we wanted.

Multithreading

Then we fixed the queues issue. With Go’s multithreading model is was so simple that it’s almost not worth mentioning – we even accidentally got a free message pre-fetching in a Go channel (another thread pools Redis and pushes messages to a buffered channel). And this was a huge kick – now we could handle more than 16,000,000 jobs per day per job manager.

We could have pushed it harder, but we still hadn’t even started profiling our new Go code at this point. Golang has great tools for profiling, so rather quickly we were able to go through the bottlenecks (it was database almost all the time). When we decided that it’s enough, we started testing… And we just couldn’t get enough EC2 instances to reach manager’s limit. We ended at about a bit less than 80,000,000 jobs per day and even a sign of sweat wasn’t visible on manager.

The graph below shows the number of videos per day projected from the number of videos processed within the last 30 minutes. We started at a bit more than 1,000,000, then switched to the Golang manager and got to the 80,000,000 limit – but there were no more jobs (we reached our EC2 spot limits while performing the benchmark!), so we might have processed even more (but it should be a safe number for some time).

The end result of this phase is a technical architecture that clears queues much faster, and for the same encoder price, delivers better throughput and greatly enhanced encoder bursting (especially good during the holiday season where we often have customer that ratchet up activity by 100x!), and more automation. We’re not done yet – and we have some fantastic features coming in 2015 that the new back-end enables us to deliver.

PS. Kudos should also go to Redis – it’s a fantastic, very stable and battle-tested piece of software. Big thanks, Antirez!

Do you have a suggestion or have some knowledge you’d like to share with us? We’d love to hear from you – get in touch support@pandastream.com anytime (we’re 24×7).

Apple released its flagship device, the iPhone 6 and iPhone 6 plus a few weeks ago, and according to Tim Cook, it’s their biggest iPhone month ever.

Most analysts, fanboys, and tech reviewers are keen on the larger screen size, new processor, and how thin it is.

Here at Panda on the other hand, were delightfully surprised that on their specs page both the iPhone 6 and 6 plus are said to utilize H.265 for encoding and decoding FaceTime.

As we’ve said in our previous blog post, H.265 or High Efficiency Video Coding (HEVC) is said to match the quality of H.264, but at half the bit rate. This would be a massive help for cellular networks, by reducing bandwidth by up to 50%.

Interestingly, in today’s Apple event they announced the new iPad Air 3, but that device does not support H.265.

H.265 has yet to see wide adoption on the consumer device market, so perhaps the iPhone can blaze another trend, as it has done so well so far.

Send us a note to support@copper.io if you want to get started with H.265 video encoding.

 

 

 

 

What is HEVC

H.265 or High Efficiency Video Coding (HEVC) is the next generation of H.264 which is commonly used in blu-ray encodings. It’s goal is to improve compression – not just add more – up to 50% over it’s predecessor,  while attaining the same level of picture quality. It can also support resolutions up to 8192×4320 (8K).

HEVC Background

There are two key groups that are helping move this industry forward, The Motion Picture Experts Group, and the International Telecommunication Union’s Telecommunication Standardization Sector (ITU-T). Side note: Could you please find an easier name ? Someone here is being a troll.

Their goal is to reduce the average bit-rate by 50% for fixed video quality, and higher quality at the same bit-rate, while remaining interoperable and network friendly.

Since the majority of internet bandwidth is video (I’m looking at you netflix), one can imagine by reducing the bit-rate of video while keeping quality high could significantly reduce the strain on current networks.

HEVC Frame Types

Similar to H.264 and MPEG-2, there are three types of frames: I, P, and B. These frame types are the core to video compression, but in newer codecs such as H.265, the algorithms used are becoming more sophisticated.

• I Frame (Intra-coded picture): Like a static image, these frames are often used as references for decoding other frames. They are usually the biggest, with the most data, but are used as references for other frames to be smaller.

• P Frame (Predicted picture): This frame uses data from the previous frame that is unchanged, and only updates the areas that have changed. This frame can use image data and/or motion vector displacements to create the frame.

• B Frame (Bi-predictive picture): A more advanced version of P, as it looks at the frame before and after to create a frame. These frames are the most efficient for final file size, but significantly slow the encoding process.

How does H.265 work and how is it different

HEVC breaks down each frame into coding units (CU) which are small blocks ranging from 4×4 pixels, all the way up to 64×64 pixels. The old maximum size was 16×16. These blocks are then used to compare which areas of the frame to change, and which areas can be referenced from I-Frames.

There is also an increased number of modes for intra prediction, from 9 in H.264 to 35 in H.265. While that will be much more processor intensive, the larger blocks will be more efficient.

Image credit: elementaltechnologies.com

All these improvements sound great, but it needs a great deal of computational power, in some cases up to ten times. This is one of the reasons we have introduced multi-core encoders into our infrastructure. They can handle these calculations, and increased resolutions.

Encoding Tools

Each encoder can vary depending on it’s implementation and use of tools available. This includes but not limited to:

• Intra prediction
• Motion compensation
• Motion vector prediction
• Filters
• Parallel processing

Heads up vs. VP9

There have been preliminary tests executed by the Fraunhofer Heinrich-Hertz-Institute on performance comparisons of H.264/MPEG-AVC and H.265/MPEG-HEVC and VP9.

In similar encoding configurations, H.265 saw bit-rate savings up to 43% over VP9 and 39% over H.264.

Encoding times were a totally different story, where VP9 outperformed H.265 by 7.3% and H.264 by 130%.

We’ve done our own tests, and in the example shown below, we’ve been able to get H.265 almost 50% smaller than H.264 with the same quality. Since there is no browser support for H.265 yet, you can download a chrome plugin, or play it with VLC player.

You can view the H.264 video below, which is 1.7MB, and download the H.265 video, which is 964KB.

Where the chips fall

Our initial tests show that VP9 and H.265 have similar file sizes, VP9 in conjunction with WebM seem to be more reliable for streaming. However, H.265 seems to have better image quality.

While this isn’t turning into a Blu-ray vs HD-DVD competition, VP9 does have a leg up being royalty free. Most companies have announced support for both formats, but YouTube has yet to support H.265, and is encoding most high res videos with VP9.

H.265 and Panda

The complexity and increased processing power needed for HEVC are well matched to infrastructure and software that Panda provides. We’ve recently added multi-core encoders just for this reason.

A few customers have had early H.265 access and we’re now opening up broader access. If your business is interested in being on the leading edge, email us to be a part of the private beta at mark@copper.io

References

http://en.wikipedia.org/wiki/Video_compression_picture_types
http://en.wikipedia.org/wiki/High_Efficiency_Video_Coding
http://www.elementaltechnologies.com/
http://iphome.hhi.de/marpe/download/Performance_HEVC_VP9_X264_PCS_2013_preprint.pdf

Two new exciting features to tell you about this week! Some of you may have noticed while using our GUI of Panda – we are on a roll with new features and improvements. This week we are bringing priority on which cloud, and select and deselect all on profiles.

Priority on Clouds

Panda has been built as FIFO – First In First Out, and we encode most of your videos this way. But sometimes you have some more important videos that you need to jump the queue with.

Now with priority on clouds the videos that you need encoded asap are processed outside of the queue. Any videos uploaded to that cloud will be encoded before any other video uploaded previously.

We’ve configured this to be a little flexible, as there are three settings: Low, Normal, and High. You can build a hierarchy to get important videos delivered quick, and others just when there’s availability.

Remember, you can upload files up to 20GB in size, so maybe make those larger size videos low priority, and the small ones high priority.

Select and Deselect all Profiles

This one is pretty straight forward – if you have a number of profiles in your cloud you will be able to select/deselect all of them with one click. Sometimes it’s just easier to pick the ones you don’t want instead of selecting them manually, one by one. Just small tweak to make life easier.

Keep track of all your encoding jobs

We’ve recently launched a new great way to keep track of your encoding queue, rather than logging into the app. The Panda activity web widget is an easy glance at what videos are being encoded, and those that are pending.

Data includes the filename, the profile, encoding process, and number of jobs left in the queue.

Configure it!

It’s really easy to get the widget on your website:

1. Log into your account at pandastream.com
2. Click on a cloud
3. Under Cloud Settings, change “Enable Web Widget” to Yes.
4. Copy and Paste the code to your website.
5. Click Save Changes at the bottom of the Cloud Settings page.