Skip to main content

SD television formats in Panda

The edge of video transmission is moving quickly, just to mention HD television being mainstream for some time and 4K getting traction; H264 being ubiquitous, and HEVC entering the stage. Yet most people still remember VHS. It’s good to be up with the latest tech, but unfortunately the world is lagging behind most of the time.

Television is a different universe than Internet transmission. The rules are made by big (usually government) bodies and rarely change. Although most countries have switched to digital transmission, standard definition isn’t gone yet – SD channels are still very popular, which forces content providers to support SD formats too.

Recently, we’ve helped a few clients to craft transcoding pipelines that support all these retiring-yet-still-popular formats. We’ve noticed that it’s a huge nuisance for content makers to invest in learning old technology and that they would love to shed the duty on someone else; so we made sure that Panda (both the platform and the team) can deal with these flawlessly.

There’s a huge variability among requirements pertaining SD: for example, you have to decide how the image should be fitted into the screen. High-quality downsampling is always used, but you have to decide what to do when the dimensions are off: should you use letterboxing, or maybe stretch the image?

Fiordland National Park, New Zealand (Nathan Kaso)
Fiordland National Park, New Zealand (Nathan Kaso)

Another decision (which usually is not up to you) is what exact format should be used. This almost always depends on the country the video is for. Although the terms NTSC, PAL and SECAM come from the analog era (digital TV uses standards like ATSC and DVB-T), they are still used to describe parameters of encoding in digital transmission (e.g. image dimensions, display aspect ratio and pixel aspect ratio). Another thing the country affects is the compression format, the most popular are MPEG-2 and H.264, though they are not the only ones.

Standard television formats also have specific requirements on frame rate. It’s a bit different than with Internet transmission, where the video is effectively a stream of images. In SD TV, transmission is interlaced, and instead of frames it uses fields (which contain only half the information that frames do, but allow to save up bandwidth).

Frame rate is therefore not a very accurate term here, but the problem is still the same – we have exact number of frames/fields to display per unit of time, and the input video might not necessarily match that number. In such case the most popular solution is to drop and duplicate frames/fields according to the needs, but quality of videos produced this way is not great.

There is a solution, though, but it’s so complicated that we’ll just mention it here – it’s motion compensation. It’s a technique originally used for video compression, but it also gives great results in frame rate conversions. It’s not only useful for SD conversions, we use it for different things at Panda, but it helps here too.

Well, it’s definitely not the end of the story. These are the basics, but the number of details that have to be considered is unfortunately much bigger. Anyway, if you ever happen to have to support SD television, we’re here to help! Supporting SD can be as easy as creating a profile in Panda:

Adding SD profile in Panda
Adding SD profile in Panda

Video Marketing Can Be the Most Effective Way to Reach Your Audience

PolarBears

For the last decade, content marketing has been dominated by video streaming. Whether you’re a comedian posting funny videos to build a following or a business creating an informative product demo to help your viewers, choosing the right type of video message is crucial to boosting views and rankings. Here is a closer look at some of the most popular forms of video marketing for various content types.

Social Videos for Individual Messages or Projects

Streaming media is the foundation on which the social Internet runs. Over the last two years, Twitter and Instagram have piggybacked on the social video marketing success of YouTube and Facebook. Twitter released Vine, which allows the user to post and share six-second videos, while Instagram added video-streaming capabilities to their regular feeds.

The benefit of choosing social video is that it has the ability to reach many people in a short amount of time. If your video is only 30 seconds to a minute long and designed to capture your viewer’s attention within the first five seconds, there’s a better chance of getting more views, likes, and shares.

This type of content marketing is great for short messages, entertainment (i.e., funny videos), and sales messages.

The Birth of the Online Film Is Giving New Life to Video Marketing

There is a misconception that people won’t take time out of their daily routines to watch a video that is more than three minutes long. YouTube was built around this belief and, up until a few years ago, was dominated by it. The Coca-Cola Company has paved the way for those needing to deliver a message that cannot be adequately expressed in under five minutes, but still want to reach their streaming media audiences.

The seven-minute animated video for their new car line was a mixture of important information, humor, and entertaining visuals. It was quickly embraced by viewers and sent across the social media world. Other companies have also taken this route. Some videos have breached the 10-minute mark, reaching up to almost half an hour in length.

This type of video marketing can be tough, but for those with an important message to deliver and a penchant for creativity, the online film may be the answer to avoiding traditional marketing routes. Online films are best suited for extended product sales pitches or for providing a visual checklist of content that appeals to your audience.

Panda Is The Foundation Of A Psychological Assessment Tool Used For The Department of Defense

Seeking a Developer-Friendly Video Encoding Solution

Adam Hasler builds digital products. He’s the lead developer at The Big Studio, a design-focused consultancy based in Boston. Not only does Adam engage in a lot of design, but he also does all the coding. Like other digital product leaders, Adam Hasler is first and foremost a developer and a designer. When it comes to other projects like video encoding, it’s usually out of scope for a typical day’s work.

Adam’s been working on an app that’s used by psychologists as an assessment mechanism. It’s the project of a psychologist, who’d been applying this assessment framework on paper, and administering it to people that way.

Adam’s task was to build a video quiz where subjects could click on a video and give their feedback. The video quiz component would then record where they clicked, and allow them to give feedback on why that moment resonated for them. Each subject’s feedback would then be compared to that of experts to assess whether or not they could read a situation as well.

adam-hasler-small

“I needed to build a tool where subjects watching a video could say, ‘There, right there, that thing that happened is what I think is important,’” explains Adam Hasler. “In my first test build, I used a solution that involved uploading a video and running it through a script. It didn’t work. It was a disaster.”

Panda Is The Best Solution

To complete the project, Adam needed to build both a testing and an authoring component. Psychologists needed to be able to write the tests, so there were two user personas in that sense: a tester and a test taker. The tester would always be a psychologist, who wasn’t technologically savvy, so Adam had to make a really good test editing interface. Because of the nature of the project, he needed to:

  1. Upload videos
  2. Have them appear in different formats depending on the browser being used

“I discovered Panda through Heroku, and it ended up being the best solution,” says Adam. “With Panda, psychologists can author an assessment video by dragging and dropping it into a container I built. Panda uploads it, and collects the feedback. We don’t have to worry about uploading 4 different video file types, because Panda encodes the videos to work on different browsers.”

shadowbox-screenshot

Panda Delivers Encoded Video To The Department Of Defense

 Thanks to Panda, the project has been highly successful. One of the key user personas is the Department of Defense, which is testing subjects for their response to conflicts.

“Because of the interactivity, I needed more than a video on a page,” explains Adam. “With Panda, I get that beautiful little jSON object back with all the information I would need to make all the difference for this very little, key component. I love Panda! It made my life so much easier. I think it’s so cool.”

Re-architecting for *real* scale

7 minute read

On the surface, Panda is a pretty simple piece of software – upload a video, encode it into various formats, add a watermark or change frame rate, and deliver it to a data store.

Once you spend some time with it, it begins to show how complex each component can be – and how important it is to continuously improve each one.

Lucy Production Line

When Panda was first built, it worked beautifully, and it was quick! But as time went on, and the volume of videos encoded per day increased, it became obvious that to keep pace with increasing speed requirements from customers, and maintain growth – core parts of the platform were going to need to be rethought.

We started looking at each component piece by piece, to find bottlenecks, optimize throughput and keep a fair operating expense so we could retain our price leadership. Panda might be a software platform – but having read the ‘The Goal‘ by Eli Goldratt about a manufacturing plant really reminded us of the process. (It’s a great read btw).

In July we updated to the most current versions of Ruby and Go – and added a memory cache to tasks that were maxing out our instances. Then we tackled the big scale bottleneck – the job manager.

Our biggest bottleneck: the Job Manager

The Job Manager is built to ensure that our customers video queues get processed as close to real-time as possible, and distributes transcoding jobs to the encoder clusters. Whether it’s 2000 encoders on 8 CPU cores each, or 1 encoder on 1 CPU core it’s important it’s allocated correctly.

It monitors all encoding servers running within an environment, receives new jobs, and assigns them to instance pools.

The Panda Job Manager was a single thread Ruby process, which worked well for quite some time. We noticed it would start struggling during peaks, and we had to do something about it. We started looking at where we could optimize it, by identifying each bottleneck one by one.

It was obvious that events processing was too slow in general, but before we even fired up a profiler, we managed to find a huge one just by looking at logs and comparing timestamps.

Redis Queue Architecture

Short digression: We use Redis queues for internal communication, and there was one such queue where all messages for the manager were being sent. The manager was constantly pooling this queue and most of its work was based on messages it received. Each encoding server had a queue in Redis too, and all these queues were used for communication between the manager and encoders.

Image 2014-12-04 at 12.24.46 PM

Because a single Redis queue was used for new jobs as well as manager/encoders communication, huge numbers of the former were causing delays in the latter. And a slow down in internal communication meant that some servers were waiting unnecessarily long for jobs to be assigned.

Is Ruby and Redis the Answer?

The obvious solution was to split the communication into two separate queues: one for new jobs and another one for internal messaging. Unfortunately, Redis doesn’t allow blocking reads from more than one queue on a single connection.

We were forced either to implement Redis client that would use non-blocking IO to handle more that one connection in a single thread, or resort to multiple threads or processes. Writing our own client seemed like a lot of work, and Ruby isn’t especially friendly if you’d like to write multithreaded code (well, unless you use Rubinius).

Before trying to solve that, we launched manager within a profiler to get a clearer picture. It turned out that roughly 30% of time was spent at querying the database (jobs were saved, updated and deleted from the DB), and the remaining 70% was just running the Ruby code. Because we were a few orders of magnitude slower that we wished, optimizing neither just the database nor the Ruby code would be enough (and we still had to solve the queues issue). We needed something more thorough that a simple fix. 

Go baby, GO!

gopherWe started by rewriting the manager in Go. We didn’t want to waste time on premature optimization, so it roughly was a 1:1 rewrite, just a few things were coded differently to be more Go-idiomatic – but the mechanics stayed the same.

The result? Those 70% that were previously spent on Ruby code dropped to about 1%! That was great, we got almost 70% speed-up, but we were still nowhere near where we wanted.

Multithreading

Then we fixed the queues issue. With Go’s multithreading model is was so simple that it’s almost not worth mentioning – we even accidentally got a free message pre-fetching in a Go channel (another thread pools Redis and pushes messages to a buffered channel). And this was a huge kick – now we could handle more than 16,000,000 jobs per day per job manager.

We could have pushed it harder, but we still hadn’t even started profiling our new Go code at this point. Golang has great tools for profiling, so rather quickly we were able to go through the bottlenecks (it was database almost all the time). When we decided that it’s enough, we started testing… And we just couldn’t get enough EC2 instances to reach manager’s limit. We ended at about a bit less than 80,000,000 jobs per day and even a sign of sweat wasn’t visible on manager.

The graph below shows the number of videos per day projected from the number of videos processed within the last 30 minutes. We started at a bit more than 1,000,000, then switched to the Golang manager and got to the 80,000,000 limit – but there were no more jobs (we reached our EC2 spot limits while performing the benchmark!), so we might have processed even more (but it should be a safe number for some time).

YC4FoQlSGCzci5043QJO

The end result of this phase is a technical architecture that clears queues much faster, and for the same encoder price, delivers better throughput and greatly enhanced encoder bursting (especially good during the holiday season where we often have customer that ratchet up activity by 100x!), and more automation. We’re not done yet – and we have some fantastic features coming in 2015 that the new back-end enables us to deliver.

PS. Kudos should also go to Redis – it’s a fantastic, very stable and battle-tested piece of software. Big thanks, Antirez!

Do you have a suggestion or have some knowledge you’d like to share with us? We’d love to hear from you – get in touch support@pandastream.com anytime (we’re 24×7).

Apple’s iPhone 6 and 6 plus boast support for H.265

Image 2014-10-16 at 4.09.16 PMApple released its flagship device, the iPhone 6 and iPhone 6 plus a few weeks ago, and according to Tim Cook, it’s their biggest iPhone month ever.

Most analysts, fanboys, and tech reviewers are keen on the larger screen size, new processor, and how thin it is.

Here at Panda on the other hand, were delightfully surprised that on their specs page both the iPhone 6 and 6 plus are said to utilize H.265 for encoding and decoding FaceTime.

As we’ve said in our previous blog post, H.265 or High Efficiency Video Coding (HEVC) is said to match the quality of H.264, but at half the bit rate. This would be a massive help for cellular networks, by reducing bandwidth by up to 50%.

Interestingly, in today’s Apple event they announced the new iPad Air 3, but that device does not support H.265.

H.265 has yet to see wide adoption on the consumer device market, so perhaps the iPhone can blaze another trend, as it has done so well so far.

Send us a note to support@copper.io if you want to get started with H.265 video encoding.

 

 

 

 

Panda introduces support for H.265

What is HEVC

H.265 or High Efficiency Video Coding (HEVC) is the next generation of H.264 which is commonly used in blu-ray encodings. It’s goal is to improve compression – not just add more – up to 50% over it’s predecessor,  while attaining the same level of picture quality. It can also support resolutions up to 8192×4320 (8K).

HEVC Background

There are two key groups that are helping move this industry forward, The Motion Picture Experts Group, and the International Telecommunication Union’s Telecommunication Standardization Sector (ITU-T). Side note: Could you please find an easier name ? Someone here is being a troll.

Their goal is to reduce the average bit-rate by 50% for fixed video quality, and higher quality at the same bit-rate, while remaining interoperable and network friendly.

Since the majority of internet bandwidth is video (I’m looking at you netflix), one can imagine by reducing the bit-rate of video while keeping quality high could significantly reduce the strain on current networks.

HEVC Frame Types

Similar to H.264 and MPEG-2, there are three types of frames: I, P, and B. These frame types are the core to video compression, but in newer codecs such as H.265, the algorithms used are becoming more sophisticated.

  • I Frame (Intra-coded picture): Like a static image, these frames are often used as references for decoding other frames. They are usually the biggest, with the most data, but are used as references for other frames to be smaller.

  • P Frame (Predicted picture): This frame uses data from the previous frame that is unchanged, and only updates the areas that have changed. This frame can use image data and/or motion vector displacements to create the frame.

  • B Frame (Bi-predictive picture): A more advanced version of P, as it looks at the frame before and after to create a frame. These frames are the most efficient for final file size, but significantly slow the encoding process.

How does H.265 work and how is it different

HEVC breaks down each frame into coding units (CU) which are small blocks ranging from 4×4 pixels, all the way up to 64×64 pixels. The old maximum size was 16×16. These blocks are then used to compare which areas of the frame to change, and which areas can be referenced from I-Frames.

There is also an increased number of modes for intra prediction, from 9 in H.264 to 35 in H.265. While that will be much more processor intensive, the larger blocks will be more efficient.

coding-units

Image credit: elementaltechnologies.com

All these improvements sound great, but it needs a great deal of computational power, in some cases up to ten times. This is one of the reasons we have introduced multi-core encoders into our infrastructure. They can handle these calculations, and increased resolutions.

Encoding Tools

Each encoder can vary depending on it’s implementation and use of tools available. This includes but not limited to:

  • Intra prediction
  • Motion compensation
  • Motion vector prediction
  • Filters
  • Parallel processing

Heads up vs. VP9

There have been preliminary tests executed by the Fraunhofer Heinrich-Hertz-Institute on performance comparisons of H.264/MPEG-AVC and H.265/MPEG-HEVC and VP9.

In similar encoding configurations, H.265 saw bit-rate savings up to 43% over VP9 and 39% over H.264.

Encoding times were a totally different story, where VP9 outperformed H.265 by 7.3% and H.264 by 130%.

We’ve done our own tests, and in the example shown below, we’ve been able to get H.265 almost 50% smaller than H.264 with the same quality. Since there is no browser support for H.265 yet, you can download a chrome plugin, or play it with VLC player.

You can view the H.264 video below, which is 1.7MB, and download the H.265 video, which is 964KB.

Where the chips fall

Our initial tests show that VP9 and H.265 have similar file sizes, VP9 in conjunction with WebM seem to be more reliable for streaming. However, H.265 seems to have better image quality.

While this isn’t turning into a Blu-ray vs HD-DVD competition, VP9 does have a leg up being royalty free. Most companies have announced support for both formats, but YouTube has yet to support H.265, and is encoding most high res videos with VP9.

H.265 and Panda

The complexity and increased processing power needed for HEVC are well matched to infrastructure and software that Panda provides. We’ve recently added multi-core encoders just for this reason.

A few customers have had early H.265 access and we’re now opening up broader access. If your business is interested in being on the leading edge, email us to be a part of the private beta at mark@copper.io

References

http://en.wikipedia.org/wiki/Video_compression_picture_types
http://en.wikipedia.org/wiki/High_Efficiency_Video_Coding
http://www.elementaltechnologies.com/
http://iphome.hhi.de/marpe/download/Performance_HEVC_VP9_X264_PCS_2013_preprint.pdf

Priority on Clouds and Select/Deselect all profiles

Two new exciting features to tell you about this week! Some of you may have noticed while using our GUI of Panda – we are on a roll with new features and improvements. This week we are bringing priority on which cloud, and select and deselect all on profiles.

Priority on Clouds

priority-cloudsPanda has been built as FIFO – First In First Out, and we encode most of your videos this way. But sometimes you have some more important videos that you need to jump the queue with.

Now with priority on clouds the videos that you need encoded asap are processed outside of the queue. Any videos uploaded to that cloud will be encoded before any other video uploaded previously.

We’ve configured this to be a little flexible, as there are three settings: Low, Normal, and High. You can build a hierarchy to get important videos delivered quick, and others just when there’s availability.

Remember, you can upload files up to 20GB in size, so maybe make those larger size videos low priority, and the small ones high priority.

select-deselect-all

Select and Deselect all Profiles

This one is pretty straight forward – if you have a number of profiles in your cloud you will be able to select/deselect all of them with one click. Sometimes it’s just easier to pick the ones you don’t want instead of selecting them manually, one by one. Just small tweak to make life easier.

Panda activity widget

Keep track of all your encoding jobs

We’ve recently launched a new great way to keep track of your encoding queue, rather than logging into the app. The Panda activity web widget is an easy glance at what videos are being encoded, and those that are pending.

Data includes the filename, the profile, encoding process, and number of jobs left in the queue.

Configure it!

It’s really easy to get the widget on your website:

  1. Log into your account at pandastream.com
  2. Click on a cloud
  3. Under Cloud Settings, change “Enable Web Widget” to Yes.
  4. Copy and Paste the code to your website.
  5. Click Save Changes at the bottom of the Cloud Settings page.

Encode Faster with Panda Multi-Core Encoders

We’ve been experimenting with ways to increase the speed of encoding times and we’ve developed a feature that allows you to turn the knob, slam the gas pedal, or push on the thrust lever. Multi-Core encoders.

Panda’s value proposition is buy an encoder and use it as much as you’d like. Our customers love that. If you have more volume of encoding jobs, you would add encoders. Adding encoders doesn’t increase the speed of jobs – you simply can encode more at once.

Video Encoding Kryptonite

Enter Panda Multi-Core. We think we have found the kryptonite to video encoding by enabling the use of multi-core encodings. With Panda Multi-Core, you now have the ability to turn on 2-32 cores on your video encoding processes.

At this point we are testing how much faster you can encode your videos using our most popular codecs, like WebM. We are still testing how fast we can increase the speed, but it is theoretically possible to increase encoding speed by 2x, 10x or even more.

Scenarios explained

Lets say you’re currently on our Crane plan, which is four encoders. That gives the capability of encoding four videos at the same time. With Panda Multi-Core you can turn up the speed on each of those encoders by as many cores as added.

multicore

Those 4 encoders can tap into our multi-core system, and you can choose to have as many cores on each of those 4 encoders. For example, you can encode four videos at once, but with up to 32 cores each!

Its kinda like being in a water slide race against your best buds, but you just put on a jet pack.

Apply for Beta

Currently we are testing out multi-core encoders with a few of our customers and seeing great results. Do you want to reduce your encoding times by 2, 10 or even 32 times? Give us a shout!

Timestamps in Panda

timestamp

A few weeks back we showed you how you could implement timestamps on your encoded videos, and we had a lot of great feedback – thank you!

A few of you also asked if this was available in the GUI as well. So, we did it! You can login to your Panda account, and select or create a new profile that you want to use for timestamps. You can check the timestamps box, and that will turn on the feature.

Currently, the options are limited, simply turning it on and off. If you are looking for more options in the GUI for timestamps, please let us know!

The video encoder adds the timestamp with FFMPG and works best with FPS of 24, 25, 30, 50 and 60.

timestamps-onvideo