IntelQuickSyncVideo

Differences between revisions 36 and 38 (spanning 2 versions)
Revision 36 as of 2017-08-09 01:44:21
Size: 11489
Editor: vanvugt
Comment:
Revision 38 as of 2017-08-29 09:34:19
Size: 11419
Editor: vanvugt
Comment:
Deletions are marked like this. Additions are marked like this.
Line 141: Line 141:
Many people cite YouTube as their main reason for wanting hardware video decoding. However it is important temper expectations here. YouTube primarily uses the VP9 codec when displaying video on a desktop browser. You can verify the codec in use by right-clicking on a video and selecting "Stats for nerds".

Hardware-accelerated VP9 decoding only exists for Intel CPUs [[http://wiki.webmproject.org/hardware/socs|in Kaby Lake and later]]. And that's only 8-bit VP9 support. If you want HDR or 10-bit VP9 then that requires a [[https://github.com/01org/intel-vaapi-driver/releases/tag/1.8.3|Gemini Lake CPU]], which Intel has not released yet.
Many people cite YouTube as their main reason for wanting hardware video decoding. However it is important temper expectations here. YouTube primarily uses the VP9 codec when displaying video on a desktop browser. You can verify the codec in use by right-clicking on a video and selecting "Stats for nerds". Hardware-accelerated VP9 decoding only exists for Intel CPUs [[http://wiki.webmproject.org/hardware/socs|in Kaby Lake and later]].
Line 147: Line 145:

== Overall Architecture ==

{{https://docs.google.com/drawings/d/1V_28_gdfkQYdjeuvhZu3svJ5pWNJa7nH46uz-9139oM/pub?w=967&h=944}}

Hardware-Accelerated Video Playback on Intel CPUs in Ubuntu

Note this document covers Intel CPU/GPUs only.

There's a lot of complicated stuff written on this topic. This document aims to get you working efficient video playback on Ubuntu, as simply as possible, as well as outlining future plans.

It's important to know the difference between working and not working. If it's working then you should be able to play modern 4K or full HD videos with around 20% CPU usage or less on a laptop and 10% or less on a desktop. If it's much more than that then something has gone wrong. For example, in testing these instructions a properly configured Haswell desktop uses only 3% CPU to play 4K 60FPS H.264 video.

Alternate approaches do exist, but those have been observed to increase rather than decrease CPU usage when compared to the default software decoders. So remember to measure your CPU. A working solution should play the video and decrease your CPU usage significantly.

Quick Start

Here's how to make things work quickly. What "works" well is limited to a couple of particular video players. If you need web browser video, or MKV files for example, then you should look at the Future Roadmap instead.

  1. Install Ubuntu. You may find that 17.10 or 17.04 is required in order to get a current acceleration driver that supports your hardware.

  2. Preferably you're using X and not Wayland. This means either Unity 7 or GNOME is OK. Just don't use GNOME on Wayland because it's not yet fully working. That is; unless you're willing to use some ugly workarounds outlined below.

  3. Install the Intel acceleration driver and tools:
     $ sudo apt install vainfo i965-va-driver
  4. Check that acceleration is installed and supported:
     $ vainfo
    You should see a list of video codec profiles that your system is capable of playing with full hardware acceleration.
  5. Reboot. It's not yet clear why, but you might need to reboot before the best performance is achievable.

Option 1: Accelerated playback using GStreamer

  1. Install the missing packages:
     $ sudo apt install gstreamer1.0-vaapi gstreamer1.0-libav gstreamer1.0-plugins-ugly gstreamer1.0-plugins-bad
  2. Play your video using gst-play-1.0 for the best performance:

     $ gst-play-1.0 myvideo.mp4
    or if you are using a Gnome Shell Wayland session then the required syntax is:
     $ env GST_GL_WINDOW=wayland gst-play-1.0 --videosink glimagesink myvideo.mp4

    You can also use totem to play videos accelerated, but compared to gst-play-1.0 or mpv it is more power hungry, less smooth and corrupt:

     $ env GST_GL_WINDOW=wayland totem myvideo.mp4

Option 2: Accelerated playback using MPV

  1. Install the missing packages:
     $ sudo apt install mpv
  2. Play your video with mpv hardware acceleration turned on (it's not turned on by default!):

     $ mpv --hwdec=vaapi myvideo.mp4
    or using autodetection:
     $ mpv --hwdec myvideo.mp4

    The reason why mpv is recommended over other FFmpeg-based players is because FFmpeg hardware acceleration support is not automatically integrated and has to be implemented per application. It is known to be implemented and working properly in mpv only.

  3. MPV can also work fully accelerated on Wayland, but again it doesn't work automatically. With Gnome Shell on Wayland you might need to force it:

     $ env -uDISPLAY mpv --hwdec --opengl-backend=wayland myvideo.mp4

Other video player apps won't work anywhere near as efficiently. VLC, Mplayer, Google Chrome and Firefox for example, will only use software decoding (much higher CPU) since they use FFmpeg directly rather than GStreamer. See the Future Roadmap below where we will try to fix those issues.

And TS files

Video files with the .ts extension are also playable with the same hardware acceleration after a few extra steps:

  1. Install the ffmpeg command-line tool:
     $ sudo apt install -f ffmpeg
  2. Make an MP4 file (this is fast and does not change the video or audio contents, just the file format):
     $ ffmpeg -i INFILE.ts -vcodec copy -acodec copy OUTFILE.mp4
  3. Play your new file OUTFILE.mp4 using the above instructions.

Know Your Limits

If video playback appears frozen or causes high CPU usage, then it's likely you're trying to play a certain profile of video your CPU is a bit too old to handle (or you made a mistake in following the instructions).

The simplest answer is to run 'vainfo' which will list all the video codecs that your CPU/GPU can handle in hardware.

The complicated answer is to look up your CPU model (from 'lscpu') on http://ark.intel.com, find out what generation it is and then using that information you can check your CPU generation against this list or this table.

Medium Term: A more polished quick start

The above Quick Start section has a lot of caveats. Most of them are mentioned as links to bugs that need fixing. But there is also the problem of poor MKV and TS container file support in GStreamer that needs fixing. After some or all of these issues are addressed we will be able to recommend Totem as the efficient video player of choice.

Long Term: Future Roadmap

In future we would like all video playback in Ubuntu to be accelerated by default (on modern Intel chips at least), out of the box. There are two good ways to get there. Both of them involve the FFmpeg because the FFmpeg libraries are the core decoding libraries that almost all video players and web browsers use by default.

For video player apps, we will need to modify the standard Ubuntu/Debian ffmpeg packages to add acceleration. For web browsers however, they will also each need to be modified as each browser has it's own private copy of FFmpeg libraries. So FFmpeg is the focus of all future work.

You should also know that the Intel Quick Sync Video is what we're aiming to use. Quick Sync Video is the Intel hardware feature that accelerates video handling efficiently. So if you see "QSV" referred to in software instructions then those instructions are misleading you, and actually talking about something else that indirectly uses your Intel Quick Sync Video hardware. Also note that historically the video decoding functions have been referred to by Intel as Intel Clear Video HD, and this is still true for Intel technical documents. But generally the internet community refers to it under the heading of Quick Sync Video (QSV).

Future Approach #1: VA-API ("LibVA") decode

FFmpeg's wiki claims that there is no support for a VA-API decoder. This appears to be contradicted using the above mpv instructions until you realize that mpv reimplements VA-API support from scratch. But it also seems to be disproven by the existence of vaapi_decode.c and lots of related logic in the FFmpeg source code starting in version 3.3.

The trouble is that FFmpeg's hwaccel architecture (v1) requires per-app implementation, and that implementation is missing in common players like Mplayer, Chrome/Chromium and Firefox. Even FFmpeg's own ffplay tool doesn't support it yet. However rather than implementing the v1 hwaccel architecture in each app, it is probably advisable to take (or wait for) the approach the FFmpeg themselves have in mind: Hardware Acceleration (hwaccel) API v2.

Future Approach #2: Media SDK ("MSDK")

(This is the approach that the FFmpeg documentation rather confusingly refers to as "Intel QSV")

Intel's Media SDK is the company's professional offering with the aim of providing the highest performance and the most video features all accelerated on Intel chips.

It sounds great. You can download the SDK for free, and they have also published a first open source release of it too. But there are hidden reasons why we can't use any of this yet in Ubuntu. The problem is that pieces are missing. Both of them have a dependency on a custom proprietary fork of LibVA (VA-API). While this "missing" code is actually provided in the commercial download, it requires that you:

  1. Overwrite official Ubuntu files, possibly breaking your system; and
  2. Recompile all your video players yourself; and
  3. Don't redistribute what you've made.

That won't work for us right now. And we don't recommend you try. It's very complicated and can easily break your Ubuntu installation.

So what about their open source version of Media SDK on Github? Well, that won't work either. It actually has a dependency on the commercial download in order to build at all, and with that has all the same restrictions as the commercial download.

Why is this a "Future Approach" then? Because in future we might be able to use the Media SDK after Intel removes the proprietary prerequisite problem.

YouTube

Many people cite YouTube as their main reason for wanting hardware video decoding. However it is important temper expectations here. YouTube primarily uses the VP9 codec when displaying video on a desktop browser. You can verify the codec in use by right-clicking on a video and selecting "Stats for nerds". Hardware-accelerated VP9 decoding only exists for Intel CPUs in Kaby Lake and later.

Even after you have satisfied the hardware requirements, web browsers like Chrome still don't have hardware acceleration support coded in for desktop Linux yet. But the work is in progress at this very moment.

Overall Architecture

https://docs.google.com/drawings/d/1V_28_gdfkQYdjeuvhZu3svJ5pWNJa7nH46uz-9139oM/pub?w=967&h=944

What about NVIDIA and AMD?

They have different acceleration architectures with different instructions. So for the sake of simplicity, NVIDIA and AMD are not covered in this document. Still, long term future efforts for all CPU/GPU types should be focussed on FFmpeg for the same reasons outlined above.

http://bbb3d.renderfarming.net/download.html

IntelQuickSyncVideo (last edited 2018-08-28 04:00:15 by vanvugt)