A mixture of emotions came across as I realized I wasted an entire day on a software decoder when a hardware solution was available for $30 to $40. The software solution was fairly disappointing, so I decided to try one of these new cards, an nVidia GeForce 8500 GT, to see if it provided a better solution. It took some work with my setup, but the results were worth it.

The new interface from nVidia for hardware video decoding on Linux is known as the Video Decode and Presentation API for UNIX or VDPAU. It’s been available officially since the release of the 180.22 nVidia Linux drivers in January of 2009, but has been available in beta drives since last November starting with 180.06. Drivers which support VDPAU will install special header files in /usr/include/vdpau/ which are required to compile media players with VDPAU support.

The installation process is fairly straightforward. The patches for mplayer can be found on the nVidia FTP site. The tarball contains not only the patch sets, but also a build script named checkout-patch-build.sh. As the name suggestes, the script checks out all the appropriate revisions of mplayer and its dependencies (libavcodec, libdvdread, etc) from a subversion repository, applies nVidia specific patches and compiles mplayer. The README.txt file in the package also suggests several free sample videos that can be used to test the new mplayer build.

To prevent conflicts with the copy of mplayer that may come with the Linux distribution and that is typically controlled by a package manager, I suggest manually installing the VDPAU version of mplayer by itself with its own name. The following are the commands I ran for both compiling and installation. Be sure to replace the tarball with the latest version available on the ftp site.

wget ftp://download.nvidia.com/XFree86/vdpau/mplayer-vdpau-3402051.tar.bz2
tar xvfj mplayer-vdpau-3402051.tar.bz2
cd mplayer-vdpau-3402051
./checkout-patch-build.sh
su 
cp ./mplayer /usr/local/bin/mplayer-vdpau

As stated earlier, the README.txt providers several sample files available to download to test out the decoder. I’d also suggest the following options when playing H.264 files:

mplayer-vdpau -vo vdpau -vc ffh264vdpau -framedrop -nocorrect-pts <filename>

Both the -framedrop and -nocorrect-pts will help provider for smoother playback and may be helpful on older CPUs. On machines that can handle the playback, they shouldn’t degrade the video quality.

You may get an error such as the following along with no video window being displayed:

Cannot find codec matching selected -vo and video format 0x31637661

This is most likely the result of having a ~/.mplayer/codecs.conf. Current versions of mplayer do not require a codecs.conf as one is compiled into the player itself. A stray configuration file will override the built-in defaults and remove the nVidia provided codecs. You may see this error if you had followed my previous tutorial on implementing software H.264 decoding.

Fixing this issue involves either deleting the file, renaming the file or combining the codecs.conf in your home directory with the patched one located in nVidia’s mplayer-vdpau build directory.

Another issue can arise if you have a multi-seat system; that is a system with multiple X servers each with their own monitor, keyboard and mouse configuration. In such an environment, it is essential that the X server you want to use the VDPAU card with comes up first. The video card does not need to be in the first PCI-E slot. Mine resides in the second:

 # lspci | grep nVidia
01:00.0 VGA compatible controller: nVidia Corporation NV42 [GeForce 6800 XT] (rev a2)
02:00.0 VGA compatible controller: nVidia Corporation GeForce 8500 GT (rev a1)

However in the greeter configuration file, the server that holds the GeForce 8500 card I want to use for decoding must be listed first. In my case, I use gdm on Gentoo Linux which uses /etc/X11/gdm/custom.conf to define server startup:

[servers]
0=Plasma
1=Console

[server-Plasma]
name=Plasma server
command=/usr/X11R6/bin/X -layout Plasma -novtswitch -sharevts -isolateDevice PCI:2:0:0
flexiable=true

[server-Console]
name=Console server
command=/usr/X11R6/bin/X  -layout Console -novtswitch -sharevts -isolateDevice PCI:1:0:0
flexiable=true

When I got the VDPAU mplayer up and running, I was very impressed by the results. Some video still jumped and sputtered in placed, but nothing too distracting from the video considering the high bitrates and quality. There was also occasional screen tearing where one frame partially overlapped a previous one. I’ve read turning off the X composite extension may help alleviate this problem although I haven’t tried this yet.

I’m glad nVidia is realizing the importance of capitalizing on the Linux market. ATI is trying to catch up with their own RadeonHD drivers. Linux support will be essential as it’s an ideal platform for embedded systems such as nVidia’s Ion. It could open up an entire realm of possibilities for small, affordable, set-top box and media PC solutions.

Unfortunately, the installation documents for coreavc-for-linux were months old, out-of-date and had few corrections for new bugs. The following are some of the common errors I found as well as the solutions I’ve found to get the trail version of the CoreAVC codec working on my Gentoo Linux system.

Installation

To start the installation, I created a new working directory within my home directory and checked out both the latest versions of mplayer and coreavc-for-linux (revision 82 for coreavc-for-linux and revision 28381 for mplayer). I then ran the configuration script for mplayer, applied the direct show patch from the coravc-for-linux project and proceeded to build mplayer:

mkdir coreavc
cd coreavc
svn checkout http://coreavc-for-linux.googlecode.com/svn/trunk/ coreavc-for-linux
svn checkout svn://svn.mplayerhq.hu/mplayer/trunk mplayer
cd mplayer
./configure
patch -p0 < ../coreavc-for-linux/mplayer/dshowserver.patch
make

At this point the new version of mplayer is ready. Rather than use the make install command, I wanted to keep this version of mplayer totally separate. So I copied the two executables I wanted directly to /usr/local/bin with new names like so:

su
cp mplayer /usr/local/bin/mplayer-coreavc
cp mencoder /usr/local/bin/mencoder-coreavc

At this point we have mplayer installed, but not the codec itself or the directshow server. We’ll start with the codec first. You can signup for a free trial on the official CoreAVC website. I would suggest the trial version first to make sure the codec works on your system. The download contains an executable installer, so you can either install it onto a windows system or install it using wine. After the install, copy the installed codec to the /usr/lib/win32 directory. The following example assumes you used wine. Your directory names may be different.

cd ~
su
cp ~/.wine/drive_c/Program\ Files/CoreCodec/CoreAVC\ 14\ Day\ Trial\ Edition/CoreAVCDecoder.ax /usr/lib/win32/

Before we go further we’ll install the directshow server. The wiki has instructions for compiling the server or using a precompiled binary. If you’re like me, you’re running a 64-bit system and can therefor not compile the binary from source. Doing so will give you a pretty massive compile error with several warning: ‘stdcall‘ attribute ignored messages from interfaces.h.

If you have a 32-bit linux system available, either real or a VM, or if you have somehow mastered the magic of cross compiling (I gave up on this years ago when I tried building a cross compiler for PPC), you can probably make this binary yourself. If you used the latest precompiled binary in the downloads section (r63 for Gentoo at the time of this writing) along with the 1.8.5 release of CoreAVC, you will get the following fault when trying to run it:

/usr/local/bin/dshowserver -c CoreAVCDecoder.ax -s 1280x720 -g 09571a4b-f1fe-4c60-9760de6d310c7c31 -b 12 -f 0x34363248 -o 0x30323449
No id specified, assuming test mode
Opening device
len: 992
ProductVersion: 1.8.5
Called unk_LoadImageA
Segmentation fault

This is due to a know issue with the 1.8.5 release of CoreAVC adding a dialog box in the initialization which is documented in Issue #62 and in a posting on Acmelabs’ Blog. In a later post, Acmelab gives his readers a precompiled version of release 77 which I’ve made a copy of for download: dshowserver-ia32-r77-acme.tar.bz2

Untar this file and copy the dshowserver/dshowserver and loader/registercodec binaries to your /usr/local/bin.

Now that you have the directshow server installed, you’ll need to add the trial serial number into mplayer’s virtual windows registry. The following command is known to work with version 1.7.0 and 1.8.5 of CoreAVC. Additional registry options can be found on the wiki

export REGISTRY=$HOME/.mplayer/registry32
registercodec -r $REGISTRY -k 
"HKLM\\Software\\CoreCodec\\CoreAVC Trial\\Serial" -v "55555-55555-CORE-55555-55555"

If you’re using the full version, you’d replace the word Trial with Pro. At this point you should be able to run the following command to test the codec:

/usr/local/bin/dshowserver -c CoreAVCDecoder.ax -s 1280x720 -g 09571a4b-f1fe-4c60-9760de6d310c7c31 -b 12 -f 0x34363248 -o 0x30323449

You should get the following response:

No id specified, assuming test mode
Opening device
Called unk_IsDebuggerPresent
len: 992
ProductVersion: 1.8.5
Decoder supports the following YUV formats: YUY2 UYVY YV12 I420 
Decoder is capable of YUV output (flags 0x2b)
Setting fmt
Starting
Initialization is complete

You may get a serial number issue here. If you do, go back and check your registry key. Now the only thing left is the ~/.mplayer/codec.conf

Using the 28381 build of mplayer, I constantly got the following error using the default instructions for the codec.conf:

==========================================================================
Forced video codec: coreserve
Cannot find codec matching selected -vo and video format 0x3267706D.
Read DOCS/HTML/en/codecs.html!
==========================================================================

Also, using the default instructions on a current built of mplayer can also lead to seeing this error in the mplayer output:

This codecs.conf is too old and incompatible with this MPlayer release!

This is because newer version of the codecs.conf must have a release line at the top of the file to prevent conflicts with other codec configurations as stated in issue #61

After playing with my configuration file a lot and with no results, I eventually found a configuration file in comment 28 on issue 14 that worked. I’ve made the configuration file available here as well: codecs.conf

Playback

When running mplayer-coreavc, you made need the following options to force mplayer to use coreavc and also to have smooth playback. Obviously, replace the example file I have with an HD movie you have on your file system.

/usr/local/bin/mplayer-coreavc -vc coreserver -framedrop -nocorrect-pts  /media/files/movies_HD/Wall-E_720p_DTS_BluRay.mkv

If you’re like me, you don’t use a frontend to mplayer and prefer to have the terminal output. If so, you can create these little scripts that will pop-up mplayer in a terminal and then use them from the “open with” menu in your favorite file manager:

/usr/local/bin/coreavc-ac3

#!/bin/sh
gnome-terminal -e "mplayer-coreavc -vc coreserver -framedrop -nocorrect-pts -ac hwac3 \"$1\""

/usr/local/bin/coreavc-dts

#!/bin/sh
gnome-terminal -e "mplayer-coreavc -vc coreserver -framedrop -nocorrect-pts -ac hwdts \"$1\""

The -ac options for AC3 or DTS allow for media that has embedded multi-channel audio in either Dolby Digital (AC3) or Sony (DTS) format to go through the digital output of your sound card and directly to a 5.1 surround sound receiver (if you have both a digital out on your sound card and such a receiver).

Conclusions

I made this guide because many of the guides I’ve seen were out of date and the steps I had to undertake spanned multiple websites, blogs and wikis. Keep in mind that if you’re reading this six months from now, this guide is probably out of date itself and there are probably never versions of CoreAVC, coreavc-for-linux and mplayer, all with new an fun interesting bugs that will take you an hour to figure out. If you find any other helpful information, please post a comment.

As far as the actual performance of the CoreAVC codec, it is considerably faster than ffmpeg. There is somewhat of a lag or frame delay effect I see with movies which I don’t see in ffmpeg. I’m guessing this is coming from the directshow server. However with 1080p content, the video decodes much faster and is very smooth without constant bumps and clicks, even when going through the considerable amounts of translation required to run a windows 32-bit native codec in a 64-bit Linux environment.

There are still a couple of artifacts I’ve found and distorted/garbage video during fast motion scenes. This may be due to the codec specific registry settings which I’ll need to play with, or it may be I’m exceeding the limitations of my processor. Your own mileage may vary.