FFmpeg vs libav: A distribution maintainer point of view almost two years after the split.

January 18, 2013

It’s been a while since I wanted to write about this and since there recently has been a sort of hijack without any kind of discussion to let libav be the default implementation for Gentoo, this motivated me.

Exactly two years ago, a group consisting of the majority of FFmpeg developers took over its maintainership. While I didn’t like the methods, I’m not an insider so my opinion stops here, especially since if you pay attention to who was involved: Luca was part of it. Luca has been a Gentoo developer since probably most of us even used Gentoo and I must admit I’ve never seen him heating any discussion, rather the contrary, and it’s always been a pleasure to work with him. What happened next, after a lot of turmoil, is that the developers split in two groups: libav formed by the “secessionists” and FFmpeg.

Good, so what do we chose now? One of the first things that was done on the libav side was to “cleanup” the API with the 0.7 release, meaning we had to fix almost all its consumers: Bad idea if you want wide adoption of a library that has an history of frequently changing its API and breaking all its consumers. Meanwhile, FFmpeg maintained two branches: the 0.7 branch compatible with the old API and the 0.8 one with the new API. The two branches were supposed to be identical except for the API change. On my side the choice was easy: Thanks but no thanks sir, I’ll stay with FFmpeg.
FFmpeg, while having its own development and improvements, has been doing daily merges of all libav changes, often with an extra pass of review and checks, so I can even benefit from all the improvements from libav while using FFmpeg.

So why should we use libav? I don’t know. Some projects use libav within their release process, so they are likely to be much more tested with libav than FFmpeg. However, until I see real bugs, I consider this as pure supposition and I have yet to see real facts. On the other hand, I can see lots of false claims, usually without any kind of reference: Tomáš claims that there’s no failure that is libav specific, well, some bugs tend to say the contrary and have been open for some time (I’ll get back to XBMC later). Another false claim is that FFmpeg-1.1 will have the same failures as libav-9: Since Diego made a tinderbox run for libav-9, I made the tests for FFmpeg 1.1 and made the failures block our old FFmpeg 0.11 tracker. If you click the links, you will see that the number of blockers is much smaller (something like 2/3) for the FFmpeg tracker. Another false claim I have seen is that there will be libav-only packages: I have yet to see one; the example I had as an answer is gst-plugins-libav, which unfortunately is in the same shape for both implementations.

In theory FFmpeg-1.1 and libav-9 should be on par, but in practice, after almost two years of disjoint development, small differences have started to accumulate. One of them is the audio resampling library: While libswresample has been in FFmpeg since the 0.9 series, libav developers did not want it and made another one, with a very similar API, called libavresample that appeared in libav-9. This smells badly as a NIH syndrome, but to be fair, it’s not the first time such things happen: libav and FFmpeg developers tend to write their own codecs instead of wrapping external libraries and usually achieve better results. The audio resampling library is why XBMC being broken with libav is, at least partly, my fault: While cleaning up its API usage of FFmpeg/libav, I made it use the public API for audio resampling, initially with libswresample but made sure it worked with libavresample from libav. At that time, this would mean it required libav git master since libav-9 was not even close to be released, so there was no point in trying to make it compatible with such a moving target. libswresample from FFmpeg was present since the 0.9 series, released more than one year ago. Meanwhile, XBMC-12 has entered its release process, meaning it will probably not work with libav easily. Too late, too bad.

Another important issue I’ve raised is the security holes: Nowadays, we are much more exposed to them. Instead of having to send a specially crafted video to my victim and make him open it with the right program, I only have to embed it in an HTML5 webpage and wait. This is why I am a bit concerned that security issues fixed 7 months ago in FFmpeg have only been fixed with the recently released libav-0.8.5. I’ve been told that these issues are just crashes are have been fixed in a better way in libav: This is probably true but I still consider the delay huge for such an important component of modern systems, and, thanks to FFmpeg merging from libav, the better fix will also land in FFmpeg. I have also been told that this will improve on the libav side, but again, I want to see facts rather than claims.

As a conclusion: Why is the default implementation changed? Some people seem to like it better and use false claims to force their preference. Is it a good idea for our users? Today, I don’t think so (remember: FFmpeg merges from libav and adds its own improvements), maybe later when we’ll have some clear evidence that libav is better (the improvements might be buggy or the merges might lead to subtle bugs). Will I fight to get the default back to FFmpeg ? No. I use it, will continue to use and maintain it, and will support people that want the default back to FFmpeg but that’s about it.

Alternatives for core TeX binaries

October 6, 2008

If you have been following packages additions you may have noticed the addition of a couple of packages some weeks ago:

  • dev-tex/pdftex
  • dev-tex/mplib
  • app-admin/eselect-mpost
  • app-admin/eselect-pdftex

While our TeX distributions already ship them (pdftex and mpost), the first two are standalone packages and the last two are eselect modules to easily switch and use them.
With the bump to TeX Live 2008 I’ve made it possible to switch between different versions instead of being bound to what has been shipped with the release. This means that they will get bumped more frequently than they used to.

For the pdftex case, it is currently the same version that comes with TeX Live 2008 with the difference that the standalone version is patched to use poppler instead of its bundled xpdf library. I’ve prefered to leave the xpdf library as default for TeX Live because that is what upstream is using but some people asked for it to use poppler instead and a lot of other distributions (debian, fedora and their derivatives) are using it as the only way. That will probably make it easier to get improvements and security fixes on that side. If you wish to try it, install pdftex and eselect-pdftex and use “eselect pdftex” to set the standalone package as your default pdftex!

As for mplib, it is a reworked and rewritten version of the good old metapost. It can be used as a library (as luatex does) but you can also switch your default mpost interpreter to be it via the eselect-mpost package. Upstream says it is a work in progress but so far I have not seen any problem with it.

By the way, I have unmasked TeX Live 2008 today so that means ~arch users will upgrade to this release and can try these alternatives out of the box.

TeX Live 2008 hits the tree

September 21, 2008

More than one week has passed since I’ve added TeX Live 2008 to the portage tree; I wanted to write a blog entry about it but have had absolutely no free time since.

So, what’s new with this release?

Of course there is the usual packages updates from CTAN, TeX 3.1415926 and METAFONT 2.718281 (yes, there is one more decimal!), pdfTeX 1.40.9 and more…

The most important addition would be LuaTeX, an extended version of pdfTeX using Lua as an embedded scripting language. Even if TeX is Turing-complete this makes much easier and nicer performing advanced tasks in your documents.
With LuaTeX comes ConTeXt Mark IV, the LuaTeX-aware part of ConTeXt. Unfortunately it comes unconfigured in TeX Live 2008 which means you’ll have to manually configure it. It does not poke well with the SELFAUTO* variables we have everywhere in our default configuration files so you’ll have to specify the paths for the texmf trees there and not rely on the texmf.cnf file.

TeX Live 2008 is still masked, awaiting for some keywording and some testing. I think I will unmask it very soon, moving the relevant masks to the various profiles that have problems with one or another package in this release.

While with TeX Live 2007, in my opinion, the urgency was more to have an up-to-date TeX distribution instead of our unmaintained and every day older teTeX, for TeX Live 2008 I’ve focused on cleaning up the tree: we have a bunch of packages in the dev-tex category, most of which were included in TeX Live 2007, so I’ve taken the liberty of removing most of those we had there from the “non core” packages of TeX Live, updating the dev-tex packages and using them from the app-text/texlive meta-ebuild (which should reflect what you can find in the official TeX Live distribution). Some other dev-tex packages had been masked or uninstallable and unmaintained for a while, those have been last rited and should be gone soon. If you find (la?)TeX packages that fall in one or the other category, it’s time to shout and file a bug about it! because it is not unlikely that I have forgotten some.

From the packaging point of view, with TeX Live 2007 we had .zip files for the CTAN packages included in TeX Live and a .tar.bz2 file for the source code of the binaries; with TeX Live 2008 everything is now in .tar.lzma files, what makes them smaller and faster to unpack. But, more importantly, each CTAN package is now split in “run file”, “doc files” and “source files” tarballs: that is, respectively, the package itself, the doc and source useflags in the dev-texlive ebuilds. As opposed to what was done in the 2007 version, the source of the packages isn’t installed anymore by default which saves some diskspace, the doc and source files tarballs are downloaded only if you want to install them what saves some bandwidth.

We now also have ways of updating core TeX packages (pdfTeX and mpost/mplib) independently of the TeX distribution via eselect modules, but that will be for another blog entry.

Introducing texmfind

March 1, 2008

During the past months TeX Live has been in ~arch and seems to be working quite well for everybody since I do not receive that much bug reports, and the ones I’ve had are now fixed. However I’ve seen lots of people asking me: “Hey, where do I find the latex package foo ? I need it to compile my latex document.”.
The reason for this is that TeX Live 2007 ebuilds are split into several smaller ebuilds rather than having one single ebuild, like that was the case for teTeX which in the end was very hard to update and forced everybody to have hundreds of TeX packages that they would never use.

For that “problem”, Alexandre ( ‘Untux’ on the forums) has written a very nice script: texmfind. He has collected the list of files from most of TeX packages we have in portage, based on a TeX Live installation, and written a grep wrapper that digs into the data file to give you what package you should install in order to have what file.

For example:

$ texmfind moderncv.cls
dev-texlive/texlive-latexextra [1 file]
moderncv.cls

Found 1 texmf file in 1 ebuild.

Of course, it is now in portage as dev-tex/texmfind and it comes with a nice man page for a more advanced usage than this toy example.

Now it seems most people have no reason not to switch to TeX Live, and I really recommend everybody to migrate. Moreover, TeX Live should be ready to go stable now and I would really appreciate some feedback from stable users running TeX Live as a TeX distribution instead of teTeX. This can also include filling a stable request on bugs.gentoo.org as this will probably ensure that I have not messed up anything with the dependencies and that I’m not mistaken when saying it should be ready to go stable.

TeXLive modular ebuilds ready(?) for the main portage tree

September 30, 2007

For those who do not read -dev ml and might be interested in testing / helping:

As some of you may or may not know, I’ve been working on modular texlive ebuilds[1], based on work found on b.g.o and pclouds work [2]. I wanted to send a mail earlier but time was lacking to fix the few remaining bugs.

I’ve had several success reports, and fixed the remaining (known) bugs there. I was thinking that it might be time to integrate this to the official tree, as a first shoot under package.mask.

The layout I’ve been using is a modular one:
- texlive-core package that builds the required binaries. That’s the only one that should be system dependant.
- several texmf modules based on upstream “collection”s (that’s how they call them) that agregates packages from CTAN in some more or less independant categories.

I’ve tried to remove as much as possible programs from the -core package, as long as they had their independant ebuilds equivalent and added the independant packages as dependencies of the texlive meta ebuild.

As you might guess it, having a modular layout can give dependencies problems. I was thinking about adding some (new style) virtuals to handle them :
- virtual/tex-base : programs that need only standard tex binaries or libraires (like kpathsea) but do not need it to compile latex files for example. There are a very very few of such packages and are ok with the next virtual, so I dunno if that one is really necessary, apart for reducing deps to the minimal set.
- virtual/latex-base : packages that need a (basic) latex, for example to compile their documentation. This virtual will help preventing from having circular dependencies between ebuilds (esp. the meta ebuild and its dependencies)
- virtual/latex-full : a full latex distribution installation, what other tex distributions like tetex provide. This one can use the current old style virtual (virtual/tetex) instead of being a new one, but the name is better imho.
So in the end, only latex-base is strictly required to merge this. tex-base and latex-full have their improvements but can benefit from discussion here.

Everything in [1] could still benefit of any kind of reviewing, especially the eclasses. I’ll also need to write a more decent guide on how to use/switch to those ebuilds, so that it can be put on the website.

The only (known) bug still left so far is that metapost (mpost) isn’t useable on hardened kernel, it gets killed. It is not a regression from tetex, but apparently nobody ever noticed that. Now that ebuilds generate the format files themselves in src_compile (this way we can improve QA imho), instead of having texmf-update doing it in pkg_postinst, some ebuilds will fail to install instead of silently not creating the format file. (though texmf-update will still recreate the formats so that they get updated with the texmf tree)
Something that annoys me is the license : there is [3], [4] and [5], so GPL-2 might probably be fine, but I’m definitely not a lawyer…

In the (very hypothetic) case where nobody would have anything to add there, I’ll start merging this to the main tree, but definitely not until the next week as I’ll be away from wednesday to sunday.

And of course, many thanks to all the people who helped there: the early testers when documentation was lacking but not bugs, the people who suggested improvements (be it to ebuilds, the packaging layout or documentation), bug fixes, reported bugs, or just mailed me about a successful installation.

Now a question to arch teams : Should I keyword this for systems I’ve tested it or just add without keywords and let you do another layer of checks ? I’ve been using it on ~x86 (and hardenend but mpost had problems), ~amd64 and ~ppc64 (this one has some missing deps, but don’t worry I’ll poke you as soon as I’ll have done extra checks ;) ).

As a side note, I’ll have to send 1.3k+ files to distfiles-local as upstream does not provide versionned names of the source files, for a total of ~700-800M. Since this is huge, I hope infra has no particular objection to it.

[1] http://overlays.gentoo.org/dev/aballier/
[2] http://dev.gentoo.org/~pclouds/texlive/overlay/
[3] http://www.tug.org/svn/texlive/trunk/Master/LICENSE.TL?view=markup
[4] http://www.tug.org/svn/texlive/trunk/Master/LICENSE.CTAN?view=markup
[5] http://www.tug.org/texlive/copying.html

Splitting x264-svn ebuilds

December 7, 2006

There are now two x264-svn ebuilds in ~arch :

media-libs/x264-svn : this one installs the library, headers and pkgconfig files needed by the other packages (for ex. ffmpeg)

media-video/x264-svn-encoder : x264 encoder that was previously installed with x264-svn and a gtk use flag with which you can install a gtk frontend to this encoder.

Why that ? Eh, let’s have a look at it :
- x264 encoder can use gpac for mp4 output
- gpac can use ffmpeg
- ffmpeg can use x264 libs

If you want all of these features, then you hit a circular dependency problem.

Moreover, previously if you wanted to install x264 gtk frontend then libx264.so would have been linked to x11… why ? I don’t know, it’s not needed. Now this is no longer a problem since they are different packages.

Test

November 20, 2006

My first blog post using drivel

My first blog post

November 20, 2006

Hi,

this is my first blog post, trying to use categories

Test

November 20, 2006

This is a test

Hello world!

November 20, 2006

Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!


Follow

Get every new post delivered to your Inbox.