Discussion:
Reintroducing FFmpeg to Debian
(too old to reply)
Andreas Cadhalpun
2014-07-27 23:20:59 UTC
Permalink
Hi all,

some of you may have noticed a weird ffmpeg package in the NEW queue[1].
Let me explain:

In 2011 Libav[2] was forked from FFmpeg[3]. It was a time of great
uncertainty, the fork happened with much drama that didn't help making a
technical cut, and at that peculiar time Debian switched to Libav.

Since then the two projects evolved differently, and we have now a
clearer view.

Some short answers to questions you might have now:

* Why is FFmpeg needed in Debian?
- It has features our users are asking for (mostly support for more
codecs, formats and filters)[4-6].
- Some applications break when built against Libav on Debian,
because they are developed using FFmpeg[7-10].
- There are issues[11-12] in Libav's command line tools, that can't
be reproduced with FFmpeg's tools.
- It has a big and active user and developer community. Those of
them who want to use Debian currently need to install FFmpeg from
third parties or compile their own version from source.

* Do you intend to replace Libav by FFmpeg in Debian?
No, there is no need to replace anything as long as it is maintained.
Currently the main goal is to give multimedia maintainers a choice
between the two sets of libraries to link against, and our users the
choice to use the 'ffmpeg' utility. That is possible, because the
packages are co-installable. (Only the *-dev packages conflict.)

* But I thought they were forks, why don't you just create conflicting
packages?
- Because it can't really work! If we do that, packages built with
FFmpeg won't be installable next to packages built with Libav
unless we have full binary compatibility.
- Binary compatibility can only be achieved with some tradeoffs:
a) Not all soversions of the libraries match, so we would have
to patch that away.
b) FFmpeg would have to be compiled with the configure option
--enable-incompatible-libav-abi, resulting in less tested
code paths.
c) FFmpeg and Libav would need to be updated at the same time.
d) The biggest problem is that this would allow applications only
to use the minimal set of the ABI supported by both.

* I do not believe you, explain that voodoo to me: How is it that it
won't break all of Debian and make kittens cry?
(aka: How is FFmpeg packaged for Debian?)
- We built the packages in a way that avoids filename conflicts.
The sonames of the FFmpeg libraries are suffixed with '-ffmpeg',
e.g. libavcodec-ffmpeg.so.55 instead of libavcodec.so.55.
This also means that only if a package uses pkg-config to detect
the correct linker flags, you can simply install e.g. the
libavcodec-ffmpeg-dev package and it will work transparently.
(About half the packages build with no further change, see
stats below.)
Reinhard Tartler
2014-07-28 00:05:22 UTC
Permalink
On Sun, Jul 27, 2014 at 7:20 PM, Andreas Cadhalpun
* Does it make sense for me to switch my package?
The rule of thumb is, if your upstream uses FFmpeg for development
you probably want to switch to using it, too.
In [1], Moritz from the security team clearly stated that he is more
than uncomfortable with having more than one copy of libavcodec in
debian/testing. In consequence this means that any package that builds
against the ffmpeg packages currently in NEW won't make it into
testing either. I am therefore surprised about the given answer to the
question above.

I think it would be best if ftp-master left the ffmpeg package in NEW
until an answer to this problem has been found.

[1] https://lists.debian.org/debian-devel/2014/02/msg00668.html
The FFmpeg version currently in NEW has been there for more than
2 months and is thus outdated. If you want to test the current
packages, you can build them from the repository on Alioth [17]
(e.g. using git-buildpackage).
Furthermore, we'd like to move the FFmpeg packaging under the umbrella
of the pkg-multimedia team, because this would facilitate future FFmpeg
transitions.
I am curious why this is your first email about this matter to
pkg-multimedia, and why do you write this email only now?

Moreover, I am curious why I haven't seen you working on libavcodec
bugs in Debian before, and why do you believe you can do a better job
with the ffmpeg package currently on NEW?
--
regards,
Reinhard
Marco d'Itri
2014-07-28 00:09:09 UTC
Permalink
Post by Reinhard Tartler
Moreover, I am curious why I haven't seen you working on libavcodec
bugs in Debian before, and why do you believe you can do a better job
with the ffmpeg package currently on NEW?
Why should he work on libavcodec when he (along with many other people)
wants ffmpeg instead?
--
ciao,
Marco
Norbert Preining
2014-07-28 00:40:47 UTC
Permalink
Hi,
Post by Reinhard Tartler
In [1], Moritz from the security team clearly stated that he is more
than uncomfortable with having more than one copy of libavcodec in
debian/testing. In consequence this means that any package that builds
against the ffmpeg packages currently in NEW won't make it into
testing either. I am therefore surprised about the given answer to the
"More than uncomfortable" does not mean "will not be included"
Post by Reinhard Tartler
I think it would be best if ftp-master left the ffmpeg package in NEW
until an answer to this problem has been found.
Why, only because libav gets a more powerful and efficient
competition?
Post by Reinhard Tartler
I am curious why this is your first email about this matter to
pkg-multimedia, and why do you write this email only now?
Moreover, I am curious why I haven't seen you working on libavcodec
bugs in Debian before, and why do you believe you can do a better job
with the ffmpeg package currently on NEW?
As Marco said, why should he fix bugs in av which have been fixed since
ages in ffmeg in many (most?) cases?

I am all for having a good ffmpeg set of cmd line progs and libs in
Debian. It is too long that this sad and useless split was made.

Norbert

------------------------------------------------------------------------
PREINING, Norbert http://www.preining.info
JAIST, Japan TeX Live & Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------
Henrique de Moraes Holschuh
2014-07-28 11:52:09 UTC
Permalink
Post by Norbert Preining
Post by Reinhard Tartler
In [1], Moritz from the security team clearly stated that he is more
than uncomfortable with having more than one copy of libavcodec in
debian/testing. In consequence this means that any package that builds
against the ffmpeg packages currently in NEW won't make it into
testing either. I am therefore surprised about the given answer to the
"More than uncomfortable" does not mean "will not be included"
Yes, it does.

Someone will have to convince the security team somehow, likely by offering
to do the work themselves _and_ convincing them that these new members will
be around for long enough.

However:

The change in Debian-specific symbol versioning and sonames being done to
ffmpeg so that it is co-installable with libav *is* a problem.

It has to be done in coordination with the Canonical guys, so that both
Debian and Ubuntu do the same thing re. ffmpeg sonames and symbol
versioning. Otherwise, the ffmpeg packages will be of very limited use
(useless to run third-party binary-only games ;-p).

I understand perfectly that the soname and symbol versioning clash with
libav is not ffmpeg's fault, but that's water (well, sewage) under the
bridge. We have to deal with it. Here's an alternative proposal that
should be less painful [to our users] in the long run:

You need one of the two upstreams to do a *large* major soname bump (at
least one order of magnitude higher than what they're currently using), so
that both projects can keep evolving with little chance of soname clashes.

Symbol versioning will take care of the rest, since both libs carry over
their major soname into the symbol version. As it was done upstream,
cross-distro/third-party compatibility problems are not increased.

Debian will have to package this new "bumped" upstream release, and get rid
of anything built against the old one. It will be easier for Debian if it
is ffmpeg upstream that does the soname bump, otherwise we're talking about
a huge number of binNMUs.

But this is all academic if the security team is not prepared to deal with
both libav and ffmpeg at the same time. That effectively forces a choice of
either libav, or ffmpeg, and not both.
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
Niv Sardi
2014-07-28 13:40:59 UTC
Permalink
Post by Henrique de Moraes Holschuh
The change in Debian-specific symbol versioning and sonames being done to
ffmpeg so that it is co-installable with libav *is* a problem.
It has to be done in coordination with the Canonical guys, so that both
Debian and Ubuntu do the same thing re. ffmpeg sonames and symbol
versioning. Otherwise, the ffmpeg packages will be of very limited use
(useless to run third-party binary-only games ;-p).
Not really, ffmpeg is packaged as a secondary multimedia library, the
default one still being libav. If these packages get traction, we can think
of moving ffmpeg to be the default (and ship if we wish libav with the
soname change).

The package will be of use for the ffmpeg command line tools, and for the
maintainers that decide to make the switch.

For now, your binary third party games will have to link against libav.
Post by Henrique de Moraes Holschuh
I understand perfectly that the soname and symbol versioning clash with
libav is not ffmpeg's fault, but that's water (well, sewage) under the
bridge. We have to deal with it. Here's an alternative proposal that
You need one of the two upstreams to do a *large* major soname bump (at
least one order of magnitude higher than what they're currently using), so
that both projects can keep evolving with little chance of soname clashes.
Symbol versioning will take care of the rest, since both libs carry over
their major soname into the symbol version. As it was done upstream,
cross-distro/third-party compatibility problems are not increased.
Debian will have to package this new "bumped" upstream release, and get rid
of anything built against the old one. It will be easier for Debian if it
is ffmpeg upstream that does the soname bump, otherwise we're talking about
a huge number of binNMUs.
That's been discussed and ruled out in favour of not overstepping libav's
namespace.
Post by Henrique de Moraes Holschuh
But this is all academic if the security team is not prepared to deal with
both libav and ffmpeg at the same time. That effectively forces a choice of
either libav, or ffmpeg, and not both.
That is premature, let's deal with this issue when we have more data.
Andreas Cadhalpun
2014-07-28 14:05:46 UTC
Permalink
Post by Henrique de Moraes Holschuh
Post by Norbert Preining
Post by Reinhard Tartler
In [1], Moritz from the security team clearly stated that he is more
than uncomfortable with having more than one copy of libavcodec in
debian/testing. In consequence this means that any package that builds
against the ffmpeg packages currently in NEW won't make it into
testing either. I am therefore surprised about the given answer to the
"More than uncomfortable" does not mean "will not be included"
Yes, it does.
Someone will have to convince the security team somehow, likely by offering
to do the work themselves _and_ convincing them that these new members will
be around for long enough.
Michael Niedermayer from FFmpeg upstream volunteered "to help with any
future security issues in FFmpeg packages in debian" [1].
Post by Henrique de Moraes Holschuh
The change in Debian-specific symbol versioning and sonames being done to
ffmpeg so that it is co-installable with libav *is* a problem.
It has to be done in coordination with the Canonical guys, so that both
Debian and Ubuntu do the same thing re. ffmpeg sonames and symbol
versioning. Otherwise, the ffmpeg packages will be of very limited use
(useless to run third-party binary-only games ;-p).
I don't think coordination with Ubuntu will be a problem.
In comment #7 in the corresponding bug at launchpad [2] Dimitri John
Ledkov wrote that Ubuntu won't introduce FFmpeg on it's on, but instead:
"If you wish to see a supported ffmpeg stack in both Debian and Ubuntu,
please become a developer and start maintaining it in Debian."

Best regards,
Andreas


1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=729203#528
2: https://bugs.launchpad.net/ubuntu/+source/libav/+bug/1263278
Michael Niedermayer
2014-07-28 17:35:37 UTC
Permalink
Post by Andreas Cadhalpun
Post by Henrique de Moraes Holschuh
Post by Norbert Preining
Post by Reinhard Tartler
In [1], Moritz from the security team clearly stated that he is more
than uncomfortable with having more than one copy of libavcodec in
debian/testing. In consequence this means that any package that builds
against the ffmpeg packages currently in NEW won't make it into
testing either. I am therefore surprised about the given answer to the
"More than uncomfortable" does not mean "will not be included"
Yes, it does.
Someone will have to convince the security team somehow, likely by offering
to do the work themselves _and_ convincing them that these new members will
be around for long enough.
Michael Niedermayer from FFmpeg upstream volunteered "to help with
any future security issues in FFmpeg packages in debian" [1].
Yes, i do!

[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Opposition brings concord. Out of discord comes the fairest harmony.
-- Heraclitus
Dimitri John Ledkov
2014-07-29 01:12:30 UTC
Permalink
On 28 July 2014 15:05, Andreas Cadhalpun
Post by Andreas Cadhalpun
Post by Henrique de Moraes Holschuh
Post by Norbert Preining
Post by Reinhard Tartler
In [1], Moritz from the security team clearly stated that he is more
than uncomfortable with having more than one copy of libavcodec in
debian/testing. In consequence this means that any package that builds
against the ffmpeg packages currently in NEW won't make it into
testing either. I am therefore surprised about the given answer to the
"More than uncomfortable" does not mean "will not be included"
Yes, it does.
Someone will have to convince the security team somehow, likely by offering
to do the work themselves _and_ convincing them that these new members will
be around for long enough.
Michael Niedermayer from FFmpeg upstream volunteered "to help with any
future security issues in FFmpeg packages in debian" [1].
Post by Henrique de Moraes Holschuh
The change in Debian-specific symbol versioning and sonames being done to
ffmpeg so that it is co-installable with libav *is* a problem.
It has to be done in coordination with the Canonical guys, so that both
Debian and Ubuntu do the same thing re. ffmpeg sonames and symbol
versioning. Otherwise, the ffmpeg packages will be of very limited use
(useless to run third-party binary-only games ;-p).
I don't think coordination with Ubuntu will be a problem.
In comment #7 in the corresponding bug at launchpad [2] Dimitri John Ledkov
"If you wish to see a supported ffmpeg stack in both Debian and Ubuntu,
please become a developer and start maintaining it in Debian."
I don't have an opinion about ffmpeg vs libav, apart from how hard the
soname transitions are, especially in ubuntu where we somehow ended up
with ex-multimedia packages around that either never were in debian,
or have been long removed from testing and/or unstable. Thankfully, we
have worked to make sure libav is in universe only and thus is not a
security maintenance burden. Nonetheless, libav10 transition is still
not complete in utopic today. I haven't checked, but now abi
compatible/incompatible the two stacks are? cause it would be a pain
if they are not drop in replacements, and it would also be a pain if
higher up packages link-in both ffmpeg & libav and some clashing
symbols are present... and people start requesting to have build
variants against both. Has a rebuild of all deps been done? How many
build failures there are? (On both debian & ubuntu, ideally) Is
hardening flags / toolchain enabled in both, or either of the two?
--
Regards,

Dimitri.
IOhannes m zmölnig (Debian/GNU)
2014-07-29 07:10:20 UTC
Permalink
Post by Dimitri John Ledkov
if they are not drop in replacements, and it would also be a
pain if
Post by Dimitri John Ledkov
higher up packages link-in both ffmpeg & libav and some
clashing symbols are present...
This is why the new ffmpeg will use different symbols. Again, read
the first message.
according to the first message, this is *not* true.
the packages will have different libraries-names / sonames, but this
does not mean that they don't have clashing symbols.
if both library foo (/usr/lib/libfoo.so.3.21) and library bar
(/usr/lib/i386-linux-gnu/libbar.so.4.1) export the symbol "knarzifax",
then how do you make sure that an application that is linked against
both libraries for different functionality always chooses the korrect
"knarzifax"?

this becomes a real world issue, as soon as plugins are involved
(which i find a common practice to access multimedia frameworks).
application "flurp" has a both "flurp-plugin-libav" and
"flurp-plugin-ffmpeg" installed.
whichever plugin is loaded first, will pull in a library that shadows
the symbol "knarzifax" for the *other* plugin.

fgamsdr
IOhannes
Andreas Cadhalpun
2014-07-29 16:10:27 UTC
Permalink
Hi Dimitri,
Post by Dimitri John Ledkov
I don't have an opinion about ffmpeg vs libav, apart from how hard the
soname transitions are, especially in ubuntu where we somehow ended up
with ex-multimedia packages around that either never were in debian,
or have been long removed from testing and/or unstable.
There are only 6 additional reverse-build-dependencies of src:libav in
utopic. Two build against lib*-ffmpeg-dev without further changes, one
needs a simple patch to use pkg-config, one needs a patch to adapt to
newer API (also needed for Libav 10), one is BD-uninstallable and one
fails for unrelated reasons, but its build-dependencies on libav*-dev
seem to be unnecessary anyway.

Per package list:

alsa-plugins-extra: OK
bombono-dvd: PATCH CodecID
dvdstyler: Unmet build dependencies: libwxsvg-dev (>= 2:1.0.9)
gstreamer-vaapi: error: unsupported GStreamer API version 1.4
kffmpegthumbnailer: OK
libdlna: PATCH pkg-config

The patches are attached to this mail.
Post by Dimitri John Ledkov
Thankfully, we
have worked to make sure libav is in universe only and thus is not a
security maintenance burden. Nonetheless, libav10 transition is still
not complete in utopic today.
Is bombono-dvd the last blocker?
Post by Dimitri John Ledkov
I haven't checked, but now abi
compatible/incompatible the two stacks are? cause it would be a pain
if they are not drop in replacements, and it would also be a pain if
higher up packages link-in both ffmpeg & libav and some clashing
symbols are present...
As Marco already wrote, FFmpeg is packaged to be ABI incompatible with
Libav, having different sonames and different symbol versions.
Post by Dimitri John Ledkov
and people start requesting to have build
variants against both.
This could theoretically be done, but I don't think anyone wants to
maintain such a thing, especially because it would need two different
source packages, as the development packages of FFmpeg and Libav have to
conflict.
Post by Dimitri John Ledkov
Has a rebuild of all deps been done? How many
build failures there are? (On both debian & ubuntu, ideally) Is
hardening flags / toolchain enabled in both, or either of the two?
I did a rebuild of all the reverse-build-dependencies in sid and the
results can be found in my original mail.
For Ubuntu, see the beginning of this mail.

I'm not sure what you want to know about hardening.
The packages are otherwise unchanged, so use hardening flags if they
already do.
If that question was about FFmpeg/Libav, then yes, FFmpeg uses
--toolchain=hardened and Libav hardening flags.

Best regards,
Andreas
Pau Garcia i Quiles
2014-07-29 19:44:13 UTC
Permalink
On Tue, Jul 29, 2014 at 6:10 PM, Andreas Cadhalpun <
Post by Dimitri John Ledkov
I don't have an opinion about ffmpeg vs libav, apart from how hard the
Post by Dimitri John Ledkov
soname transitions are, especially in ubuntu where we somehow ended up
with ex-multimedia packages around that either never were in debian,
or have been long removed from testing and/or unstable.
There are only 6 additional reverse-build-dependencies of src:libav in
utopic. Two build against lib*-ffmpeg-dev without further changes, one
needs a simple patch to use pkg-config, one needs a patch to adapt to newer
API (also needed for Libav 10), one is BD-uninstallable and one fails for
unrelated reasons, but its build-dependencies on libav*-dev seem to be
unnecessary anyway.
alsa-plugins-extra: OK
bombono-dvd: PATCH CodecID
dvdstyler: Unmet build dependencies: libwxsvg-dev (>= 2:1.0.9)
gstreamer-vaapi: error: unsupported GStreamer API version 1.4
kffmpegthumbnailer: OK
libdlna: PATCH pkg-config
In addition to this, I would like to note there is a lot of closed-source
software which uses ffmpeg instead of libav.

Not saying it doesn't exist but I don't know a single piece of
closed-source software which has moved from ffmpeg to libav.

I know, I know "non DFSG-free software, we don't care". Well, I do. E. g.
I'm having trouble with Qt right now because I'm using the commercial SDK
which indirectly uses ffmpeg to provide some codecs on Linux.
--
Pau Garcia i Quiles
http://www.elpauer.org
(Due to my workload, I may need 10 days to answer)
Andreas Cadhalpun
2014-07-28 01:39:29 UTC
Permalink
Hi Reinhard,
Post by Reinhard Tartler
On Sun, Jul 27, 2014 at 7:20 PM, Andreas Cadhalpun
* Does it make sense for me to switch my package?
The rule of thumb is, if your upstream uses FFmpeg for development
you probably want to switch to using it, too.
In [1], Moritz from the security team clearly stated that he is more
than uncomfortable with having more than one copy of libavcodec in
debian/testing.
I discussed this with Moritz in the ITP bug. Moritz ended this
discussion [a], and as I wasn't convinced by his arguments, I continued
my work. If in the end really only one copy is allowed in the next
stable release, I think it should be FFmpeg.
Post by Reinhard Tartler
In consequence this means that any package that builds
against the ffmpeg packages currently in NEW won't make it into
testing either. I am therefore surprised about the given answer to the
question above.
It remains to be seen, what the release team prefers: frustrated users
and developers or both forks in jessie.
Post by Reinhard Tartler
I think it would be best if ftp-master left the ffmpeg package in NEW
until an answer to this problem has been found.
I fail to see how this would help anyone, it only makes testing the
package more difficult. Whether or not the package is acceptable for the
next stable release is not to be decided by the ftp-masters, but rather
by the release team.
Post by Reinhard Tartler
[1] https://lists.debian.org/debian-devel/2014/02/msg00668.html
The FFmpeg version currently in NEW has been there for more than
2 months and is thus outdated. If you want to test the current
packages, you can build them from the repository on Alioth [17]
(e.g. using git-buildpackage).
Furthermore, we'd like to move the FFmpeg packaging under the umbrella
of the pkg-multimedia team, because this would facilitate future FFmpeg
transitions.
I am curious why this is your first email about this matter to
pkg-multimedia, and why do you write this email only now?
In the last discussion on debian-devel it was suggested to get the
FFmpeg packages into experimental first [b], before further discussion,
so I tried to achieve that.

As the package has been in NEW for a rather long time and the freeze is
getting closer, sending this mail now seemed appropriate.
Post by Reinhard Tartler
Moreover, I am curious why I haven't seen you working on libavcodec
bugs in Debian before,
It would be great if I could fix every bug in Debian, but unfortunately
my time is limited. Therefore, when I encounter a problem that cannot
immediately be fixed, I try to work around it. The workaround for
practically all problems I had with the Libav packages in Debian could
be solved by installing FFmpeg binaries from third parties. Therefore I
finally decided to work on a more sustainable solution, i.e. a FFmpeg
package in Debian.
Post by Reinhard Tartler
and why do you believe you can do a better job
with the ffmpeg package currently on NEW?
It is a lot more likely that I work on fixing a bug that affects me, if
there is no easy workaround.

Best regards,
Andreas


a: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=729203#568
b: https://lists.debian.org/debian-devel/2014/02/msg00714.html
Julien Cristau
2014-07-28 08:44:30 UTC
Permalink
Post by Andreas Cadhalpun
Hi Reinhard,
Post by Reinhard Tartler
On Sun, Jul 27, 2014 at 7:20 PM, Andreas Cadhalpun
* Does it make sense for me to switch my package?
The rule of thumb is, if your upstream uses FFmpeg for development
you probably want to switch to using it, too.
In [1], Moritz from the security team clearly stated that he is more
than uncomfortable with having more than one copy of libavcodec in
debian/testing.
I discussed this with Moritz in the ITP bug. Moritz ended this discussion
[a], and as I wasn't convinced by his arguments, I continued my work. If in
the end really only one copy is allowed in the next stable release, I think
it should be FFmpeg.
Post by Reinhard Tartler
In consequence this means that any package that builds
against the ffmpeg packages currently in NEW won't make it into
testing either. I am therefore surprised about the given answer to the
question above.
It remains to be seen, what the release team prefers: frustrated users and
developers or both forks in jessie.
The release team is likely to let the people involved in multimedia foo
fight it out among themselves and pick a winner. We're not going to
ship both and hand that mess over to the security team.

Cheers,
Julien
Alessio Treglia
2014-07-28 09:50:09 UTC
Permalink
Ciao,
Post by Julien Cristau
The release team is likely to let the people involved in multimedia foo
fight it out among themselves and pick a winner. We're not going to
ship both and hand that mess over to the security team.
Personally I don't feel like dropping libav in favor of ffmpeg now at
this stage. It's too late for Jessie.
Rather I'd suggest to start reconsidering such switch for Jessie+1.

Cheers.
--
Alessio Treglia | www.alessiotreglia.com
Debian Developer | ***@debian.org
Ubuntu Core Developer | ***@ubuntu.com
0416 0004 A827 6E40 BB98 90FB E8A4 8AE5 311D 765A
Marco d'Itri
2014-07-28 09:55:21 UTC
Permalink
Post by Alessio Treglia
Personally I don't feel like dropping libav in favor of ffmpeg now at
this stage. It's too late for Jessie.
Except that, for a lot of the depending packages, there would be an
immediate benefit in the number of bugs fixed.

Personally I feel that we have inflicted libav on our users for way more
time than it was sensible to do.
--
ciao,
Marco
IOhannes m zmölnig (Debian/GNU)
2014-07-28 11:12:52 UTC
Permalink
personally i would welcome if both libav and ffmpeg could co-exist
within Debian¹.
as i see it, libav and ffmpeg have diverged, and as such i would like
to have the choice which one to use.
Post by Marco d'Itri
Post by Alessio Treglia
Personally I don't feel like dropping libav in favor of ffmpeg
now at this stage.
+ 1
i don't think that dropping libav is appropriate at all.
Post by Marco d'Itri
Except that, for a lot of the depending packages, there would be an
immediate benefit in the number of bugs fixed.
at least in theory.
Post by Marco d'Itri
Personally I feel that we have inflicted libav on our users for way
more time than it was sensible to do.
i would appreciate it, if you (and anybody else) used a less flammable
&| touchy language.


fgmadr
IOhannes



¹ but then i'm not a member of the security team :-)
Alessio Treglia
2014-07-28 11:24:57 UTC
Permalink
On Mon, Jul 28, 2014 at 12:12 PM, "IOhannes m zmölnig (Debian/GNU)"
Post by IOhannes m zmölnig (Debian/GNU)
Post by Marco d'Itri
Except that, for a lot of the depending packages, there would be an
immediate benefit in the number of bugs fixed.
at least in theory.
Plus I would definitely appreciate to see some bug stats supporting
such a theory.

Cheers.

(IOhannes et Multimedia guys, please let's keep debian-devel in the
loop, I feel this is much more of general interest than a thing that
needs to be addressed internally in pkg-multimedia)
--
Alessio Treglia | www.alessiotreglia.com
Debian Developer | ***@debian.org
Ubuntu Core Developer | ***@ubuntu.com
0416 0004 A827 6E40 BB98 90FB E8A4 8AE5 311D 765A
Andreas Cadhalpun
2014-07-28 13:26:40 UTC
Permalink
Post by Alessio Treglia
On Mon, Jul 28, 2014 at 12:12 PM, "IOhannes m zmölnig (Debian/GNU)"
Post by IOhannes m zmölnig (Debian/GNU)
Post by Marco d'Itri
Except that, for a lot of the depending packages, there would be an
immediate benefit in the number of bugs fixed.
at least in theory.
Plus I would definitely appreciate to see some bug stats supporting
such a theory.
My original mail mentioned some examples.

Once FFmpeg is in the archive, each maintainer of a multimedia package
could test build it against FFmpeg and see which, if any, of the bugs
reported against said package vanish.

Best regards,
Andreas
Andreas Cadhalpun
2014-07-28 11:10:18 UTC
Permalink
Hi Julien,
Post by Julien Cristau
Post by Andreas Cadhalpun
It remains to be seen, what the release team prefers: frustrated users and
developers or both forks in jessie.
The release team is likely to let the people involved in multimedia foo
fight it out among themselves and pick a winner.
I am not interested in a "fight" and would prefer it very much if this
discussion remained purely technical.
Having a fresh memory of the last fight that took place on debian-devel,
I do not think that repeating a similar disaster is a good idea.
Post by Julien Cristau
We're not going to ship both and hand that mess over to the security team.
Could you please explain what "mess" you are talking about?

According to the changelog[1], there have been 8 security updates for
ffmpeg in squeeze. Two of them (4:0.5.6-2 and 4:0.5.6-3) do not contain
security related fixes, but rather fix build failures of the previous
security upload, so they do not really count.
That makes about 6 security fix uploads in about 3 years for squeeze,
i.e. 1 upload per 6 month.

If there were both forks in Jessie, this might double the number of
uploads to 12 in 3 years, but probably some of them could also go
through stable-updates instead of stable-security.

Is that an unbearable burden?

A lot of other software in Debian has already alternatives, like desktop
environments, web browsers, text editors and even init systems.

Why should this not be the case for a multimedia framework?

There is also one particularly similar case, as in the packages are
forks and require many security updates:
MySQL and MariaDB are currently in Debian testing.

Just for comparison, MySQL in squeeze had 3 uploads to stable-security
and 3 to oldstable(-security) [2].

As I mentioned this particular example in my discussion with Moritz, he
said that the security team will "be working with the release
team to sort this out for jessie"[3].

Now, 5 months later, he seems to have changed his mind, as I am not
aware of any such attempt, but instead Moritz seems to support both [4][5].

Thanks in advance for taking the time to answer these questions.

Best regards,
Andreas


1:
http://metadata.ftp-master.debian.org/changelogs//main/f/ffmpeg/ffmpeg_0.5.10-1_changelog

2:
http://metadata.ftp-master.debian.org/changelogs//main/m/mysql-5.1/mysql-5.1_5.1.73-1_changelog
3: https://bugs.debian.org/729203#435
4: https://bugs.debian.org/754940
5: https://bugs.debian.org/754941
Raphael Geissert
2014-07-29 07:47:51 UTC
Permalink
Post by Andreas Cadhalpun
According to the changelog[1], there have been 8 security updates for
ffmpeg in squeeze.
There would have been more but the code has evolved too much for it to be
feasible to backport the patches. Not to mention that some bugs that are being
fixed are, for example, for incomplete checks - checks that don't exist in the
0.5 branch.



Cheers,
--
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net
Andreas Cadhalpun
2014-07-29 16:43:17 UTC
Permalink
Hi Raphael,
Post by Raphael Geissert
Post by Andreas Cadhalpun
According to the changelog[1], there have been 8 security updates for
ffmpeg in squeeze.
There would have been more
You're right, my calculation is slightly flawed.
Post by Raphael Geissert
but the code has evolved too much for it to be
feasible to backport the patches.
That is likely true for some, but not for others.

The real reason that there have not been more security updates for
ffmpeg in squeeze is that since 0.5.6 this is actually Libav and Libav
upstream stopped providing backports to 0.5 after 0.5.10 in February
2013 [1]. FFmpeg upstream released 0.5.14 in July 2014 [2] with some
more fixes [3].

So had both been in squeeze, there would have been four more, i.e. 16
security updates.
Post by Raphael Geissert
Not to mention that some bugs that are being
fixed are, for example, for incomplete checks - checks that don't exist in the
0.5 branch.
What do you mean here? If the affected code is not there, then that's
nice, because a backport is not needed.

Best regards,
Andreas

1: https://www.libav.org/releases/
2: https://www.ffmpeg.org/releases/
3:
https://git.videolan.org/?p=ffmpeg.git;a=shortlog;h=refs/heads/release/0.5
Raphael Geissert
2014-07-29 19:59:22 UTC
Permalink
Post by Andreas Cadhalpun
Post by Raphael Geissert
Post by Andreas Cadhalpun
According to the changelog[1], there have been 8 security updates for
ffmpeg in squeeze.
There would have been more
You're right, my calculation is slightly flawed.
That was my point, so please don't use it as an argument.
Post by Andreas Cadhalpun
Post by Raphael Geissert
Not to mention that some bugs that are being
fixed are, for example, for incomplete checks - checks that don't exist
in the 0.5 branch.
What do you mean here? If the affected code is not there, then that's
nice, because a backport is not needed.
Let me rephrase it: the fix is for an incomplete check, but in 0.5 the check
is missing - while the rest of the code is there. Which is kinda... worse.

Cheers,
--
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net
Andreas Cadhalpun
2014-07-29 21:15:06 UTC
Permalink
Post by Raphael Geissert
Post by Andreas Cadhalpun
Post by Raphael Geissert
Post by Andreas Cadhalpun
According to the changelog[1], there have been 8 security updates for
ffmpeg in squeeze.
There would have been more
You're right, my calculation is slightly flawed.
That was my point, so please don't use it as an argument.
Maybe I didn't make my point clear enough, for which the actual number
of the security uploads is not important, only the order of magnitude.

Given the amount of software in Debian and thus the amount of security
fixes necessary for a stable release, I think that the additional
stable-security uploads for FFmpeg in the order of 10 per release will
be hardly noticeable.

While I understand and agree with the general idea of reducing code
duplication, I have a really hard time trying to understand why the
security team has such a strong opposition to the idea of having both
FFmpeg and Libav in Debian stable.

One argument against code duplication is the risk that security issues
get fixed in one, but not in the other. But in this particular case
FFmpeg upstream merges all security fixes from Libav, so an FFmpeg
package in a stable release won't have that problem.

What is particularly hard for me to understand is why e.g. MySQL and
MariaDB can be in testing at the same time without much resistance from
the security team, but FFmpeg and Libav can apparently not.
Post by Raphael Geissert
Post by Andreas Cadhalpun
Post by Raphael Geissert
Not to mention that some bugs that are being
fixed are, for example, for incomplete checks - checks that don't exist
in the 0.5 branch.
What do you mean here? If the affected code is not there, then that's
nice, because a backport is not needed.
Let me rephrase it: the fix is for an incomplete check, but in 0.5 the check
is missing - while the rest of the code is there. Which is kinda... worse.
Now I see, what you mean. Indeed that's worse, but if one notices
something like that, then the whole check can be backported instead of
the change in the check.
Though it probably would have been better to backport already the
incomplete check, when it was introduced in the development branch.

Best regards,
Andreas
Raphael Geissert
2014-07-30 10:28:37 UTC
Permalink
Post by Andreas Cadhalpun
Given the amount of software in Debian and thus the amount of security
fixes necessary for a stable release, I think that the additional
stable-security uploads for FFmpeg in the order of 10 per release will
be hardly noticeable.
They are surely noticeable to the security team: the release process of a
security update is more than just a "throw and forget".
Tracking every single vulnerability for each copy of the code consumes time.
Every single update also consumes team's time, and that of many organisations
external to Debian.
Post by Andreas Cadhalpun
What is particularly hard for me to understand is why e.g. MySQL and
MariaDB can be in testing at the same time without much resistance from
the security team, but FFmpeg and Libav can apparently not.
There is resistance - we only want one, not two, not three (percona).

IMH (and personal) O, if you want to see ffmpeg in Jessie or later, you should
replace libav - i.e. no silly one binary + libraries that won't work for
anything else.

Cheers,
--
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net
Ian Jackson
2014-08-01 12:09:24 UTC
Permalink
Based purely on security evaluations by others that I was able to find on
the web, FFmpeg appears to be better at the moment than libav on the
security front (although libav appears to be trying to catch up).
Hello everybody
At that point, and given the impressive number of users who
expressed interest for having FFmpeg in Debian (see
http://bugs.debian.org/729203), I think that it would be fair to ask
the libav maintainers to provide arguments on whether to distribute
both libraries or make a choice, even if it is against their own
interest since they may see their packages removed at the end.
Otherwise we are in that kind of frequent Debianesque situation where the
winning strategy is to stay silent as long as possible.
CCing ***@packages.d.o.

Ian.
Thorsten Glaser
2014-07-28 10:20:59 UTC
Permalink
Post by Andreas Cadhalpun
* Do you intend to replace Libav by FFmpeg in Debian?
No, there is no need to replace anything as long as it is maintained.
Currently the main goal is to give multimedia maintainers a choice
between the two sets of libraries to link against, and our users the
choice to use the 'ffmpeg' utility. That is possible, because the
packages are co-installable. (Only the *-dev packages conflict.)
Hm, I wonder if this will work, see the JPEG discussion.

But I'd *love* to see ffmpeg replace libav and to get
my mplayer back, which is currently on hold.

Thanks,
//mirabilos
Andreas Cadhalpun
2014-07-28 13:20:49 UTC
Permalink
Post by Thorsten Glaser
Post by Andreas Cadhalpun
* Do you intend to replace Libav by FFmpeg in Debian?
No, there is no need to replace anything as long as it is maintained.
Currently the main goal is to give multimedia maintainers a choice
between the two sets of libraries to link against, and our users the
choice to use the 'ffmpeg' utility. That is possible, because the
packages are co-installable. (Only the *-dev packages conflict.)
Hm, I wonder if this will work, see the JPEG discussion.
Well, it would work, if the security and release teams would agree with it.
Post by Thorsten Glaser
But I'd *love* to see ffmpeg replace libav and to get
my mplayer back, which is currently on hold.
Once FFmpeg is back in the archive, it'll be easy to reintroduce
MPlayer. It has been removed from sid, since it fails to build against
Libav, but it builds fine against FFmpeg.
(It uses some of the features only provided by FFmpeg.)

Best regards,
Andreas
Matthias Urlichs
2014-08-08 11:13:29 UTC
Permalink
Hi,
Once FFmpeg is back in the archive, it'll be easy to reintroduce MPlayer. It
has been removed from sid, since it fails to build against Libav, but it
builds fine against FFmpeg.
(It uses some of the features only provided by FFmpeg.)
Yet another reason why solely depending on libav is a bad idea.

That leaves the question of the "official" opinion of the libav
maintainers (pkg-multimedia-***@lists.alioth.debian.org).
Did none of them write anything in "defense" of libav, or have I simply
missed it?

IMHO the best idea at this point would be to toss out libav, and rebuild
the rdeps with ffmpeg. Now, before it's too late for jessie.
--
-- Matthias Urlichs
Marco d'Itri
2014-08-08 12:25:01 UTC
Permalink
Post by Matthias Urlichs
IMHO the best idea at this point would be to toss out libav, and rebuild
the rdeps with ffmpeg. Now, before it's too late for jessie.
Agreed. The interested parties should really raise this with the CTTE
ASAP.
--
ciao,
Marco
Reinhard Tartler
2014-08-08 12:29:59 UTC
Permalink
Post by Matthias Urlichs
Hi,
Once FFmpeg is back in the archive, it'll be easy to reintroduce MPlayer. It
has been removed from sid, since it fails to build against Libav, but it
builds fine against FFmpeg.
(It uses some of the features only provided by FFmpeg.)
Yet another reason why solely depending on libav is a bad idea.
That leaves the question of the "official" opinion of the libav
Did none of them write anything in "defense" of libav, or have I simply
missed it?
I intended to come up with a more timely full response, but I just
didn't get to it so far.

For now, please refer to http://lwn.net/Articles/607662/,
http://codecs.multimedia.cx/?p=370 (rather old, but still true), and
http://codecs.multimedia.cx/?p=674 (recent update on that matter)

Regarding Marco's argument that libav had few friends, well, let me
point to https://medium.com/libre-parole/libav-is-beautiful-4c1a2c886948
for instance.
Post by Matthias Urlichs
IMHO the best idea at this point would be to toss out libav, and rebuild
the rdeps with ffmpeg. Now, before it's too late for jessie.
I think that is totally out of question: Uploading FFmpeg to our
archive will break the Debian archive so hard that it will take months
to get everything back to testing, if it works at all.

To the best of my knowledge, there are only two high-profile projects
that play hardball to require FFmpeg: Mplayer and mythtv. Neither of
those do that (again to the best of my knowledge) mainly because of
technical but rather very political reasons. The case of mplayer has
been elaborated extensively in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732159#79 (see the
following "discussion" with Reimar - my conclusion from that is while
technically possible, nobody wants to make mplayer work with Libav -
and that's why it was removed, not because of the FFmpeg dependency).
For Mythtv I can only say that they didn't even bother engaging any
discussion.

(All) other high-profile downstream projects, including VLC or XBMC
(now Kodi) work demonstrably just fine with Libav. Sure, FFmpeg may
provide them with extra features, but why on earth would anyone want 3
different implementations of a ProRes encoder (seriously, and FFmpeg
seems to claim to provide "security support" for each of them), or
support for fringe codecs such as Wing Command 3 (yes, you can decode
the videos from that video game).

There are a number of legitimate requested backports, such as for some
of FFmpeg's additional filters in libavfilter, and similar. All of
them are rather easy to backport, and this is done on a regular basis.
However, with the due diligence and attention such a widely used and
high-profile library requires.

Which brings me to my next point: Release frequency. FFmpeg has an
insane frequency of releases, and still seems to "support" (at least
providing updates) to all of them, including 0.5 which is currently in
oldstable (needless to say none of these patches made it to
oldstable-security, why is still a mystery to me). The effect of that
is that downstream projects have a hard time to choose what version of
FFmpeg they want to support. This brings effectively back to the
situation I was in when I took over maintenance of the package many
years ago: Back then, Michael Niedermeyer strongly recommended all
downstreams to avoid shared libraries and just link statically against
libavcodec.a to avoid problems regarding "incompatible" library
versions. I see exactly this problem arising again: Both mythtv and
mplayer upstream (btw, xbmc as well) bundle some copy of ffmpeg in
their sources and recommend everyone to just use the internal copy to
avoid problems.

Needless to say that this is not acceptable for use in Debian.

Yes, I agree that this situation is a mess. A big mess. My fear is
that switching to FFmpeg will bring us back to the mess we where 8
years ago, and I worked on for years.

Libav is far from being perfect. That's true. I don't know why exactly
FFmpeg seems to have more manpower, but it's not all black or white.
For instance, there are a number of developers that actively
contribute to both projects and are essential in keeping both projects
in good health. Also I don't really buy the security argument. True,
FFmpeg has more security updates, but backporting them to Libav turned
out to be difficult because many if not most of them turn out to be
either incomplete, invalid or require further clarification. Libav
developers prefer to go the unpleasant road of fully understanding the
issue, which takes extra time. For all these reasons, I do not see the
necessity to do any drastic and dangerous steps right now.


I now seriously wonder if the last 45 minutes in which I wrote this
email wouldn't have been better spent with preparing the next upload
for stable-security. YMMV.
--
regards,
Reinhard
Jonas Smedegaard
2014-08-08 13:59:26 UTC
Permalink
Quoting Reinhard Tartler (2014-08-08 14:29:59)
Post by Reinhard Tartler
For now, please refer to http://lwn.net/Articles/607662/,
http://codecs.multimedia.cx/?p=370 (rather old, but still true), and
http://codecs.multimedia.cx/?p=674 (recent update on that matter)
Regarding Marco's argument that libav had few friends, well, let me
point to https://medium.com/libre-parole/libav-is-beautiful-4c1a2c886948
for instance.
Post by Matthias Urlichs
IMHO the best idea at this point would be to toss out libav, and rebuild
the rdeps with ffmpeg. Now, before it's too late for jessie.
I think that is totally out of question: Uploading FFmpeg to our
archive will break the Debian archive so hard that it will take months
to get everything back to testing, if it works at all.
To the best of my knowledge, there are only two high-profile projects
that play hardball to require FFmpeg: Mplayer and mythtv. Neither of
those do that (again to the best of my knowledge) mainly because of
technical but rather very political reasons.
[snip]
Post by Reinhard Tartler
I now seriously wonder if the last 45 minutes in which I wrote this
email wouldn't have been better spent with preparing the next upload
for stable-security. YMMV.
Thanks a lot for your input, Reinhard.

Even if the loud ones in this thread may not, I doubt I am the only one
finding value in your references and arguments.


- Jonas
--
* Jonas Smedegaard - idealist & Internet-arkitekt
* Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private
Dimitri John Ledkov
2014-08-08 14:47:43 UTC
Permalink
Post by Reinhard Tartler
Post by Matthias Urlichs
Hi,
Once FFmpeg is back in the archive, it'll be easy to reintroduce MPlayer. It
has been removed from sid, since it fails to build against Libav, but it
builds fine against FFmpeg.
(It uses some of the features only provided by FFmpeg.)
Yet another reason why solely depending on libav is a bad idea.
That leaves the question of the "official" opinion of the libav
Did none of them write anything in "defense" of libav, or have I simply
missed it?
I intended to come up with a more timely full response, but I just
didn't get to it so far.
For now, please refer to http://lwn.net/Articles/607662/,
http://codecs.multimedia.cx/?p=370 (rather old, but still true), and
http://codecs.multimedia.cx/?p=674 (recent update on that matter)
Regarding Marco's argument that libav had few friends, well, let me
point to https://medium.com/libre-parole/libav-is-beautiful-4c1a2c886948
for instance.
Post by Matthias Urlichs
IMHO the best idea at this point would be to toss out libav, and rebuild
the rdeps with ffmpeg. Now, before it's too late for jessie.
I think that is totally out of question: Uploading FFmpeg to our
archive will break the Debian archive so hard that it will take months
to get everything back to testing, if it works at all.
To the best of my knowledge, there are only two high-profile projects
that play hardball to require FFmpeg: Mplayer and mythtv. Neither of
those do that (again to the best of my knowledge) mainly because of
technical but rather very political reasons. The case of mplayer has
been elaborated extensively in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732159#79 (see the
following "discussion" with Reimar - my conclusion from that is while
technically possible, nobody wants to make mplayer work with Libav -
and that's why it was removed, not because of the FFmpeg dependency).
For Mythtv I can only say that they didn't even bother engaging any
discussion.
So in ubuntu, at the time we were doing libav9 transition we also
still had mplayer and pleora of packages inheritance from
deb-multimedia or some such repositories. I was very reluctant to
remove mplayer and all the reverse-deps, as I felt it is valuable to
ship mplayer. I believe it was based on unreleased experimental
packaging in git and other bits [1] Later Colin Watson also provided
minimal port to make it build on arm64. Unfortunately libav10
transition got entangled in too many ways and we didn't manager to
port mplayer to libav10. This was attempted based on top of mplayer
trunk snapshot / last stable tarball, patches from gentoo and own
porting efforts from myself and sil2100/robru but albeit
unsuccessfully. Under pressing wait of too many things stuck in
proposed migration (britney migration) we did remove mplayer and
reverse dependencies and pointed / filed bugs to port to mplayer2 or
just libav-tools where appropriate. I did ponder about compiling
mplayer with an embedded copy of libav9 statically linked, but
ultimately decided that it's unnecessary evil and majority of
use-cases are served by the current libav stack. Either of ffmpeg and
libav is a big security maintenance burden, simply from nature of
inherently handling complex and large streams of untrusted data. So in
trusty, I did work to unsplit the packages such that libav moved from
main to universe, and can be synced from debian unmodified to minimize
net-total maintenance burden as now updates and packaging can be fully
shared. I see moving to mplayer2 as a net positive benefit for Debian.
MythTV alone, whilst important package, I don't see as special enough
to grant use of an embedded copy or forcing an uncertain cost of
switching back to ffmpeg. If we need to port that project, then in
Debian/Ubuntu/UbuntuStudio there are enough interested people to get
it done.

[1] https://launchpad.net/ubuntu/+source/mplayer/2:1.1+dfsg1-0ubuntu1

Regards,

Dimitri.
Post by Reinhard Tartler
(All) other high-profile downstream projects, including VLC or XBMC
(now Kodi) work demonstrably just fine with Libav. Sure, FFmpeg may
provide them with extra features, but why on earth would anyone want 3
different implementations of a ProRes encoder (seriously, and FFmpeg
seems to claim to provide "security support" for each of them), or
support for fringe codecs such as Wing Command 3 (yes, you can decode
the videos from that video game).
There are a number of legitimate requested backports, such as for some
of FFmpeg's additional filters in libavfilter, and similar. All of
them are rather easy to backport, and this is done on a regular basis.
However, with the due diligence and attention such a widely used and
high-profile library requires.
Which brings me to my next point: Release frequency. FFmpeg has an
insane frequency of releases, and still seems to "support" (at least
providing updates) to all of them, including 0.5 which is currently in
oldstable (needless to say none of these patches made it to
oldstable-security, why is still a mystery to me). The effect of that
is that downstream projects have a hard time to choose what version of
FFmpeg they want to support. This brings effectively back to the
situation I was in when I took over maintenance of the package many
years ago: Back then, Michael Niedermeyer strongly recommended all
downstreams to avoid shared libraries and just link statically against
libavcodec.a to avoid problems regarding "incompatible" library
versions. I see exactly this problem arising again: Both mythtv and
mplayer upstream (btw, xbmc as well) bundle some copy of ffmpeg in
their sources and recommend everyone to just use the internal copy to
avoid problems.
Needless to say that this is not acceptable for use in Debian.
Yes, I agree that this situation is a mess. A big mess. My fear is
that switching to FFmpeg will bring us back to the mess we where 8
years ago, and I worked on for years.
Libav is far from being perfect. That's true. I don't know why exactly
FFmpeg seems to have more manpower, but it's not all black or white.
For instance, there are a number of developers that actively
contribute to both projects and are essential in keeping both projects
in good health. Also I don't really buy the security argument. True,
FFmpeg has more security updates, but backporting them to Libav turned
out to be difficult because many if not most of them turn out to be
either incomplete, invalid or require further clarification. Libav
developers prefer to go the unpleasant road of fully understanding the
issue, which takes extra time. For all these reasons, I do not see the
necessity to do any drastic and dangerous steps right now.
I now seriously wonder if the last 45 minutes in which I wrote this
email wouldn't have been better spent with preparing the next upload
for stable-security. YMMV.
--
regards,
Reinhard
--
--
Regards,

Dimitri.
Andreas Cadhalpun
2014-08-08 18:06:08 UTC
Permalink
Hi Reinhard,
Post by Reinhard Tartler
I intended to come up with a more timely full response, but I just
didn't get to it so far.
Thanks for explaining your point of view here.
Post by Reinhard Tartler
For now, please refer to http://lwn.net/Articles/607662/,
Have a look at: http://lwn.net/Articles/608040/

I was not there, when the FFmpeg/Libav split happened and I don't want
to judge, whether it was a good thing or not. But it clearly caused a
lot of bad blood between the developers involved, which in my opinion
hurts the development of the multimedia framework in recent times.
Post by Reinhard Tartler
http://codecs.multimedia.cx/?p=370 (rather old, but still true), and
If these features weren't used, they wouldn't have been developed.
And many new features have been added to FFmpeg after that post.
Post by Reinhard Tartler
http://codecs.multimedia.cx/?p=674 (recent update on that matter)
Let me quote from there:
"But that’s not what kills the fun, “security holes” do.

With an advance of automatic fuzz tools it’s easy to generate millions
of damaged files that crash your decoder and yet there are no tools for
generating correct patches. Fixing those crashes is tedious, requires a
lot of thinking (should I disable it? will it affect decoding correct
files? etc.) and in other words not fun at all."

That seems as if he doesn't want to continue Libav development, because
fixing security issues is tedious work.
It has basically nothing to do with whether FFmpeg is of good quality
security wise or not, or a good or bad competitor against Libav.
Post by Reinhard Tartler
Regarding Marco's argument that libav had few friends, well, let me
point to https://medium.com/libre-parole/libav-is-beautiful-4c1a2c886948
for instance.
One person thinking that the code is 'beautiful' is no convincing
argument. The number of people expressing their interest in having
FFmpeg back in Debian is a lot more convincing.
Post by Reinhard Tartler
Post by Matthias Urlichs
IMHO the best idea at this point would be to toss out libav, and rebuild
the rdeps with ffmpeg. Now, before it's too late for jessie.
I think that is totally out of question: Uploading FFmpeg to our
archive will break the Debian archive so hard that it will take months
to get everything back to testing, if it works at all.
Let me repeat what I wrote in my initial mail in this thread:
Having FFmpeg in the Debian archive breaks absolutely *nothing*, but it
gives developers and users a choice between the two.

Even if Libav were to be removed, a transition to FFmpeg could be rather
smooth, as all the necessary patches already exist.
But you're right that the time to test the resulting packages is getting
rather short for the coming freeze.

Still it can make sense for packages that are tested with FFmpeg
upstream and have known issues with Libav.

So if you and the other Libav maintainers were to support the idea of
having both, the security and release teams could perhaps be convinced
to allow that. This has been my goal from the beginning and I hoped that
would be appreciated.
Post by Reinhard Tartler
To the best of my knowledge, there are only two high-profile projects
that play hardball to require FFmpeg: Mplayer and mythtv. Neither of
those do that (again to the best of my knowledge) mainly because of
technical but rather very political reasons. The case of mplayer has
been elaborated extensively in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732159#79 (see the
following "discussion" with Reimar - my conclusion from that is while
technically possible, nobody wants to make mplayer work with Libav -
and that's why it was removed, not because of the FFmpeg dependency).
For Mythtv I can only say that they didn't even bother engaging any
discussion.
That FFmpeg has more features is a rather technical argument.

Note that also many other projects are developed mainly with FFmpeg,
they just happen to not break completely when compiled against Libav.

For example, mpv prefers FFmpeg for good reasons:
"Although mpv attempts to work well with both FFmpeg and Libav, FFmpeg
is preferred in general. This is simply because FFmpeg merges from
Libav, and seems to have more features and fewer bugs than Libav." [1]

These features are actually requested by users, see e.g. [2].
Post by Reinhard Tartler
(All) other high-profile downstream projects, including VLC or XBMC
(now Kodi) work demonstrably just fine with Libav. Sure, FFmpeg may
Just fine? Did you see the bugs I mentioned in my initial mail?

And how does it come that XBMC needs the '--enable-libav-compat'
configure option, which then uses code from lib/xbmc-libav-hacks, to
build against Libav?
Post by Reinhard Tartler
provide them with extra features, but why on earth would anyone want 3
different implementations of a ProRes encoder (seriously, and FFmpeg
seems to claim to provide "security support" for each of them), or
support for fringe codecs such as Wing Command 3 (yes, you can decode
the videos from that video game).
One of those ProRes encoders comes from Libav and is provided in FFmpeg
for compatibility with Libav. Supporting both encoders provides the user
with additional choice.
Also security issues in an encoder are less likely, given that it
_encodes_ bitmaps into a bitstream and doesn't have to do complicated
parsing of attacker provided bitstreams into bitmaps like a decoder.

If FFmpeg wouldn't care so much about compatibility with Libav, a lot
less programs could be built with both.
Post by Reinhard Tartler
There are a number of legitimate requested backports, such as for some
of FFmpeg's additional filters in libavfilter, and similar. All of
them are rather easy to backport, and this is done on a regular basis.
However, with the due diligence and attention such a widely used and
high-profile library requires.
I'm not sure such ports to Libav would be easy, as I'm under the
impression that many filters in FFmpeg use features not available in Libav.

As far as I can tell, only 3 filters have been ported in the last two
years, including one submission by an external developer. Keeping in
mind that more than 100 additional filters have been added to FFmpeg
this is only a very small number.

The external submission came from a developer of a music player who uses
Debian/Ubuntu and therefore has to use Libav.
"I went with libav simply because it is what is in the Debian and Ubuntu
package managers, and one of my goals is to get this music player
backend into their package managers.
[...]
In order to have this functionality, I ported the "compand" audio filter
from ffmpeg." [3]
Post by Reinhard Tartler
Which brings me to my next point: Release frequency. FFmpeg has an
insane frequency of releases, and still seems to "support" (at least
providing updates) to all of them,
Generally FFmpeg supports all releases still widely used.
Post by Reinhard Tartler
including 0.5 which is currently in
oldstable (needless to say none of these patches made it to
oldstable-security, why is still a mystery to me).
It was a mystery to me, but I assumed you would know the reasons [4]:
ffmpeg (4:0.5.6-1) stable-security; urgency=low
[...]
* Note while the source package name reads 'ffmpeg', this is actually
the libav-0.5.6 release.

-- Reinhard Tartler <***@tauware.de> Mon, 26 Dec 2011 00:14:25 +0100

So even if the package in oldstable is called ffmpeg, it is actually
Libav and thus only got the security updates provided by Libav.

Nothing prevents you from uploading ffmpeg-0.5.14 to squeeze-lts.
Post by Reinhard Tartler
The effect of that
is that downstream projects have a hard time to choose what version of
FFmpeg they want to support.
What do you mean?
Every distribution includes the latest possible version and upstream
tracks how long they need to be supported [5].
Post by Reinhard Tartler
This brings effectively back to the
situation I was in when I took over maintenance of the package many
years ago: Back then, Michael Niedermeyer strongly recommended all
downstreams to avoid shared libraries and just link statically against
libavcodec.a to avoid problems regarding "incompatible" library
versions. I see exactly this problem arising again: Both mythtv and
mplayer upstream (btw, xbmc as well) bundle some copy of ffmpeg in
their sources and recommend everyone to just use the internal copy to
avoid problems.
Needless to say that this is not acceptable for use in Debian.
Both MPlayer and XBMC can be built against a system version of FFmpeg.
Post by Reinhard Tartler
Yes, I agree that this situation is a mess. A big mess. My fear is
that switching to FFmpeg will bring us back to the mess we where 8
years ago, and I worked on for years.
How can switching to FFmpeg undo the development of the last 8 years?
Especially since commits from Libav get merged on a daily basis.
Post by Reinhard Tartler
Libav is far from being perfect. That's true. I don't know why exactly
FFmpeg seems to have more manpower, but it's not all black or white.
For instance, there are a number of developers that actively
contribute to both projects and are essential in keeping both projects
in good health.
Indirectly all Libav contributors also contribute to FFmpeg, because
changes in Libav get merged into FFmpeg.
Post by Reinhard Tartler
Also I don't really buy the security argument. True,
FFmpeg has more security updates, but backporting them to Libav turned
out to be difficult because many if not most of them turn out to be
either incomplete, invalid or require further clarification. Libav
developers prefer to go the unpleasant road of fully understanding the
issue, which takes extra time.
If that extra time is about a year or more (CVE-2011-3946,
CVE-2013-0851, CVE-2013-0852, CVE-2013-0868) [6], I think it is clearly
far too long.
Post by Reinhard Tartler
For all these reasons, I do not see the
necessity to do any drastic and dangerous steps right now.
I now seriously wonder if the last 45 minutes in which I wrote this
email wouldn't have been better spent with preparing the next upload
for stable-security. YMMV.
It would be nice, if you could also spent some time thinking about the
possibility of having both FFmpeg and Libav in Debian.

Best regards,
Andreas


1: https://github.com/mpv-player/mpv/wiki/FFmpeg-versus-Libav
2: http://lwn.net/Articles/608012/
3: http://andrewkelley.me/post/quest-build-ultimate-music-player.html
4: https://tracker.debian.org/media/packages/f/ffmpeg/changelog-4%3A0.5.10-1
5: https://trac.ffmpeg.org/wiki/Downstreams
6: https://tracker.debian.org/media/packages/liba/libav/changelog-6%3A10.3-1
Bálint Réczey
2014-08-09 09:38:54 UTC
Permalink
Hi,
Post by Andreas Cadhalpun
Hi Reinhard,
Post by Reinhard Tartler
I intended to come up with a more timely full response, but I just
didn't get to it so far.
Thanks for explaining your point of view here.
Post by Reinhard Tartler
For now, please refer to http://lwn.net/Articles/607662/,
Have a look at: http://lwn.net/Articles/608040/
I was not there, when the FFmpeg/Libav split happened and I don't want to
judge, whether it was a good thing or not. But it clearly caused a lot of
bad blood between the developers involved, which in my opinion hurts the
development of the multimedia framework in recent times.
Post by Reinhard Tartler
http://codecs.multimedia.cx/?p=370 (rather old, but still true), and
If these features weren't used, they wouldn't have been developed.
And many new features have been added to FFmpeg after that post.
Post by Reinhard Tartler
http://codecs.multimedia.cx/?p=674 (recent update on that matter)
"But that’s not what kills the fun, “security holes” do.
With an advance of automatic fuzz tools it’s easy to generate millions of
damaged files that crash your decoder and yet there are no tools for
generating correct patches. Fixing those crashes is tedious, requires a lot
of thinking (should I disable it? will it affect decoding correct files?
etc.) and in other words not fun at all."
That seems as if he doesn't want to continue Libav development, because
fixing security issues is tedious work.
It has basically nothing to do with whether FFmpeg is of good quality
security wise or not, or a good or bad competitor against Libav.
Post by Reinhard Tartler
Regarding Marco's argument that libav had few friends, well, let me
point to https://medium.com/libre-parole/libav-is-beautiful-4c1a2c886948
for instance.
One person thinking that the code is 'beautiful' is no convincing argument.
The number of people expressing their interest in having FFmpeg back in
Debian is a lot more convincing.
Post by Reinhard Tartler
Post by Matthias Urlichs
IMHO the best idea at this point would be to toss out libav, and rebuild
the rdeps with ffmpeg. Now, before it's too late for jessie.
I think that is totally out of question: Uploading FFmpeg to our
archive will break the Debian archive so hard that it will take months
to get everything back to testing, if it works at all.
Having FFmpeg in the Debian archive breaks absolutely *nothing*, but it
gives developers and users a choice between the two.
Even if Libav were to be removed, a transition to FFmpeg could be rather
smooth, as all the necessary patches already exist.
But you're right that the time to test the resulting packages is getting
rather short for the coming freeze.
Still it can make sense for packages that are tested with FFmpeg upstream
and have known issues with Libav.
So if you and the other Libav maintainers were to support the idea of having
both, the security and release teams could perhaps be convinced to allow
that. This has been my goal from the beginning and I hoped that would be
appreciated.
Post by Reinhard Tartler
To the best of my knowledge, there are only two high-profile projects
that play hardball to require FFmpeg: Mplayer and mythtv. Neither of
those do that (again to the best of my knowledge) mainly because of
technical but rather very political reasons. The case of mplayer has
been elaborated extensively in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732159#79 (see the
following "discussion" with Reimar - my conclusion from that is while
technically possible, nobody wants to make mplayer work with Libav -
and that's why it was removed, not because of the FFmpeg dependency).
For Mythtv I can only say that they didn't even bother engaging any
discussion.
That FFmpeg has more features is a rather technical argument.
Note that also many other projects are developed mainly with FFmpeg, they
just happen to not break completely when compiled against Libav.
"Although mpv attempts to work well with both FFmpeg and Libav, FFmpeg is
preferred in general. This is simply because FFmpeg merges from Libav, and
seems to have more features and fewer bugs than Libav." [1]
These features are actually requested by users, see e.g. [2].
Post by Reinhard Tartler
(All) other high-profile downstream projects, including VLC or XBMC
(now Kodi) work demonstrably just fine with Libav. Sure, FFmpeg may
Just fine? Did you see the bugs I mentioned in my initial mail?
And how does it come that XBMC needs the '--enable-libav-compat' configure
option, which then uses code from lib/xbmc-libav-hacks, to build against
Libav?
Being the xbmc maintainer I feel being addressed and let me share my
POV regarding XBMC. :-)
XBMC works with Libav for most use-cases while it fails in the rest,
notably it can not use VDPAU acceleration which is being
(understandably) complained about very often (#742896). Another issue
is Libav crashing on bad input which makes XBMC/Libav unusable in PVR
configurations when signal is corrupted sometimes (#741170). Upstream
makes sure all their use-cases work well with FFmpeg and not
interested in Libav-related issues. I can't fix the problems because I
don't have any HW reproducing them, and I don't get help from Libav
upstream/maintainers either in fixing those issues.
Media requiring codecs not supported by Libav will not play with
Debian's pacakaged XBMC, but I have not seen any file encoded in such
way.

To overcome the situation I had to set up a "PPA" at
https://people.debian.org/~rbalint/ppa/xbmc-ffmpeg/ because I would
like to serve Debian users who would like to use VDPAU or PVR.
Maintaining this costs me extra time whenever I update XBMC in Debian
because I have to build and update those packages as well.

Regarding the '--enable-libav-compat' configure option and
lib/xbmc-libav-hacks they exist(ed) because XBMC's codebase is written
with FFmpeg API in mind and the compatibility with Libav 10 was
"hacked in" this way. I wrote "existed", because upstream removed the
support completely in their master branch. Most of my time spent to
XBMC packaging is spent on making sure that XBMC works with Libav and
many upstream developers oppose having XBMC packaged in Debian at all
because they want to see only fully functional XBMC builds everywhere.

I would like to have flawlessly working XBMC in Debian as well, but it
can't happen without fixing the Libav issues I mentioned above or
letting FFmpeg entering Debian.

IMO Andreas did a very good job and made re-introducing FFmpeg to
Debian as painless as possible. The package deserves to enter at least
experimental. Keeping it out of Debian just because and other
well-known fork is in Debian is totally unfair. We claim to create The
Universal Operating System not an App Store.

Cheers,
Balint

PS: I think Libav made a strategic mistake by not following Linux's
strategy of creating a "staging" area and not allowing less than
stellar code entering their codebase.
Post by Andreas Cadhalpun
Post by Reinhard Tartler
provide them with extra features, but why on earth would anyone want 3
different implementations of a ProRes encoder (seriously, and FFmpeg
seems to claim to provide "security support" for each of them), or
support for fringe codecs such as Wing Command 3 (yes, you can decode
the videos from that video game).
One of those ProRes encoders comes from Libav and is provided in FFmpeg for
compatibility with Libav. Supporting both encoders provides the user with
additional choice.
Also security issues in an encoder are less likely, given that it
_encodes_ bitmaps into a bitstream and doesn't have to do complicated
parsing of attacker provided bitstreams into bitmaps like a decoder.
If FFmpeg wouldn't care so much about compatibility with Libav, a lot less
programs could be built with both.
Post by Reinhard Tartler
There are a number of legitimate requested backports, such as for some
of FFmpeg's additional filters in libavfilter, and similar. All of
them are rather easy to backport, and this is done on a regular basis.
However, with the due diligence and attention such a widely used and
high-profile library requires.
I'm not sure such ports to Libav would be easy, as I'm under the impression
that many filters in FFmpeg use features not available in Libav.
As far as I can tell, only 3 filters have been ported in the last two years,
including one submission by an external developer. Keeping in mind that more
than 100 additional filters have been added to FFmpeg this is only a very
small number.
The external submission came from a developer of a music player who uses
Debian/Ubuntu and therefore has to use Libav.
"I went with libav simply because it is what is in the Debian and Ubuntu
package managers, and one of my goals is to get this music player backend
into their package managers.
[...]
In order to have this functionality, I ported the "compand" audio filter
from ffmpeg." [3]
Post by Reinhard Tartler
Which brings me to my next point: Release frequency. FFmpeg has an
insane frequency of releases, and still seems to "support" (at least
providing updates) to all of them,
Generally FFmpeg supports all releases still widely used.
Post by Reinhard Tartler
including 0.5 which is currently in
oldstable (needless to say none of these patches made it to
oldstable-security, why is still a mystery to me).
ffmpeg (4:0.5.6-1) stable-security; urgency=low
[...]
* Note while the source package name reads 'ffmpeg', this is actually
the libav-0.5.6 release.
So even if the package in oldstable is called ffmpeg, it is actually Libav
and thus only got the security updates provided by Libav.
Nothing prevents you from uploading ffmpeg-0.5.14 to squeeze-lts.
Post by Reinhard Tartler
The effect of that
is that downstream projects have a hard time to choose what version of
FFmpeg they want to support.
What do you mean?
Every distribution includes the latest possible version and upstream tracks
how long they need to be supported [5].
Post by Reinhard Tartler
This brings effectively back to the
situation I was in when I took over maintenance of the package many
years ago: Back then, Michael Niedermeyer strongly recommended all
downstreams to avoid shared libraries and just link statically against
libavcodec.a to avoid problems regarding "incompatible" library
versions. I see exactly this problem arising again: Both mythtv and
mplayer upstream (btw, xbmc as well) bundle some copy of ffmpeg in
their sources and recommend everyone to just use the internal copy to
avoid problems.
Needless to say that this is not acceptable for use in Debian.
Both MPlayer and XBMC can be built against a system version of FFmpeg.
Post by Reinhard Tartler
Yes, I agree that this situation is a mess. A big mess. My fear is
that switching to FFmpeg will bring us back to the mess we where 8
years ago, and I worked on for years.
How can switching to FFmpeg undo the development of the last 8 years?
Especially since commits from Libav get merged on a daily basis.
Post by Reinhard Tartler
Libav is far from being perfect. That's true. I don't know why exactly
FFmpeg seems to have more manpower, but it's not all black or white.
For instance, there are a number of developers that actively
contribute to both projects and are essential in keeping both projects
in good health.
Indirectly all Libav contributors also contribute to FFmpeg, because changes
in Libav get merged into FFmpeg.
Post by Reinhard Tartler
Also I don't really buy the security argument. True,
FFmpeg has more security updates, but backporting them to Libav turned
out to be difficult because many if not most of them turn out to be
either incomplete, invalid or require further clarification. Libav
developers prefer to go the unpleasant road of fully understanding the
issue, which takes extra time.
If that extra time is about a year or more (CVE-2011-3946, CVE-2013-0851,
CVE-2013-0852, CVE-2013-0868) [6], I think it is clearly far too long.
Post by Reinhard Tartler
For all these reasons, I do not see the
necessity to do any drastic and dangerous steps right now.
I now seriously wonder if the last 45 minutes in which I wrote this
email wouldn't have been better spent with preparing the next upload
for stable-security. YMMV.
It would be nice, if you could also spent some time thinking about the
possibility of having both FFmpeg and Libav in Debian.
Best regards,
Andreas
1: https://github.com/mpv-player/mpv/wiki/FFmpeg-versus-Libav
2: http://lwn.net/Articles/608012/
3: http://andrewkelley.me/post/quest-build-ultimate-music-player.html
4: https://tracker.debian.org/media/packages/f/ffmpeg/changelog-4%3A0.5.10-1
5: https://trac.ffmpeg.org/wiki/Downstreams
6: https://tracker.debian.org/media/packages/liba/libav/changelog-6%3A10.3-1
Jonas Smedegaard
2014-08-09 11:41:55 UTC
Permalink
Quoting Bálint Réczey (2014-08-09 11:38:54)
Post by Bálint Réczey
XBMC works with Libav for most use-cases while it fails in the rest,
notably it can not use VDPAU acceleration which is being
(understandably) complained about very often (#742896). Another issue
is Libav crashing on bad input which makes XBMC/Libav unusable in PVR
configurations when signal is corrupted sometimes (#741170).
Ok, so you know factually that some things are broken when linking XBMC
with Libav instead of XBMC-FFmpeg.
Post by Bálint Réczey
Upstream makes sure all their use-cases work well with FFmpeg and not
interested in Libav-related issues.
According to XBMC, they only make sure their code works with
XBMC-FFmpeg.
Post by Bálint Réczey
I can't fix the problems because I don't have any HW reproducing them,
and I don't get help from Libav upstream/maintainers either in fixing
those issues.
That sounds to me like you do not factually know if XBMC will be broken
also when linked against FFmeg (you only really know about XBMC-FFmpeg).
Post by Bálint Réczey
I would like to have flawlessly working XBMC in Debian as well, but it
can't happen without fixing the Libav issues I mentioned above or
letting FFmpeg entering Debian.
I do understand how it is easier for you to link XBMC against FFmpeg
than against Libav, since FFmpeg has similar/same API as XBMC-FFmpeg,
but it seems to me that you are jumping to conclusions when stating that
linking against (non-XBMC-)FFmpeg will make XBMC work "flawlessly".


- Jonas
--
* Jonas Smedegaard - idealist & Internet-arkitekt
* Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private
Bálint Réczey
2014-08-09 12:39:09 UTC
Permalink
Quoting Bálint Réczey (2014-08-09 11:38:54)
Post by Bálint Réczey
XBMC works with Libav for most use-cases while it fails in the rest,
notably it can not use VDPAU acceleration which is being
(understandably) complained about very often (#742896). Another issue
is Libav crashing on bad input which makes XBMC/Libav unusable in PVR
configurations when signal is corrupted sometimes (#741170).
Ok, so you know factually that some things are broken when linking XBMC
with Libav instead of XBMC-FFmpeg.
Well, it depends on the definition of factually. I could not test the
VDPAU issues myself. :-)
Post by Bálint Réczey
Upstream makes sure all their use-cases work well with FFmpeg and not
interested in Libav-related issues.
According to XBMC, they only make sure their code works with
XBMC-FFmpeg.
Post by Bálint Réczey
I can't fix the problems because I don't have any HW reproducing them,
and I don't get help from Libav upstream/maintainers either in fixing
those issues.
That sounds to me like you do not factually know if XBMC will be broken
also when linked against FFmeg (you only really know about XBMC-FFmpeg).
Since XBMC switched to vanilla FFmpeg from their internal fork I would
be really surprised if XBMC did not work with the proposed new FFmpeg
packages. -dmo packages are built with external FFmpeg, too...
If this test is a deal-breaker for accepting FFmpeg into experimental
I can provide binaries for testing but probably most curious DD-s
having the right HW would be able to test if my theory is right.
Post by Bálint Réczey
I would like to have flawlessly working XBMC in Debian as well, but it
can't happen without fixing the Libav issues I mentioned above or
letting FFmpeg entering Debian.
I do understand how it is easier for you to link XBMC against FFmpeg
than against Libav, since FFmpeg has similar/same API as XBMC-FFmpeg,
but it seems to me that you are jumping to conclusions when stating that
linking against (non-XBMC-)FFmpeg will make XBMC work "flawlessly".
I would say it is not a mathematically correct reasoning, but a strong
expectation supported by observations.
Prove me wrong if you really think I'm missing something, I will stand
corrected. I made falsifiable statements.

By "flawlessly" I mean very close to upstream's expectations and
specifically fixing VDPAU and PVR issues I mentioned earlier.

Cheers,
Balint
Jonas Smedegaard
2014-08-09 19:41:55 UTC
Permalink
Quoting Bálint Réczey (2014-08-09 14:39:09)
Post by Bálint Réczey
Post by Jonas Smedegaard
Quoting Bálint Réczey (2014-08-09 11:38:54)
Post by Bálint Réczey
Upstream makes sure all their use-cases work well with FFmpeg and
not interested in Libav-related issues.
According to XBMC, they only make sure their code works with
XBMC-FFmpeg.
Post by Bálint Réczey
I can't fix the problems because I don't have any HW reproducing
them, and I don't get help from Libav upstream/maintainers either in
fixing those issues.
That sounds to me like you do not factually know if XBMC will be
broken also when linked against FFmeg (you only really know about
XBMC-FFmpeg).
Since XBMC switched to vanilla FFmpeg from their internal fork I would
be really surprised if XBMC did not work with the proposed new FFmpeg
packages.
Whoops - I confused XBMC and MythTV. Sorry for the noise.


- Jonas
--
* Jonas Smedegaard - idealist & Internet-arkitekt
* Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private
Alessio Treglia
2014-08-08 14:06:20 UTC
Permalink
Hi,
Post by Matthias Urlichs
That leaves the question of the "official" opinion of the libav
Did none of them write anything in "defense" of libav, or have I simply
missed it?
IMHO the best idea at this point would be to toss out libav, and rebuild
the rdeps with ffmpeg. Now, before it's too late for jessie.
This is an eminently bad idea. As I said before, it's too late for
Jessie already.

Many Jessie's multimedia framework and packages have been developed
and QA'd with libav.

We've spent a lot of time over the past months talking to upstreams,
forwarding them our patches and make sure their programs and libraries
work with libav.
We've spent ***months*** in making the whole thing work, and dropping
libav in favour of FFmpeg at this point, roughly few weeks from the
freeze deadline, would be definitely insane.

Cheers.
--
Alessio Treglia | www.alessiotreglia.com
Debian Developer | ***@debian.org
Ubuntu Core Developer | ***@ubuntu.com
0416 0004 A827 6E40 BB98 90FB E8A4 8AE5 311D 765A
Matthias Urlichs
2014-08-08 19:22:38 UTC
Permalink
Hi,
Post by Alessio Treglia
We've spent a lot of time over the past months talking to upstreams,
forwarding them our patches and make sure their programs and libraries
work with libav.
We've spent ***months*** in making the whole thing work, and dropping
libav in favour of FFmpeg at this point, roughly few weeks from the
freeze deadline, would be definitely insane.
Yes, that work might be "wasted". But I don't think that it's a good idea
to base the decision of whether or not X is better for Debian on the fact
that somebody did a lot of work for Y instead.

Yes, the freeze is not that far away. But frankly, how much effort would it
be to switch now? As far as I can tell from this discussion, the answer is
"not a whole lot". The bulk of ffmpeg/libav's reverse dependencies is just
a simple binNMU away, and as the libraries seem to be co-installable we
don't even need a big transition.

We'd also benefit from the fact that Upstream tends to use FFmpeg. I'd
hate to report some intractable codec bug which Upstream closes with an
"it works with FFmpeg" comment -- what would you recommend me to do in
that situation, next year -- install the ffmpeg libs from Experimental
and rebuild the offender?
--
-- Matthias Urlichs
Peter B.
2014-08-10 09:16:59 UTC
Permalink
Post by Matthias Urlichs
We'd also benefit from the fact that Upstream tends to use FFmpeg. I'd
hate to report some intractable codec bug which Upstream closes with
an "it works with FFmpeg" comment
Oh, btw, just a few days ago, that's exactly what happened on kdenlive [1].
A developer posted the following statement on an issue which is open for
more than 1.5 years now:

[quote]
Confirm this is a libav problem, use builds with ffmpeg or wait debian
(&derivatives) to bring ffmpeg back
[/quote]

Just thought this might fit here...


Regards,
Pb


== References:
[1] http://www.kdenlive.org/mantis/view.php?id=2943#c10195
Reinhard Tartler
2014-08-10 13:10:23 UTC
Permalink
Hi,
Including rename of constants (code enums id for example).
Another nail in libav's coffin, then.
That's one way to see it. To me, this makes mythtv unsuitable for
IMHO it's reasonable to expect core APIs to be upwards-compatible and keep
deprecated interfaces around for another release or two.
This is exactly what Libav is doing: The deprecation process for
symbols, APIs, enums, etc. takes *years*, because so many software
packages in Debian and else where use them, and it is so believably
painful to change them. Just have a look at the last two Libav
transitions, and the massive amount of patches it took to get packages
fixed and eventually to get Debian to the new Libav release.

Now enter FFmpeg.

FFmpeg has a significant higher release frequency, (it seems to me
about every 3-4 months), so that you would get a deprecation cycle
that is considerably less than a year. In practice, the deprecation
cycle more or less seems to match Libav's cycle, because at least
right now, FFmpeg tracks Libav's API. If that were not the case (and
I promise you FFmpeg would stop tracking Libav as soon as it replaces
Libav in Debian), I can almost guarantee [1] you that FFmpeg would
very much prefer to resume to the deprecation cycle the project
before: None, i.e., every piece of software is expected to keep up
with FFmpeg's master branch for reasons Jean-Yves outlines.

[1] at least statements such as
https://ffmpeg.org/pipermail/ffmpeg-devel/2014-August/160876.html
strongly suggest this (at least if you have followed the
libavresample/libswresample mess).
Keeping your own static version is the only reasonable approach.
That may be OK on Windows. However, a proper Linux distribution is supposed
to be an integrated whole and not a haphazard collection of programs, each
bringing along their own copy of core libraries and their own un- or
semi-fixed security problems.
BTW Jean-Yves outlines an approach that used to be very common on the
past: Pick some particular snapshot of FFmpeg and maintain that in a
downstream project, and expect users to use that because it is too
much effort to keep up with FFmpeg's release frequency. Prominent
examples of projects that did this (and actually, still do) include
xine-lib, mplayer, xbmc, and many more. This lead exactly to the mess
I was talking about in my previous email: dozens of embedded
code-copies that were accepted into the Debian archive.

Over many years, I've spearheaded a significant effort in Debian with
packaging and in upstream with introducing a release culture in FFmpeg
(as release manager) to get to somewhat same release frequencies, so
that downstream projects at least had a chance to agree on common
versions of FFmpeg. At the time of the split, I was worried that this
work would have been in vain. Considering that most active developers
of FFmpeg at that time (which coincidentally supported my approach to
release management and frequency) joined what is now known as Libav, I
continued my work as upstream release manager in Libav, because I
consider Libav as much more suited for Debian than FFmpeg. Today, I
still firmly believe that this was the right move for Debian as a
project.

I do strongly believe that projects that require people to use FFmpeg
actually mean to use their own private fork (cf. the mythtv debacle),
and given the amount of packages in Debian, it would significantly
much more effort to "port" (or "patch" as Andreas is phrasing it) them
to some common version of FFmpeg than doing what we are doing now:
Making sure they work with the version of Libav's libavcodec.so
implementation. This thread has shown a couple of examples that
support this argument: Mythtv, but also mplayer that claims to work
with a system libavcodec.so, which is true as long as it matches the
version that is was built against. This is what makes mplayer so hard
to package, and was ultimately the reason why I requested mplayer's
removal (which is more than ironic, given that back then, I fought
with ftp-master for many years to get it included into Debian in the
first place).

On a related note: Most Libav developers are very tired of the
constant flamewars and defamation attempts that arises from FFmpeg.
Over years, Libav tries to convinced everyone by providing usable
software releases. Nevertheless, this particular debate is very
worrisome: The silence from the Libav camp seems to not to be taken as
consent. Quite the contrary is true.

How to proceed from here? TBH, I'm not sure. Ideally, both projects
would find some common ground and "just merge" (however that would
technically look like). However, this very debate within Debian shows
that this is unlikely to happen anytime soon: There is way to much
disagreement on very fundamental questions that require agreement
within a free software project, and the hostile and aggressive tone
the majority of participants in this debate exhibit does not help with
making progress on that front either.
--
regards,
Reinhard
Andreas Cadhalpun
2014-08-10 21:02:52 UTC
Permalink
Hi Reinhard,
Post by Reinhard Tartler
IMHO it's reasonable to expect core APIs to be upwards-compatible and keep
deprecated interfaces around for another release or two.
This is exactly what Libav is doing: The deprecation process for
symbols, APIs, enums, etc. takes *years*, because so many software
packages in Debian and else where use them, and it is so believably
painful to change them. Just have a look at the last two Libav
transitions, and the massive amount of patches it took to get packages
fixed and eventually to get Debian to the new Libav release.
Now enter FFmpeg.
FFmpeg also has a deprecation process and keeps deprecated features
around longer than Libav. For example, avcodec_encode_video,
av_close_input_file and avcodec_decode_audio3 are still present in
FFmpeg, but already removed from Libav.
Post by Reinhard Tartler
FFmpeg has a significant higher release frequency, (it seems to me
about every 3-4 months), so that you would get a deprecation cycle
that is considerably less than a year.
The deprecation cycle is not related to the release frequency, as many
FFmpeg release are API/ABI backwards compatible with the previous one.
Post by Reinhard Tartler
In practice, the deprecation
cycle more or less seems to match Libav's cycle, because at least
right now, FFmpeg tracks Libav's API. If that were not the case (and
I promise you FFmpeg would stop tracking Libav as soon as it replaces
Libav in Debian), I can almost guarantee [1] you that FFmpeg would
very much prefer to resume to the deprecation cycle the project
before: None, i.e., every piece of software is expected to keep up
with FFmpeg's master branch for reasons Jean-Yves outlines.
I think you won't be able to keep that promise, because it wouldn't make
much sense to stop merging changes from Libav (especially, if they are
useful) after doing it for such a long time. Even in the unlikely event
that this might happen, there is no reason to change the handling of
deprecations.
Post by Reinhard Tartler
[1] at least statements such as
https://ffmpeg.org/pipermail/ffmpeg-devel/2014-August/160876.html
strongly suggest this (at least if you have followed the
libavresample/libswresample mess).
I'm understanding this mail differently: What Michael is explaining is
that it is more difficult for FFmpeg to change things in libavresample
than in libswresample, because Libav is unlikely to merge back changes,
but FFmpeg tries to be compatible with Libav.

In reality, there hasn't been any backwards incompatible change in
libswresample (still soversion 0), but there has been one in
libavresample (now at soversion 1).
Post by Reinhard Tartler
Keeping your own static version is the only reasonable approach.
That may be OK on Windows. However, a proper Linux distribution is supposed
to be an integrated whole and not a haphazard collection of programs, each
bringing along their own copy of core libraries and their own un- or
semi-fixed security problems.
BTW Jean-Yves outlines an approach that used to be very common on the
past: Pick some particular snapshot of FFmpeg and maintain that in a
downstream project, and expect users to use that because it is too
much effort to keep up with FFmpeg's release frequency.
It is easy to 'keep up' with releases that are API/ABI compatible, which
many FFmpeg releases are.
One doesn't even have to recompile dependent programs (if they are not
buggy), one can just install the new version of the libraries.
Post by Reinhard Tartler
Prominent
examples of projects that did this (and actually, still do) include
xine-lib, mplayer, xbmc, and many more. This lead exactly to the mess
I was talking about in my previous email: dozens of embedded
code-copies that were accepted into the Debian archive.
As you must know, xine-lib and xbmc are - and mplayer was - compiled
against the system version of Libav in Debian.
Post by Reinhard Tartler
Over many years, I've spearheaded a significant effort in Debian with
packaging and in upstream with introducing a release culture in FFmpeg
(as release manager) to get to somewhat same release frequencies, so
that downstream projects at least had a chance to agree on common
versions of FFmpeg. At the time of the split, I was worried that this
work would have been in vain.
It appears your work has not have been in vain, as FFmpeg's current
release culture takes into account that any backwards incompatible API
change means a lot of work for everyone using it. I believe this is
handled now much better than in the times before the 0.5 release.

If you are unhappy with how the releases are managed in FFmpeg, you can
always send your concerns to ffmpeg-devel (and I think you still have
commit rights for FFmpeg's git repository as well).
Post by Reinhard Tartler
Considering that most active developers
of FFmpeg at that time (which coincidentally supported my approach to
release management and frequency) joined what is now known as Libav, I
continued my work as upstream release manager in Libav, because I
consider Libav as much more suited for Debian than FFmpeg. Today, I
still firmly believe that this was the right move for Debian as a
project.
Conversely, I firmly believe, that the absence of FFmpeg from the
archive is bad for Debian's users. The reasons for that can be found in
my initial mail in this thread.
Post by Reinhard Tartler
I do strongly believe that projects that require people to use FFmpeg
actually mean to use their own private fork (cf. the mythtv debacle),
As far as I know, MythTV is the only example for a program that cannot
be compiled with a system version of FFmpeg and instead uses its own fork.
Post by Reinhard Tartler
and given the amount of packages in Debian, it would significantly
much more effort to "port" (or "patch" as Andreas is phrasing it) them
As I explained in my initial mail in this thread [1], most of these
patches are only necessary for switching the packages to pkg-config to
detect the FFmpeg linker flags.
If the FFmpeg in Debian would use the upstream library names, i.e. the
same as Libav, these patches would not be necessary. One still can avoid
soname clashes by using the --enable-raise-major configure option and
thus bumping all major soversion by 100, but this is not done in order
to not use the same namespace that Libav in Debian currently uses and
because some programs (I only know of vlc) check for a maximum version
of the libraries.

There are only a few other changes necessary, and most of them are also
needed for Libav 10 (or even totally unrelated to FFmpeg/Libav).
For the benefit of everyone, here is a comprehensive list of those:
* acoustid-fingerprinter: Still uses CodecID in a codepath not used,
when compiled against Libav.
* bino: Currently fails to build due to a gettext version mismatch [2].
* dvswitch: Still uses CodecID (and also avcodec_encode_video, but
that is still present in FFmpeg.) [3]
* ffdiaporama: The libav10.patch introduces incompatible API
(avfilter_graph_parse).
* k3b: Still uses CodecID (and also av_close_input_file and
avcodec_decode_audio3, but these are still present in FFmpeg.)
Therefore the FFmpeg plugin has been disabled. [4]
* libextractor: Misses build-dependencies for the FFmpeg plugin. [5]
* miro: Still uses CodecID (and also fails to build with Libav 10). [6]
* mplayer2: MP_INPUT_BUFFER_PADDING_SIZE has to be increased to 32,
because FFmpeg needs 32-byte padding.
* netgen: Still uses CodecID, av_get_pict_type_char,
avcodec_alloc_context and avcodec_open, but the linking with libav
libraries has been disabled, see [7].
* renpy: Still uses AVCODEC_MAX_AUDIO_FRAME_SIZE in a codepath not
used, when compiled against Libav.
* xbmc: The '--enable-libav-compat' needs to be removed, when compiling
against FFmpeg.
* zoneminder: Misses a build-dependency on libgcrypt11-dev. [8]

As you seem to not believe me, I'm going to send all the patches to the
BTS, so that everyone can judge the amount of work for themselves.
Post by Reinhard Tartler
Making sure they work with the version of Libav's libavcodec.so
implementation.
This seems like more work, because, for example, I don't know about any
patches fixing the problems XMBC has with Libav and they don't seem to
be straightforward to fix.
Post by Reinhard Tartler
This thread has shown a couple of examples that
support this argument: Mythtv, but also mplayer that claims to work
with a system libavcodec.so, which is true as long as it matches the
version that is was built against.
MythTV shows nothing, because it uses its own fork and MPlayer works
fine with a system FFmpeg. There is no need to recompile it, except for
a major soversion bump, unless it has bugs I don't know about. Do you
know of such bugs?

I'm using a version of MPlayer compiled against FFmpeg 2.2 with the
FFmpeg libraries from 2.3 and it works for me.
Post by Reinhard Tartler
This is what makes mplayer so hard
to package, and was ultimately the reason why I requested mplayer's
removal (which is more than ironic, given that back then, I fought
with ftp-master for many years to get it included into Debian in the
first place).
I have read MPlayer's removal bug and it shows clearly that the only
reason for its removal was, that it can't be compiled with Libav and
upstream is not interested in working on this, because of the missing
features in Libav.
Post by Reinhard Tartler
On a related note: Most Libav developers are very tired of the
constant flamewars and defamation attempts that arises from FFmpeg.
I'm also very tired of flamewars in general and happy that this
discussion hasn't become one.
Post by Reinhard Tartler
Over years, Libav tries to convinced everyone by providing usable
software releases.
That didn't succeed very well, at least not for me and all the other
people for which Libav doesn't work correctly.
Post by Reinhard Tartler
Nevertheless, this particular debate is very
worrisome: The silence from the Libav camp seems to not to be taken as
consent. Quite the contrary is true.
Do you want to say here, that there are many people preferring Libav,
who haven't participated in this discussion?

This might or might not be true, but I would welcome anyone, who makes a
technical argument.
Post by Reinhard Tartler
How to proceed from here? TBH, I'm not sure. Ideally, both projects
would find some common ground and "just merge" (however that would
technically look like).
Indeed, that would be the ideal solution, ...
Post by Reinhard Tartler
However, this very debate within Debian shows
that this is unlikely to happen anytime soon: There is way to much
disagreement on very fundamental questions that require agreement
within a free software project, and the hostile and aggressive tone
the majority of participants in this debate exhibit does not help with
making progress on that front either.
... but I'm also under the impression that there is still too much bad
blood between both upstreams for this to happen.
Post by Reinhard Tartler
On Fri, Aug 8, 2014 at 2:06 PM, Andreas Cadhalpun
It would be nice, if you could also spent some time thinking about
the possibility of having both FFmpeg and Libav in Debian.
Please believe me that I did spend a lot of time thinking about this.
I firmly do not think that this is a reasonable approach. Please see
my other email on debian-devel as explanation.
To me it seems your other mail (above) doesn't mention the possibility
of having both, only:
"FFmpeg would stop tracking Libav as soon as it replaces Libav in Debian"

But if you are opposed to having both and also opposed to letting FFmpeg
replace Libav, it seems that it will be impossible to get a consensus here.

Therefore the technical committee will have to decide between:
a) having both FFmpeg and Libav in Debian
b) only having FFmpeg in Debian
c) only having Libav in Debian

As I haven't seen any convincing technical argument for only having
Libav, I doubt the technical committee's decision will be c).

Best regards,
Andreas


1: I'm getting tired of this phrase...
2: https://bugs.debian.org/757280
3: https://bugs.debian.org/747868
4: https://bugs.debian.org/739312
5: https://bugs.debian.org/755810
6: https://bugs.debian.org/748861
7: https://bugs.debian.org/751344
8: https://bugs.debian.org/745819
Ben Hutchings
2014-08-10 22:28:05 UTC
Permalink
On Sun, 2014-08-10 at 23:02 +0200, Andreas Cadhalpun wrote:
[...]
Post by Andreas Cadhalpun
* dvswitch: Still uses CodecID (and also avcodec_encode_video, but
that is still present in FFmpeg.) [3]
[...]

dvswitch was also broken by the removal of support for downscaled
decoding of DV video. I don't know whether that change is specific to
libav or was also made in FFmpeg.

Ben.
--
Ben Hutchings
Any sufficiently advanced bug is indistinguishable from a feature.
Holger Levsen
2014-08-11 08:27:07 UTC
Permalink
Hi,
Post by Ben Hutchings
dvswitch was also broken by the removal of support for downscaled
decoding of DV video. I don't know whether that change is specific to
libav or was also made in FFmpeg.
dvswitch is still broken and there is no dvswitch in jessie...

We have a daily job testing against libav from git, but that was alwayys
broken everyday in the last half year or so. Maybe it would be useful to setup
building against ffmpeg.


cheers,
Holger
Michael Niedermayer
2014-08-11 21:56:27 UTC
Permalink
Hi
[...]
Post by Reinhard Tartler
IMHO it's reasonable to expect core APIs to be upwards-compatible and keep
deprecated interfaces around for another release or two.
This is exactly what Libav is doing: The deprecation process for
symbols, APIs, enums, etc. takes *years*, because so many software
packages in Debian and else where use them, and it is so believably
painful to change them. Just have a look at the last two Libav
transitions, and the massive amount of patches it took to get packages
fixed and eventually to get Debian to the new Libav release.
Now enter FFmpeg.
FFmpeg has a significant higher release frequency, (it seems to me
about every 3-4 months), so that you would get a deprecation cycle
that is considerably less than a year. In practice, the deprecation
cycle more or less seems to match Libav's cycle, because at least
right now, FFmpeg tracks Libav's API. If that were not the case (and
I promise you FFmpeg would stop tracking Libav as soon as it replaces
Libav in Debian), I can almost guarantee [1] you that FFmpeg would
very much prefer to resume to the deprecation cycle the project
before: None, i.e., every piece of software is expected to keep up
with FFmpeg's master branch for reasons Jean-Yves outlines.
These fears are unfounded and these predictions not only do not match
reality they also lack any reason or motive for FFmpeg. We would be
shooting us in our own foot if we randomly broke API or stopped
integrating improvments

It has always been my wish to provide the best multimedia software
to the world. And sure us including all improvments and bugfixes from
Libav is in line with that.

also you have write access to FFmpeg git ...

and iam happy to work together with andreas and anyone else on
debian lifecycle releases.
And you are certainly welcome too


[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

When the tyrant has disposed of foreign enemies by conquest or treaty, and
there is nothing more to fear from them, then he is always stirring up
some war or other, in order that the people may require a leader. -- Plato
Jose Luis Rivas
2014-07-25 11:27:54 UTC
Permalink
Post by Andreas Cadhalpun
Hi all,
some of you may have noticed a weird ffmpeg package in the NEW queue[1].
In 2011 Libav[2] was forked from FFmpeg[3]. It was a time of great
uncertainty, the fork happened with much drama that didn't help making a
technical cut, and at that peculiar time Debian switched to Libav.
Hi Andreas and everyone,

FWIW, my experience with this is that I had to make my own FFmpeg
package a while ago [0] because I needed it for a project I was working
on at the moment [1].

[0] https://github.com/ghostbar/FFmpeg.deb
[1] https://github.com/ghostbar/RTSP-Streaming.js

The reason for having to package my own FFmpeg is the current libav
which is taking the space of ffmpeg seemed to conflict with every other
ffmpeg package out there, including marillat's and for my project I
actually needed ffmpeg, not libav since it didn't had the functionality.
(More specifically: the ability to take still images from an rtsp
stream).

Not having FFmpeg available in the debian repositories is a nuissance,
and certainly having libav instead which seems to be a fork yet not
having the full FFmpeg functionality and using the same package name is
worst. I didn't figured this out at first because the binary said
`ffmpeg`. Of course, I'm talking about [2] since now that seems to not
be an issue yet remains the lack of functionality.

[2] https://packages.debian.org/wheezy/ffmpeg

If the issue is that this would mean having to fix security bugs twice
then it would be reasonable to stop shipping libav and instead ship
ffmpeg, since has more functionality and AFAICS from their repos bunch
of active bug-fixing.

I honestly do not understand why ffmpeg is not in the repos nor why
there seems to be an active movement to block it.

Kind regards.
--
Jose Luis Rivas · ghostbar <http://ghostbar.co>
The Debian Project · <http://www.debian.org>
GPG · D278 F9C1 5E54 61AA 3C1E 2FCD 13EC 43EE B9AC 8C43
Niv Sardi
2014-08-05 14:49:59 UTC
Permalink
I feel the debate is going a bit on a tangent in this thread, so I'd like
to take an opportunity to recenter it a tad.​

​We have many issues that were risen in this thread, ​
​but ​
I believe that the cut has to be made by the people that we have in special
roles for; -security for security concerns, -release for release scheduling
and required transitions and our many maintainers of multimedia related
packages to know what they want to link their packages against
​​
​, and well, as it has been named, tech-ctte for technical matters that we
can't resolve in -devel.​

​In few words​
,
​this is how I understand​
the core of Andreas' plan, and has motivated many technical decisions in
the packaging.

​Many ​
​​
sensible
​ technical decisions that lead to a polite and correct way to go.

Now, our main blocker to get onto the next steps Andreas has exposed with
great detail in his original mail is that the package has been sitting in
NEW for the last
​3​
months.

As people voiced their feelings, it seems the vast majority does not
opposes having FFmpeg in experimental
​​.

So my point here is:
​Shouldn't we first get it in the archive ? ​
how can we help speed​ing
​ ​
it out of NEW ?

​The current packaging is a 'low-conflict' one, it can be easily
transitioned into replacing libav if that's what gets decided further down
the line. Hence, I see no blocker​ in getting it in.

The licencing has an ancestor with libav, and the new files have been
following the same inclusion pattern. So my guess is that the blocker may
be packaging quality, as the sponsor, I have checked it, but maybe we can
help reconfort ftpmasters with more eyes on it ?
​Thanks,​
--
Niv
Continue reading on narkive:
Loading...