• 1 Solving the real problems of Multimedia
  • 1.1 Defining the goal
  • 1.2 How GStreamer address the true needs of developers
  • 1.3 What still needs to be done
  • 1.4 Other issues
  • 1.5 What others are doing

Warning this text is meant as a bit of a controversial debate starter about what should be the goals for something like the The Linux Foundation multimedia working group. It attacks the issue of multimedia deficiencies under Linux from a completely different direction than what has been done so far.

Be aware that I, Christian Schaller, the author of this text has been involved with the development of GStreamer for the last seven years and is currently working for Fluendo, a company doing various kinds of bussiness around the GStreamer framework. So I am not pretending to be 'neutral' on this issue..

The discussion so far have either been oriented around sound systems or about vague notions of shared API's and of course talks about standarisation.

Personally I feel that that debate is not the correct debate to have and that we are instead ending up doing a lot of talk and work around solutions of little interest. The current approach only lets us look backwards instead of forwards. The focus I will advocate here is very different as it is made on the assumption that the problem is not that we do not know what is needed to 'solve' multimedia, cause I think we do, but it is simply a question of resources and focus.

My basic claim is that application developers today want more than low level system API's. Further clarifying the API's of ALSA, V4L2 or OSS do have value, but I honestly do not think that this answers the needs of developers out there. Instead what developers today do want is a feature complete and unified high level multimedia API, like DirectShow and Quicktime.

In the GStreamer community we are trying to address these issues every day by adding support for more and more related technologies, with improved RTP and DVB support being among the most recent. In addition to implementing the functionality people report missing, we are working with the community to implement a long range of applications ranging from music synthesizers, audio and video playback applications, browser plugins, media center solutions, audio editors, video editors, streaming servers and so on. All using GStreamer. Because in order to meet the challenges of tomorrow we need a framework that can address all these types of applications, not just deal with playback for instance.

In the GStreamer community we do continuous license vetting and checking and work with both upstream and downstream developer on re-licensing, to ensure that our stack stays LGPL licensed and that applications using GStreamer can be shipped even with binary-only plugins or plugins which are open source, but have non-GPL compatible licenses.

We also work with embedded systems makers trying to ensure that GStreamer stays small and modular enough to fit their use cases, be that set-top boxes or mobile devices like cell phones.

We have been working with the major Linux distributions for many years now to ensure we package our software in a way that allow them to easily include GStreamer without having to worry about patent and copyright issues. This has produced the the current set of packages with GStreamer core, base and good being things we feel should be safe from known patent and licensing issues. While modules which have licensing or potential patent issues are packaged in the ugly, bad and ffmpeg modules.

If we look at the Linux/Unix desktop market today I feel confident to say that there are two kind of multimedia applications being currently made. There are those that do 'everything' themselves and those that use GStreamer. The few competitors GStreamer might have, are either much more narrowly targeted (ie. playback only) or only used by a very few application and are thus highly immature.

GStreamer is the only widely used framework out there which has made any serious attempts at integrating with the linux low-level driver and API stack and also tried to make it easier for the configuration of the system to be integrated with that of the desktop. The now widely shared GConf keys under GNOME is a testament to that.

While GStreamer is the best linux/unix solution out there today that doesn't mean we are great for every usecase yet. The robustness of the framework in regards to editing applications for instance still needs work, but we are confident that our current effort in the area with the aid of the communities surrounding projects like Jokosher and Pitivi will help us nail those down.

Our playback autoplugger needs to be even more flexible than it is today in order to address situations like supporting output options requiring encoded data like certain DSP decoder or the common usecase of S/PDIF ports.

We still need to design and implement a good way to automatically deal with interlaced media in GStreamer. Currently the application developer has to manually plug a deinterlacer into the pipeline.

In the context of interoperability we are writing a GStreamer backend for Phonon to ensure that KDE4 can use GStreamer for its basic media playback needs. This work has so far gone much slower than we hoped, but hopefully a first working version can be released soon.

But these issues and more is not the problem of lack of standards, but the lack of implementations.

So are there other issues that needs addressing outside GStreamer? Sure, there are a few concrete problems in the sound driver layer for instance, like dmix being bad, but once again this is a problem of implementation and coding and not of lack of standards.

At Fluendo, the company I work for we try to solve the business issues surrounding the use of multimedia on Linux by licensing and developing codec plugins for GStreamer to ensure corporations that they have a legally vetted solution available.

I attended the CE Linux forum some weeks ago and the question asked there was 'what projects could we invest some money/developer time in to make things progress further in regards to embedded linux multimedia'. Which is the question I think the DTL Sound and Multimedia workgroup need to start asking too, if it wants to make a difference.

If the DTL can get even more people and corporations to help out with GStreamer development in one way or the other we can reach our shared goal much quicker. The goal of having a competitive Linux multimedia stack. A good example here is our recently much improved RTP/RTSP support which have come about through an indirect collaboration between Collabora, Fluendo, Intel, Nokia and Axis.