I have now been working on the GStreamer integration in Fennec for some time, and it is time for a status update on it.
The integration is going well, but has been haunted by some issues, mainly to do with the different behavior of GStreamer on the PC and in the device – it is currently unknown to me, how much of this can be contributed to the fact that the version of the GStreamer library is different in the two cases, and how much is due to other factors.
My target device, the Nokia N810, comes with version 0.10.13 of the library, while the current version on my PC is 0.10.22
The first iteration of the integration was based on work done by doublec (see Bug 22540, that holds the history for the work) using playbin as the decoder.
The result in the device of using playbin, was that the audio part of the video played back as expected, but the video was following too slowly (e.g. didn’t play back at the proper frame-rate) – another issue with that solution was that the playbin used the GStreamer network routines to fetch data from the Internet, where we in Fennec/Firefox would like to use necko as the source of data.
What I did was that I wrote a native element to communicate first with Necko, and later abstracting it away from the basic necko interface to use nsMediaStream as the source of data. This element functions as the data source in the GStreamer pipeline that is build when media content should be played.
During the time I have also moved from using decodebin to decodebin2, as the folks over at #gstreamer told me that decodebin won’t be able to handle the audio playback as needed in the version found in the N810 (as a side note, the original playbin solution also uses decodebin2 internally).
Talking about audio, let me explain some of the audio issues that I have noticed – During development I have been almost exclusively testing with mpeg clips as these were the first that came up when I was looking for something to test with. There is no default GStreamer element on the N810 that can decode the audio part of these to a raw format, this means that decodebin and decodebin2 if left alone will just send a “unknown-type” signal and leave the source bin with the audio stream dangling.
I haven’t found a way to link the “unknown-type” pads to anything – but with the help from one of the guys on the #gstreamer IRC channel on freenode.net I got it working by using decodebin2 and the “autoplug-continue” signal.
The “autoplug-continue” signal is emitted every time a new source pad is found and depending on the return type from your handler of the signal it will either continue to try and decode the stream or link the pad to itself and inform you about this with the “new-decoded-pad” signal.
Different behavior and the problem with volume control
A difference in the behavior on the PC and in the target is that while on the PC elements are found by decodebin2 that can decode the audio part of an mpeg stream to a raw format such a decoder isn’t found on the N810.
On the N810 the decoding of the audio/mpeg stream is done by a special element “dspmp3sink” that also takes care of the actual audio playback – this sink isn’t considered by decodebin2 – so the trick in target is to use the “autoplug-continue” signal as described above and abandoning the autoplug process when an audio stream is detected.
There is one important difference between the PC and the N810 here thou… on the PC we get a raw audio stream that can be linked to different audio manipulating elements, like volume control etc. on the N810 it’s the mpeg audio stream we get out of the decodebin2 element, and you can’t link this stream to the volume control element (it’s expecting a raw stream of numbers it can scale, not a compressed stream).
I’m sure there is a way around this, but I’m also sure that I haven’t found it yet 🙂
Another thing about audio is that the version of the integration that I have on my computer currently is hard-coded to using the “dspmp3sink” element, if the audio format isn’t supported by this sink element, the playback will fail.
Drawing video frames
Initially I forwarded an invalidate event to the main thread for each video frame that was decoded by decodebin2 – this had an unwanted effect as the decoding and the displaying engine ran in two different threads.
The unwanted effect was that it might decode a handful of frames before the drawing thread started to draw, it would then invalidate the screen as many times as there were piled up invalidate events -not the best use of CPU cycles 🙂 Btw I’m not to say if it actually resulted in the same number of redraws as they should be coalesced until the screen is actually redrawn.
The current solution ensures that there is ever only one invalidate pending, but it looks like GStreamer is still trying to decode every video frame, which in turn takes it’s part of the CPU cycles, it would be better to skip at least the color space conversion for the frames that aren’t going to be showed anyway (in order to keep the CPU load low enough to keep the duration of the video correct and the video in sync with the audio).
One could argue that the above is expected as using a fakesink which is currently the way the video frames are extracted from the pipeline is considered a “hack” in the GStreamer docs, the recommended solution is to write a dedicated element to do this – which much be my next task 🙂
This might also fix an issue I see sometimes, where the audio is playing back, but there is absolutely no update of the screen until the very last frame of the video.