Ubuntu Wiki

Ubuntu Opportunistic Developers Week March 2010 - Building multimedia into your app with GStreamer - Laszlo Pandy - Mar 2 2010

(03:01:51 PM) laszlok: Alright I guess that's me
(03:02:02 PM) laszlok: I'm going to be talking about GStreamer
(03:02:35 PM) laszlok: I'm not sure how much you guys know already, so if you have any questions or I missed something, please ask in #ubuntu-classroom-chat
(03:02:47 PM) laszlok: [SLIDE 1]
(03:03:18 PM) laszlok: Seems the schedule in lernid isn't working
(03:04:13 PM) laszlok: I'll get a link for those who aren't using lernid as well
(03:05:08 PM) laszlok: For those that do still see the gwibber slides, is the schedule list loaded? On my lernid neither is there
(03:05:42 PM) laszlok: Lets do it manually then: http://laszlopandy.com/files/opportunistic-developer-gstreamer.pdf
(03:07:11 PM) laszlok: Thanks for reporting guys, it must be a server issue
(03:08:01 PM) laszlok: I'm gonna start now and you can follow along manually from the link above
(03:08:10 PM) laszlok: First slide!
(03:08:41 PM) laszlok: So I'm talking about adding gstreamer support to your app
(03:08:51 PM) laszlok: This means multimedia stuff
(03:08:58 PM) laszlok: audio and video all that cool jazz
(03:09:25 PM) laszlok: GStreamer can do pictures too, but most apps use separate libraries for images because of performance
(03:09:39 PM) laszlok: However I am not talking about playing event sounds like an alert
(03:10:02 PM) laszlok: those sounds which are really short and you just play, never pause, seek, etc
(03:11:01 PM) laszlok: GStreamer is overkill for that, and there are better libraries for that (which could be covered in another talk)
(03:11:14 PM) laszlok: Second slide!
(03:12:08 PM) laszlok: GStreamer is a framework for putting decoders, filters, encoders, file readers, etc. together
(03:12:35 PM) laszlok: if you are writing a decoder to integrate with ubuntu you are going to want to know about the framework aspects
(03:12:47 PM) laszlok: [SLIDE 2]
(03:13:51 PM) laszlok: but for us, in most cases we only care about GStreamer as a library
(03:14:16 PM) laszlok: and GStreamer is a very good library, it takes very little code to achieve a great deal in your application
(03:14:36 PM) laszlok: Slide three!
(03:15:57 PM) laszlok: GStreamer can automatically detect which plugin to use depending on the protocol (file://, http://, etc) the media container, codec
(03:16:14 PM) laszlok: you just have to tell it where to get the data and it works like magic
(03:17:13 PM) laszlok: another important aspect is that it spawns its own threads for each stream and provides a nice interface so we don't have to deal with it
(03:17:58 PM) laszlok: when you are playing video there is a thread for audio and one for video, but all the signals come in to your main thread like gtk callbacks; nice and clean
(03:18:09 PM) laszlok: QUESTION: How portable is it? Can I make gstreamer apps for Win/Mac?
(03:18:17 PM) laszlok: yes you can
(03:18:24 PM) laszlok: its supports many platforms
(03:19:05 PM) laszlok: I am part of the Jokosher project which uses GStreamer for an audio multitracker
(03:19:16 PM) laszlok: and we have been doing windows releases for more than a year i think
(03:20:04 PM) laszlok: the official releases don't have builds for any platform, but like the GStreamer PPA for ubuntu there is a win-builds which has all the latest releases for windows
(03:20:25 PM) laszlok: I don't have any experience using it on the Mac, but I know it is supported
(03:20:40 PM) laszlok: [SLIDE 4]
(03:20:48 PM) laszlok: okay now we get into the real stuff
(03:21:08 PM) laszlok: today's talk is mostly a high level usage of gstreamer, but i think it is important to understand a few concepts
(03:21:42 PM) laszlok: QUESTION: This is a really specific one, and may want to wait, but can I seek into a movie and display a still thumbnail?
(03:22:00 PM) laszlok: yes, remind me about that at the end and I will tell you specifically how to do it
(03:22:30 PM) laszlok: So Gstreamer is concerned with moving data
(03:22:50 PM) laszlok: data flow happens inside a pipeline
(03:23:03 PM) laszlok: there are a bunch of elements all linked together
(03:23:28 PM) laszlok: the source(s) provide data to the pipeline, and the sink(s) take it out and put it somewhere
(03:24:29 PM) laszlok: an example pipeline would start with a file source, move it through a decoder, pass it to audioconvert (which makes sure it is in the right format) and that would pass it to the Pulseaudio sink
(03:25:04 PM) laszlok: A bin is another important concept. It is just a collection of elements within a pipeline
(03:25:14 PM) laszlok: you can think of it as a pipeline inside a pipeline
(03:25:38 PM) laszlok: [SLIDE 5]
(03:26:03 PM) laszlok: here are some example sources
(03:26:12 PM) laszlok: gstreamer has lots
(03:26:49 PM) laszlok: giosrc is important because it provides all the protocols of the gnome virtual file system
(03:27:21 PM) laszlok: if you run the debugging tool gst-inspect-0.10 with no arguments in the terminal it will tell you alll the plugins you have installed
(03:27:24 PM) w1nGNUtz_ is now known as w1nGNUtz
(03:27:54 PM) laszlok: gst-inspect-0.10 pulsesrc will give you all the info about the pulse audio source
(03:28:30 PM) laszlok: [SLIDE 6]
(03:28:35 PM) laszlok: same deal with sinks
(03:30:12 PM) laszlok: QUESTION: how does changing framerates fit into the pipline?
(03:31:05 PM) laszlok: since its been brought up i will quickly cover another topic
(03:31:23 PM) laszlok: the link between each element has a thing called caps (short for capabilities)
(03:31:41 PM) laszlok: it specifies what kind of data is allowed to be sent from one element to another
(03:32:08 PM) laszlok: for example pulsesink does not want to receive ogg vorbis data, it wants raw audio
(03:32:26 PM) laszlok: so we have to feed it through the ogg vorbis decoder first
(03:33:05 PM) laszlok: it is possible to manually specify the caps of each link, so you can say you want this audio converted to 44100Hz before sending to pulsesink
(03:33:21 PM) laszlok: in the same way you can force the framerate of a video stream
(03:33:52 PM) laszlok: but if you get the caps wrong, the elements will complain that they can't link
(03:34:05 PM) laszlok: so its a bit more advanced, we won't be doing caps today
(03:34:18 PM) laszlok: [SLIDE 7]
(03:34:33 PM) laszlok: so here is an example pipeline
(03:34:49 PM) laszlok: the gstreamer language syntax has the ! meaning link to
(03:35:11 PM) laszlok: there is a debugging tool which uses this syntax, and you can test out all sorts of cool pipelines
(03:35:20 PM) laszlok: every can try:
(03:35:45 PM) laszlok: gst-launch-0.10 audiotestsrc wave=ticks ! pulsesink
(03:36:15 PM) laszlok: gst-inspect-0.10 audiotestsrc will tell you near the bottom what other types of waves you can use
(03:36:43 PM) laszlok: [SLIDE 8]
(03:37:10 PM) laszlok: as I have mentioned there is all sorts of complicated stuff going on in gstreamer
(03:37:39 PM) laszlok: but playback is a pretty simple use case, so they invented playbin2
(03:38:01 PM) laszlok: it is used by totem and many gstreamer apps who just want to play audio and video
(03:38:27 PM) laszlok: it internally manages almost everything for us
(03:39:05 PM) laszlok: reading from the right protocol, finding decoders, it even handles subtitles and can do gapless playback
(03:39:22 PM) laszlok: [SLIDE 9]
(03:39:34 PM) laszlok: so lets try a really simple playbin example using some python
(03:39:52 PM) laszlok: http://laszlopandy.com/files/playbin.html
(03:40:28 PM) laszlok: as you can see in the code, we have to import gst and create a new playbin2 object
(03:41:03 PM) laszlok: all gstreamer plugins are dynamically loaded, so you have to use the factory to create it
(03:41:30 PM) laszlok: gst.element_factory_make('playbin2') will fail if the plugin 'playbin2' is not installed
(03:41:51 PM) laszlok: the next step we set the URI
(03:42:11 PM) laszlok: then we get the bus, which allows us to monitor the pipeline through signals, just like in GTK
(03:42:35 PM) laszlok: we can attach a callback to the "message::eos" signal (EOS is end of stream)
(03:42:52 PM) laszlok: then we set the playbin to the playing state
(03:43:11 PM) laszlok: there are four states in gstreamer: NULL, READY, PAUSED and PLAYING
(03:43:32 PM) laszlok: NULL is for when you want to free the resources and destroy the object
(03:44:14 PM) laszlok: after we set the state we start the mainloop so we get receeve signals
(03:44:21 PM) laszlok: *receive
(03:44:34 PM) laszlok: any questions about this code?
(03:44:53 PM) laszlok: [SLIDE 10]
(03:45:13 PM) laszlok: QUESTION: what if it's a file not a url?
(03:45:29 PM) laszlok: do it like file:///home/laszlo/... (it has to be an absolute path)
(03:45:50 PM) laszlok: QUESTION: can you elaborate on states? what state a pipeline should have if we don't want to destroy it, but rather play next source?
(03:46:03 PM) laszlok: you can have it in whichever you like
(03:46:16 PM) laszlok: gstreamer allows the pipeline to be changed while its playing
(03:46:42 PM) laszlok: though its probably best to move it to paused, switch the uri, and put it back to playing
(03:47:08 PM) laszlok: QUESTION: what other proteries can we set with playbin.set_property()? And sourcefile is it allway an uri?
(03:47:16 PM) laszlok: gst-inspect-0.10 playbin2
(03:47:21 PM) laszlok: that will tell you about the properties
(03:47:43 PM) laszlok: playbin2 only deals with URIs, but those can be file:/// ones too
(03:47:52 PM) laszlok: QUESTION: and if we want to stop playing and restart it from beginning?
(03:48:31 PM) laszlok: you should set it to PAUSED and seek to the start (seeking is slide 17 if we get there)
(03:48:58 PM) laszlok: to clarify PAUSED means the same thing as PLAYING except that the data is not moving
(03:49:09 PM) laszlok: the data is there on the edge, but the damn is closed
(03:49:18 PM) laszlok: if you have a video in PAUSED you will see the first frame
(03:49:59 PM) laszlok: QUESTION: does using playbin also handle installing packages for missing codecs like with mp3's? Or that's something that should still be manually programmed?
(03:50:21 PM) laszlok: i dont think playbin does this itself
(03:50:40 PM) laszlok: GStreamer will send an error on the pipeline ("missing codec!")
(03:51:01 PM) laszlok: and you can catch the error and use another module (import gst.pbutils) to launch the codec install window
(03:51:30 PM) laszlok: okay everyone got that code from slide 10?
(03:51:44 PM) laszlok: its a GTK window, pretty simpel
(03:51:51 PM) laszlok: its needs some gstreamer love
(03:51:58 PM) laszlok: [SLIDE 11]
(03:52:34 PM) laszlok: http://laszlopandy.com/files/playback_interface.html
(03:52:48 PM) laszlok: [SLIDE 12]
(03:53:35 PM) laszlok: http://upload.wikimedia.org/wikipedia/commons/d/df/Hurricane_Connie_1955.ogg
(03:53:46 PM) laszlok: and back to the code http://laszlopandy.com/files/playback_interface.html
(03:54:19 PM) laszlok: so we are gonna take the playbin2 example and put it into the GUI class
(03:54:58 PM) laszlok: just make sure all code in __init__() is before the call to self.main_window.show_all()
(03:55:03 PM) laszlok: [SLIDE 13]
(03:55:35 PM) laszlok: on_finish becomes a class method, and we update the GUI when it is called
(03:55:47 PM) laszlok: whoever was asking about to restart from the beginning, this is it
(03:56:02 PM) laszlok: seek_simple, FORMAT_TIME, to position 0
(03:56:19 PM) laszlok: [SLIDE 14]
(03:56:37 PM) laszlok: this is straightforward
(03:56:56 PM) laszlok: in the button handler we either set the state to PLAYING or PAUSED
(03:57:26 PM) laszlok: right before we do gtk.main_quit() we should set the state to NULL otherwise gstreamer will print an error on the console about resources not being cleaned up properly
(03:57:34 PM) laszlok: [SLIDE 15]
(03:57:54 PM) laszlok: if you guys are building this code, it should sorta work now
(03:58:08 PM) laszlok: QUESTION: the uris in playbin need to be absolute, or I can write them like relative paths for files?
(03:58:23 PM) laszlok: I have not figured out a way to make file:/// work without absolute paths
(03:58:46 PM) laszlok: you should get the current working durectory using the os.path.abspath('.') command in python
(03:59:06 PM) laszlok: [SLIDE 16]
(03:59:28 PM) laszlok: heres the tricky part, we have to attach the video to our GTK window
(03:59:56 PM) laszlok: playbin2 has a video-sink property which allows us to switch where it sends the video
(04:00:31 PM) laszlok: we should wait until the widget is realized before connecting it, cause it might not be on the screen and video_area.window will be None
(04:00:54 PM) laszlok: then we just set the xwindow id
(04:01:14 PM) laszlok: and we're done
(04:01:22 PM) laszlok: [SLIDE 17]
(04:01:44 PM) laszlok: heres how to seek from the value on the slider
(04:01:50 PM) laszlok: [SLIDE 18]
(04:02:14 PM) laszlok: if there any questions about that, ask me
(04:02:20 PM) laszlok: [SLIDE 19]
(04:02:43 PM) laszlok:  QUESTION: How do I seek into a movie and display a still thumbnail?
(04:02:52 PM) laszlok: thanks for reminding me
(04:03:18 PM) laszlok: you should keep the state in PAUSED so that the first frame shows
(04:03:24 PM) laszlok: and then seek with FLUSH and KEY
(04:03:43 PM) laszlok: this is important because if it is plaused, there is data in the pipeline thats not moving
(04:04:03 PM) laszlok: FLUSH will tell the elements to throw out the old data and only use the data from the new position
(04:04:27 PM) laszlok: KEY_UNIT will tell the decoder to seek to a key frame, so you don't get a partially decoded blocky frame from the video
(04:04:50 PM) laszlok: QUESTION: what do that seek flags mean?
(04:05:05 PM) laszlok: is it clear from what i said on slide 18?
(04:05:17 PM) laszlok: [SLIDE 18]
(04:05:50 PM) laszlok: [SLIDE 19]
(04:06:32 PM) laszlok: if you have not seen gobject.timeout_add() before, it will call your function at the time interval given as long as the mainloop is still running
(04:06:52 PM) laszlok: 100 means 100 milliseconds, or 10 times per second
(04:07:20 PM) laszlok: if you have some other intensive operation which is blocking the mainloop it will be called much less often and your GUI will be unresponsive
(04:07:41 PM) laszlok: so every 100ms we query the position and update the slider, make sense?
(04:07:45 PM) laszlok: [SLIDE 20]
(04:08:16 PM) laszlok: here I am querying both the position and duration, because sometimes the duration changes
(04:08:34 PM) laszlok: for many audio types like mp3, the duration is an estimate
(04:09:10 PM) laszlok: there is no way to know the exact length without playing the file to the end and saving the result
(04:09:29 PM) laszlok: QUESTION: Why would the duration change? for streaming files?
(04:09:48 PM) laszlok: often with streaming files you don't know how long it is, the duration query will fail
(04:10:26 PM) laszlok: other files like mp3 i mentioned the duration is estimated by calculating it form the size of the file and the bitrate
(04:10:47 PM) laszlok: if the file is a poorly encoded variable bitrate file, the duration estimate may change
(04:11:08 PM) laszlok: gstreamer requires you specify the format you want
(04:11:21 PM) laszlok: but almost always we want TIME (which comes back in nanoseconds)
(04:11:36 PM) laszlok: you can call query bytes or percent for example
(04:11:52 PM) laszlok: [SLIDE 21]
(04:12:08 PM) laszlok: so once we have the values from the query here is how we update the slider
(04:12:26 PM) laszlok: make sure you block the on_slider_change function or that will be called with you do set_value()
(04:12:38 PM) laszlok: [SLIDE 22]
(04:12:54 PM) laszlok: http://laszlopandy.com/files/playback_interface_complete.html
(04:12:59 PM) laszlok: heres the completed code
(04:13:31 PM) laszlok: I added a little thing there to always read the file from the current directory
(04:13:50 PM) laszlok: 120 lines and you can do the basics of what totem does
(04:14:14 PM) laszlok: thanks guys, i'm just gonna answer questions now
(04:14:22 PM) laszlok: QUESTION: possible to extract/record a slice of audio/video from a file to a file ? faster than real-time ? (yes/no answer is enough)
(04:14:31 PM) laszlok: yes it is possible
(04:15:01 PM) laszlok: if you don't have an element which requires real time (like going from files to files) gstreamer will go as fast as possible
(04:15:47 PM) laszlok: to extract slices of audio/video files the elements you want are in the gnonlin package
(04:15:54 PM) laszlok: they are used by pitivi for example
(04:16:18 PM) laszlok: QUESTION: is it possible to seek to/display a still of a non-keyframe?
(04:16:36 PM) laszlok: yes, i believe this is what happens when you leave out the KEY_UNIT
(04:16:55 PM) laszlok: it will take longer to seek, because the decoder has to find a key frame and decode everything after that to get the full frame
(04:17:09 PM) laszlok: but this is what totem does for its frame-by-frame step feature
(04:17:48 PM) laszlok: QUESTION: Can you help me to find "wmap" codec for .wmv files?
(04:17:56 PM) laszlok: find me later in #jokosher and i'll see
(04:18:03 PM) laszlok: QUESTION: If I have a video in HD and only want to display it as a thumbnail, is this simple, and does gstreamer do clever stuff to provide low memory usage?
(04:18:38 PM) laszlok: gstreamer is fairly good with memory usage
(04:19:08 PM) laszlok: i believe the nautilus thumbnailer which gets those pictures of your videos uses gstreamer and just sets it to paused to extract a single frame
(04:19:24 PM) laszlok: sometimes it does use a lot of IO though
(04:20:14 PM) laszlok: okay so if the movie is playing, and you want a smaller copy of it
(04:20:41 PM) laszlok: this is just the same as when you scale down the totem window
(04:21:14 PM) laszlok: there is an element called tee to make copies of streams
(04:21:25 PM) laszlok: so you can have one small copy and one large playing at the same time even
(04:21:44 PM) laszlok: but for an HD video it sill have to decode the entire thing, then scale it down which will require a lot of cpu
(04:22:31 PM) laszlok: however the gstreamer magicians have some more tricks which are currently only prototype and use cairo to draw the scaled video directly instead of copying the HD video around
(04:22:55 PM) laszlok: you may have seen it on the planet, it sounds pretty much like what mattmole is looking for