== App Developer Week -- GStreamer+Python: Multimedia Swiss Army Machete -- jderose -- Mon, Apr 12th, 2011 == {{{#!irc [19:01] thanks seiflotfy and m4n1sh for a great session! Next up: jderose will tell us all about using python and gstreamer in many interesting ways :) [19:01] Logs for this session will be available at http://irclogs.ubuntu.com/2011/04/12/%23ubuntu-classroom.html following the conclusion of the session. [19:01] Thanks everyone [19:03] okay, should i start? [19:03] yes :) [19:03] Hi everyone, hope you're having a great Ubuntu App Developer Week so far! [19:03] Okay, lets get started... [19:03] I have a short prepared intro to get you pumped up about GStreamer + Python. [19:03] Then we'll spend rest of the hour getting our hands dirty with real code, and I'll do my best to answer all of your questions. [19:04] During the intro, why don't you make sure the packages needed for the code examples are installed: [19:04] sudo apt-get install gstreamer0.10-tools python-gst0.10 gstreamer0.10-plugins-good [19:04] Aside from `gstreamer0.10-tools`, you probably already have the rest installed, but it's good to make sure we're on the same page package-wise. [19:04] I'm running Natty, but the examples should work fine under Maverick and Lucid too, and even older release. [19:05] == INTRO == [19:05] First, I'm going to share why I think GStreamer is *the* multimedia framework, is going to totally dominate in *everything* from simple playback to big production video editing. [19:05] I hope you're a bit surprised as to why I think this, because it's exciting, and I want to get you exited! [19:05] If I don't surprise you, then I assume you're already as excited as I am :) [19:05] Second, I'm going to share why I think Python is *the* language for building GStreamer apps, and correct some misconceptions I frequently hear about Python threading and GStreamer. [19:06] -- Why GStreamer? -- [19:06] Ah, I should introduce myself. My name is Jason Gerard DeRose, and I started writing pygst apps 7 years ago, back when gstreamer0.8 was the hot newness. [19:06] So I have a longtime love affair with GStreamer. [19:06] But recently I had to pick the multimedia framework for Novacut, my distributed (ala bzr/git/hg) video editor project. [19:07] Novacut isn't just a project, it's a startup, so I needed to pick something that makes good longterm strategic sense. [19:07] GStreamer was my gut feeling, but I played devils advocate with myself and looked at a number of other options. [19:07] I looked most seriously at Media Lovin' Toolkit (MLT), as Jonathan Thomas originally was using GStreamer + Gnonlin for OpenShot, and then switched to MLT out of frustration. [19:07] I believe Jonathan pointed out some legitimate weaknesses in Gnonlin, and the OpenShot development pace has been impressively quick, so you can't argue with that. [19:07] However, I still chose GStreamer without hesitation. Why? [19:08] * jderose makes "drumroll" sounds... [19:08] * GStreamer is on the Kindle [19:08] * GStreamer is on the Nokia N900 [19:08] * GStreamer is on webos phones and tablets [19:08] * GStreamer is what's getting attention from those wonderful Linaro folks [19:08] * GStreamer is on every Ubuntu desktop, along with most other desktop Linux distros [19:08] In short, I choose GStreamer because of it's economy of scale. [19:09] GStreamer is already running on everything from small to medium, and although running it at industrial scale (big) might not be that common right now... it's inevitable. [19:09] And doesn't that sound strikingly similar to something? [19:09] It does to me: the Linux kernel, running on everything from smart phones to supercomputers, everything from consumer grade to pro grade. [19:09] Once you reach that economy of scale, you're pretty unbeatable. And I believe that over the past several years GStreamer has reached that tipping point. [19:09] Nonlinear editing is easily exercising 90% of the same code paths as playback. [19:09] And from a business perspective, I'd choose something where I knew that 90% would be getting serious investment across the industry... [19:10] even if the other 10% might currently have some shortcomings compared to other options. [19:10] I believe Edward Hervey has built an excellent foundation in Gnonlin. It just needs more developers, more apps uses it, more users abusing it. [19:10] -- Why Python? -- [19:11] Why not? GStreamer gives you a lot of power, you can build arbitrarily complex pipelines. [19:11] And that's exactly the place when a simple, clear language like Python is perfect. [19:11] You want to be able to iterate quite, and write tons of tests without a lot of friction. [19:11] Now if you want to write new GStreamer plugins (say some new video filter), those should of course be written in C. [19:12] But the job of assembling a GStreamer Pipeline can get surprisingly complex, and that's a great place for Python. [19:12] Q: But wont Python make my GStreamer application slow because Python only allows on thread to run at once because of the Global Interpreter Lock (GIL)? [19:12] A: No :) [19:12] The Python GIL means only one thread at a time can *manipulate Python state*. [19:12] But an arbitrary number of threads can run at once assuming those threads are't manipulating Python state (aka pretty much everything GStreamer does). [19:13] So repeat after me: [19:13] "Python wont make my GStreamer application slow, because after I assemble and start the pipeline, Python just sits there waiting for signals from GStreamer, and GStreamer with exactly the same performance it would have it the pipeline were assembled and started in C!" [19:13] :) [19:14] == LEARNING BY DOING == [19:14] okay, is everyone ready to play with some code? [19:15] anyone needs a moment to catch up, at any point, please say so in #ubuntu-classroom-chat, which is also where you ask questions [19:15] QUESTION: So, that's because the threads are just running code from an external lib, then, rather then Python code? [19:15] chadadavis: basically, yes. [19:16] gstreamer can/will create quite a few different threads, say for video playback [19:17] and unless you wrote gstreamer plugins in python (which is possible, and handy for prototyping) [19:17] python wont actually be doing anything in any of those threads [19:17] python will just be sitting idle waiting for events from gstreamer [19:17] the normal way to use gstreamer is all asyncronous [19:18] okay, do doing stuff with multimedia, you always need a test video to work with: [19:18] http://cdn.novacut.com/jorge.ogv [19:18] :) [19:19] everyone go ahead and grab the example code here: [19:19] bzr branch lp:~jderose/+junk/machete [19:20] or you can browse it here - http://bazaar.launchpad.net/~jderose/+junk/machete/files [19:20] i didn't quite have time to get all the minimal python examples together i wanted, so i'm ganna wing it a bit, but thats okay :) [19:21] gstreamer is a graph based pipeline, very genric at it's core [19:22] the `gst-launch-0.10` command is very handy for quickly testing a pipeline, so lets look at ex1-audio-flac.sh - http://bazaar.launchpad.net/~jderose/+junk/machete/view/head:/ex1-audio-flac.sh === kevin7060 is now known as seidos [19:23] i know, not python yet, but this is a good way to see what gstreamer is going conceptually :) [19:23] so the first element in this pipeline is `filesrc`... which reads from a file, in this case "jorge.ogv" [19:24] the next element is `oggdemux`... ogg is a containing that can contain many different types of data inside: theroa video, vp8 video, vorbis audio, flac audio, etc [19:25] so a demuxer will take a container as split out individual elementary streams [19:25] in this example, were just going to split out the vorbis audio, transcode to flac [19:26] now gst-launch has some magic it dose behind the scences, so it's a bit more complex from python, where you're doing everything very explicity [19:27] now, let me introduce you to handy cool you'll use all the time if you do much with gstreamer [19:27] in a terminal, run: [19:27] gst-inspect-0.10 vorbisdec [19:28] sudo apt-get install gstreamer0.10-tools [19:28] you might have to install that ^^^ [19:28] that work for everyone? [19:29] if you scroll up in the output, you'll see something like this: [19:29] SINK template: 'sink' [19:29] Availability: Always [19:29] Capabilities: [19:29] audio/x-vorbis [19:30] vorbisdec can receive 'audio/x-vorbis', only [19:31] gstreamer has "caps" (capabilities) that describe what an element can consume (at its src pads), and what an element can produce (at its sink pads) [19:31] so when you assemble and start a pipeline, the elements do some pretty amazing dynamic negotiation [19:32] okay, back to example - http://bazaar.launchpad.net/~jderose/+junk/machete/view/head:/ex1-audio-flac.sh [19:33] the `audiorate` rate element will duplicate or drop samples in order to make the buffer timestamps match whatever the global clock of the pipeline is [19:33] it can also correct badly constructed files, or deal with issues where on formats idea of time is different than anothers [19:34] this stuff gets trick to make work all the time because so many of the media files in the wild are often slightly broken, don't comply with a spec totally [19:34] `audioconvert`, okay, now we go back to gst-inspect-0.10 [19:35] gst-inspect-0.10 vorbisdec [19:35] SRC template: 'src' [19:35] Availability: Always [19:35] Capabilities: [19:35] audio/x-raw-float [19:35] rate: [ 1, 2147483647 ] [19:35] channels: [ 1, 256 ] [19:35] endianness: 1234 [19:35] width: 32 [19:35] gst-inspect-0.10 flacenc [19:35] SINK template: 'sink' [19:35] Availability: Always [19:35] Capabilities: [19:35] audio/x-raw-int [19:35] endianness: 1234 [19:35] signed: true [19:35] width: 8 [19:35] depth: 8 [19:35] rate: [ 1, 655350 ] [19:35] channels: [ 1, 8 ] [19:36] so vorbisdec produces audio/x-raw-float, but flacenc consumes audio/x-raw-int [19:36] you might try removing the `audioconvert` from that pipeline, and you'll see that things wont work [19:37] so audioconvert sees that on one side there is audio/x-raw-flow, the other audio/x-raw-int, and it converts between the two [19:37] make sense? [19:38] http://bazaar.launchpad.net/~jderose/+junk/machete/view/head:/ex2-transcode.py [19:38] i didn't have time to trim this down, but here we go [19:39] http://bazaar.launchpad.net/~jderose/+junk/machete/view/head:/ex2-transcode.py#L236 [19:39] look at the AudioTranscoder class [19:39] this is a common pattern in pygst [19:40] there is a step you need, like trancoding audio, and you want it to me reusuable [19:40] so you but the only process into a gst.Bin, and use that element abstractly [19:40] very handy [19:40] gst.element_factory_make('queue') [19:40] this deserves special mention [19:41] when you have something like a jorge.ogv, which has audio and video, you need to use queues like this: [19:42] audio side: demux => inq => dec => enc => outq => mux [19:42] video side: audio side: demux => inq => dec => enc => outq => mux [19:43] this is because the audio and video are interleaved in the container, and if you don't do this, things will just hang because there wont be exactly enough to keep all the consumers happy [19:44] -- Getting Signals/Events from pygst -- [19:44] http://bazaar.launchpad.net/~jderose/+junk/machete/view/head:/ex2-transcode.py#L308 [19:45] chadadavis: QUESTION: So, a queue can be a mux or a demux, How does it know what's what? [19:45] will, a queue itself is neither, a queue is a type of gstreamer element [19:45] gst-inspect-0.10 queue [19:46] a queue just means that buffers can be added before the last was consumed [19:46] most of the gstreamer elements are 1-to-1: consume a buffer, do stuff, produce a buffer [19:47] self.bus = self.pipeline.get_bus() [19:47] you get messages from pygst using a "bus" [19:47] this is quite nice because it takes care of a threading issue that can be a pain... [19:48] messages from the bus are only emitted in the main thread [19:48] so your UI code can always safely manipulate the UI state based on the signal [19:49] self.bus.connect('message::eos', self.on_eos) [19:49] this signal is fired when the pipeline has completed, when say an entire file has been transcoded, rendered, played back, etc [19:49] self.bus.connect('message::error', self.on_error) [19:50] and this one when gstreamer encounters an error.... any time you build a pipeline, you'll probably have those two signals [19:50] at least those two, that is [19:50] http://bazaar.launchpad.net/~jderose/+junk/machete/view/head:/ex2-transcode.py#L331 [19:51] a Pipeline is sort of the main containing for all the gstreamer elements you chain together [19:51] There are 10 minutes remaining in the current session. [19:51] so any element that is linked into the chain *must* be in the pipeline [19:52] murphy: QUESTION: what about progress events? [19:52] good question :) [19:52] so gstreamer doesn't have intrinsic progress events [19:53] so what you do is great a gobject timeout that fires every 1 second or whatever [19:53] and then you query gstreamer to figure out where it is the the pipeline [19:53] you would to this for a seek bar for audio/video playback [19:53] or to get progress for transcoding [19:54] http://bazaar.launchpad.net/~jderose/+junk/machete/view/head:/ex2-transcode.py#L348 [19:54] i know the times about up, but i want to talk about states a bit [19:54] gst.STATE_NULL - no resources have been alocated at all [19:55] gst.STATE_READY - plugins are ready, but they haven't actually touched any data, allocated buffers [19:56] gst.STATE_PAUSED - the first buffers have been consumed, pipeline is negotiated [19:56] There are 5 minutes remaining in the current session. [19:56] gst.STATE_PLAYING - the loop is running, all the elements are consuming, producing, doing their thing [19:57] so to query the pipeline at all, it must be in at least gst.STATE_PAUSED [19:57] well, that's about time [19:57] sorry if this was a bit rough - this is my first time doing a session like this :) [19:58] i'm going to continue to work on that example repo, make it more useful [19:58] so thanks everyone, and enjoy all the rest of the sessions! :) }}}