This page is here to collect together proceedings from sessions as part of the 'Multimedia' track at the Natty UDS in Orlando, Florida.

Please add proceedings by doing the following:

• Add a new section with the name of the session.
• Add the outcomes of the session as a collection of bullet points. The goal of the proceedings is to really focus on decided outcomes, so please try to keep them crisp.

Thanks!

Proceedings

Optimize JPEG Decoding

• Agreed Major Usescases for JPEG Decoder: Browser (many small images), Image Gallery (Large Images)
• Selection of Software JPEG Decoder for optimization: IJG's Libjpeg

• JPEG Interface to be exposed: libJPEG (& OpenMAX possibly)

• Combining H/W & S/W Decoder: Decided to do it with simple GLUE layer over libJPEG.

• Runtime Selection of SoC specific execution path: Agreed to have only compile time support.
• ARM+NEON Optimizations Ideas: Agreed to not do SMP-aware codec, Agreed for doing no optimization for ARM11 and older processors.

Linaro Seeds for Multimedia WG 11.05

• If it makes sense, seed(s) could be based on the upcoming developer seed
• Research should be done to find and include tools for analysis of video/audio streams/content
• nfsclient needs to be included for easy access to test content
• Support for application logging needs to be included for those applications that have a logging feature
• There should be enough content for test / validation either as part of the image or accessible
• It was suggested that the landing teams should be the team to assemble demo materials for specific boards with the help of the Multimedia WG
• There should be enough support included to support automated testing both running and reporting results

Ubuntu One integration with Shotwell

• Make Ubuntu One's publish functionality available in Shotwell's publish options
• Make Ubuntu One's mobile photos available by default in Shotwell

Cairo backend and backend selection

https://blueprints.edge.launchpad.net/linaro-graphics-wg/+spec/multimedia-linaro-cairo-1105

• cairo is not the right place to handle runtime selection. It is up to the higher-level code to decide and use the best cairo facilities.
• We should implement a GLES2 backend, by preferably making the current experimental GL backend GL/GLES2 "agnostic". We should measure and publish the performance benefits.

Linaro Seeds for Graphics WG 11.05

• developer tools should be included in a seed with all dev packages, compilers, build essentials, automake, enough to build gles, graphical applications

• alf suggestioned the following seed layout : x11-base + dev-base -> gfx-dev + gfx-packags

• There should be two flavours, dev and test/demo (meant to boot, run simple demo)
• spandex - python based framework for gles benchmarks by nokia
• packages should include enough to build gles, applications and run them dev versions of mesa, x, freeglut, extention wrangler, dri experimental (for gallium), meego-touch (maybe), cairo, cairo-perf, gtk-perk, mesa-demos
  • could have multiple window managers etc installed but allow for easy switching for the purposes of development
• size constraints are a consideration for the purposes of a head (such as alip) which has a goal to fit into 64M flash
• automated test images should support automated test running and result reporting in general
• [action] connect with team defining development seed (tgall-foo)
• [action] need to push fix for germinate for mulitple PPAs (alf)
• [action] Jesse Barker to push graphics dev packages to tgall-foo
• [action] Jesse Barker will work gles performance tools list and supply input
• [action] check / remove SOC specific headers / images from headless, move to hwpack (tgall-foo)

Qt backend and backend selection

https://blueprints.edge.launchpad.net/linaro-graphics-wg/+spec/multimedia-linaro-qt-1105

• Some runtime selection already implemented in Qt, specifically for NEON paths.
• options of automatic selection of backends to be discussed with qt upstreams
• proof of concept to be developed that shows shows whether conflicting selection of GLES/GL in backend approach to be discussed with qt upstreams
• develop an abstraction layer that supports runtime backend selection for multiple toolkits

Skia backend and backend selection

https://blueprints.edge.launchpad.net/linaro-graphics-wg/+spec/multimedia-linaro-skia-1105

• Runtime selection of NEON will use hardware capabilities from /proc/self/auxv (recommended method), and NEON support in toolchain should also be considered for implementation.
• Proposal will be presented to upstream Skia team.
• Barring objections from the team, solution will be implemented and submitted upstream.
• GLES backend selection deemed out of scope for this topic (see Qt proceedings).

Direct rendering support for ARM-based GPUs

https://blueprints.edge.launchpad.net/linaro-graphics-wg/+spec/multimedia-linaro-dri-1105

• Kernel support for ARM-based (non-PCI-attached in general) GPUs in upstream (great news).
• Additional memory managers (most hardware vendors have their own) are potentially repellent to upstream DRM.
• Linaro graphics WG will contact maintainer for first-hand account.
• Vendors should be urged to extend existing DRM memory management framework rather than adding their own.
• No real user space DRI issues apart from dependence upon proprietary kernel interfaces.
• Any integration of vendor Xorg video driver code will be serious uphill struggle without full release (most hardware vendors maintain closed source components to their driver stacks which prevents much upstream acceptace).
• Gallium holds much promise and will continue to be tracked for performance criteria.

Optimization opportunities in compositors

https://blueprints.edge.launchpad.net/linaro-graphics-wg/+spec/multimedia-linaro-compositors-1105

• The Linaro priority list is already using OpenGL ES, and, crucially, the EGL_KHR_image_pixmap extension (avoids custom extensions or waiting for Khronos to ratify one).
• Linaro graphics WG will confirm that EGL_KHR_image_pixmap is functionally equivalent to GLX_EXT_texture_from_pixmap.
• Compiz and kwin upstreams are working on OpenGL ES backends as well.

Special Features for Touch-based Systems

The session was focused on recording brainstorms/ideas on creative ways in which we might use touch interfaces.

• using an iPad to control an Ubuntu installation (i.e., a large trackpad)
  • "lots" of hardware that can support 2-touch (1 is probably all you need for a remote) and ubuntu
    • ideacom (4 touch), joojoo (egalax), wacom, cando, mozart
  • lots of crazy ideas that are possible...
  • advanced remotes (e.g., home automation, light shows, serious multimedia)
• creating a touch app for Ubuntu to act as a remote control for TVs
• enabling auto-flip for the Dell XT2 (and similar form-factors with screens that rotate around)
• enabling auto-rotate for tablets with the appropriate hardware
  • tabuntu? lp project (tablet-screen-rotation-support)
  • support is getting better
• separate cursors for interacting with touch and mouse simultaneously
  • you can configure X to do this
  • supported via MPX in XInput 2.0
  • ideally, the cursor for the touch source would not be visible; just the mouse cursor
• Equivalent to MS Touch Pack?
  • we don't have this
  • we do have PyMT sample apps -- a good place to start
  • any ideas? share them with PyMT mail list!

Improving audio apport symptom

• Decided to go ahead and make audio apport symptom more detailed, ask more initial questions to aid triaging and in some cases help users to help themselves
• Investigate whether we can add a shortcut to the audio apport symptom from Gnome's volume control

Secure, low-latency audio that just works

• Integrate Jack and PulseAudio even more, so that PulseAudio adds ports to Jack whenver Jack is started

• Investigate feasibility of integrating a currently existing command-line prototype of recording VoIP calls

• Investigate whether we can help FFADO (FireWire Audio) to complete their work moving FFADO into standard ALSA at kernel level and how much it would take to complete the effort.

• Investigate if it's possible to make RealTimeKit hand out memlock permissions in a safe way, to keep people from giving themselves memlock permissions (which is bad from a security point of view).

Functional and performance graphics benchmarks

https://blueprints.edge.launchpad.net/linaro-graphics-wg/+spec/multimedia-linaro-benchmark-1105

• Base performance will be tracked using glmark[-es2], cairo-perf-trace, gtkperf, x11perf.
• Additional test cases (both performance and functionality) will be added based upon user experience problem reports.
• Further investigation into spandex framework for additional directed functional tests.

General Xorg Planning

• Version planning:
  • Mesa 7.10 (planned release Dec 2010)
  • Decision on staying with Xserver 1.9 or switching to 1.10 (planned release Feb 2011) postponed pending discussion with Chase Douglas about multitouch requirements.
  • libdrm / xserver-xorg-video-ati / xserver-xorg-video-intel / xserver-xorg-video-nouveau:
    • No big changes on the horizon, so plan on latest upstream releases at Feature Freeze.
• Gallium discussion:
  • Upstream has switched default r300 DRI driver from the classic r300c to the gallium r300g driver. r300g requires KMS support, so this presents problems with 3D when using usermodesetting.
  • UMS is still useful - there's an (obscure) feature missing from KMS, UMS is a useful debugging fallback and workaround for bugs in KMS.
  • Also r600c / r600g is interesting, but r600g isn't yet default upstream.
  • PoR: Support 3D by shipping both r300c+r300g and r600c+r600g with a distro-patch to select appropriately based on xorg.conf and available KMS support.
• Input redirection:
  • Compiz would like input-redirection support in the X server to support input to transformed windows (zoom, exposé, etc).
    • Currently has hacks to make at least zoom work properly, but it's not possible to make these hacks work with touch.
  • Patches which apply to current X server exist.
  • Chase to investigate patches and decide whether to push for upstream.
• xvfb discussion:
  • xvfb is possibly broken in Maverick, and this breaks build-time testsuites for some packages.
  • Security team needs this to work all the time.
  • PoR: Investigate whether xvfb tests can be added to xserver build-time checks. Talk with Debian upstream about integrating these tests into the xserver packaging.
• Unity:
  • Unity team commits to providing a Unity test-suite for X team to run before mesa / X / driver uploads.

Kubuntu and X.org

http://piratepad.net/multimedia-kubuntu-n-x

• natty: mesa 7.10
• galium
• xserver stable release, 1.9 or 1.10 or some mixture of the input layer
• x driver features are enabled at 90% ready but the driver authors can't test all edge cases
• kwin would rather have 0% of features than 90%, API call for a "completely supported list" would be good
• no emulation of shaders on CPU. fine for full screen games, really bad for kwin
• automated testing very hard for X
• QA has lots of test machines but many headless
• full screen testing needed
• kwin has scripting interface, could have javascript to move windows around etc
• if people could install test framework packaged in e.g. PPA and the framework is more granular than "it does" needs to report "dies while using this ARB shader" that would be helpful
• xorg edgers PPA available for testing. martin got a few from that and ignored them because assumed it was fault of it being development drivers. will consider differently in coming cycle.
• ensure drkoni downloads debug symbols for drivers as well as applications
• could project neon and xorg edgers be interesting to test at the same time?
• ppa purge works well to remove packages, X has no crazy maintainer scripts, downgrading not too dangerous
• hard for distros to add to blacklists
• finding problems after string freeze means can't add options to UI
• KDM crashes on logout for some drivers, respawn ought to have some way to test if needed
• can we disable compositing before logging out? best to just fix the bug in the resource code
• also happens in gnome but they don't use server reset they expect server to quit anyway so not a problem

Actions

• RAOF action: work out how to test for crash on logout bug, tell KDM to respawn if affected
• shadeslayer action: include in project neon docs how to test with xorg edgers

Colour Management in Ubuntu

• - Investigate can we get profiles for all certified hardware as a starting point?

  - [iainfarrell] Get in touch again with http://www.copyrightimage.com and talk again to Stephen Johnson - [iainfarrell] Find out how we can get in touch with right people to get ICC profiles from OEMs - [Sciri] write up summary of conversation with Alan into LPad - [iainfarrell] and [otto-chaotic] Talk to open source design agency in Miami Policus.com - [iainfarrell] How does Blender do it? When they're working as a team to ensure colour matches?

Multi-monitor experience

• Use cases:
  • Require a list of use cases. Perhaps can obtain from Intel or other providers.
  • **Zero** monitors is a use case
  • [Action] Peter Goodall to try to obtain existing use cases
• Desired behaviours:
  • Default on monitor plug-in event: suggested clone mode when connecting a second monitor - OS X and W7 appear to default to clone
  • Default to resolution of smaller device (within sane ranges), letterbox on larger device.
  • On display plugin event, pop up new GUI with a few, graphical, sane defaults presented to user.
  • Allowing user to define their preference is desirable
    • Allowing per-specific-monitor preference is desirable
      • e.g. connecting to office monitor wants replace, but connecting to projectors wants extend or clone
  • Hot-key should bring up display chooser - preferably it can be configured on the dialog what the hot key does. User can select to have the hotkey switch between a chosen set of configurations.
• Current Problems
  • With disconnection - Monitor dialog (and other windows) sometimes remain on disconnected monitor. VGA disconnect is hard to detect (reportedly OS X and W7 manage it)
    • Not all hardware/driver combinations supports this. Investigate what hardware does support it; can we turn VGA detect on here?
  • Monitors dialog often pops up on secondary (external) display of laptop. Defaulting to internal display (if not cloned) would be more logical.
  • Unity will always set the leftmost monitor to be primary.
    • Monitor dialog does not allow user to explicitly choose primary.

  • Zero monitors causes server initialization to fail. No bug found - AlanBell on irc said will test/file.

Use of type in the desktop: Semantic use of fonts in the interface

[otto-chaotic] and [mpt] to review guidelines from the competition [otto-chaotic] and [mpt] to look at selection of common apps and test sharing resulting findings [DanRabbit] and [mpt] speak to Cody about whether some sort of GTK version of the DOM inspector exists/ could be created to help with this.

Pulseaudio for ARM improvements for 1105

• improve support for low power audio in pulseaudio by adding support for big buffers that allow arm to go to sleep

GLES support in ubuntu archive

• switch armel architecture to use GLES 2.0 instead of GL by default where possible
• work on automatic GL/GLES/software backend rendering selection to allow better tuning based on hardware run on

Consolidate zero copy for gstreamer/xvideo

• define a common kernel API for allocating contiguous buffers
• improve the gstreamer and xvideo framework to allow zero copying
• add a consolidated gstreamer-omax tree and ensure zero copy is properly done

Software Codec optimization

• optimize one video decoder using fixed and NEON (VP8)
• optimize an audio encoder (AAC)

free multimedia test content

• make available test content for video and audio performance and validation

graphics and multimedia metrics gathering app

• make easy to use system metrics tool based on perf and trace events
• add support for annotating ftrace events in kernel

Produce an image suitable for set-top boxes for armel

• It makes no sense to use something unaccelerated.
• Strong desire to use something gstreamer-based rather than mythtv.
• New task will be created to define a supported set-up box application for ARM
• Remote controls via bluetooth or IR
• Requirements:
  • video play
  • media library
  • remote control UI
  • hardware acceleration
  • remote filesystem support
  • recording is target of opportunity
  • don't worry about youtube, but nice if it works