MaverickWeb20WorkloadsSpec

Revision 2 as of 2010-05-24 22:06:17

Launchpad Entry: server-maverick-uds-web20-workloads
Created: 2010-05-24
Contributors: ClintByrum
Packages affected:

Summary

This specification outlines improvements to Maverick's support of common scalable web application (aka "Web 2.0") stacks such as memcache and gearman.

Release Note

Better support for memcached is now included in main. Packages: Memcache service (memcached), language bindings for C (libmemcached2), PHP (php5-memcache, php5-memcached), python (python-memcache), perl (libcache-memcached-perl), ruby (libmemcache-client-ruby).

gearman bindings for python, php, and java have been added to universe.

Rationale

In order to assist users deploying scalable web applications on top of Ubuntu Server we should offer strong support of common backend server technologies.

User stories

As a WebOps developer I want to deploy memcache servers and memcache enabled applications quickly with a minimum of configuration and packaging effort, knowing that they will be supported through the life cycle of the product. I identify a need for caching, install on my development environment, and then roll out to production using identical packages.
As a WebOps developer I want to write my applications to use gearman to distribute load to backend servers and complete unrelated tasks without blocking. I use apt to install the appropriate language binding and gearman-job-server.

Assumptions

n/a

Design

memcached

The memcached server is extremely stable and in widespread use. It should be promoted to main.

language bindings

libmemcached2 - low level C library, considered the best client implementation
- php5-memcached - promote to main, considered higher quality than php5-memcache
- python-libmemcached - needs packaging for universe in maverick, main in later releases
- libmemcached-ruby - needs packaging for universe in maverick, main in later releases
- libmemcached-libmemcached-perl - Memcached::libmemcached CPAN module, needs packaging, universe now, main later
  - libcache-memcached-libmemcached-perl - Cache::Memcached::libmemcached in CPAN, High level abstraction CPAN module, needs packaging, universe now, main later
non libmemcached based language bindings
- php5-memcache - PECL module for memcache access (does not conflict with php5-memcached), popularity would suggest promoting to main even though php5-memcached is considered "better".
- python-memcache - pure python memcache library - leave in universe
- libcache-memcached-perl - pure python memcache library - leave in universe
- libmemcache-client-ruby - pure ruby memcache library - leave in universe

gearman

gearman-job-server is already in universe. Perl bindings are also already in universe.

python-libgearman

pypi package. Package and add to universe.

php5-gearman

PECL module, add to universe.

gearman-java

Java client libraries for gearman, add to universe.

varnish

This will remain in universe. Latest version needs to be synced to deal with ESI bugs in 2.1.0 which was shipped with Lucid.

Implementation

Please see: server-maverick-uds-web20-workloads whiteboard for Work Items

Test/Demo Plan

"This need not be added or completed until the specification is nearing beta."

Unresolved issues

rabbitmq-stomp

The rabbitmq-stomp adapter is still considered experimental and is only available as a checkout from a mercurial server. We should produce packages when an official (even if just beta/alpha) release is made from the project.

UDS Session agenda and discussion

Discussion plan:

Web accelerators:
- - varnish: All good!
Key/value store in RAM:
- memcached (in universe dapper+):
  - promote to main: yes
  - Client libraries to promote to main: php5-memcache, python-memcache, libmemcached-tools, libcache-memcached-perl, libmemcache-client-ruby, libmemcached2.
  - only if based on libmemcached2, demote others
    - libmemcache (without the D) is stale & old - should be deleted
  - libmemcachedN is the right thing to package
    - libmemcached4 is coming
    - if we update, we need to rebuild
  - Java client library to package in universe:
- spymemcached
  - Package spymemcached in universe ?
    - not in debian either
  - compares how to Ehcache?
    - back Ehcache
web frameworks and their stacks (deployment tools):
- ruby: RoR
  - TBD with lucas
- python:
  - - django
    - already in main
    - pylons (in universe)
    - following debian, uptodate
    - turbogears (in universe, built on top of pylons)
    - uptodate, following debian
- perl: catalyst: uptodate, follow debian
- java
  - refer to seperate java session
- php
  - - symfony (1.0 in universe, deprecated upstream, clint already updating to 1.4) - zend framework (in universe, up to date) (we do have two versions: libzend-framework-php and zendframework) - cakephp (in universe, up to date)
Batch systems:
- gearman (universe):
  - lucid 0.10, debian 0.12, latest 0.13
  - gearman-job-server is the current supported upstream
  - promotions/packaging only if they depend on libgearman TBD: - python client library - php client library - java client library - perl client library
Messaging:
- AMQP: rabbitmq (main)
  - do we package the StOMP adapter? If not, do so
  - perl, ruby, python client libraries available
- ZeroMQ (0MQ)
  - In Debian since somewhat recently
  - Ensure we're following debian
HA Proxy/Nginx/Lighttpd
- Loadbalancing / director specialty
- Security concerns?
- Lighttpd
  - spare time project, community driven
  - ubuntu uptodate
- HA proxy
  - no url rewrites currently
  - outperforms apache's mod_proxy
  - 1.3.22, maverick is up-to-date
- Nginx:
  - async, very performant
  - closed development - no public source control repository, no security backports/test suite available
  - currently in universe at 0.7.65 (latest upstream)
Search/indexing:
- Solr/Lucene (discussed in java stacks) - up-to-date in universe
- sphinx popular in MySQL world
  - packaged as sphinxsearch in Universe 0.9.8 (0.9.9 current & in maverick)
- Xapian (loosing market share).
Hadoop (discussed in separate session)

CategorySpec

Ubuntu Wiki