##(see the SpecSpec for an explanation)

 * '''Launchpad Entry''': UbuntuSpec:cloud-server-n-distributed-logging
 * '''Created''':
 * '''Contributors''': 
 * '''Packages affected''':

== Summary ==

== Release Note ==

## This section should include a paragraph describing the end-user impact of this change.  It is meant to be included in the release notes of the first release in which it is implemented.  (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)

## It is mandatory.

== Rationale ==

== User stories ==

As a UEC admin I can view all logs from every physical system include in my UEC deployment in one central place so that I can easily find out and debug why some components of my UEC deployment are failing.

== Assumptions ==

== Design ==

## You can have subsections that better describe specific parts of the issue.

{{attachment:overview.png}}

Central logging is built around rsyslog with the relp protocol to support reliable message delivery. Puppet recipes are used to configure all rsyslog daemons to connect to their relevant peers.


== Implementation ==

## This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

See Work Items section of the blueprint whiteboard.

== Test/Demo Plan ==

## It's important that we are able to test new features, and demonstrate them to users.  Use this section to describe a short plan that anybody can follow that demonstrates the feature is working.  This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

## This need not be added or completed until the specification is nearing beta.

== Unresolved issues ==

## This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

== BoF agenda and discussion ==

## Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarizing what was discussed and note any options that were rejected.

=== UDS Natty discussion ===

{{{
Distributed logging for physical deployment (for UEC and more)


User story:
  As a UEC admin I can view all logs from every physical system include in my UEC deployment in one central place so that I can easily find out and debug why some components of my UEC deployment are failing.
  
 * flume:
   - java component
   - centrally configured (redundant via zookeeper) 
   - agents / forwarders / sinks / management server
   - not packaged -- low complexity to do so
   
 * central syslog: rsyslog, relp (MIR librelp), rsyslog output modules (sql Database, hdfs).
   - rsyslog
     * agent -> * can use [ udp / tcp / relp ] to forward
       * RELP Reliable Event Logging Protocol - http://www.rsyslog.com/doc/relp.html
         - supports reliable delivery end-to-end
     * "sinks" have output plugins
       * existing plugins: sql & files
     * centralized configs == config management, i.e. puppet
     * TLS enabled for TCP
     
 * Reconoiter
   - Created by OmniTI 
   - Log aggregation and introspection
   - Highly Polished
   - can be used for reporting and data presentation
   - experimental debian packaging. http://github.com/skorgu/reconnoiter-debian
   - Reasonable dependencies -- python, pcre, postgres, etc. (for the web ui)
   - Minimal dependencies for the agents.
   
   
 * real-time debugging
   - tail + grep on log "sink" -- for UEC, CLC
     - tail -f /var/log/uec/* | grep $INSTANCE_ID
     - highlighting tools with UEC knowledge -- swatch, etc.
     - Web interface for watching/querying logs
       - leverage text searching rather than grep
       - http://www.linuxjournal.com/content/centralized-logging-web-interface
       
 * scribe:
   - c++ binary
   - built on top of thrift
   
 * Swatch
 * 
 
ACTIONS:
 * Support relp in main (MIR librelp)
 * write puppet recipes to automatically configure rsyslog
 * Integrate in UEC:
   * configure central rsyslog on the ClC
   * configure aggegrator rsyslog on the CC
   * configure central logging via rsyslog on the NC, SC, Walrus
   * use syslog for all UEC components
   * write a script (grep++) to automatically track messages related to an InstanceId
 * package Reconoiter in ubuntu to use it for reporting and presentation
}}}
----
CategorySpec