Launchpad Entry: cloud-server-n-distributed-logging
Created:
Contributors:
Packages affected:
Summary
Release Note
Rationale
User stories
As a UEC admin I can view all logs from every physical system include in my UEC deployment in one central place so that I can easily find out and debug why some components of my UEC deployment are failing.
Assumptions
Design
Central logging is built around rsyslog with the relp protocol to support reliable message delivery. Puppet recipes are used to configure all rsyslog daemons to connect to their relevant peers.
Implementation
See Work Items section of the blueprint whiteboard.
Test/Demo Plan
Unresolved issues
BoF agenda and discussion
UDS Natty discussion
Distributed logging for physical deployment (for UEC and more) User story: As a UEC admin I can view all logs from every physical system include in my UEC deployment in one central place so that I can easily find out and debug why some components of my UEC deployment are failing. * flume: - java component - centrally configured (redundant via zookeeper) - agents / forwarders / sinks / management server - not packaged -- low complexity to do so * central syslog: rsyslog, relp (MIR librelp), rsyslog output modules (sql Database, hdfs). - rsyslog * agent -> * can use [ udp / tcp / relp ] to forward * RELP Reliable Event Logging Protocol - http://www.rsyslog.com/doc/relp.html - supports reliable delivery end-to-end * "sinks" have output plugins * existing plugins: sql & files * centralized configs == config management, i.e. puppet * TLS enabled for TCP * Reconoiter - Created by OmniTI - Log aggregation and introspection - Highly Polished - can be used for reporting and data presentation - experimental debian packaging. http://github.com/skorgu/reconnoiter-debian - Reasonable dependencies -- python, pcre, postgres, etc. (for the web ui) - Minimal dependencies for the agents. * real-time debugging - tail + grep on log "sink" -- for UEC, CLC - tail -f /var/log/uec/* | grep $INSTANCE_ID - highlighting tools with UEC knowledge -- swatch, etc. - Web interface for watching/querying logs - leverage text searching rather than grep - http://www.linuxjournal.com/content/centralized-logging-web-interface * scribe: - c++ binary - built on top of thrift * Swatch * ACTIONS: * Support relp in main (MIR librelp) * write puppet recipes to automatically configure rsyslog * Integrate in UEC: * configure central rsyslog on the ClC * configure aggegrator rsyslog on the CC * configure central logging via rsyslog on the NC, SC, Walrus * use syslog for all UEC components * write a script (grep++) to automatically track messages related to an InstanceId * package Reconoiter in ubuntu to use it for reporting and presentation