LinuxLogFiles

Revision 2 as of 2006-02-23 23:21:58

Clear message

Introduction

One of the things which makes GNU/Linux a great operating system is that virtually anything, and everything happening on, and to the system may be logged in some manner. This information is invaluable to operating the system in an informed manner, and should be one of the first resources used in trouble-shooting system, and application issues. The Ubuntu logs can tell you almost anything you want to know, so long as you have an idea where to look first.

Your Ubuntu system provides vital information concerning events, operation, and other functions in various system log files. These log files are typically plain ASCII text in a standard log file format, and the majority of them are located in the traditional system log subdirectory /var/log. Many of these log files are generated by the system log daemon, syslogd on behalf of the system, and certain applications, while some applications generate their own logs by writing to log files located in the /var/log subdirectory.

This guide discusses several of these system log files, describes their content with examples, and further presents examples on extracting useful information from these system log files with such command-line utilities as grep and less. This guide will also discuss the system logging daemon, syslogd, its configuration, and the concept of log rotation. Additional information will be provided in the Resources section of this guide.

Target Audience

This guide is for anyone with sufficient experience with the GNU/Linux command-line, and particularly experience in executing command-line utilities, and editing system configuration files with a preferred editor.

System Logs

This section of the guide addresses so-called system logs, or logs which deal primarily with the functioning of the Ubuntu system, and not necessarily with additional applications added by the System Administrator, or users of the system. Examples of such logs include the logging of system authorization, running system daemons, system messages, and the all-encompassing system log itself, or syslog as it is known. (not to be confused with the syslogd daemon which we'll cover later).

Authorization Log

The Authorization Log tracks usage of authorization systems, that is, all of the Ubuntu mechanisms for authorizing users which prompt for user passwords, such as the Pluggable Authentication Module (PAM) system, the sudo command, remote logins to sshd, and so on. The Authorization Log file may be accessed as: /var/log/auth.log. This log is useful for learning about user logins, and usage of the sudo command on your Ubuntu system, for example.

You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt:

less /var/log/auth.log

Press the SPACE BAR to advance to the next page, or the ENTER key to advance one line at-a-time. The b key will scroll backwards one full page, and the q quits the less utility.

Specific information may be accessed from the Authorization Log by using such commands as grep, and less. For example, to see only information in the Authorization Log pertaining to sshd logins, you might use a command such as the following at a terminal prompt:

grep sshd /var/log/auth.log | less

Daemon Log

The Daemon Log exists at: /var/log/daemon.log and contains information specific to running system, and application daemons such as the Gnome Display Manager daemon, (gdm) the Bluetooth HCI daemon, (hcid) or the MySQL Relational Database Management System (RDBMS) daemon, (mysqld). This log is useful for gaining information about running daemons, and for trouble-shooting problems with a particular daemon.

You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt:

less /var/log/daemon.log

Specific information may be accessed from the Daemon Log by using such commands as grep, and less. For example, to see only information in the Daemon Log pertaining to the MySQL daemon (mysqld), you might use a command such as the following at a terminal prompt:

grep mysqld /var/log/daemon.log | less

Debug Log

The Debug Log exists at: /var/log/debug and provides detailed, debug messages from the Ubuntu system, and applications which log to syslogd at the DEBUG level. These messages are useful for debugging issues with everything from hardware drivers, to server daemons.

You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt:

less /var/log/debug

Specific information may be accessed from the Debug Log by using such commands as grep, and less. For example, to see only information in the Debug Log pertaining to the Advanced Configuration and Power Interface (ACPI), you might use a command such as the following at a terminal prompt:

grep ACPI /var/log/debug | less

Kernel Log

The Kernel Log at /var/log/kern.log provides a detailed log of messages from the Ubuntu Linux kernel. These messages may prove useful for trouble-shooting a new, or custom-built kernel for example.

You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt:

less /var/log/kern.log

Specific information may be accessed from the Kernel Log by using such commands as grep, and less. For example, to see only information in the Kernel Log pertaining to the computer's Central Processing Unit (CPU), you might use a command such as the following at a terminal prompt:

grep CPU /var/log/kern.log | less

Kernel Ring Buffer

The Kernel Ring Buffer is not really a log file per se, but rather an area in the running kernel which may be queried for kernel bootup messages via the dmesg utility. You may see all kernel bootup messages by using a command such as the following at a terminal prompt:

dmesg | less

You may also use the dmesg utility to examine specific information from the kernel bootup messages such as Plug and Play (pnp) messages by using a command such as the following at a terminal prompt:

dmesg | grep pnp | less

In relation to the Kernel Ring Buffer is the default behavior of the /etc/init.d/bootmisc.sh system initialization script is to use the dmesg command to log all bootup messages to the file: /var/log/dmesg as well. This file may be used as any other log file for examining Kernel bootup messages via commands grep, less, and others.

Messages Log

The Messages Log contains informational messages from applications, and system facilities, and is available at: /var/log/messages. This log is useful for examining message output from applications, and system facilities which log to the syslog/sysklog daemon at the INFO level.

You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt:

less /var/log/messages

Specific information may be accessed from the Messages Log by using such commands as grep, and less. For example, to see only information in the Messages Log pertaining to the Gnome Configuration Daemon (gconfd), you might use a command such as the following at a terminal prompt:

grep gconfd /var/log/messages | less

System Log

The System Log typically contains the greatest deal of information by default about your Ubuntu system. It is located at: /var/log/syslog, and may contain information other logs do not. You may wish to consult the System Log when you are unable to locate the desired log information in another log.

Application Logs

In addition to the myriad of system-specific logs available on your Ubuntu system, you may also access individual logs which may be used by certain applications. If you list the contents of your /var/log subdirectory, you will see familiar names of applications you may have installed, such as /var/log/apache2 representing the logs for the Apache 2 Hyper Text Transport Protocol (HTTP) server, or /var/log/samba, which contains the logs for the Samba Server Message Block (SMB) server. This section of the guide introduces some specific examples of application logs, and information contained within them.

Apache HTTP Server Logs

The default installation for Apache2 on Ubuntu creates a log subdirectory: /var/log/apache2, and within this subdirectory, are two log files with two distinct purposes:

  • /var/log/access.log : Contains records of all access to the HTTP server by clients

  • /var/log/error.log : Contains records of all error conditions reported by the HTTP server

With this information in mind, and a command of the tools grep, and less basic information gathering from these logs becomes possible.

For example, and in terms of access, if you wished to see log records for every recorded access to your Apache2 server from the client IP address 82.211.81.166, and display the results as one page per screen, you would simply use a command such as the following at a terminal prompt:

grep "82.211.81.166 /var/log/apache2/access.log | less

Or, if you wished to determine if any clients using your Apache2 server were using Mac OS X, a simple command such as the following, typed into a terminal window would suffice:

grep "Mac OS X /var/log/apache2/access.log | less

On the other side of the coin, suppose you wished to see information from the /var/log/apache2/error.log. This log can be used to search for instances of the Apache2 server being shut down, by using a command such as the following from a terminal prompt:

grep "shutting down" /var/log/apache2/error.log | less

You may also see all log entries which are considered errors by Apache2 with a command such as the following entered at a terminal prompt:

grep error /var/log/apache2/error.log | less

CUPS Print System Logs

The Common Unix Printing System (CUPS) uses the default error log file: /var/log/cups/error_log to store informational and error messages. If you need to solve a printing issue in Ubuntu, then this log may be a good place to start. You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt:

less /var/log/cups/error_log

Specific information may be accessed from the CUPS Log by using such commands as grep, and less. For example, to see only information in the CUPS Log pertaining to Full reloads, you might use a command such as the following at a terminal prompt:

grep reload /var/log/cups/error_log | less

Rootkit Hunter Log

The Rootkit Hunter utility (rkhunter) checks your Ubuntu system for backdoors, sniffers, and so-called "rootkits", which are all signs of compromise of your system. The log which rkhunter uses is located at: /var/log/rkhunter.log You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt:

less /var/log/rkhunter.log

Specific information may be accessed from the Rootkit Hunter Log by using such commands as grep, and less. For example, to see only information in the Rootkit Hunter Log pertaining to Warnings, you might use a command such as the following at a terminal prompt:

grep WARNING /var/log/rkhunter.log | less

Samba SMB Server Logs

The Server Message Block Protocol (SMB) server, Samba is popularly used for sharing files from your Ubuntu computer to other computers which support the SMB protocol. Samba keeps three distinct types of logs in the subdirectory /var/log/samba:

  • log.nmbd : Logs all messages related to Samba's NETBIOS over IP functionality (the network stuff)

  • log.smbd : Logs all messages related to Samba's SMB/CIFS functionality (the file, print, etc. sharing stuff)

  • log.[IP_ADDRESS] : Logs messages related to requests for services from the IP address contained in the log file name, for example, log.192.168.1.1

To view all information related to Samba's networking, you could use a command such as the following at a terminal prompt:

less /var/log/samba/log.nmbd

If you wanted only to see information logged about Master Browsers, then you might use a command like this at a terminal prompt:

grep "master browser" /var/log/samba/log.nmbd | less

If you would like to see details related to the SMB functionality of Samba, you can view the appropriate log in its entirety with a command similar to the following at a terminal prompt:

less /var/log/samba/log.smbd

To see only messages related to the start up of the Samba server, a command like the following at a terminal prompt may be used:

grep started /var/log/samba/log.smbd | less

To view all the details on connections from the client system with IP address 192.168.99.99 you could use a command such as the following at a terminal prompt:

less /var/log/samba/log.192.168.99.99

X11 Server Log

The default X11 server in use with Ubuntu is the Xorg X11 server, and assuming your computer has but one display defined, it stores log messages in the file: /var/log/Xorg.0.log. This log is helpful for diagnosing issues with your X11 environment. You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt:

less /var/log/Xorg.0.log

Specific information may be accessed from the Xorg Log by using such commands as grep, and less. For example, to see only information in the Xorg Log pertaining to the freetype font engine, you might use a command such as the following at a terminal prompt:

grep freetype /var/log/Xorg.0.log | less

Non-Human-Readable Logs

Some log files found in the /var/log subdirectory are designed to be readable by applications, and not necessarily by humans. Some examples of such log files which appear in /var/log follow.

Login Failures Log

The Login Failures Log located at: /var/log/faillog is actually designed to be parsed, and output by the faillog command. For example, to print recent login failures using the faillog command, simply enter the following at a terminal prompt:

faillog

Last Logins Log

The Last Logins Log at: /var/log/lastlog should not typically be parsed, and examined by humans, but rather should be used in conjunction with the lastlog command. For example to see a listing of logins with the lastlog command, displayed one page per screen with the less command, use the following command at a terminal prompt:

lastlog | less

Login Records Log

The file /var/log/wtmp contains login records, but unlike /var/log/lastlog above, /var/log/wtmp is not used to show a list of recent logins, but is instead used by other utilities such as the who command to present a listed of currently logged in users. For example, if you wish to see who is currently logged in on the machine you are currently using, issue the following at a terminal prompt:

who

System Logging Daemon (syslogd)

The System Logging Daemon, or syslogd (also known as sysklogd) is a system daemon, or special application which executes silently, in the background and does good things for your system. Specifically, syslogd awaits logging messages from numerous system, and application sources, and routes the messages to their proper target, be it a standard log file, a First In First Out (FIFO) pipe for use by a log analyzation application, or even across the network to another system's syslogd.

Messages logged to syslogd usually contain important common elements, such as system hostnames, and time-stamps in addition to the specific log information from a system source, the Linux kernel, or a user application.

Configuration of syslogd

The detailed configuration of syslogd is beyond the scope of this guide, and the reader is encouraged to seek additional information via the Resources section of this guide for information on correctly configuring, and modifying the configuration of syslogd. The file which configures the behavior of the syslogd daemon is /etc/syslog.conf and consists primarily of two fields, the selector, and the action. The selector field consists of a facility, to be logged, such as for example the auth facility which deals with authorization, and a priority, or level to log such information at, such as info, or warning which would log all messages at the informational priority and higher, or only at the warning level and higher respectively. The action field consists of a target for the log information, such as a standard log file (i.e. /var/log/syslog), or the hostname of a remote computer to send the log information to (e.g. @myotherubuntu).

The configuration file is very flexible, and powerful in nature, allowing a seemingly infinite combination of logging to take place to fit the particular requirements your installation may have.

Echoing Messages to syslogd With Logger

A neat utility exists in the logger tool, which allows one to place messages into the System Log (i.e. /var/log/syslog) arbitrarily. This is a powerful tool, which you may use in Administrative scripts, such as Perl, or shell scripts to provide them with standard logging capabilities, or you may use it just to place things in the system log as needed. For example, assume your user name is buddha and you would like to enter a message into the syslog about a particularly delicious pizza you're eating, you could use a command such as the following at a terminal prompt:

logger This Pizza from Vinnys Gourmet Rocks

and you would find a line in the /var/log/syslog file such as this afterward:

Jan 12 23:17:02 localhost buddha: This Pizza from Vinnys Gourmet Rocks

Used in a little more professional manner in shell scripts, you can even specify a tag the messages come from, and redirect the output standard error too. This lets you have excellent error logging in a script, such as in this example snippet:

#
# sample logger error jive
#
logmsg="/usr/bin/logger -s -t MyScript "

# announce what this script is, even to the log
$logmsg "Directory Checker FooScript Jive 1.0"

# test for the existence of Fred's home dir on this machine
if [ -d /home/fred ]; then
   $logmsg "I. Fred's Home Directory Found"
else
   $logmsg "E. Fred's Home Directory was NOT Found. Boo Hoo."
   exit 1
fi

Executing this script as chkdir.sh on the machine butters where Fred does not have a home directory, /home/fred, gives the following results:

bumpy@butters:~$./chkdir.sh
MyScript: Directory Checker FooScript Jive 1.0
MyScript: E. Fred's Home Directory was NOT Found. Boo Hoo.
bumpy@butters:~$tail -n 2 /var/log/syslog
Jan 12 23:23:11 localhost MyScript: Directory Checker FooScript Jive 1.0
Jan 12 23:23:11 localhost MyScript: E. Fred's Home Directory was NOT Found. Boo Hoo.

So, as you can see, we received the messages both via standard error, at the terminal prompt, and they also appear in our syslog!

Log Rotation

When viewing directory listings in /var/log or any of its subdirectories, you may encounter log files with names such as daemon.log.0, daemon.log.1.gz, and so on. What are these log files? They are rotated log files. That is, they have automatically been renamed after a predefined time-frame, and a new original log started. After even more time the log files are compressed with the gzip utility as in the case of the example daemon.log.1.gz. The purpose of log rotation is to archive and compress old logs so that they consume less disk space, but are still available for inspection as needed. What handles this functionality? Why, the logrotate command of course! Typically, logrotate is called from the system-wide cron script /etc/cron.daily/logrotate, and further defined by the configuration file /etc/logrotate.conf.

This guide will not cover the myriad of ways logrotate may be configured to handle the automatic rotation of any log file on your Ubuntu system, but rather the reader is encouraged to use the Resources section of this guide, and study the requisite manual pages to determine how to configure logrotate for a particular log file, and needs.

attachment:IconsPage/IconNote.png NOTE: You may also rotate system log files via the cron.daily script /etc/cron.daily/sysklogd instead of using logrotate. Actually, the utility savelog may produce unexpected results on log rotation which configuring logrotate seems to have no effect on. In those cases, you should check the cron.daily sysklogd script in /etc/cron.daily/sysklogd and read the savelog manual page to see if savelog is not in fact doing the rotation in a way that is not what you are specifying with logrotate.

Additional Tips

Some additional tips for quickly viewing logs manually, (i.e. without a log file analyzer application) which may help you expediently locate the information you require.

Just the Beginning

You may look at just the beginning of any log file by using the head command. by default, head shows the first ten lines of any text file, so for example, if you wished to see the oldest entries in your Authorization Log file, a command such as the following could be used at a terminal prompt:

head /var/log/auth.log

If ten lines is not enough, and you need to see the first twenty-five (25) lines, then use head with the -n switch as such:

head -n 25 /var/log/auth.log

Just the End

The compliment to head of course is none other than the tail command. Can you guess what tail allows you to do? Say you need the last ten lines of the Kernel log for important messages from the kernel of late. A command such as the following entered into a terminal prompt should do:

tail /var/log/kern.log

Again, and as with head, you may get more than the default ten lines of output with tail by specifying the -n switch as such:

tail -n 30 /var/log/kern.log

to see the last thirty (30) lines of the Kernel log instead.

Real-Time Tail

Another neat use for the tail command is to use it for watching a log in real-time by specifying the -f switch. For example, if you wished to watch in real-time as clients access your Apache2 server, issuing a command such as this from a terminal prompt would allow you to do so:

tail -f /var/log/apache2/access.log

You will see the log spit out, then stop, and as the Apache2 server is accessed, log entries will fly by in real-time! If you have a very busy server, they will fly by too fast for you to read them! You can use the -f switch to view any log file in this manner.

Resources

Additional information on system, and application logs, and syslogd is available via the following resources:

Local System Resources

  • man dmesg - manual page for the dmesg kernel ring buffer utility

  • man failllog - manual page for the faillog command (and also the faillog configuration file via man 5 faillog)

  • man grep - manual page for the grep pattern searching utility

  • man head - manual page for the head utility

  • man klogd - manual page for the kernel log daemon (klogd)

  • man last - manual for the last command which shows last logged in users

  • man less - manual page for the less paging utility

  • man logger - manual page for the logger command-line interface to syslog utility

  • man logrotate - manual page for the the logrotate utility

  • savelog - manual page for the savelog log file saving utility

  • man syslogd - manual page for the system log daemon (syslogd)

  • man syslog.conf - manual page for the syslogd configuration file

  • man tail - manual page for the tail utility

WWW Resources

[http://www.samag.com/documents/s=1146/sam0109m/0109m.htm Checking Your System Logs with awk]

[http://www.linuxvoodoo.com/resources/howtos/syslog/ Syslog - Watching Your Logs]

[http://www-128.ibm.com/developerworks/linux/library/l-roadmap5/ Windows-to-Linux roadmap: Part 5. Linux logging (IBM)]

[http://www.linuxsecurity.com/content/view/116430/151/ Sawing Linux Logs With Simple Tools]


CategoryDocumentation