LinuxLogFiles
ContentsBRTableOfContents(3) |
Introduction
One of the things which makes GNU/Linux a great operating system is that virtually anything, and everything happening on, and to the system may be logged in some manner. This information is invaluable to operating the system in an informed manner, and should be one of the first resources used in trouble-shooting system, and application issues. The Ubuntu logs can tell you almost anything you want to know, so long as you have an idea where to look first.
Your Ubuntu system provides vital information concerning events, operation, and other functions in various system log files. These log files are typically plain ASCII text in a standard log file format, and the majority of them are located in the traditional system log subdirectory /var/log. Many of these log files are generated by the system log daemon, syslogd on behalf of the system, and certain applications, while some applications generate their own logs by writing to log files located in the /var/log subdirectory.
This guide discusses several of these system log files, describes their content with examples, and further presents examples on extracting useful information from these system log files with such command-line utilities as grep and less. This guide will also discuss the system logging daemon, syslogd, its configuration, and the concept of log rotation. Additional information will be provided in the Resources section of this guide.
Target Audience
This guide is for anyone with sufficient experience with the GNU/Linux command-line, and particularly experience in executing command-line utilities, and editing system configuration files with a preferred editor.
System Logs
This section of the guide addresses so-called system logs, or logs which deal primarily with the functioning of the Ubuntu system, and not necessarily with additional applications added by the System Administrator, or users of the system. Examples of such logs include the logging of system authorization, running system daemons, system messages, and the all-encompassing system log itself, or syslog as it is known. (not to be confused with the syslogd daemon which we'll cover later).
The Authorization Log tracks usage of authorization systems, that is, all of the Ubuntu mechanisms for authorizing users which prompt for user passwords, such as the Pluggable Authentication Module (PAM) system, the sudo command, remote logins to sshd, and so on. The Authorization Log file may be accessed as: /var/log/auth.log. This log is useful for learning about user logins, and usage of the sudo command on your Ubuntu system, for example. You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt: less /var/log/auth.log Press the SPACE BAR to advance to the next page, or the ENTER key to advance one line at-a-time. The b key will scroll backwards one full page, and the q quits the less utility. Specific information may be accessed from the Authorization Log by using such commands as grep, and less. For example, to see only information in the Authorization Log pertaining to sshd logins, you might use a command such as the following at a terminal prompt: grep sshd /var/log/auth.log | less
The You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt: less /var/log/daemon.log Specific information may be accessed from the Daemon Log by using such commands as grep, and less. For example, to see only information in the Daemon Log pertaining to the MySQL daemon (mysqld), you might use a command such as the following at a terminal prompt: grep mysqld /var/log/daemon.log | less
The You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt: less /var/log/debug Specific information may be accessed from the Debug Log by using such commands as grep, and less. For example, to see only information in the Debug Log pertaining to the Advanced Configuration and Power Interface (ACPI), you might use a command such as the following at a terminal prompt: grep ACPI /var/log/debug | less
The You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt: less /var/log/kern.log Specific information may be accessed from the Kernel Log by using such commands as grep, and less. For example, to see only information in the Kernel Log pertaining to the computer's Central Processing Unit (CPU), you might use a command such as the following at a terminal prompt: grep CPU /var/log/kern.log | less
The dmesg | less You may also use the dmesg utility to examine specific information from the kernel bootup messages such as Plug and Play (pnp) messages by using a command such as the following at a terminal prompt: dmesg | grep pnp | less In relation to the Kernel Ring Buffer is the default behavior of the
The You may examine the log, from its most recent rotation point, (if applicable) one page per screen, using a command such as the following from a terminal prompt: less /var/log/messages Specific information may be accessed from the Messages Log by using such commands as grep, and less. For example, to see only information in the Messages Log pertaining to the Gnome Configuration Daemon (gconfd), you might use a command such as the following at a terminal prompt: grep gconfd /var/log/messages | less
The
In addition to the myriad of system-specific logs available on your Ubuntu system, you may also access individual logs which may be used by certain applications. If you list the contents of your
The default installation for Apache2 on Ubuntu creates a log subdirectory: With this information in mind, and a command of the tools grep, and less basic information gathering from these logs becomes possible. For example, and in terms of access, if you wished to see log records for every recorded access to your Apache2 server from the client IP address grep "82.211.81.166 /var/log/apache2/access.log | less Or, if you wished to determine if any clients using your Apache2 server were using Mac OS X, a simple command such as the following, typed into a terminal window would suffice: grep "Mac OS X /var/log/apache2/access.log | less On the other side of the coin, suppose you wished to see information from the grep "shutting down" /var/log/apache2/error.log | less You may also see all log entries which are considered errors by Apache2 with a command such as the following entered at a terminal prompt: grep error /var/log/apache2/error.log | less
The Common Unix Printing System (CUPS) uses the default error log file: less /var/log/cups/error_log Specific information may be accessed from the CUPS Log by using such commands as grep, and less. For example, to see only information in the CUPS Log pertaining to Full reloads, you might use a command such as the following at a terminal prompt: grep reload /var/log/cups/error_log | less
The less /var/log/rkhunter.log Specific information may be accessed from the Rootkit Hunter Log by using such commands as grep, and less. For example, to see only information in the Rootkit Hunter Log pertaining to Warnings, you might use a command such as the following at a terminal prompt: grep WARNING /var/log/rkhunter.log | less
The Server Message Block Protocol (SMB) server, Samba is popularly used for sharing files from your Ubuntu computer to other computers which support the SMB protocol. Samba keeps three distinct types of logs in the subdirectory log.nmbd : Logs all messages related to Samba's NETBIOS over IP functionality (the network stuff) log.smbd : Logs all messages related to Samba's SMB/CIFS functionality (the file, print, etc. sharing stuff) log.[IP_ADDRESS] : Logs messages related to requests for services from the IP address contained in the log file name, for example, log.192.168.1.1 To view all information related to Samba's networking, you could use a command such as the following at a terminal prompt: less /var/log/samba/log.nmbd If you wanted only to see information logged about Master Browsers, then you might use a command like this at a terminal prompt: grep "master browser" /var/log/samba/log.nmbd | less If you would like to see details related to the SMB functionality of Samba, you can view the appropriate log in its entirety with a command similar to the following at a terminal prompt: less /var/log/samba/log.smbd To see only messages related to the start up of the Samba server, a command like the following at a terminal prompt may be used: grep started /var/log/samba/log.smbd | less To view all the details on connections from the client system with IP address less /var/log/samba/log.192.168.99.99
The default X11 server in use with Ubuntu is the Xorg X11 server, and assuming your computer has but one display defined, it stores log messages in the file: less /var/log/Xorg.0.log Specific information may be accessed from the Xorg Log by using such commands as grep, and less. For example, to see only information in the Xorg Log pertaining to the freetype font engine, you might use a command such as the following at a terminal prompt: grep freetype /var/log/Xorg.0.log | less
Some log files found in the
The faillog
The lastlog | less
The file who
The Messages logged to syslogd usually contain important common elements, such as system hostnames, and time-stamps in addition to the specific log information from a system source, the Linux kernel, or a user application.
The detailed configuration of syslogd is beyond the scope of this guide, and the reader is encouraged to seek additional information via the Resources section of this guide for information on correctly configuring, and modifying the configuration of syslogd. The file which configures the behavior of the syslogd daemon is The configuration file is very flexible, and powerful in nature, allowing a seemingly infinite combination of logging to take place to fit the particular requirements your installation may have.
A neat utility exists in the logger tool, which allows one to place messages into the System Log (i.e. logger This Pizza from Vinnys Gourmet Rocks and you would find a line in the Used in a little more professional manner in shell scripts, you can even specify a tag the messages come from, and redirect the output standard error too. This lets you have excellent error logging in a script, such as in this example snippet: Executing this script as So, as you can see, we received the messages both via standard error, at the terminal prompt, and they also appear in our syslog!
When viewing directory listings in Authorization Log
Daemon Log
Debug Log
Kernel Log
Kernel Ring Buffer
Messages Log
System Log
Application Logs
Apache HTTP Server Logs
/var/log/access.log : Contains records of all access to the HTTP server by clients CUPS Print System Logs
Rootkit Hunter Log
Samba SMB Server Logs
X11 Server Log
Non-Human-Readable Logs
Login Failures Log
Last Logins Log
Login Records Log
System Logging Daemon (syslogd)
Configuration of syslogd
Echoing Messages to syslogd With Logger
Jan 12 23:17:02 localhost buddha: This Pizza from Vinnys Gourmet Rocks
#
# sample logger error jive
#
logmsg="/usr/bin/logger -s -t MyScript "
# announce what this script is, even to the log
$logmsg "Directory Checker FooScript Jive 1.0"
# test for the existence of Fred's home dir on this machine
if [ -d /home/fred ]; then
$logmsg "I. Fred's Home Directory Found"
else
$logmsg "E. Fred's Home Directory was NOT Found. Boo Hoo."
exit 1
fi
bumpy@butters:~$./chkdir.sh
MyScript: Directory Checker FooScript Jive 1.0
MyScript: E. Fred's Home Directory was NOT Found. Boo Hoo.
bumpy@butters:~$tail -n 2 /var/log/syslog
Jan 12 23:23:11 localhost MyScript: Directory Checker FooScript Jive 1.0
Jan 12 23:23:11 localhost MyScript: E. Fred's Home Directory was NOT Found. Boo Hoo.
Log Rotation
This guide will not cover the myriad of ways logrotate may be configured to handle the automatic rotation of any log file on your Ubuntu system, but rather the reader is encouraged to use the Resources section of this guide, and study the requisite manual pages to determine how to configure logrotate for a particular log file, and needs.
attachment:IconsPage/IconNote.png NOTE: You may also rotate system log files via the cron.daily script /etc/cron.daily/sysklogd instead of using logrotate. Actually, the utility savelog may produce unexpected results on log rotation which configuring logrotate seems to have no effect on. In those cases, you should check the cron.daily sysklogd script in /etc/cron.daily/sysklogd and read the savelog manual page to see if savelog is not in fact doing the rotation in a way that is not what you are specifying with logrotate.
Additional Tips
Some additional tips for quickly viewing logs manually, (i.e. without a log file analyzer application) which may help you expediently locate the information you require.
Just the Beginning
You may look at just the beginning of any log file by using the head command. by default, head shows the first ten lines of any text file, so for example, if you wished to see the oldest entries in your Authorization Log file, a command such as the following could be used at a terminal prompt:
head /var/log/auth.log
If ten lines is not enough, and you need to see the first twenty-five (25) lines, then use head with the -n switch as such:
head -n 25 /var/log/auth.log
Just the End
The compliment to head of course is none other than the tail command. Can you guess what tail allows you to do? Say you need the last ten lines of the Kernel log for important messages from the kernel of late. A command such as the following entered into a terminal prompt should do:
tail /var/log/kern.log
Again, and as with head, you may get more than the default ten lines of output with tail by specifying the -n switch as such:
tail -n 30 /var/log/kern.log
to see the last thirty (30) lines of the Kernel log instead.
Real-Time Tail
Another neat use for the tail command is to use it for watching a log in real-time by specifying the -f switch. For example, if you wished to watch in real-time as clients access your Apache2 server, issuing a command such as this from a terminal prompt would allow you to do so:
tail -f /var/log/apache2/access.log
You will see the log spit out, then stop, and as the Apache2 server is accessed, log entries will fly by in real-time! If you have a very busy server, they will fly by too fast for you to read them! You can use the -f switch to view any log file in this manner.
Resources
Additional information on system, and application logs, and syslogd is available via the following resources:
Local System Resources
man dmesg - manual page for the dmesg kernel ring buffer utility
man failllog - manual page for the faillog command (and also the faillog configuration file via man 5 faillog)
man grep - manual page for the grep pattern searching utility
man head - manual page for the head utility
man klogd - manual page for the kernel log daemon (klogd)
man last - manual for the last command which shows last logged in users
man less - manual page for the less paging utility
man logger - manual page for the logger command-line interface to syslog utility
man logrotate - manual page for the the logrotate utility
savelog - manual page for the savelog log file saving utility
man syslogd - manual page for the system log daemon (syslogd)
man syslog.conf - manual page for the syslogd configuration file
man tail - manual page for the tail utility
WWW Resources
[http://www.samag.com/documents/s=1146/sam0109m/0109m.htm Checking Your System Logs with awk]
[http://www.linuxvoodoo.com/resources/howtos/syslog/ Syslog - Watching Your Logs]
[http://www-128.ibm.com/developerworks/linux/library/l-roadmap5/ Windows-to-Linux roadmap: Part 5. Linux logging (IBM)]
[http://www.linuxsecurity.com/content/view/116430/151/ Sawing Linux Logs With Simple Tools]