Logging, Log File Rotation, and Syslog Tutorial

Logging Overview:

To identify problems and trends, and to trouble-shoot them, requires observing events over a period of time (historical monitoring).  Since it is generally impossible to observe all events as they occur, most daemons record important events to files known as log files.  Log files are used for audits, for evidence in legal actions, for incident response, to reduce liability, and for various legal and regulatory compliance reasons.  Email logs can alert you to spam problems.  Web logs may be useful for marketing and website design, and so on.  When upgrading or deploying new (or newly configured) services, log data can be valuable in finding problems quickly.

Early services managed their own files.  Today most (but not all!) can use the logging daemon syslog to collect, identify (i.e., host, command name, and process ID), time-stamp, filter, store, alert, and forward logging data.  Syslog has the added benefit of somewhat standardizing log file formats, making it much easier to examine log data with various standard tools.  (Syslog was developed first for BSD, but there was no standard for it, resulting in many incompatible “syslog”s.)  However some daemons may need to have their default logging configuration changed to have them take advantage of syslog (or indeed, to provide any log data at all).

An important part of a system administrator's job is to check regularly various log files.  Be sure to learn where your hosts keep their log files, as the directory is different for different flavors of *nix.

Log files need to be examined or they are useless.  However it would be foolish to try to read all log data, all the time.  Since it is impossible to know in advance what log data will be useful, you end up collecting far more than any human (or sys admin) can possibly read and understand.  This can be partially managed with log alerting and parsing tools.  Such tools often alert you to unusual log entries, and merely summarize the rest.  Usually summaries of the data are sufficient to alert you to potential problems, at which time you would then examine the relevant log entries.  Some of these tools (also known as data reduction tools) include logwatch, logcheck, swatch, logsurfer, and SEC.  Other utilities include GUI tools to examine and mange log files, and standard text processing tools such as grep, tail, tail -f, and less (for example:  grep service logFile |grep date |less).

The default logwatch setup on Fedora works fine, but doesn't examine all logfiles.  Which service log files are monitored is controlled by /etc/logwatch/conf/* files.  Each service to be monitored (other than the defaults) have conf files in conf/services.

A bunch of sample conf files you can use can be found in /usr/share/logwatch/default.conf/services/*.conf.  Try copying sendmail.conf to /etc/logwatch/conf/services/.  To get more accurate details, crank up sendmail's LogLevel setting from (the default of) 9 to 15 (see /etc/mail/sendmail.{cf,mc}).  To get more detail, change the value of “detail” in logwatch's sendmail.conf file from (the default of) 3 to 4 (max is 10 for most detail possible).

Syslog standards are over 20 years old (RFC-3164) and many issues have surfaced with them (see below for a discussion of these).  New IETF standards (RFCs) for syslog are being developed (RFC-3164 was replaced in 2009 with RFC-5424) to address security issues and other syslog shortcomings.

A sys admin must address the problems of the current syslog.  Most hosts today ship with a basic syslog daemon only, but a number of replacement versions (some are compatible) include many newer features.  See syslog-ng, module syslog, SDSC-syslog, rsyslog (used by Red Hat and others since it is backward compatible with old syslog, has many new features available, and comes with an open license), and nsyslog.  (There is also a log aggregator, which can combine logs from *nix, Windows, Android, or whatever; see nxlog.)

Note that most network devices today can produce syslog data.  Don't forget to collect logging data from all important sources, including network devices (routers, switches, firewalls), printers, workstations, and Windows servers.  Use SNMP/RMON to monitor network devices that don't support syslog.

Common Linux Log Files and their Location

For most distributions of Linux, you should examine various log files in /var/log, especially the main (default) log file messages.  Use the dmesg command (and /var/log/boot.log) to see boot problems, hardware issues (and identification).  Copy /tmp/install.log to someplace safe.  (Must do this on first boot as /tmp gets erased on reboot.)  The secure log is also very important to monitor.  Other log files include audit/*, for SE Linux and related log messages.

There are additional, non-syslog files maintained you should know about.  wtmp is a log of who logged in and when (This is a binary file, so view with last command and manage with the Linux sessreg command).  utmp is a binary file (not a log) of who's logged in now.  Two related files may exist: btmp (a log of failed login attempts) and lastlog (not a log file but a sparse file — examine it with ls -l and du), which shows the last login per user id (view with finger and lastlog commands).  You must manually create btmp and lastlog via touch if you want Linux to use them.

Keep in mind you don't want to rotate binary log files!  (You may wish to back them up occasionally.)  Log rotation is discussed, below.

Logging Issues:

Syslog Overview:

What gets logged by syslogd and where it goes is controlled by /etc/syslog.conf.  In the past (and to a small extent today), servers had hard-coded filenames to use for their log files.  This is a very inflexible scheme, and log files would wind up scattered all over the disk.  A modern system uses syslog to centralize logging.  A single configuration file can control what gets logged (and what gets ignored), and where the log messages should go. 

What gets logged also depends on what a server (daemon) sends to syslog.  Most services have configuration setting to increase or reduce the amount of log data then generate.  There are two common setups: have syslog save everything and let the sys admin control the amount of log data by configuring each and every service (each service configuration file may use a different syntax), or have the services generate lots of logging data, and let the sys admin control what gets saved to log files by configuring only syslog.  The first approach is less wasteful of CPU and RAM resources, but more demanding of the sys admin.

Here's how it works:  A program uses the syslog API function (via a DLL; or uses the logger program for shell scripts) to send a log message to syslogdSyslogd will also read log messages from sockets (by default just /dev/log, but others can be used), and if started with the right option, also from the network.

The information passed to syslogd includes the source of the log message (called a facility) and the priority of the log message.  Syslogd then matches the source and priority against selectors (combinations of facilities and priorities) in its configuration file, and if a log message matches a selector(s), the message is sent to the corresponding destination(s).  This is a primitive form of log message filtering, especially considering that syslog trusts programs to set the facility and priority accurately.

Syslog-ng and other recent syslog replacements allow more sophisticated filtering, using facilities, priorities, and arbitrary regular expressions.

Note that many PAM modules send log messages to syslogd.  Also, some systems use a separate log daemon for kernel messages, often called klogd that you may need to configure.  (With Fedora, klogd just passes messages to syslog via the “kern” facility.)

Syslog.conf Syntax:

Aside from blank lines and comment lines, syslog.conf has rule lines, with two parts:

When a log message is processed by syslog, the message is compared with each selector in turn.  If the selector matches the message, the action is done.  So a given message may be handled by multiple actions, if multiple selectors match.

In many older syslog daemons, the selector and action must be separated with a TAB, and not just spaces!

Syslog Selectors — Facilities and Priorities:

The source of a log message is referred to as a facility.  For example any email related program that sends a log message uses the mail facility no matter what the name of the program actually was.  When a daemon sends a log message to syslog, it includes the facility syslog should use.  Note that syslog trusts daemons to use the correct facility when sending a log msg.  Since this is defined by the programmer, in some cases the facility may not be the one a sys admin would expect.

There is no way to define your own facilities but there are many predefined ones (up to 23 in all, depending on which syslog you use):

Note that syslog trusts the software to use the correct facility when sending a log message.

Due to the limited number of facilities available, it is inevitable that some services will wind up using the same facility for their log messages.  Syslog allows programs to supply an identifying string, known as a tag, that syslog will prepend to each line of the log messages.  This permits easy selection using grep or other tools, to filter only the log messages of interest.

(You'd think that modern syslog replacements would add new facilities.  A few have, such as for NTP, but most just stick with the standard facilities.)

Priorities:

The priority is one of the following eight levels, which are ranked in order from high to low priority:

When specifying a priority, that and all higher ones are selected too.  A selector is one or more facilities (separated by commas), a dot, then the priority.  More complex selectors are possible too; one such is shown below.)  Some example selectors:

mail.*       mail facility, any priority
mail.debug   mail facility, debug or higher priority (same as *)
mail,news.*  all messages from mail or news
auth.warning all security messages of warning or higher priority
*.info       all messages from any facility except debug msgs
*.=info      any facility, info msgs only (and not higher)
*.!err       any facility, pri <= err only
*.!=alert    any facility, any priority except alert
*.info;mail,news,authpriv.none
             all msgs with info or higher priority except
             mail, news, and authpriv

That last one is tricky.  Using multiple selectors on a single line this way allows you to specify a general category first, then for the matching log messages you can specify exceptions.  Always go from most general selector to most specific or your setup may not log what you think it should!

Syslog Actions — files, users, pipes:

Log messages don't only have to go to files, you can direct them to user terminals, run them through other programs (with a pipe, to email, pager, or just a log file analyzer), or send them to another host running syslogd.

(This last is handy if you have a network of computers you must monitor.  Besides consolidating many log files, there is great security in using a remote log server that has no other services on it.  This is because when a server is hacked the attacker usually destroys the log files.  This scheme protects against disk crashes too.)

Here's the syntax for the actions:

Using logger:

   logger [-p facility.priority] [-t tag] message

The default selector is user.info, and the default tag is logger.

You can also copy a file to the logs.  Here's an example of copying some-file to the system logs:

logger -t "backup script" -f some-file #or <file, no -f

This will send all lines of some-file as individual log messages.

Log File Rotation:

One problem with log files is that over time they grow.  When a system is experiencing problems the log files can grow very large, very quickly.  Periodically trimming or removing log files is necessary.  This is known as log file rotation and is a service usually run via cron.

The most popular scheme is to rename a log file log as log.1 and to start a new log file.  Next time, log.1 is renamed to log.2, log is renamed to log.1, and a new log file is started.  This continues for N previous files.  An even better scheme is the similar, but use the date the file was rotated as the extension, rather than a simple number.

Instead of discarding old log entries, consider archiving them to some cheap backup media.  You never can tell when old log records will come in handy.  (But, be careful with privacy and security issues!)

Since dealing with log file rotation is a common problem most Unix systems have a standard way to deal with it.  On Solaris 9, you have /usr/sbin/logadm (/usr/lib/newsyslog on earlier Solaris).  On Linux you have the logrotate command.  This command runs via the cron facility.

Using logrotate, you can set your log rotation policy for any log file by editing the file logrotate.conf, or editing files in the /etc/logrotate.d directory.  Here's a sample logrotate.conf file:

#Global settings:

# rotate log files weekly
weekly
# keep 4 weeks' worth of backlogs
rotate 4
# Create new (empty) log files after rotating old ones
create 0644 root root

# Per log file settings:

/var/log/cups/*_log {
    missingok
    notifempty
    errors root
    postrotate
        /etc/init.d/cups condrestart >/dev/null 2>&1 || true
    endscript
}

You can configure logrotate to email to someone the old log files it would otherwise delete, handy for automatic archiving.

When adding or enabling a new server, remember to configure syslog and/or logrotate to manage that server's log messages.  If possible, configure the service to use syslog (and not its own log files).  Remember that syslog and logrotate are independent; even when not using syslog, you still need to configure log rotation for new daemons.

Consider always rotating logs on a weekly or monthly basis.  This makes it much easier to guess which log file to examine when looking for an old event.

Note: With Debian systems, the /etc/cron.daily/sysklogd script reads the syslog.conf file and automatically rotates any log files it finds configured there.  This eliminates the need to use logrotate for the common system log files, but not for any daemon that doesn't use syslog.

There is another important reason always to rotate your log files: the default syslog log file format timestamps don’t include a year.  If a system runs for longer than one year, tools such as logwatch will start reporting the old events again!  Always rotate all log files at least once per year.

Integrating Windows (and Macintosh) System Logs

Non Unix/Linux systems also maintain log files, but usually not in syslog format.  This may be a problem for the sys admin who must deal with a mix of Windows and *nix servers.  Windows systems keep detailed event logs.  Windows event log files are binary (not text like syslog).  They are also fixed in size; when full, they erase themselves and start over, losing valuable data!  (This policy can be changed from the control panel, and may not be the default in current Windows versions.)  Although the logs are binary, the format is publicly available, and a number of Perl and other tools exist to convert these to text.

Windows logs are consistent across all Windows versions and services (e.g., Event ID 529 always means a failed login).  And since event logging is built into the OS, it is generally more secure than syslog.

Windows provides no mechanism to forward events to a central loghost.  Instead, there are a number of third party tools for this, such as Kiwi syslog for Windows, EventReporter, Snare for Windows, and even roll-your-own with the Perl module Win32::EventLog.

The Windows event log is really three logs: the system log, the security log, and the application log.  (Think of these as three syslog facilities.)  Each log is stored in a separate file: ...\system32\conf\SysEvent.Evt, ...\SecEvent.Evt, and ...\AppEvent.Evt.  Applications must register themselves to be able to use the event log service (see registry key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Eventlog\Application).

System and service event logging is controlled by the Windows Audit Policy (Control Panel→Administrative Tools→Local Security Policy→Audit Policy).

Windows provides logevent (equivalent to the Unix/Linux logger command line tool) to create event log messages.

For older Macintosh systems (OS9 and earlier), you can use the syslog compatible netlogger tool.  Modern Macintosh is built on BSD Unix, and thus supports syslog directly.

Links: