Logging, Syslog, and Log File Rotation Tutorial

To identify problems and trends and to trouble-shoot them requires observing events over a period of time (historical monitoring).  Since it is generally impossible to observe all interesting events as they occur, most services (daemons) record important events to files known as log files (or logfiles).  Log files are used for debugging, for audits, for evidence in legal actions, for incident response, to reduce liability, and for various legal and regulatory compliance reasons.  For example, email logs can alert you to spam problems, web logs may be useful for marketing and website design, and database logs can show inefficiencies with popular queries.  When upgrading or deploying new (or newly configured) services, log data can be valuable in finding problems quickly.  Log files can show daemon resource usage (CPU, memory use, disk and network utilization), useful for capacity planning and (for developers) efficiency tuning.

In almost all cases, log files are plain, ASCII or UTF-8 text files with one record per line.  The more modern journald logging system (part of systemd) uses binary log files that cannot be directly examined with standard text tools such as less and grep.  You should use the journalctl tool.  (I am old school and prefer plain text log files!)  Other operating systems have logging deamons that use binary files as well, such as Microsoft's Windows event logs.

In the early days of the Internet, service daemons managed their own files.  That meant system administrators needed to configure logging and manage the log files for each service separately.  That is a hassle.  Also, each daemon's log files had different formats, making it difficult to find and correlate events.

Today most (but not all!) daemons can use a standard logging daemon called syslog to collect, identify (i.e., host, command name, and process ID), add time-stamps, filter, store, alert, and forward logging data.  Using a system logging daemon has the added benefit of somewhat standardizing log file formats, making it much easier to examine log data with various standard tools.

Jan 28 06:17:54 YborStudent sshd[14588]: Invalid user test from 125.243.249.194
Jan 28 06:17:56 YborStudent sshd[14588]: Failed password for invalid user test
    from 125.243.249.194 port 51440 ssh2
Jan 28 06:17:58 YborStudent sshd[14590]: Invalid user guest from 125.243.249.194
Jan 28 06:18:00 YborStudent sshd[14590]: Failed password for invalid user guest
    from 125.243.249.194 port 51586 ssh2
Jan 28 06:18:02 YborStudent sshd[14593]: Invalid user admin from 125.243.249.194
Jan 28 19:19:50 YborStudent sshd[18313]: Accepted publickey for wpollock from
    72.186.232.55 port 62487 ssh2
Jan 29 07:12:54 YborStudent sshd[18052]: Accepted password for ub09 from
    170.12.45.53 port 1403 ssh2

Daemons are usually configurable with the level of detail they output as log data.  Too much data can obscure important events!  Generally, you only need to log errors and security events.  When debugging some service, you will increase the level of detail in the logs to help find the problem.  Once solved, the level is lowered again.

Using a system logging daemon such as syslog provides another way to work.  You can have the various service daemons output maximum level of detail all the time, and have syslog filter out most of them and only store errors and security events.  This means when troubleshooting some service, you increase the retained log level using syslog and you don't need to reconfigure the service daemons at all.  This is usually easier, but is less efficient than not having daemons produce unneeded log data in the first place.

Logging is related to, but distinct from, monitoring.  Monitoring services means collecting metrics about their health and performance.  Monitoring can be useful to alert you to issues, but won't help with troubleshooting them or recording security events.  So generally, both logging and monitoring systems are used.  (It can be confusing since log data has timestamps in each message and can thus be used to produce some metric data; also metric data can be used to produce some log data.  Generally the two systems are independent.)

Syslog was developed first for BSD, but there was no standard for it, resulting in many somewhat incompatible “syslog”s.)  Most daemons send logging data to syslog by default.  However some daemons may need to have their logging configuration changed to have them take advantage of syslog (or indeed, to provide any log data at all).

Log files need to be examined or they are useless.  However it would be foolish to try to read all log data, all the time.  Since it is impossible to know in advance what log data will be useful, you end up collecting far more than any human (system administrator, or just “sys admin”) can possibly read or understand.  This can be managed with log alerting and parsing (a.k.a. data reduction) tools.  Such tools will alert you to unusual log entries and inform you of the number of logs of each type.  Usually these alerts and summaries of the log data are sufficient to show potential problems, at which time you would then examine the relevant log entries.  Some data reduction tools include logwatch, logcheck, swatch, logsurfer, and SEC.  Other utilities include GUI tools to examine and mange log files, and standard text processing tools such as grep, tail (especially tail -f), and less.  (For example, to find log entries for some service on some date, in some logfile(s):  grep service logfile |grep date |less).  More sophisticated tools will try to understand the log data and provide alerts based on their meaning, however these tools usually cost money.

Today, monitoring systems are preferred for alerting.  Using logs for alerting is the second-best solution but useful if you don't have monitoring in place.  Once alerted to some issue, then you examine the logs.

Syslog standards are over 20 years old (RFC-3164) and many issues have surfaced with them (see below for a discussion of these).  A sys admin must address the problems of their installed syslog.  New IETF standards (RFCs) for syslog are being developed (RFC-3164 was replaced in 2009 with RFC-5424) to address security issues and other syslog shortcomings.  Some distributions still ship with a basic, old syslog daemon only, but a number of replacement versions (some are compatible) include many newer features.  You should see which syslog daemon your hosts have and replace them with newer ones, if at all possible.  See syslog-ng, module syslog, SDSC-syslog, rsyslog (used by Red Hat and others since it is backward compatible with old syslog, has many new features available, and comes with an open license), and nsyslog.  However, most of them use similar configuration to syslog, so that will be described here.

There are also log aggregators, which can combine logs from *nix, Windows, Android, or whatever; see (for example) nxlog.)

Note that most network devices today can produce syslog data.  Don't forget to collect logging data from all important sources, including network devices (routers, switches, firewalls), printers, workstations, and Windows servers.  Use SNMP/RMON to monitor network devices that don't support syslog compatible log format.

For most distributions of Linux, you should examine various log files in /var/log, especially the main (default) log file messages.  To see boot problems, hardware issues (and identification) use the dmesg command (and look in /var/log/boot.log).  After your first boot, copy /tmp/install.log to someplace safe.  (You must do this on first boot as /tmp gets erased on reboot.)  The secure log is also very important to monitor.  Other log files include audit/*, for SE Linux and related log messages.

Some distributions of Linux and most Unix systems keep log files in other locations.

There are additional, non-syslog files maintained you should know about in /var/log.  wtmp is a log of who logged in and when (This is a binary file, so view with last command and manage with the Linux sessreg command).  utmp is a binary file (not a log) of who's logged in now.  Two related files may exist: btmp (a log of failed login attempts) and lastlog (not a log file but a sparse file — examine it with ls -l and du), which shows the last login per user id (view with finger and lastlog commands).  You must manually create btmp and lastlog via touch if you want Linux to use them.

With the introduction of systemd, logging was changed to a binary format viewable with journalctl.  These journals are configured separately from syslog files and have advantages over traditional per-host logging daemons.  However, not all services use these journals and still use syslog.  The easy solution is to configure systemd's journalling system to send log data to syslog too.  You can then use journalctl on those services that support it and use your traditional syslog tools on the rest.  (This does mean some log data will be duplicated; fortunately disk space is very cheap nowadays.)

Keep in mind you don't want to rotate binary log files!  (You may wish to back them up occasionally; note journald handles that and log file truncation for systemd jounal files.)  Log rotation is discussed, below.

• HIDS and NIDS (collectively just intrusion detection systems, a.k.a. alerting systems) aren't enough!  Don't rely on these tools exclusively.  The alerts they generate are often meaningless without log data to see if the unusual event has resulted in a failure or a security breach.  Only log data can tell you that.

  Common IDSs include file integrity checkers such as tripwire, osiris, and samhain.  Snort is a common NIDS scanner.

• When managing a network you will have many hosts and devices that generate log data.  It is usually best to funnel all the log data to a single host.  This loghost is often a dedicated host just for logging, and should be secured by removing all unused services and access, and generally hardened.  Commercial log management software will parse the syslog messages and store the data in some sort of database, allowing sophisticated queries, reports, and dashboards (showing alerts and recent log entries).
• With loghosts listening for remote syslog data, make sure you open up the syslog port in all firewalls between you and the remote host: UDP/514 for traditional syslog.  Note!  UDP is not secure and you should only allow selected hosts (a white-list) to use that port.  It is much better to use a modern syslog replacement with secure transport.  (Note syslog must be started with the “-r” option or it won't listen to the network.)
• Configure your services to use syslog.  Many services ship with logging off by default, or attempt to use a custom log file.  Each service should send all messages to syslog.  If this is too much data, you can configure syslog to filter it out.  This means you don't need to continually remember how to configure every service when you need to change its log-level.  Just configure syslog for any service.
• When initially installing a new server (or updating one) it will pay to crank up the amount of log data collected for that service until you are confident it is working correctly.
• As mentioned, you should send all log data to a single central loghost.  If your network is spread out geographically, each location can have a local loghost and these in turn send data to the central loghost.  These relay loghosts should also store the log data, giving you a backup in case the central loghost fails or if a network connection goes down.

  Harvard University collects syslog data from each switch, hub, router, firewall, and server.  Reportedly (;login: 4/2011), they collect data from 3,500 devices and 400 servers.  All the data goes to a central loghost, running special data collection/indexing/monitoring/reporting software, called Splunk.  The loghost collects about 18 GB of log data per day, which is why Perl scripts (e.g., logwatch) and other “home-grown” solutions don't work.

• One problem with a central loghost is that the default syslog daemon (but not all newer versions) only reports the last hop IP address in the log, so the central loghost will lose the IP address of the original host that generated the message.  A good solution is to use a more modern syslog replacement, such as syslog-ng.

  Loghosts are attacked by hackers, so when using the old BSD syslog, it is sometimes worth the work to build a stealth loghost.  Such a host has one NIC connected to an inner network and is used for SSH connections by the administrator or to relay log data.  The other NIC is on the LAN from which you are collecting the log data.  This NIC is unnumbered and set in promiscuous mode.  You use various tools (netcat, a.k.a. nc) to monitor the network for UDP log data sent to a fictitious address.  The various hosts on the LAN will try to send their log data to that fictitious host.  Log data is sent via UDP, so the sending hosts won't know the difference.  An attacker can't access such a stealth host.  (Be sure to turn off ARP on that NIC as well, or attackers can find it.)  A stealth loghost doesn't prevent log injection, but since this makes it impossible to access remotely the loghost from the public LAN, it does prevent some attacks.

• Log data is sent via UDP with syslog.  Modern syslog replacements (and hopefully syslog itself someday) will be able to use TCP, SSH/SSL/TLS encryption, and digitally signed log messages, to provide extra security.
• Log files contain sensitive information, so:
  • Set the file permissions accordingly!  This includes the files and (usually) syslog's /dev/log device (and similar ones in chroot jails).  Note, some daemons will refuse to write to log files if they have insecure permissions.
  • When a log file is closed (when rotated for instance) consider digitally signing and/or encrypting the log file to prevent tampering and/or unauthorized access.  This prevents attackers from dumpster diving for your old log files on discarded backup tapes.  Especially if not encrypted, you should log all access to your log files!  Without special hardware support, encryption and digital signatures can take a long time.  This can cause a busy log server to fall behind, eventually losing log entries, or crashing altogether!  (Some modern syslog replacements report such lost log entry statistics, so you can test your system under a simulated load to make sure you won't lose valuable data.)  Modern hardware supports fast cryptography operations, so it may not be too slow to use.  You should definitely try encryption/signing, and see if you lose any log data (or if logging is taking too much CPU time; if so, you can always turn it off.

  If possible, digitally sign each log entry when added to the log file.  Before a party may move for admission of a computer record or any other evidence in any U.S. court of law, the proponent must show that it is authentic.  Some syslog replacements do this already.

  A different key should be used to digitally sign the whole log; either after every entry is added or after the log file is closed/rotated.  This is an example of a dual control that prevents a single person working alone from falsifying data (e.g., hiding financial transactions to embezzle funds).  Digital signatures can do this (along with a copy of your logging policy, often part of the security policy, that shows your data handling policies).  See justice.gov/criminal/cybercrime/usamarch2001_4.htm (a PDF copy).

  • When possible, don't collect data you don't need! The best way to keep data private is not to store it at all.  For example turn down the log level (amount of detail) produced after a daemon is setup completely.
  • Only keep data as long as you need to.  Regulated industries have limits on how long to keep different types of data.  Most governments have data retention laws.  These regulations specify both minimum and maximum data retention times.  Follow those guidelines!

    As a rule of thumb, syslog data can be kept on-line up to a year; 3 months (or 6 months in some cases) is also commonly used policies.  Older data can be summarized for baselining purposes, and only the summaries kept on-line.  After that, you need to archive the old logs according to law and your data retention policy.  In part, how much old data you keep on-line depends on how much will fit on one backup tape/CD-ROM/DVD.  If 4 months nearly fills one DVD, than 4 months may be a better policy than 6 months.

  • Of the data you do keep, decide which data can be blinded, which data should be encrypted, and which data can be safely left open:
    • Blinding data means that it is destroyed, but in a way that makes it unique.  A hash (one-way) function is a good technique for this.
    • Blinding is a useful way to store personal data such as credit card numbers, phone numbers, addresses, customer ID numbers, etc.  Note that blinded data can still be used as a primary key for a database table, as a hash of a unique value is still unique.
    • If possible randomize the order of such data, to hide the sequence of data (not a good idea for system log files, but useful for, say, financial transaction logs).
    • Blinded data can't be recovered to its original form.  So if there is a requirement for sensitive (private) data that must be recoverable use encryption instead of data blinding.

    Note that blinding, encrypting, and not keeping data are general techniques, not well suited to system log files.  However it still pays to consider these issues.

• Always sanitize external data before logging it.  (This applies mostly to developers, but also shell script writers.)  You should for example replace/remove any newlines or an attacker can generate fake log entries.  You may also wish to replace passwords (and maybe usernames) in log files with something like “*****”.*nbsp; Replace any non-ASCII with question marks.  If using Unicode instead, always normalize all log data not generated by syslog itself.  (Again, this is mostly a developer issue, but you can certainly ask them!)  See the Unicode Normalization FAQ for more information.
• PCI (the Payment Card Industry; really the banks and credit card companies) have mandated security standards for handling credit card data.  Companies must certify each year (with an audit) that they comply with PCI-DSS or pay hefty fines (or lose the right to collect payments via credit cards).  One point of PCI-DSS is that you can only store the first 6 and last 4 digits of a credit card number in an unencrypted file.  Keep this in mind if you don't encrypt your log files.
• HIPPA, FISMA, and other standards require logging and specific log data handling procedures to be in compliance.  Depending on your organization, one or more of these regulations and laws will require compliance.
• Log data can grow to fill even large disks quickly.  You must make sure large log files won't crash your system or result in some DoS (when sending log information across your network).  (See log rotation, below.)  Consider various data reduction techniques: summarizing repetitive log entries, discarding low utility log messages, and data compression.
• Old log data, at least some of it, has lasting value for baselining, auditing, and (if blinded) training.  Consider archiving old log data, or filtering the logs and archiving important events and summary data (e.g., 208 SSH logins successful in past day).  One useful design would be to have old log data sent to a database.  Then you could easily get reports and make queries using SQL.
• Syslog doesn't restrict which hosts can send log data or what data is sent.  This make all log data unreliable.  Hackers can easily generate fake log data and send it to your loghost.  This is called log injection, and the possibility will reduce the evidentiary value of your data.  This can be partially mitigated using a careful firewall configuration, but this issue is better addressed by using one of the modern syslog replacements.
• Some syslog replacements use TCP instead of UDP and support IPSec or SSL/TLS tunnels for the transport.  (This makes stealth loghosts both impossible and unnecessary.)  Digitally signing log entries can also reduce fake log entries (but not if the evil-doer has access to the remote system and can cause it to generate the correctly signed log entries!)
• Newer syslog protocols may use syslog-sign and syslog-reliable for safer transport and sender authentication, and are supported by some syslog replacements.  These standards are old and don't seem to be going anywhere, so I recommend simply using TCP with TLS.
• If you're not going to replace an older syslog with a more secure one, consider using SSH or SSL/TLS tunnels (see stunnel) for transport of syslog data.  Set up tunnel on each remote host so data sent to localhost port 9999 (or any unprivileged port) gets automatically forwarded to the loghost via the secure tunnel.  Here's an example using netcat (nc) for the tunnels:

  On client:     nc -l -u -p syslog | nc locahost 9999
  On loghost:    nc -l -p 999 | nc localhost -u syslog
  

• Old syslog time-stamps don't contain the year, so make sure the archived log files' names include the year.  So if you keep your logs online for over a year, your alerting tools may send alerts based off of last year's events!  (This happened to me; it was embarrassing!)
• Old syslog time-stamps don't contain time zone information.  When data is sent to a central loghost from a wide geographical area, it is important to know the time zone of the original host that generated the log message.  Consider using one of the newer syslog replacements that does provide this information, or otherwise take care of this issue when performing data reduction and data parsing.
• Accurate network-wide time is vital to correlating log events and for preserving their value as an audit trail and evidence in court.  Use NTP to synchronize your hosts and network devices to the same time.  If an external time server is not feasible for some reason, pick one (well-connected) host to act as your organization's time server; it is more important that all hosts agree on the time, not as important that they all keep the correct time.  For about $100 you can install a GPS or radio-controlled clock.

What gets logged by syslogd and where it goes is controlled by /etc/syslog.conf.  In the past (and to a small extent today), servers had hard-coded filenames to use for their log files.  This is a very inflexible scheme, and log files would wind up scattered all over the disk.  A modern system uses syslog to centralize logging.  A single configuration file can control what gets logged (and what gets ignored), and where the log messages should go.  The log data is simply text.  Syslog will add some additional data to that: a timestamp, the hostname, the process name and PID number, and optionally a string called the tag.

What gets logged also depends on what a server (daemon) sends to syslog.  Most services have configuration setting to increase or reduce the amount of log data then generate.  There are two common setups: have syslog save everything and let the sys admin control the amount of log data by configuring each and every service (each service configuration file may use a different syntax), or have the services generate lots of logging data, and let the sys admin control what gets saved to log files by configuring only syslog.  The first approach is less wasteful of CPU and RAM resources, but more demanding of the sys admin.

Here's how it works:  A program uses the syslog API (library) function or uses the logger program for shell scripts) to send a log message to syslogd.  (Some modern service daemons use Systemd's journal API instead.  However it is possible to have journald send any received log data to syslog too.)  Syslogd will also read log messages from sockets (by default just /dev/log, but others can be used), and if started with the right option, also from the network.

The information passed to syslogd includes the source of the log message (called a facility) and the priority of the log message.  Optionally, a a short string called a tag is also passed.  Syslogd then matches the facility and priority against selectors (combinations of facilities and priorities) in its configuration file, and if a log message matches a selector(s), the message is sent to the corresponding destination(s).  This is a primitive form of log message filtering, especially considering that syslog trusts programs to set the facility and priority accurately.

There's only a small list of facilities in syslog, with no prevision for adding more.  Data reduction tools can also filter on the tag string, but not old syslog.

Syslog-ng and other recent syslog replacements allow more sophisticated filtering, using facilities, priorities, and arbitrary regular expressions.

Note that many PAM modules send log messages to syslogd, and use only one or two facilities for all daemons.  So even if you have a dedicated facility for some daemon, some of its log messages may end up in a different log file than the one you expect.

Some systems use a separate log daemon for kernel messages, often called klogd that you may need to configure.  (With Fedora, klogd just passes messages to syslog via the “kern” facility.)

(The original syslog syntax is described here.  It is compatible with rsyslog without change (except for the filename, rsyslog.conf), but not with other syslog replacements.  See the tutorials for your specific syslog daemon for their syntax and extra features.)

Aside from blank lines and comment lines, syslog.conf has rule lines, with two parts:

• The first part says what to log (that is, it is a filter called the selector)
• The second part says where it goes (called the action for some strange reason).

When a log message is processed by syslog, the message's facility and priority is compared with each selector in turn.  If the selector matches the message, the action is done.  So a given message may be handled by multiple actions, if multiple selectors match.

In many older syslog daemons, the selector and action must be separated with a TAB, and not just spaces!  If using one of those, make sure your text editor doesn't replace the tabs when you edit the file.

The source of a log message is referred to as a facility.  For example any email related program that sends a log message uses the mail facility no matter what the name of the program actually was.  When a daemon sends a log message to syslog, it includes the facility syslog should use.  Note that syslog trusts daemons to use the correct facility when sending a log message.  Since this is defined by the programmer, in some cases the facility may not be the one a sys admin would expect.

There is no way to define your own facilities but there are many predefined ones (up to 23 in all, depending on which syslog you use):

• auth (Security events get logged with this)
• authpriv (user access messages use this)
• cron (for cron, at, and anacron, but not for the programs started by them)
• daemon (other daemon programs without a facility of their own)
• kern (kernel messages)
• lpr (print system; the name comes from line printer)
• mail
• mark (used by syslogd to produce timestamps in log files, to show syslog is working if there's no log messages for a long period)
• news (NNTP usenet/netnews/newsgroups)
• syslog (errors from syslog itself)
• user (for user programs)
• uucp (obsolete form of networking)
• local0 – local7 (any use; RH uses local7 for boot messages)
• * (for all)

Note that syslog trusts the software to use the correct facility when sending a log message.

Due to the limited number of facilities available, it is inevitable that some services will wind up using the same facility for their log messages.  Syslog allows programs to supply an identifying string, known as a tag, that syslog will prepend to each line of the log messages.  This permits easy selection using grep or other tools, to filter only the log messages of interest.  However, older syslog cannot use the tag in a selector.

(You'd think that modern syslog replacements would add new facilities.  A few have, such as for NTP, but most just stick with the standard facilities.)

The priority is one of the following eight levels, which are ranked in order from high to low priority:

• emerg
• alert
• crit
• err
• warning
• notice
• info
• debug (or “*”)

When specifying a priority, that and all higher ones are selected too.

A selector is one or more facilities (separated by commas), a dot, then the priority.  More complex selectors are possible too; one such is shown below.)  Some example selectors:

mail.*       mail facility, any priority
mail.debug   mail facility, debug or higher priority (same as *)
mail,news.*  all messages from mail or news
auth.warning all security messages of warning or higher priority
*.info       all messages from any facility except debug msgs
*.=info      any facility, info msgs only (and not higher)
*.!err       any facility, pri <= err only
*.!=alert    any facility, any priority except alert
*.info;mail,news,authpriv.none
             all msgs with info or higher priority except
             mail, news, and authpriv

That last one is tricky.  Using multiple selectors on a single line this way allows you to specify a general category first, then for the matching log messages you can specify exceptions using the special priority of none.  Always go from most general selector to most specific or your setup may not log what you think it should!

Log messages don't only have to go to files, you can direct them to user terminals, run them through other programs (with a pipe, to email, to a pager or as a text message to your cell phone, or to a log file analyzer/alerter), or send them to another host running syslogd (a central loghost).

(This last is handy if you have a network of computers you must monitor.  Besides consolidating many log files, there is great security in using a remote log server that has no other services on it.  This is because when a server is hacked the attacker usually destroys the log files.  This scheme protects against disk crashes too.)

Here's the syntax for the actions:

• /complete/path/of/some/file
• /dev/console
  (This is a link to the system console)
• -/complete/path/of/some/file
  (Don't flush file each time; better performance but risks loss of some log info.)
• username1[,username2 ...]
  (Send the log message to the named users, if they are logged on and have the appropriate permissions enabled)
• *
  (all logged in users)
• @remotehost
  (e.g., @log.example.com; start the remote syslogd with -r option.)
• |/path/to/named/pipe
  (To send output to a command you must create a named pipe, say /var/lib/cmd.pipe with the mkfifo command.  Then start the command with cmd < /var/lib/cmd.pipe.)

   logger [-p facility.priority] [-t tag] message

The default selector is user.info, and the default tag is logger.

You can also copy a file to the logs.  Here's an example of copying some-file to the system logs:

logger -t "backup script" -f some-file #or < file, no -f

This will send all lines of some-file as individual log messages.

One problem with log files is that over time they grow.  When a system is experiencing problems the log files can grow very large, very quickly.  Periodically trimming or removing log files is necessary.  This is known as log file rotation.

The most popular scheme is to rename a log file log as log.1 and to start a new log file.  Next time, log.1 is renamed to log.2, log is renamed to log.1, and a new log file is started.  This continues for N previous files, and older log files are deleted or archived.  An even better scheme is similar, but use the date the file was rotated as the extension, rather than a simple number.

Instead of discarding old log entries, consider archiving them to some cheap backup media.  You never can tell when old log records will come in handy.  (But, be careful with privacy and security issues!)

Since dealing with log file rotation is a common problem, most systems have a standard way to deal with it.  On many Linux distributions you have the logrotate command.  This command usually runs via one of the cron facilities (on Fedora, it's run from anacron).

Using logrotate, you can set your log rotation policy for any log file by editing the file logrotate.conf, or editing files in the /etc/logrotate.d directory.  (Using multiple configuration files is best when you have multiple rotation policies for different groups of log files.)  Here's a sample logrotate.conf file:

#Global settings:

# rotate log files weekly
weekly
# keep 4 weeks' worth of backlogs
rotate 4
# Create new (empty) log files after rotating old ones
create 0644 root root

# Per log file settings:

/var/log/cups/*_log {
    missingok
    notifempty
    errors root
    postrotate
        /etc/init.d/cups condrestart >/dev/null 2>&1 || true
    endscript
}

You can configure logrotate to email to someone the old log files it would otherwise delete, handy for automatic archiving.

When adding or enabling a new service, remember to configure syslog and/or logrotate to manage that service's log messages.  If possible, configure the service to use syslog (and not its own log files).  Remember that syslog and logrotate are independent; even when not using syslog, you still need to configure log rotation for new daemons that create log files.

Consider always rotating logs on a weekly or monthly basis.  This makes it much easier to guess which log file to examine when looking for an old event.

Note: With Debian systems, the /etc/cron.daily/sysklogd script reads the syslog.conf file and automatically rotates any log files it finds configured there.  This eliminates the need to use logrotate for the common system log files, but not for any daemon that doesn't use syslog.

There is another important reason always to rotate your log files: the default syslog log file format timestamps don’t include a year.  If a system runs for longer than one year, tools such as logwatch will start reporting the old events again!  Always rotate all log files at least once per year.

Non Unix/Linux systems also maintain log files, but usually not in syslog format.  This may be a problem for the sys admin who must deal with a mix of Windows and *nix servers.  Windows systems keep detailed event logs.  Windows event log files are binary (not text like syslog).  They are also fixed in size; when full, they erase themselves and start over, losing valuable data!  (This policy can be changed from the control panel, and may not be the default in current Windows versions.)  Although the logs are binary, the format is publicly available, and a number of Perl and other tools exist to convert these to text.

Windows logs are consistent across all Windows versions and services (e.g., Event ID 529 always means a failed login).  And since event logging is built into the OS, it is generally more secure than syslog.

Windows provides no mechanism to forward events to a central loghost.  Instead, there are a number of third party tools for this, such as Kiwi syslog for Windows, EventReporter, Snare for Windows, and even roll-your-own with the Perl module Win32::EventLog.

The Windows event log is really three logs: the system log, the security log, and the application log.  (Think of these as three syslog facilities.)  Each log is stored in a separate file: ...\system32\conf\SysEvent.Evt, ...\SecEvent.Evt, and ...\AppEvent.Evt.  Applications must register themselves to be able to use the event log service (see registry key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Eventlog\Application).

System and service event logging is controlled by the Windows Audit Policy (Control Panel→Administrative Tools→Local Security Policy→Audit Policy).

Windows provides logevent (equivalent to the Unix/Linux logger command line tool) to create event log messages.

For older Macintosh systems (OS9 and earlier), you can use the syslog compatible netlogger tool.  Modern Macintosh is built on BSD Unix, and thus supports syslog directly.

• Syslog replacements
  A list of links for syslog replacement packages, with discussions and other resources for logging.  (From LogAnalysis.org at Archive.org 2010, no longer maintained.)
• LogAnalysis.org at archive.org (2010)
  A rich source of logging information, sponsored by Splunk, but no longer maintained.
• Syslog-ng
  An excellent and popular replacement for syslog.
• rsyslog
  Currently, the most popular syslog replacement.
• journalctl tutorial
  A nice, recent, and fairly complete journalctl tutorial.
• journald configuration
  Describes the millions of journald configuration options, including how to use journald with syslog.
• IETF work on syslog
  The IETF finished working on updating syslog to provide extra security features.  This web page shows the status of that work and provides links to discussions and draft RFCs. You can see all IETF RFCs related to syslog at www.rfc-editor.org.
• Syslog HOW-TO information and links.