To identify problems and trends and to trouble-shoot them requires observing events over a period of time (historical monitoring). Since it is generally impossible to observe all interesting events as they occur, most services (daemons) record important events to files known as log files. Log files are used for debugging, audits, for evidence in legal actions, for incident response, to reduce liability, and for various legal and regulatory compliance reasons. Email logs can alert you to spam problems. Web logs may be useful for marketing and website design. Database logs can show inefficiencies with popular queries. When upgrading or deploying new (or newly configured) services, log data can be valuable in finding problems quickly.
In the early days of the Internet, service daemons managed their own files.
That meant system administrators needed to configure logging and manage the
log files for each service separately.
Today most (but not all!) can use a standard logging daemon called
syslog
to collect, identify (i.e., host, command name, and
process ID), time-stamp, filter, store, alert, and forward
logging data.
Using a system logging daemon has the added benefit of somewhat standardizing
log file formats, making it much easier to examine log data with various
standard tools.
Jan 28 06:17:54 YborStudent sshd[14588]: Invalid user test from 125.243.249.194 Jan 28 06:17:56 YborStudent sshd[14588]: Failed password for invalid user test from 125.243.249.194 port 51440 ssh2 Jan 28 06:17:58 YborStudent sshd[14590]: Invalid user guest from 125.243.249.194 Jan 28 06:18:00 YborStudent sshd[14590]: Failed password for invalid user guest from 125.243.249.194 port 51586 ssh2 Jan 28 06:18:02 YborStudent sshd[14593]: Invalid user admin from 125.243.249.194 Jan 28 19:19:50 YborStudent sshd[18313]: Accepted publickey for wpollock from 72.186.232.55 port 62487 ssh2 Jan 29 07:12:54 YborStudent sshd[18052]: Accepted password for ub09 from 170.12.45.53 port 1403 ssh2
Daemons are usually configurable with the level of detail they output as log data. Generally, you only need to log errors or security events. When debugging some service, you will increase the level of detail in the logs to help find the problem. Once solved, the level is lowered again.
Using a system logging daemon such as syslog provides another way to work. You can have the various service daemons output maximum level of detail all the time, and have syslog filter out most of them and only store errors and security events. This means when troubleshooting some service, you increase the retained log level using syslog and you don't need to reconfigure service daemons at all.&nbps; This is usually easier, but is less efficient than not having daemons produce unneeded log data in the first place.
Logging is related to, but distinct from, monitoring. Monitoring services means collecting metrics about their health and performance. Monitoring can be useful to alert you to issues, but won't help with troubleshooting them or recording security events. So generally, both logging and monitoring systems are used. (It can be confusing since log data has timestamps in each message and can thus be used to produce some metric data; metric data can be used to produce some log data. Generally the two systems are independent.)
Syslog was developed first for BSD, but there was no standard for it, resulting in many incompatible “syslog”s.) However some daemons may need to have their default logging configuration changed to have them take advantage of syslog (or indeed, to provide any log data at all).
Log files need to be examined or they are useless.
However it would be foolish to try to read all log data, all the
time.
Since it is impossible to know in advance what log data will be useful,
you end up collecting far more than any human (or sys admin) can possibly
read or understand.
This can be managed with log alerting and parsing (a.k.a. data
reduction) tools.
Such tools will alert you to unusual log entries and inform you of the number
of logs of each type.
Usually these alerts and summaries of the log data are sufficient to show
potential problems, at which time you would then examine the
relevant log entries.
Some data reduction tools include
logwatch
, logcheck
,
swatch
, logsurfer
, and SEC.
Other utilities include GUI
tools to examine and mange log files, and standard text processing
tools such as grep
, tail
,
tail -f
, and less
(for example:
grep service logFile |grep date |less
).
More sophisticated tools will try to understand the log data and
provide alerts based on their meaning, however these tools usually
cost money.
Fedora uses logwatch by default and it works fine, but doesn't
examine all logfiles.
Which service log files are monitored is controlled by
/etc/logwatch/conf/*
files.
Each service to be monitored (other than the defaults) have conf files
in conf/services
.
The logwatch tool examines various log files, collects info about each, and
emails a report daily to the system administrator.
This works will for standard, well-known services where you expect few
issues.
For custom or newly deployed services you will want more immediate alerting
than logwatch provides.
A bunch of sample logwatch conf files you can use can be found in
/usr/share/logwatch/default.conf/services/*.conf
.
Suppose you decide to use Sendmail.
Try copying sendmail.conf
to
/etc/logwatch/conf/services/
.
To get more details, crank up sendmail's LogLevel
setting from (the default of) 9 to 15 (see
/etc/mail/sendmail.{cf,mc}
).
However, syslog will filter most of the resulting log messages out.
To get more detail, change the value of “detail” in
logwatch's sendmail.conf
file from (the default of)
3 to 4 (max is 10 for most detail possible).
View a Sample logwatch report.
Syslog standards are over 20 years old (RFC-3164) and many issues have surfaced with them (see below for a discussion of these). A sys admin must address the problems of the current syslog. New IETF standards (RFCs) for syslog are being developed (RFC-3164 was replaced in 2009 with RFC-5424) to address security issues and other syslog shortcomings. Most hosts today ship with a basic, old syslog daemon only, but a number of replacement versions (some are compatible) include many newer features. See syslog-ng, module syslog, SDSC-syslog, rsyslog (used by Red Hat and others since it is backward compatible with old syslog, has many new features available, and comes with an open license), and nsyslog. (There is also log aggregators, which can combine logs from *nix, Windows, Android, or whatever; see for example nxlog.) You should always use one of the newer syslog replacements. However, most of them use similar configuration to syslog, so that will be described here.
Note that most network devices today can produce syslog data. Don't forget to collect logging data from all important sources, including network devices (routers, switches, firewalls), printers, workstations, and Windows servers. Use SNMP/RMON to monitor network devices that don't support syslog.
For most distributions of Linux,
you should examine various log files in /var/log
,
especially the main (default) log file messages
.
Use the dmesg
command (and /var/log/boot.log
)
to see boot problems, hardware issues (and identification).
Copy /tmp/install.log
to someplace safe.
(Must do this on first boot as /tmp
gets erased on reboot.)
The secure
log is also very important to monitor.
Other log files include audit/*
, for SE
Linux and related log messages.
There are additional, non-syslog files maintained you should know about.
wtmp
is a log of who logged in and when
(This is a binary file, so view with last
command
and manage with the Linux sessreg
command).
utmp
is a binary file (not a log) of who's logged in
now.
Two related files may exist:
btmp
(a log of failed login attempts)
and lastlog
(not a log file but a sparse file
— examine it with ls -l
and du
),
which shows the last login per user id (view with finger
and lastlog
commands).
You must manually create btmp
and lastlog
via touch
if you want Linux to use them.
With the introduction of systemd, logging was changed to a binary format
viewable with journalctl
.
These journals are configured separately from syslog files and have advantages
over traditional per-host logging daemons.
However, not all services use these journals and still use syslog.
The easy solution is to configure systemd's journalling system to send log
data to syslog too.
You can then use journalctl
on those services that support it
and use your traditional syslog tools on the rest.
(This does mean some log data will be duplicated; fortunately disk space is
very cheap nowadays.)
Keep in mind you don't want to rotate binary log
files!
(You may wish to back them up occasionally; note journald
handles
that and log file truncation for systemd jounal files.)
Log rotation is discussed, below.
Common IDSs include file integrity checkers such as tripwire, osiris, and samhain. Snort is a common NIDS scanner.
UDP/514
for traditional syslog.
Note UDP is not secure and you should only allow selected hosts
to use that port; best is to use a modern syslog replacement with secure
transport.
(Note syslog must be started with the “-r
”
option or it won't listen to the network.)
Harvard University collects syslog data from each switch, hub, router,
firewall, and server.
Reportedly (;login: 4/2011
), they collect data from 3,500
devices and 400 servers.
All the data goes to a central loghost, running special data
collection/indexing/monitoring/reporting software, called
Splunk.
The loghost collects about 18 GB of log data per day,
which is why Perl scripts (e.g., logwatch) and other
“home-grown” solutions don't work.
Loghosts are attacked by hackers, so when using the old BSD
syslog, it is sometimes worth the work to build a stealth loghost.
Such a host has one NIC connected to an inner network and is used for
SSH connections by the administrator or to relay log
data.
The other NIC is on the LAN from which you are collecting
the log data.
This NIC is unnumbered and set in promiscuous mode.
You use various tools (netcat
, a.k.a.
nc
)
to monitor the network for UDP log data sent
to a fictitious address.
The various hosts on the LAN will try to send their log data to that
fictitious host.
Log data is sent via UDP, so the sending hosts won't
know the difference.
An attacker can't access such a stealth host.
(Be sure to turn off ARP on that NIC as well, or attackers can find it.)
A stealth loghost doesn't prevent log injection, but since this makes it
impossible to access remotely the loghost from the public LAN, it does
prevent some attacks.
/dev/log
device (and similar ones in chroot
jails).
Note, some daemons will refuse to write to log files if they have
insecure permissions. If possible, digitally sign each log entry when added to the log file. Before a party may move for admission of a computer record or any other evidence in any U.S. court of law, the proponent must show that it is authentic. Some syslog replacements do this already.
A different key should be used to digitally sign the whole log; either after every entry is added or after the log file is closed/rotated. This is an example of a dual control that prevents a single person working alone from falsifying data (e.g., hiding financial transactions to embezzle funds). Digital signatures can do this (along with a copy of your logging policy, often part of the security policy, that shows your data handling policies). See justice.gov/criminal/cybercrime/usamarch2001_4.htm (a PDF copy).
As a rule of thumb, syslog data can be kept on-line up to a year; 3 months (or 6 months in some cases) is also commonly used policies. Note older data can be summarized for baselining purposes, and only the summaries kept on-line. After that, you need to archive the old logs according to law and the data retention policy. In part, how much old data you keep on-line depends on how much will fit on one backup tape/CD-ROM/DVD. If 4 months nearly fills one DVD, than 4 months may be a better policy than 6 months.
one-way) function is a good technique for this.
Note that blinding, encrypting, and not keeping data are general techniques, not well suited to system log files. However it still pays to consider these issues.
stunnel
) for transport of syslog data.
Set up tunnel on each remote host so data send to localhost port 9999
(or any unprivileged port) gets automatically forwarded to the
loghost via the secure tunnel:
On client: nc -l -u -p syslog | nc locahost 9999 On loghost: nc -l -p 999 | nc localhost -u syslog
NTP
to
synchronize your hosts and network devices to the same time.
If an external time server is not feasible for some reason,
pick one (well-connected) host to act as your or4ganizatin's
time server.
For about $100 you can install a GPS
or radio-controlled clock.
What gets logged by syslogd
and where it goes is controlled by
/etc/syslog.conf
.
In the past (and to a small extent today), servers had
hard-coded filenames to use for their log files.
This is a very inflexible scheme, and log files would wind up
scattered all over the disk.
A modern system uses syslog to centralize logging.
A single configuration file can control what gets logged
(and what gets ignored), and where the log messages should go.
The log data is simply text.
Syslog will add some additional data to that: a timestamp, the hostname,
the process name and PID number, and optionally a string called the
tag.
What gets logged also depends on what a server (daemon) sends to syslog. Most services have configuration setting to increase or reduce the amount of log data then generate. There are two common setups: have syslog save everything and let the sys admin control the amount of log data by configuring each and every service (each service configuration file may use a different syntax), or have the services generate lots of logging data, and let the sys admin control what gets saved to log files by configuring only syslog. The first approach is less wasteful of CPU and RAM resources, but more demanding of the sys admin.
Here's how it works:
A program uses the syslog API function (via a
DLL; or uses the logger
program for shell
scripts) to send a log message to syslogd
.
(Some modern service daemons use Systemd's journal API instead.
However it is possible to have journald
send any received log
data to syslog too.)
Syslogd
will also read log messages from sockets
(by default just /dev/log
, but others can be used),
and if started with the right option, also from the network.
The information passed to syslogd
includes the source of
the log message (called a facility) and the
priority of the log message.
Syslogd
then matches the source and priority against
selectors (combinations of facilities and priorities)
in its configuration file, and if a log message matches a selector(s),
the message is sent to the corresponding destination(s).
This is a primitive form of log message filtering, especially considering
that syslog trusts programs to set the facility and priority accurately.
Syslog-ng and other recent syslog replacements allow more sophisticated filtering, using facilities, priorities, and arbitrary regular expressions.
Note that many PAM modules send log messages to syslogd
.
Also, some systems use a separate log daemon for kernel messages,
often called klogd
that you may need to configure.
(With Fedora, klogd
just passes messages to syslog via
the “kern
” facility.)
Aside from blank lines and comment lines, syslog.conf
has rule lines, with two parts:
When a log message is processed by syslog, the message is compared with each selector in turn. If the selector matches the message, the action is done. So a given message may be handled by multiple actions, if multiple selectors match.
In many older syslog daemons, the selector and action must be separated with a TAB, and not just spaces!
The source of a log message is referred to as a facility.
For example any email related program that sends a log message uses the
mail
facility no matter what the name of the program actually
was.
When a daemon sends a log message to syslog, it includes the facility
syslog should use.
Note that syslog trusts daemons to use the correct facility when
sending a log msg.
Since this is defined by the programmer, in some cases the facility
may not be the one a sys admin would expect.
There is no way to define your own facilities but there are many predefined ones (up to 23 in all, depending on which syslog you use):
auth
(Security events get logged with this) authpriv
(user access messages use this) cron
(for cron, at, and anacron, but not for the programs started by cron) daemon
(other daemon programs without a facility of their own) kern
(kernel messages) lpr
(print system) mail
mark
(used by syslogd to produce timestamps in log files) news
syslog
user
(for user programs) uucp
(obsolete form of networking) local0 – local7
(any use; RH uses local7
for boot messages) *
(for all) Note that syslog trusts the software to use the correct facility when sending a log message.
Due to the limited number of facilities available, it is
inevitable that some services will wind up using the same
facility for their log messages.
Syslog allows programs to supply an identifying string, known
as a tag, that syslog will prepend to each line of
the log messages.
This permits easy selection using grep
or other
tools, to filter only the log messages of interest.
(You'd think that modern syslog replacements would add new facilities. A few have, such as for NTP, but most just stick with the standard facilities.)
The priority is one of the following eight levels, which are ranked in order from high to low priority:
emerg
alert
crit
err
warning
notice
info
debug
(or “*
”) When specifying a priority, that and all higher ones are selected too. A selector is one or more facilities (separated by commas), a dot, then the priority. More complex selectors are possible too; one such is shown below.) Some example selectors:
mail.* mail facility, any priority mail.debug mail facility, debug or higher priority (same as *) mail,news.* all messages from mail or news auth.warning all security messages of warning or higher priority *.info all messages from any facility except debug msgs *.=info any facility, info msgs only (and not higher) *.!err any facility, pri <= err only *.!=alert any facility, any priority except alert *.info;mail,news,authpriv.none all msgs with info or higher priority except mail, news, and authpriv
That last one is tricky. Using multiple selectors on a single line this way allows you to specify a general category first, then for the matching log messages you can specify exceptions. Always go from most general selector to most specific or your setup may not log what you think it should!
Log messages don't only have to go to files, you can direct
them to user terminals, run them through other programs
(with a pipe, to email, to a pager or as an SMS message to your
cell phone, or to a log file analyzer),
or send them to another host running syslogd
(a central loghost).
(This last is handy if you have a network of computers you must monitor. Besides consolidating many log files, there is great security in using a remote log server that has no other services on it. This is because when a server is hacked the attacker usually destroys the log files. This scheme protects against disk crashes too.)
Here's the syntax for the actions:
/complete/path/of/some/file
/dev/console
-/complete/path/of/some/file
username1[,username2 ...]
*
@remotehost
@log.example.com
;
start the remote syslogd
with
-r
option.) |/path/to/named/pipe
/var/lib/cmd.pipe
with the mkfifo
command.
Then start the command with
cmd < /var/lib/cmd.pipe
.)
logger
is a command line utility you can use to send log
messages to syslog.
It is often used in shell scripts used as daemons.
The syntax is simple:
logger [-p facility.priority] [-t tag] message
The default selector is user.info
,
and the default tag is logger
.
You can also copy a file to the logs.
Here's an example of copying some-file
to the system logs:
logger -t "backup script" -f some-file #or < file, no -f
This will send all lines of some-file
as individual log messages.
One problem with log files is that over time they grow. When a system is experiencing problems the log files can grow very large, very quickly. Periodically trimming or removing log files is necessary. This is known as log file rotation and is a service usually run via cron.
The most popular scheme is to rename a log file log
as log.1
and to start a new log
file.
Next time, log.1
is renamed to log.2
,
log
is renamed to log.1
,
and a new log
file is started.
This continues for N previous files.
An even better scheme is the similar, but use the date the file
was rotated as the extension, rather than a simple number.
Instead of discarding old log entries, consider archiving them to some cheap backup media. You never can tell when old log records will come in handy. (But, be careful with privacy and security issues!)
Since dealing with log file rotation is a common problem
most Unix systems have a standard way to deal with it.
On Solaris 9, you have /usr/sbin/logadm
(/usr/lib/newsyslog
on earlier Solaris).
On Linux you have the logrotate
command.
This command runs via the cron facility.
Using logrotate
, you can set your log rotation policy
for any log file by editing the file logrotate.conf
,
or editing files in the /etc/logrotate.d
directory.
Here's a sample logrotate.conf
file:
#Global settings: # rotate log files weekly weekly # keep 4 weeks' worth of backlogs rotate 4 # Create new (empty) log files after rotating old ones create 0644 root root # Per log file settings: /var/log/cups/*_log { missingok notifempty errors root postrotate /etc/init.d/cups condrestart >/dev/null 2>&1 || true endscript }
You can configure logrotate
to email to someone
the old log files it would otherwise delete, handy for automatic archiving.
When adding or enabling a new service, remember to configure syslog and/or logrotate to manage that service's log messages. If possible, configure the service to use syslog (and not its own log files). Remember that syslog and logrotate are independent; even when not using syslog, you still need to configure log rotation for new daemons that create log files.
Consider always rotating logs on a weekly or monthly basis. This makes it much easier to guess which log file to examine when looking for an old event.
Note: With Debian systems, the /etc/cron.daily/sysklogd
script reads the syslog.conf
file and automatically
rotates any log files it finds configured there.
This eliminates the need to use logrotate for the common system
log files, but not for any daemon that doesn't use syslog.
There is another important reason always to rotate your log files:
the default syslog log file format timestamps don’t include a year.
If a system runs for longer than one year, tools such as
logwatch
will start reporting the old events again!
Always rotate all log files at least once per year.
Non Unix/Linux systems also maintain log files, but usually not in syslog format. This may be a problem for the sys admin who must deal with a mix of Windows and *nix servers. Windows systems keep detailed event logs. Windows event log files are binary (not text like syslog). They are also fixed in size; when full, they erase themselves and start over, losing valuable data! (This policy can be changed from the control panel, and may not be the default in current Windows versions.) Although the logs are binary, the format is publicly available, and a number of Perl and other tools exist to convert these to text.
Windows logs are consistent across all Windows versions and services (e.g., Event ID 529 always means a failed login). And since event logging is built into the OS, it is generally more secure than syslog.
Windows provides no mechanism to forward events to a central loghost.
Instead, there are a number of third party tools for this, such as
Kiwi syslog for Windows, EventReporter, Snare for Windows,
and even roll-your-own with the Perl module Win32::EventLog
.
The Windows event log is really three logs: the system log, the security
log, and the application log.
(Think of these as three syslog facilities.)
Each log is stored in a separate file:
...\system32\conf\SysEvent.Evt
,
...\SecEvent.Evt
, and ...\AppEvent.Evt
.
Applications must register themselves to be able to use the event log
service (see registry key
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Eventlog\Application
).
System and service event logging is controlled by the Windows Audit Policy (Control Panel→Administrative Tools→Local Security Policy→Audit Policy).
Windows provides logevent
(equivalent to the Unix/Linux
logger
command line tool) to create event log messages.
For older Macintosh systems (OS9 and earlier), you can use the syslog
compatible netlogger
tool.
Modern Macintosh is built on BSD Unix, and thus supports
syslog directly.