Building a Secure Logging Infrastructure

[Adapted from “Building a Logging Infrastructure” by Abe Singe and Tine Bird, number 12 in the SAGE “Short topics in Systems Administration” series.]

Logging is the heart of all system and network monitoring, and hence, of system management and auditing. Examining logs is one of the first steps in forensic system analysis. Using NIDS and other alerting tools are only the first step in incident response; the follow-up is to examine the log files. Logging is used for software development and system testing. Logs are also used for recovery of mis-configuration.

A proper logging infrastructure should include policies that determine what to log, how much/often to collect log data, log rotation schedules, log backups, security controls for logs, and (as much as possible) standard log entry formats.

Log data is generated by the OS, standard services/daemons, and custom software. Other sources include network components (routers, switches, SNMP/RMON sensors). A huge volume of data is potentially generated from these sources, yet only a small fraction (less than 5% typically) are security (or otherwise administratively) related. Yet you need to collect all the data you can, since there is no way to know in advance which 5% you need. A type of HIDS is used to monitor logs and to generate alerts and/or summary reports, so the admin doesn’t have to read all the log data all the time.

Every system today uses syslog to centralize logging on a host, which facilitates management of all logs on that single host. (Windows uses a different logging system.) Additionally all systems provide some log rotation facility (Windows?).

Yet syslog has issues. By default it runs on the same server as other services. So if that server has a problem (especially a security problem) the log files can become compromised, corrupted, or (if the disk runs low on space) truncated. (And if the log files fill the /var partition other services could be affected.)

Centralized Logging Architecture

Running a log server and having other server’s syslog daemons forward log entries is a more secure design. Syslog has a mode where log entries sent to the local syslog are forwarded to a central syslog, known as a loghost. This mode is enabled using a syslogd command line option. This design is called a single central log architecture. The loghost should be hardened and only run minimum services: syslogd, sshd, and ntpd.

All hosts that log to a single server must agree on the current time. Even if you don’t sync to a public time server, you should choose one well-connected host to act as the local time server.

Note there exists 3rd party Windows event log to syslog forwarders.

If you have several sites connected by a WAN it may pay to have a central log server at each site, and have that log server in turn forward to a single log server. This variation is known as a relay architecture.

The syslog protocol for forwarding log messages between servers is documented in RFC-3164. Although this logging infrastructure appears to work well, syslog has problems, especially with centralized logging:

· syslog can lose information (only the last hop IP to the log server is logged).

· By using UDP, syslog log entries can be lost.

· Attackers can forge fake log entries, polluting your logs.

· Syslog doesn’t validate log message headers or content in any way.

· Log messages may include sensitive information but syslog forwards them without encryption.

· Denial of service attacks against the log server can render all logs useless.

· Collected log data is not easily searchable by standard means. (You need to cobble together a grep/Perl script for each query against log files.

· There is no standard log entry format in the world. This is not so much a syslog problem but a general logging issue. However syslog provides only very limited control over the format of log entries.

· Syslog provides limited actions upon receipt of a log message. It can only store or forward the message. There is no way to do anything else, pass the message to alerting software, pick destinations of regular expressions matching the messages, or trigger a response to messages (e.g., turn off services, force a user to log out, ...).

· There are no restrictions on sources of log data when using a loghost, as there is not way to configure syslog to only talk to selected hosts.

Today there are a number of replacements for standard syslog. Some of these replacements use an extended version of the syslog.conf file syntax for configuration, making them suitable for drop-in replacements. Others that provide more features use their own configuration syntax (which may be easier to understand the standard syslog facility and selector syntax!).

Stealth Loghosts

Whatever logging software you use, your system can deploy a central stealth log server. Such a server listens to network traffic on the LAN using promiscuous mode. The NIC is not assigned any IP address! The other hosts must be configured to send logs to a non-existent (but valid) IP address. Note that since there is no host at this fake address, ARP won’t work and you’ll need to configure a static ARP cache entry as well.

On the loghost itself, you use tcpdump, snort, plog (“Plog [www.ranum.com] is a promiscuous syslog listener. It sucks UDP syslog packets up off a network, rips the message screaming and kicking out of the packet body, and stuffs it into /dev/log”), or some other packet logger. From this you extract the syslog entries and forward to syslogd (usually via /dev/log, or another named pipe or socket).

This stealth solution will work for any UDP syslog replacement, but not any TCP one (which requires the server at the other end to complete the handshake).

Security Measures for the Logging Infrastructure

Each piece must be secured: update the software, set owner/group and permissions correctly, set disk quotas.

For centralized logging, configure TCP wrappers (only for TCP syslog replacements), host firewalls, and routers (internal routers must forward log packets, external ones should not). Syslog listens for log data at port UDP/514. This should be locked down using local firewall so only localhost can send to this port (except for the central loghost of course).

A commonly forgotten security measure is to protect /dev/log and other named pipes used by syslog so only root and selected users can write to it:

groupadd loggers
chgrp loggers /dev/log
chmod 660 /dev/log

When first setting up a loghost you should aim to use all available disk space for log data. A compressing filesystem can help here.

You should estimate how much data you can store and configure log rotation accordingly. (Initially, it is okay to underestimate log file growth. Whenever you add or change services, you should monitor log file sizes for awhile. Use that data (and your data retention policies) to adjust log file rotation.

To ensure the log files don’t grow too large, configure a separate disk partition for log data, or configure a disk quota for syslog.

Note some filesystems (or old kernel versions or file services such as NFS) impose a limit on file sizes, often around 2 gigabytes. Also if you create 50+ GB files it will take a long time to grep through them! In fact many tools will not work with very large files. The solution is to configure log rotation by size, rather than by time (e.g. every day). You can use hybrid rotation policies as well.

After a while you will run out of space, so you must decide what to do with old log data. If possible you should archive old log data rather than throw it away. (If you do toss it, make sure to treat paper logs and backup tapes/disks as containing sensitive information.)

Although you’d like to log everything sometimes it pays to not send all available data to your loghost. This can be done by tuning (configuring) syslog logging levels to save less data or using other filtering techniques.

(Note! /etc/syslog.conf on many systems requires a tab character as a delimiter, yet vi and other editors by default insert spaces when you use the tab key. Messing up syslog.conf is not a good thing to do!)

Syslog Replacements

Syslog has a number of replacements and additional components available to address the short-comings noted above. To provide authentication and integrity of log sources when using central logging, a number of syslog protocols are being developed:

· syslog-sign provides reliable transport over UDP. It also provides authentication of the sender, includes timestamps to prevent replay attacks, and performs delivery checks (checksum/CRC???).

· syslog-reliable users TCP rather than UDP.

Another transport option is to use an SSH/SSL/TLS tunnel to transport regular syslog messages.

There are a number of syslog replacements. None seem to offer all the features you’d like to have but most are better than the standard syslog:

· syslog-ng is a popular syslog replacement. It allows TCP as the transport, remembers forwarding IP addresses so a relay architecture doesn’t lose data, and provides finer message filtering than standard syslog facility/level selectors.

· Modular syslog includes integrity checks, regular expression message filtering, and easy SQL database integration for storing log data. This is important for management and analysis: you can use a high-speed DB, stored queries, and standard reports.

· SDSC-syslog implements both syslog-sign and syslog-reliable protocols. It is designed for efficiency and robustness even under high loads.

· nsyslog uses SSL (over TCP) for message transport. SSH/SSL provides authentication, authorization, integrity, and confidentiality.

I’m not sure the TCP/SSL/SSH solutions can be used with a stealth loghost. But if the transport is secure, reliable, encrypted, authenticated, and provides message integrity checks, and the routers and host firewalls are configured tightly, then maybe you don’t need a stealth loghost!

Integrating Windows and Other System Logs

Non-Unix/Linux systems can use syslog: netlogger for old (OS9 and earlier) Macintosh systems, and Kiwi syslog for Windows. Modern Macintosh is built on BSD Unix and thus supports syslog directly.

By default logging on Windows hosts uses binary event logs, that automatically over-write themselves when full. The binary the format is publicly available and a number of Perl and other tools exists to convert these to text.

Windows logs are consistent across all Windows versions and services (Event ID 529 always means a failed login). And as event logging is built into the OS it is generally more secure than syslog.

Windows provides no mechanism to forward events to a central loghost. Instead there are a number of third party tools for this, such as Kiwi syslog for Windows, EventReporter, Snare for Windows, and even roll-your-own with Perl module Win32::EventLog.

The Windows event log has three parts: system log, Security log, and the application log. (Think of these as 3 facilities.) Each part is stored in a separate file in ...\system32\conf\SysEvent.Evt, SecEvent.Evt, and AppEvent.Evt. Applications must register themselves to be able to use the event log service (see registry key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Eventlog\Application. System and service event logging is controlled by the Windows Audit Policy (Control Panel-->Administrative Tools-->Local Security Policy-->Audit Policy)

Windows provides logevent, equivalent to Unix logger command line tool to create event log messages.