Build a centralized log management and monitoring system

Seasoned system administrators know that routinely reading system logs is an important task, but reading endless lines from logs is both time-consuming and boring, especially if you are responsible for a large number of busy servers. In this article I will show you how to set up a system that gathers and archives system logs from many network hosts and emails only important or irregular system events to administrators.
The majority of GNU/Linux distributions uses the good old syslogd system logger by default, which is based on the original 4.3BSD syslogd daemon. Syslogd is a fine system logger, but it lacks some advanced features modern alternatives offer. We will use syslog-ng instead, which provides all the functionality of the traditional syslogd along with some nice enhancements. Among others, it provides powerful filtering capabilities based on message content, and can also be used in a firewalled environment without problems.

Installation is a breeze since most distributions provide binary packages. If you prefer to manually build the program, check the INSTALL file included in the source tarball, which outlines all the necessary steps. Make sure to uninstall syslogd before installing syslog-ng.

The syntax of the configuration file might seem peculiar and complex compared to the traditional syslog.conf syntax, but it offers almost limitless customization options. Make sure to read the syslog-ng.conf man page for information on how to use it.

Since our logging system will gather logs from other hosts, we need to instruct it to listen for network connections. Syslog-ng supports both the TCP and UDP protocols. IANA has assigned the 514/udp port to the syslog service, so we will use that port for maximum compatibility with syslogd and network devices such as routers. If you use syslog-ng on all your hosts, it's better to use the TCP protocol, which is more reliable and firewall-friendly.

Add the following lines to your /etc/syslog-ng/syslog-ng.conf to the appropriate sections indicated by comments (lines starting with #) to enable listening for network connections on a specific IP address, and to archive logs from remote hosts as /var/log/$HOST/$FACILITY (e.g. /var/log/mailserver/mail):

## add this to the options section
create_dirs(yes);
long_hostnames(off);
keep_hostname(yes);

# uncomment the following line only on a LAN with working DNS
#use_dns(yes);

## add this to the source section
source s_udp {
udp ( ip(192.168.1.2) ); # replace with your system's IP address
};

## add this to the destination section
destination df_udp {
file ("/var/log/$HOST/$FACILITY");
};

## add this to the log section
log {
source(s_udp);
destination (df_udp);
};
On the remote hosts add the following lines in /etc/syslog-ng/syslog-ng.conf or /etc/syslog.conf, depending on whether they run syslog-ng or syslogd respectively:

## /etc/syslog-ng/syslog-ng.conf

## add this to the destination section
destination remote_udp { udp("192.168.1.2"); }; # replace with your log server's IP address

## add this to the log section
log { source(src); destination(remote_udp); };


## /etc/syslog.conf

# use tabs instead of space
*.* @192.168.1.2 # replace with your log server's IP address
You can add additional filters to better suit your needs or even log to a database like PostgreSQL. Note that syslog traffic between hosts is unencrypted; if you want to gather logs from remote hosts over the Internet, create SSH tunnels first, for security.

Logcheck

At this point you have configured a full-featured syslog server that gathers and archives logs from multiple servers, but so far you still have to read those logs manually. Now we'll add logcheck to the equation.

Logcheck is an excellent program that parses log files, filters out expected, normal events based on pre-defined regular expressions, then summarizes the remaining entries and emails them to the system administrator's account. Logcheck was previously part of the Sentry tools suite, but since it had been unmaintained for a long time, it was forked by Debian developers, who have done a wonderful job integrating logcheck into the system. Most network daemon packages include logcheck rules out of the box.

To install logcheck on a Debian-based system simply apt-get install the logcheck, logtail, and logcheck-database packages; the last of the three provides lots of ready-made rules for various system events. To install it on other distributions, download the source tarball and read the INSTALL file. Since it is just a shell script it does not need any compilation.

Configuration is simple. First enter which log files you want to be checked by logcheck in /etc/logcheck/logcheck.logfiles. Logcheck supports three levels of filtering: paranoid, server, and workstation. Each level uses a directory named /etc/logcheck/ignore.d.level_name that includes filtering rules files with different verbosity levels. Paranoid produces highly verbose output and should be used only on high-security systems running a minimum number of services. Server should be fine for most systems and is used by default. Workstation filters out most of the messages and thus produces the least verbose output of the three. You can define which filtering level should be used in /etc/logcheck/logcheck.conf, along with other parameters, such as the recipient email address and the subject of the emails.

Logcheck uses standard regular expressions to filter logs. It's not difficult to write custom rules; read the WRITING RULES paragraph of the docs/README.logcheck-database file in the source tarball (or /usr/share/doc/logcheck-database/README.logcheck-database.gz in Debian) for more information. If you're new to regular expressions, this guide might be useful. As an example, check this filter file for Dovecot:

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ imap-login: Login: [.[:alnum:]-]+ \[[0-9.]+\]$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ imap-login: Disconnected \[[0-9.]+\]$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ imap\([^[:space:]]+\): File isn't in mbox format: [^[:space:]]+$
Logcheck runs periodically from cron. The default cron job installed by the Debian package (/etc/cron.d/logcheck) runs logcheck every hour or when the system reboots. Unless you want 24 logcheck email messages per day, you should adjust how often it should run by editing the file. I prefer to run it on a daily basis.