Syslogd

The workhorse of the Linux logging system is the system logging daemon or syslogd. This daemon is normally started from the system start-up (rc) scripts when the system goes into run level 1. Once running, almost any part of the system, including applications, drivers, as well as other daemons can make log entries. There is even a command line interface so you can make entries from scripts or anywhere else.

With Windows NT, each system maintains its own log files. There is no central location where they are all stored. Although the Event Viewer can access event logs on other machines, this can often take a great deal of time especially when there are a lot of entries and you have a slow connection.

Instead, syslogd can be configured to send all (or just some) of the messages to a remote machine, which processes them and writes them to the necessary files. It is thus possible that all the log messages of a particular type from all machines in your network are stored in a single file, which make accessing and administering them much easier.

Another advantage is due to the fact that syslogd stores configuration information and log entries in text files. Therefore, it is a simple matter of writing a script that parses the entries and splits them into separate files, or processes them in other ways.

Part of this ability lies in the standard format of each log entry. Although it is possible that a rogue program could write information in any order, all system daemons and most programs follow the standard, which is:

date time system facility message

Here "system" is the host name which generated the message. The "facility" is a component of the system generating the message. This could be anything like the kernel itself, system daemons and even applications. Finally, there is the text of the message itself. Here are two messages on the system jupiter. One is from syslogd and the other from the kernel:

Jun  5 09:20:52 jupiter syslogd 1.3-0: restart.

Jun  5 09:20:55 jupiter kernel: VFS: Mounted root (ext2 file system) readonly.

As you can see, even if you could not separate the log entries into different files, it would be fairly easy to separate them using a script.

Configuring syslogd

What is done and when it is done is determined by the syslogd configuration file, syslog.conf, which is usually in /etc. (I have never seen it anywhere else.) This is a typical Linux configuration file with one item (or rule) per line and comment lines begin with a pound-sign (#). Each rule consists of selector portion, which determines the events to react to and the action portion, which determines what is to be done.

The selector portion is itself broken into two parts, which are separated by a dot. The facility part says what aspect of the system is to be recorded and the priority says what level of messages to react to. The selector has the general syntax:

facility.priority

You can see a list of facilities in table 1 and a list of the priorities in table 2.

For both facilities and priorities there is a "wildcard" that can be used (an asterisk - *) which means any facility or any priorities. For example, *.emerg would mean all emergency messages. mail.* would mean all messages coming from the mail facility. Logically, *.* means all priorities of messages from all facilities.

The word "none" is used to refer to no priority for the specified facility. For example, the selector mail.none would say not to perform the action for any mail event. At first, this might not make sense. Why not simply leave off that facility? The answer lies in the previous paragraph. Using the wildcard, you could say that all info messages were to be logged to a certain file. However, for obvious reasons, you want all of the security (regardless of the priority) written to another file.

Another possibility is to specify a sub-set of facilities, rather than all of them. This is done by separating the facilities with a comma and then the priority follows the last facility listed. For example, to refer to information messages for mail, uucp and news, the selector entry would look like this:

mail,uucp,news.info

One thing I need to point out here is that when you specify a priority, you are actually specifying everything at that priority or *higher*. Therefore, in this example, we are selecting all of the priorities at info and higher.

There are three primary things you can do with these events (the actions). Probably the most common action is to write them to a file. However, there is more to this than it appears. Remember that Linux (as well as other UNIX dialects) treat devices as files. Therefore, you can send the logging messages to a specific device.

Here, I not talking about sending them a tape drive (although that might not be a bad idea). Instead, I am talking about something like the system console (/dev/console). It is a common practice to send emergency messages to the system console, where someone will see the messages no matter to what console terminal they are logged on. In other cases, kernel messages are sent to one of the console terminals (e.g. /dev/tty7). You might end with something like this:

kernel.*  /dev/tty7

When writing to files, you want to consider that the system will actually write the information to the disk with each event. This ensures the entry actually makes it to the file if the system should crash. The problem is that writing to the harddisk takes time. That's why the system normally saves up a number of writes before sending them all to the disk.

If overall system performance becomes an important factor in regard to logging, you can tell syslogd **not** to sync the disk each time it writes to a log file. This is done by putting a minus sign (-) in front of the file name, like this:

lpr.info  -/var/adm/printer.log

If you disable syncing the log file like this, one important thing to remember is that you stand the chance of losing information. If the system goes down for some reason before the information is written to the file, you may lose an important clue as to why the system went down. One solution would be to have a central log server where all of the information is sent and where you do not disable syncing. That way no matter what, you have a record of what happened.

Sending the log messages to another machine is done by using an at-sign (@) in front of the machine name as the action. For example:

*.emerg   @logserver

This sends all emergency message to the machine logserver. I would suggest that you do not create a log server that is connected to the Internet. A ill-intended person might be able to bring the system to a halt or at least affect its performance by flooding it with erroneous log messages.

Another useful feature is the ability to send messages to named pipes. This is done by preceding the name of the | .', CAPTION, 'pipe',TEXTSIZE, '3', CAPTIONSIZE, '3', FGCOLOR, '#DEDED4', BGCOLOR, '#D8D8C4', CAPCOLOR, 'BLACK', STATUS, 'click me for more details');" onmouseout="nd();" target="glossary">pipe by the pipe-symbol (|). I find this a useful way of sending log messages to other programs, where I can process them further. Named pipes are created using the mkfifo(1) command and must exist prior to syslogd starting.

Another action is the ability to send messages to particular users, provided they are logged in at the moment. To do this you simply put their username as the action. To send it to multiple users, separate the names by a comma. This might give you something like this:

*.emerg   root,jimmo

By using an asterisk in place of the list of user names, you can send a message to everyone logged in.

In some cases, you want multiple actions for a specific facility or priority. This is no problem. You simply create multiple rules. One common example is broadcasting all of the emergency messages to every user, as well as writing them to a log file **and** sending them to another server in case the local machine crashes. This might be done like this:

*.emerg   /var/adm/messages

*.emerg   *

*.emerg   @logserver

Previously, I mentioned the ability to cause a single action based on the same kind of messages for multiple facilities. This is still an example of a single selector resulting in a specific action. Taking this one step further, you might want multiple selectors all to result in a specific action. Although it could be done with multiple rules, it possible to have multiple selectors all on the same line. This is done by separating the selectors with a semi-colon (;).

*.emerg;kernel.critical   root,jimmo

This would notify the users root and jimmo for all emergency messages as well as critical messages from the kernel facility.

The Linux syslogd has added a couple of functions that are not available in other versions of UNIX. By preceding a priority with an equal-sign (=), you tell syslogd only to react to that one priority. This is useful since syslogd normally reacts to everything with that priority and higher. One place where this is useful is when you want all debugging messages to be logged to a specific file, but everything logged to another file.

You can also explicitly exclude priorities by preceding them with an exclamation mark. Note that this will exclude the priorities listed as well as anything higher. You can combine the equal-sign and exclamation mark equal-sign and exclamation mark and therefore exclude a specific priority. If you do so, you need to precede the equal sign with the exclamation mark as what you are saying is not to include anything that equal a particular priority.

All of these features can be combined in many different ways. For example, you can have multiple selectors, which include as well as exclude specific priorities. For example:

*.warn;mail.=info;lpr.none;uucp.!crit   /dev/tty07

This would send warning messages from all priorities to the system console terminal /dev/tty7, plus the mail log messages at only the info priority, no printer messages at all, and finally excluding just the uucp critical messages. Granted this is a rather contrived example, but it does show you how complex you can get.

Note that multiple selectors on a single line can cause some confusion when there are conflicting components within a selector. The thing to keep in mind is that the last component takes precedence. In the previous example, we specified warning messages for all facilities and then "overwrote" portions of that for the mail, lpr and uucp facilities.