Setup a Centralized Log Server with Rsyslog in CentOS/RHEL 8

In order for a system administrators to recognize or analyze problems on a CentOS 8 or RHEL 8 server, it is important to know and view the events that occurred on the server in a particular period of time from log files found in /var/log the directory in the system.

The Syslog (System Logging Protocol) system on the server can act as a central log monitoring point over a network where all servers, network devices, switches, routers and internal services that create logs, whether linked to the particular internal issue or just informative messages can send their logs.

On a CentOS/RHEL 8 server, Rsyslog daemon is the most important log server that comes preinstalled by default, followed by Systemd Journal Daemon (journald).

Rsyslog is an open-source utility, developed as a client/server architecture service and can achieve both roles independently. It can run as a server and gather all logs transmitted by other devices over the network or it can run as a client by sending all internal system events logged to a remote Syslog server.

Requirements

1. Installation of “CentOS 8.0″ with Screenshots
2. Installation of RHEL 8 with Screenshots

In order to set up a centralized log server on a CentOS/RHEL 8 server, you need to check an confirm that the /var partition has enough space (a few GB minimum) to store all recorded log files on the system that send by other devices on the network. I recommend you to have a separate drive (LVM or RAID) to mount the /var/log/ directory.

How to Configure Rsyslog Server in CentOS/RHEL 8

1. As I said, Rsyslog service is installed and running automatically in CentOS/RHEL 8 server. In order to verify that the daemon is running in the system, run the following command.

# systemctl status rsyslog.service

If the service is not running by default, run the following command to start rsyslog daemon.

# systemctl start rsyslog.service

2. If the Rsyslog utility is not installed by default on the system that you plan to use as a centralized logging server, run the following dnf command to install the rsyslog package and start the daemon.

# dnf install rsyslog
# systemctl start rsyslog.service

3. Once Rsyslog utility installed, you can now configure rsyslog as a centralized logging server by opening the main configuration file /etc/rsyslog.conf, in order to receive log messages for external clients.

# vi /etc/rsyslog.conf

In the /etc/rsyslog.conf configuration file, find and uncomment the following lines to grant UDP transport reception to the Rsyslog server via 514 port. Rsyslog uses the standard UDP protocol for log transmission.

module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

4. The UDP protocol doesn’t have the TCP overhead, and it makes data transmission faster than the TCP protocol. On the other hand, the UDP protocol doesn’t guarantee the reliability of transmitted data.

However, if you want to use TCP protocol for log reception you must find and uncomment the following lines in the /etc/rsyslog.conf the configuration file in order to configure Rsyslog daemon to bind and listen to a TCP socket on 514 port.

module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")

5. Now create a new template for receiving remote messages, as this template will guide the local Rsyslog server, where to save the received messages send by Syslog network clients.

$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" 
*.* ?RemoteLogs

The $template RemoteLogs directive guides Rsyslog daemon to gather and write all of the transmitted log messages to distinct files, based on the client name and remote client application that created the messages based on the outlined properties added in the template configuration: %HOSTNAME% and %PROGRAMNAME%.

All received log files will be written to the local filesystem to an allocated file named after the client machine’s hostname and kept in /var/log/ directory.

The & ~ redirect rule directs the local Rsyslog server to stop processing the received log message further and remove the messages (not write them to internal log files).

The RemoteLogs is an arbitrary name given to this template directive. You can use whatever name you want that best suitable for your template.

To configure more complex Rsyslog templates, read the Rsyslog configuration file manual by running the man rsyslog.conf command or consult Rsyslog online documentation.

# man rsyslog.conf

6. After making the above configuration changes, you can restart the Rsyslog daemon in order to apply recent changes by running the following command.

# service rsyslog restart

7. Once you restarted the Rsyslog server, it should now act as a centralized log server and record messages from Syslog clients. To confirm the Rsyslog network sockets, run netstat command and use grep utility to filter rsyslog string.

# netstat -tulpn | grep rsyslog 

If netstat command not intall on CentOS 8, you can install it using the following command.

# dnf whatprovides netstat
# dnf install net-tools

8. If you have SELinux active in CentOS/RHEL 8, run the following command to allow rsyslog traffic depending on network socket type.

# semanage port -a -t syslogd_port_t -p udp 514
# semanage port -a -t syslogd_port_t -p tcp 514

If semanage command not install on CentOS 8, you can install it using the following command.

# dnf whatprovides semanage
# dnf install policycoreutils-python-utils

9. If you have a firewall active on the system, run the following command in order to add the needed rules for allowing rsyslog traffic on ports in Firewalld.

# firewall-cmd --permanent --add-port=514/tcp
# firewall-cmd --permanent --add-port=514/udp
# firewall-cmd --reload

You can also limit incoming connections on port 514 from whitelisted IP ranges as shown.

# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="tcp" accept'
# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="udp" accept'
# firewall-cmd --reload

That’s all! Rsyslog is now configured as a centralize logs server and can collect logs from remote clients. In the next article, we will see how to configure Rsyslog client on CentOS/RHEL 8 server.