Use rsyslog omprog with a Python script

If you want to filter some specific logs and redirect them to another server or to another file, you can use the module provided with rsyslog called “omprog”.

Thanks to that module, you can just ask rsyslog to execute a script on every logs you have from a specific application and process it with your own rules.

Let’s go with a simple example, I want to filter my logs from postfix and only extract few specific information.

Below is the configuration I will now push to /etc/rsyslog.d/rsyslog_postfix.conf in order to:

1. Redirect the current log to /var/log/maillog as a default path (to keep the raw log)
2. Process every log line with my python script in order to extract only some specific informations that I will write in /var/log/rsyslog_postfix.log

module(load="omprog")

mail.* {
    action(type="omfile" 
        file="/var/log/maillog")
    action(type="omprog"
        name="rsyslog_postfix"
        binary="/usr/bin/python3 /opt/rsyslog_postfix.py"
        queue.type="LinkedList"
        queue.size="20000"
        queue.saveOnShutdown="on"
        queue.workerThreads="4"
        queue.workerThreadMinimumMessages="5000"
        action.resumeInterval="5"
        output="/var/log/rsyslog_postfix.log")
    stop
}

Rsyslog script

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# python_version  :3.6
#===========================

import sys
import traceback
import datetime

if __name__ == '__main__':
    data = {}
    while True:
        try:
            line = sys.stdin.readline()
            msg = line.strip()
            if msg != "":
                data.clear()
                timestamp = msg.split(' ')[0]
                proc= msg.split(' ')[2].split('[')[0]
                pid = msg.split(' ')[2].split('[')[1].split(']')[0]
                log = msg.split(' ', 3)[3]
                if proc.startswith("postfix/"):
                    data = { 
                        "Timestamp": timestamp,
                        "Process": proc,
                        "PID": pid,
                    }
                    data["PostfixID"] = log.split(' ')[0].split(':')[0]
                print(data)
        except Exception as e:
            err = traceback.format_exc()
            print(err)

Now, I just need to restart rsyslogd process and the logs will start to be processed as expected, and you should get the raw logs in /var/log/maillog and the filtered ones in /var/log/rsyslog_postfix.log

systemctl restart rsyslog