1

I’m using syslog-ng to filter logs before sending them to Splunk, but logging suddenly stops and I don’t understand why.

systemctl status syslog-ng

says running

syslog-ng-ctl stats give me:

Error connecting control socket, socket=‘/var/lib/syslog-ng/syslog-ng.ctl’, error=‘Permission denied’

I’m running it on Ubuntu 20.04 as root.

Logs keeps getting written to the file but are not sent over UDP.

If I restart syslog-ng it still fails, if I reboot the machine logging is restored.

What could be the issue?

My config:

@version: 3.25
@include "scl.conf"
# First, set some global options.
options {
 chain_hostnames(off);
 flush_lines(0);
 use_dns(no);
 use_fqdn(no);
 dns_cache(no);
 owner("root");
 group("adm");
 perm(0640);
 create_dirs (yes);
 dir-perm(0700);
 stats_freq(0);
 bad_hostname("^gconfd$");
};
@include "/etc/syslog-ng/conf.d/*.conf"

####CONF.D##########
##########
 # Source #
##########
source s_udp {
 udp();
};
##########
 # Filter #
##########
# filters are all like that
filter remove_matching_logs { not match("string" value(MSG)); };
#...
############
 # Template #
############
template no_head {
 template ("${MSGHDR}${MSG}\n"); template_escape(no);
};

###############
 # Destination #
###############
destination d_splunk {
  udp("192.168.200.11" port(514) template(no_head));
};
destination d_file {
 file("/var/log/syslogtemp" template(no_head));
};
##############
 # Log Action #
##############
log {
 source(s_udp);
 filter(remove_matching_logs);
#...
 destination(d_splunk);
 destination(d_file);
};
9 Spice ups
2

How long does it work before it suddenly stops? Have you looked at the logs around the time it stops working?

I would say that the most likely issue is that another daemon is binding to that port, but that obviously would require something/somebody to stop syslog-ng first, and then start something else.

So another thing to try would be to see whether there is another process binding to port 514 (with netstat) when it stops working.