Syslog to New Relic Logs

2021-03-11

Dashboard

DISCLAIMER: I work for New Relic (at the time of this post). I am not being paid or asked to write this, my blog is my own personal thoughts and experiences.

I set up a personal account to use New Relic for FREE, with one goal in mind: migrate from Papertrail to NR Logs.

Why? I work there and I wanted to dog food a specific aspect of New Relic’s Logging product: TCP syslog

I’m not new to our logging product, we mostly use fluent-bit to send structured log data to the HTTP API (log-api.newrelic.com)

So why not do this at work, and keep your biased mouth shut? Because I have some devices that only speak udp syslog (networking devices and ancient SGI machines)

New Relic does not accept unaltered syslog data, and I’m not going to install Fluent Bit on IRIX 6.5.30 (or my running jails for that matter). I have decided to use one on my favorite logging tools, rsyslog.

Rsyslog is fast, feature rich and highly scalable.

This point however is my only complaint for the product. Splunk accepts udp/tcp syslog. Papertrail accepts udp/tcp syslog. ELK supports syslog.

New Relic Logging only supports its modified RFC5424. This means the following devices are incompatible:

  • Classic BSD syslog
  • Any simple syslog that doesn’t support re-writing the message
  • Journald
  • Network Devices (even my fancy new Ubiquiti Router)
  • Out of Band management devices like Dell’s iDRAC

All of these devices require a rsyslog or fluent-bit forwarder. This is additional infrastructure that needs care and feeding in an Enterprise environment.

With that, I’ll contiune with my implementation.

Creating a Jail

server :: ~ » sudo iocage create -r 12.2-RELEASE -n loghost boot=on
Password:

loghost successfully created!
* Starting loghost
  + Started OK
  + Using devfs_ruleset: 1006 (iocage generated default)
  + Using IP options: ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
  + Starting services OK
  + Executing poststart OK
server :: ~ sudo iocage set ip4_addr="re0|192.168.1.240/24" loghost
ip4_addr: none -> re0|192.168.1.240/24
server :: ~ »

Setting up using iocage console

server :: ~ » sudo iocage console loghost
FreeBSD 12.2-RELEASE-p3 GENERIC

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

Edit /etc/motd to change this login announcement.
root@loghost:~ #

I build my own packages, so I’ve set some non-default options for the rsyslog8 port:

 #_OPTIONS_READ=rsyslog-8.2012.0
_FILE_COMPLETE_OPTIONS_LIST=DBI DOCS ELASTIC GCRYPT GSSAPI HTTP JSONPARSE KAFKA MYSQL NORMALIZE PGSQL RABBITMQ RELP SNMP GNUTLS OPENSSL
OPTIONS_FILE_UNSET+=DBI
OPTIONS_FILE_SET+=DOCS
OPTIONS_FILE_UNSET+=ELASTIC
OPTIONS_FILE_SET+=GCRYPT
OPTIONS_FILE_SET+=GSSAPI
OPTIONS_FILE_SET+=HTTP
OPTIONS_FILE_SET+=JSONPARSE
OPTIONS_FILE_UNSET+=KAFKA
OPTIONS_FILE_UNSET+=MYSQL
OPTIONS_FILE_UNSET+=NORMALIZE
OPTIONS_FILE_UNSET+=PGSQL
OPTIONS_FILE_UNSET+=RABBITMQ
OPTIONS_FILE_UNSET+=RELP
OPTIONS_FILE_UNSET+=SNMP
OPTIONS_FILE_SET+=GNUTLS
OPTIONS_FILE_UNSET+=OPENSSL

With that, a simple pkg install -r pkgng rsyslog was all that was required.

Container Version

As much as I like jails, and its what I can run at home, Containers are also pretty cool.

Here is a simple Dockerfile that I’ve been using:

FROM	centos:7
RUN     yum -y install wget \
	&& cd /etc/yum.repos.d/ \
        && wget http://rpms.adiscon.com/v8-stable/rsyslog.repo
RUN	yum -y install rsyslog \
     rsyslog-gnutls \
     ca-certificates \
	   rsyslog-elasticsearch \
	   rsyslog-imptcp \
	   rsyslog-imrelp \
	   rsyslog-mmjsonparse \
	   rsyslog-omrelp \
	   rsyslog-omstdout \
	&& rm /etc/rsyslog.d/listen.conf
COPY	rsyslog.conf /etc/rsyslog.conf

ENTRYPOINT [ "rsyslogd", "-n", "-f", "/etc/rsyslog.conf" ]

Running the container:

docker run \
  --name rsyslog-fwd \
  -p 514:514/udp -p 514:514/tcp \
  -e NR_INSERT_KEY=YOURKEYHERE \
  -v /data/rsyslog:/logs \
  rsyslog-fwd-nr:latest

Noticed the env variable NR_INSERT_KEY, this is pulled in by rsyslog here:

set $/nr_insert_key=`echo $NR_INSERT_KEY`;

Config File

For simplicity sake, here is a single rsyslog config file that:

  • accepts tcp/udp syslog data
  • formats the messages to RFC-5424 and injects your NR insert key
  • uses a disk assisted queue and worker threads in case you are using this in production

New Relic’s TCP endpoint uses TLS for encryption. Your CA bundle location may vary depending on the OS.

The CentOS container image installs the ca-bundle.crt in /etc/pki/tls/certs/ca-bundle.crt, while FreeBSD’s ca_root_nss package installs a ca bundle in /etc/ssl/cert.pem -> /usr/local/share/certs/ca-root-nss.crt

$ModLoad imuxsock

module(load="imudp" threads="8" timeRequery="8" batchSize="128")
input(type="imudp" port="514" ruleset="sendToNR" )

module(load="imptcp" threads="4")
input(type="imptcp" port="514" ruleset="sendToNR" )

set $/nr_insert_key=`echo $NR_INSERT_KEY`;

template(name="newrelic-rfc5424"
         type="string"
         string="%$/nr_insert_key% <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n"
)


global(DefaultNetstreamDriver="gtls"
       DefaultNetstreamDriverCAFile="/etc/pki/tls/certs/ca-bundle.crt"
)

ruleset(name="sendToNR"
        queue.type="fixedArray"
        queue.size="250000"
        queue.dequeueBatchSize="4096"
        queue.workerThreads="4"
        queue.workerThreadMinimumMessages="60000"
        queue.spoolDirectory="/logs/queue"
        queue.filename="fwd"
        queue.saveOnShutdown="on"
        ) {
  action(type="omfwd"
       Target="newrelic.syslog.nr-data.net"
       Port="6514"
       Protocol="tcp"
       Template="newrelic-rfc5424"
       ResendLastMSGOnReconnect="on"
       StreamDriver="gtls"
       StreamDriverAuthMode="x509/name"
       StreamDriverPermittedPeers="*.syslog.nr-data.net"
       StreamDriverMode="1"
  )
}

The important NR Logs bit is this

set $/nr_insert_key=`echo $NR_INSERT_KEY`;

template(name="newrelic-rfc5424"
         type="string"
         string="%$/nr_insert_key% <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n"
)

Rsyslog is grabbing the env value for NR_INSERT_KEY, and building a custom RFC-5424 string to prepend the insert key to the log message.

The other fun bit is in the sendToNR ruleset. Which sets up queues and threads.

Disk Assisted Queues are different than disk based queues. Rsyslog has no-queues, memory queues, disk queues and disk assisted queues.

A Disk queue is the slowest, it spools to a file. So even if this is backed by crazy fast ssd or nvme devices, it is still slower than memory.

Memory queues like the above fixed array type, are held in memory. Really fast, but obviously limited in size and may not hold enough if your forward host is down. This config is not logging to persistent storage at all, so a memory only queue is not guaranteed to deliver every message.

This is what a DA queue is for. Once the memory queue is exhausted, it will spool to /logs/queue (in this case).

Client Configuration

This is pretty simple, most of my home infrastructure is running syslog, so we can just add:

*.* @loghost.home.michaelc.dev

Hey, I get 100G free a month with NR Logs so I don’t care how much I send.

For jails like Minecraft, I’ll flip that to rsyslog so I can setup a custom file input for the Java/Minecraft log.

Have fun

NRQL FTW

Despite working at New Relic for a while, I’m not an expert.

One thing I can strongly recommend is:

  • Use NRQL to query (your logs)

The default Log view in NR One is more for management types that say things like business enablment and success management. Even things like live tail are absolutely useless for data at scale.

My log query lineage originates with Splunk and grep, so Splunks query language is very natural to me. I’m not sure how (yet) to use NRQL and transform a message into a useful data type like “DNS Client” from this kind of message:

client @0x447c0c00 192.168.1.94#54663 (lastpass.com): query: lastpass.com IN A + (192.168.1.2)

Other things I will probably send to NR Logs that I may have a follow up post on:

  • Minecraft logs
  • DNS (I run two bind servers and a local zone)
  • GPS/Chrony logs
  • Other stupidly obsolete systems that I get my hands on

#Geekyness #Logging #UNIX