published on in Splunk ELK Metrics FreeBSD

ELK Stack

Needless Explanation

I have been a long time Splunk user, and I think it is great.

Still, I have had my eyes on other log aggregation and analytic tools for a while. Price is usually the pain point with Splunk, ask anyone, I also run FreeBSD, which was recently dropped as a supported server OS.

There is still a universal forwarder for FreeBSD, and I am still pleased with that.

ELK Stack is a term I hear(read) a lot these days, ELK standing for:

  • ElasticSearch (underlying database)
  • Logstash (collect and transform inputs, then output)
  • Kibana (Node web front-end)

Greylog is another similar tool but I wont dive into that in this post.

I Stood up a small VM here at Stanford to dive into this toolset. Stanford does have a central Splunk server but we (myself and John Gerth) wanted to see how ELK would compare, especially if we used it to index a very large set of data (which would cost a LOT of Splunk to do)

Up-front Conclusion

I think ELK is a great product, but it has a while to go to catch up to Splunk. There are certain things that me, a sysadmin, found much easier in Splunk:

  • Field Extractions
  • Simple time charts
  • Splunks query language
  • Any kind of report in general
  • Examples, community and documentation
  • All-in-one product. Splunk is pretty turn key

However, lets take a look at what ELK does right, and in my opinion, better than Splunk:

  • Indices! It shards data automatically
  • ElasticSearch is by its very nature, distributed. So clustering works very well
  • Open source (which is great, because it is in the FreeBSD ports tree)
  • (metric|top|file|net)beats. Collecting system metrics and piping it into ElasticSearch or Logstash (and then to ElasticSearch) is a really nice tool
  • Flexibility
  • “Free”

If you have the resources to tune ELK, and the lesiure to setup a at minimum 3 node environment, it is really worth your time to investigate it.

If not, use Splunk. The costs are up-front, and not so much the hidden cost that comes with ELK (people time). Sometimes a product costs as much as it does because it is worth it

Still, everyone has a budget.

Installation (single node)

My evaluation VM at Stanford is actually running the 5.0.0-alpha4. I was really happy to see Elastic unify the version numbers. Keeping track of which logstash version was compatible with elasticsearch and kibana could be considered intimidating for someone new.

I will show the FreeBSD install from pkgng.

pkg install elasticsearch2 logstash kibana45
vim /usr/local/etc/elasticsearch/elasticsearch.yml
vim /usr/local/etc/logstash/logstash.conf
vim /usr/local/etc/kibana.yml
sysrc elasticsearch_enable=YES
sysrc logstash_enable=YES
sysrc kibana_enable=YES

Configuration

ElasticSearch

/usr/local/etc/elasticsearch/elasticsearch.yml

 node.name: server
 path.data: /var/db/elasticsearch
 path.logs: /var/log/elasticsearch
 network.host: 127.0.0.1
 http.port: 9200
 discovery.zen.minimum_master_nodes: 1

Logstash

/usr/local/etc/logstash/logstash.conf

input {

 file {
  type => "syslog"
  path => [ "/var/log/messages", "/var/log/auth.log" ]
  start_position => "beginning"
 }

 file {
  type => "minecraft"
  path => "/var/log/minecraft-server/latest.log"
  start_position => "beginning"
 }
}

filter {
 if [type] == "syslog" {
  grok {
   match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} (%{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}|%{GREEDYDATA:syslog_message})" }
   add_field => [ "received_at", "%{@timestamp}" ]
   add_field => [ "received_from", "%{@source_host}" ]
  }
                                                                
 if !("_grokparsefailure" in [tags]) {
   mutate {
    replace => [ "@source_host", "%{syslog_hostname}" ]
    replace => [ "@message", "%{syslog_message}" ]
   }
  }
  date {
   match => [ "syslog_timestamp","MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] 
  }
  syslog_pri { }
  }
                                                                                                               if [type] == "minecraft" {
   grok {
    match => { "message" => "\[%{TIME:time}\] \[Server thread/INFO\]: %{DATA:user} %{GREEDYDATA:message2}" }
    break_on_match => false
   }
 
   grok {
    match => [ "message",  "\[%{TIME:time}\] \[Server thread/INFO\]: (?<user>\S+) joined the game$" ]
   }

   grok {
    match => [ "message",  "\[%{TIME:time}\] \[Server thread/INFO\]: (?<user>\S+) has just earned the achievement \[(?<achievement>[^\[]+)\]$" ]
   }

   grok {
    match => [ "message",  "\[%{TIME:time}\] \[Server thread/INFO\]: (?<user>\S+) left the game$" ]
   }

   grok {
    match => [ "message", "\[%{TIME:time}\] \[Server thread/INFO\]: <(?<user>\S+)> (?<chat>.*$)" ]
   }

   grok {
   }
 }
}

output {
 elasticsearch { 
  hosts => [ "localhost:9200" ] 
 }
}

Kibana + nginx

/usr/local/etc/kibana.yml

 server.port: 5601
 server.host: "127.0.0.1"
 server.basePath: "/kibana"
 elasticsearch.url: "http://localhost:9200"
 elasticsearch.preserveHost: true

Starting it up

service elasticsearch start
service logstash start
service kibana start

Resources

Examples