GrokToJSON

GrokToJSON is a formatter that applies regex filters to messages. It works by combining text patterns into something that matches your logs. See https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_grok_basics for more information about Grok.

The output format is JSON.

Parameters

Patterns

A list of grok patterns that will be applied to messages. The first matching pattern will be used to parse the message.

Parameters (from core.SimpleFormatter)

ApplyTo

This value chooses the part of the message the formatting should be applied to. Use “” to target the message payload; other values specify the name of a metadata field to target. By default this parameter is set to “”.

SkipIfEmpty

When set to true, this formatter will not be applied to data that is empty or - in case of metadata - not existing. By default this parameter is set to false

Examples

This example transforms unstructured input into a structured json output. Input:

us-west.servicename.webserver0.this.is.the.measurement 12.0 1497003802

Output:

{
  "datacenter": "us-west",
  "service": "servicename",
  "host": "webserver0",
  "measurement": "this.is.the.measurement",
  "value": "12.0",
  "time": "1497003802"
}

Config:

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.GrokToJSON:
      Patterns:
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.gauge-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_gauge:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.latency-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_latency:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.derive-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_derive:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value:float}\s*%{INT:time}