GrokToJSON

GrokToJSON is a formatter that applies regex filters to messages. It works by combining text patterns into something that matches your logs. See https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_grok_basics for more information about Grok.

The output format is JSON.

Parameters

RemoveEmptyValues

When set to true, empty captures will not be returned. By default this parameter is set to “true”.

NamedCapturesOnly

When set to true, only named captures will be returned. By default this parameter is set to “true”.

SkipDefaultPatterns

When set to true, standard grok patterns will not be included in the list of patterns. By default this parameter is set to “true”.

Patterns

A list of grok patterns that will be applied to messages. The first matching pattern will be used to parse the message.

Parameters (from core.SimpleFormatter)

ApplyTo

This value chooses the part of the message the formatting should be applied to. Use “” to target the message payload; other values specify the name of a metadata field to target. By default this parameter is set to “”.

SkipIfEmpty

When set to true, this formatter will not be applied to data that is empty or - in case of metadata - not existing. By default this parameter is set to false

Examples

This example transforms unstructured input into a structured json output. Input:

us-west.servicename.webserver0.this.is.the.measurement 12.0 1497003802

Output:

{
  "datacenter": "us-west",
  "service": "servicename",
  "host": "webserver0",
  "measurement": "this.is.the.measurement",
  "value": "12.0",
  "time": "1497003802"
}

Config:

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.GrokToJSON:
      Patterns:
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.gauge-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_gauge:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.latency-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_latency:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.derive-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_derive:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value:float}\s*%{INT:time}