Grok

Grok is a formatter that applies regex filters to messages and stores the result as metadata fields. If the target key is not existing it will be created. If the target key is existing but not a map, it will be replaced. It works by combining text patterns into something that matches your logs. See https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_grok_basics for more information about Grok.

Parameters

RemoveEmptyValues

When set to true, empty captures will not be returned. By default this parameter is set to “true”.

NamedCapturesOnly

When set to true, only named captures will be returned. By default this parameter is set to “true”.

SkipDefaultPatterns

When set to true, standard grok patterns will not be included in the list of patterns. By default this parameter is set to “true”.

Patterns

A list of grok patterns that will be applied to messages. The first matching pattern will be used to parse the message.

Parameters (from core.SimpleFormatter)

Source

This value chooses the part of the message the data to be formatted should be read from. Use “” to target the message payload; other values specify the name of a metadata field to target. By default this parameter is set to “”.

Target

This value chooses the part of the message the formatted data should be stored to. Use “” to target the message payload; other values specify the name of a metadata field to target. By default this parameter is set to “”.

ApplyTo

Use this to set Source and Target to the same value. This setting will be ignored if either Source or Target is set to something else but “”. By default this parameter is set to “”.

SkipIfEmpty

When set to true, this formatter will not be applied to data that is empty or - in case of metadata - not existing. By default this parameter is set to false

Examples

This example transforms unstructured input into a structured json output. Input:

us-west.servicename.webserver0.this.is.the.measurement 12.0 1497003802

Output:

{
  "datacenter": "us-west",
  "service": "servicename",
  "host": "webserver0",
  "measurement": "this.is.the.measurement",
  "value": "12.0",
  "time": "1497003802"
}

Config:

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - format.Grok:
      Patterns:
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.gauge-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_gauge:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.latency-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_latency:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.statsd\.derive-(?P<application>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value_derive:float}\s*%{INT:time}
        - ^(?P<datacenter>[^\.]+?)\.(?P<service>[^\.]+?)\.(?P<host>[^\.]+?)\.(?P<measurement>[^\s]+?)\s%{NUMBER:value:float}\s*%{INT:time}

       - format.ToJSON: {}