ProcessJSON

This formatter allows modification of JSON encoded data. Each field can be processed by different directives and the result of all directives will be stored back to the original location.

Parameters

GeoIPFile

Defines a GeoIP file to load. This enables the “geoip” directive. If no file is loaded IPs will not be resolved. Files can be found e.g. at http://dev.maxmind.com/geoip/geoip2/geolite2/. By default this parameter is set to “”.

TrimValues (default: true)

Allows trimming of whitespaces from the beginning and end of each value after processing took place. By default this parameter is set to true.

Directives

Defines an array of actions to be applied to the JSON encoded data. Directives are processed in order of their appearance. Directives start with the name of the field, followed by an action followed by additional parameters if necessary. Parameters, key and action are separated by using the “:” character. By default this parameter is set to an empty list.

split

<delimiter> {<key>, <key>, …} Split the field’s value by the given delimiter, store the results to the fields listed after the delimiter.

replace

<string> <new string> Replace a given string inside the field’s value with a new one.

trim

<characters> Remove the given characters from the start and end of the field’s value.

rename

<new key> Rename a given field

remove

{<value>, <value>, …}` Remove a given field. If additional parameters are given, the value is expected to be an array. The given strings will be removed from that array.

pick

<index> <key> Pick a specific index from an array and store it to the given field.

time

<from fromat> <to format> Read a timestamp with a given format compatible to time.Parse and transform it into another format compatible with time.Format.

unixtimestamp

<unit> <to format> Read a unix timestamp with a given unit (“s”,”ms” or “ns”) and transform it it into another format compatible with time.Format.

flatten

{<delimiter>} Move all keys from a nested object to new fields named field + delimiter + subfield. If no delimiter is given “.” will be used.

agent

<prefix> {<field>, <field>, …} Parse the field’s value as a user agent string and extract the given fields into new fields named prefix + “_” + field. If no fields are given all fields are returned.

mozilla

mozilla version

platform

the platform used

os

the operating system used

localization

the language used

engine

codename of the browser engine

engine_version

version of the browser engine

browser

name of the browser

version

version of the browser

ip

Parse the field as an array of strings and remove all values that cannot be parsed as a valid IP. Single-string fields are supported, too, but will be converted to an array.

geoip

{<field>, <field>, …} Parse the field as an IP and extract the given fields into new fields named prefix + “_” + field. This action requires a valid GeoIP file to be loaded. If no fields are given all fields are returned.

country

the contry code of the IP. Generates country, countryCode.

city

the city of the IP

continent

the continent of the IP. Generates continent, continentCode.

timezone

the timezome of the IP

proxy

name of the proxy if applying Generates proxy, satellite.

location

the geolocation of this IP. Generates geocoord, geohash.

Parameters (from core.SimpleFormatter)

ApplyTo

This value chooses the part of the message the formatting should be applied to. Use “” to target the message payload; other values specify the name of a metadata field to target. By default this parameter is set to “”.

SkipIfEmpty

When set to true, this formatter will not be applied to data that is empty or - in case of metadata - not existing. By default this parameter is set to false

Examples

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - format.ProcessJSON:
      Directives:
        - "host:split: :host:@timestamp"
        - "@timestamp:time:20060102150405:2006-01-02 15\\:04\\:05"
        - "client:ip"
        - "client:geoip:location:country"
        - "ua:agent:ua:os:engine:engine_version"