ProcessJSON¶
This formatter allows modification of JSON encoded data. Each field can be processed by different directives and the result of all directives will be stored back to the original location.
Parameters¶
GeoIPFile
Defines a GeoIP file to load. This enables the “geoip” directive. If no file is loaded IPs will not be resolved. Files can be found e.g. at http://dev.maxmind.com/geoip/geoip2/geolite2/. By default this parameter is set to “”.
TrimValues (default: true)
Allows trimming of whitespaces from the beginning and end of each value after processing took place. By default this parameter is set to true.
Directives
Defines an array of actions to be applied to the JSON encoded data. Directives are processed in order of their appearance. Directives start with the name of the field, followed by an action followed by additional parameters if necessary. Parameters, key and action are separated by using the “:” character. By default this parameter is set to an empty list.
split
<delimiter> {<key>, <key>, …} Split the field’s value by the given delimiter, store the results to the fields listed after the delimiter.replace
<string> <new string> Replace a given string inside the field’s value with a new one.trim
<characters> Remove the given characters from the start and end of the field’s value.rename
<new key> Rename a given fieldremove
{<value>, <value>, …}` Remove a given field. If additional parameters are given, the value is expected to be an array. The given strings will be removed from that array.pick
<index> <key> Pick a specific index from an array and store it to the given field.time
<from fromat> <to format> Read a timestamp with a given format compatible to time.Parse and transform it into another format compatible with time.Format.unixtimestamp
<unit> <to format> Read a unix timestamp with a given unit (“s”,”ms” or “ns”) and transform it it into another format compatible with time.Format.flatten
{<delimiter>} Move all keys from a nested object to new fields named field + delimiter + subfield. If no delimiter is given “.” will be used.agent
<prefix> {<field>, <field>, …} Parse the field’s value as a user agent string and extract the given fields into new fields named prefix + “_” + field. If no fields are given all fields are returned.
mozilla
mozilla versionplatform
the platform usedos
the operating system usedlocalization
the language usedengine
codename of the browser engineengine_version
version of the browser enginebrowser
name of the browserversion
version of the browserip
Parse the field as an array of strings and remove all values that cannot be parsed as a valid IP. Single-string fields are supported, too, but will be converted to an array.geoip
{<field>, <field>, …} Parse the field as an IP and extract the given fields into new fields named prefix + “_” + field. This action requires a valid GeoIP file to be loaded. If no fields are given all fields are returned.
country
the contry code of the IP. Generates country, countryCode.city
the city of the IPcontinent
the continent of the IP. Generates continent, continentCode.timezone
the timezome of the IPproxy
name of the proxy if applying Generates proxy, satellite.location
the geolocation of this IP. Generates geocoord, geohash.
Parameters (from core.SimpleFormatter)¶
ApplyTo
This value chooses the part of the message the formatting should be applied to. Use “” to target the message payload; other values specify the name of a metadata field to target. By default this parameter is set to “”.
SkipIfEmpty
When set to true, this formatter will not be applied to data that is empty or - in case of metadata - not existing. By default this parameter is set to false
Examples¶
ExampleConsumer:
Type: consumer.Console
Streams: console
Modulators:
- format.ProcessJSON:
Directives:
- "host:split: :host:@timestamp"
- "@timestamp:time:20060102150405:2006-01-02 15\\:04\\:05"
- "client:ip"
- "client:geoip:location:country"
- "ua:agent:ua:os:engine:engine_version"