ProcessJSON

ProcessJSON is a formatter that allows modifications to fields of a given JSON message. The message is modified and returned again as JSON.

Parameters

ProcessJSONDataFormatter
ProcessJSONDataFormatter formatter that will be applied before ProcessJSONDirectives are processed.
ProcessJSONGeoIPFile
ProcessJSONGeoIPFile defines a GeoIP file to load. This enables the “geoip” directive. If no file is loaded IPs will not be resolved. Files can be found e.g. at http://dev.maxmind.com/geoip/geoip2/geolite2/.
ProcessJSONDirectives

ProcessJSONDirectives defines the action to be applied to the json payload. Directives are processed in order of appearance. The directives have to be given in the form of key:operation:parameters, where operation can be one of the following.

  • split:<string>{:<key>:<key>:...} Split the value by a string and set the resulting array elements to the given fields in order of appearance.
  • replace:<old>:<new> replace a given string in the value with a new one * trim:<characters> remove the given characters (not string!) from the start and end of the value * rename:<old>:<new> rename a given field * remove{:<string>:<string>...} remove a given field. If additional parameters are given, an array is expected. Strings given as additional parameters will be removed from that array * pick:<key>:<index>:<name> Pick a specific index from an array and store it in a new field.
  • time:<read>:<write> read a timestamp and transform it into another format * unixtimestamp:<read>:<write> read a unix timestamp and transform it into another format. valid read formats are s, ms, and ns.
  • flatten{:<delimiter>} create new fields from the values in field, with new fields named field + delimiter + subfield. Delimiter defaults to ”.”. Removes the original field.
  • agent:<key>{:<field>:<field>:...} Parse the value as a user agent string and extract the given fields into <key>_<field>. (“ua:agent:browser:os” would create the new fields “ua_browser” and “ua_os”). Possible values are: “mozilla”, “platform”, “os”, “localization”, “engine”, “engine_version”, “browser”, “version”.
  • ip Parse the field as an array of strings and remove all values that cannot be parsed as a valid IP. Single-string fields are supported, too, but will be converted to an array.
  • geoip:{<field>:<field>:...} like agent this directive will analyse an IP string via geoip and produce new fields. Possible values are: “country”, “city”, “continent”, “timezone”, “proxy”, “location”.
ProcessJSONTrimValues
ProcessJSONTrimValues will trim whitspaces from all values if enabled. Enabled by default.

Example

- "stream.Broadcast":
    Formatter: "format.ProcessJSON"
    ProcessJSONDataFormatter: "format.Forward"
    ProcessJSONGeoIPFile: ""
    ProcessJSONDirectives:
        - "host:split: :host:@timestamp"
        - "@timestamp:time:20060102150405:2006-01-02 15\\:04\\:05"
        - "error:replace:°:\n"
        - "text:trim: \t"
        - "foo:rename:bar"
        - "foobar:remove"
        - "array:pick:0:firstOfArray"
        - "array:remove:foobar"
        - "user_agent:agent:browser:os:version"
        - "client:geoip:country:city:timezone:location"
    ProcessJSONTrimValues: true