ProcessTSV

This formatter allows modification of TSV encoded data. Each field can be processed by different directives and the result of all directives will be stored back to the original location.

Parameters

Delimiter (default: t)

Defines the separator used to split values. By default this parameter is set to “t”.

QuotedValue

When set to true, values that start and end with a quotation mark are not scanned for delimiter characters. I.e. those values will not be split even if they contain delimiter characters. By default this parameter is set to false.

Directives

Defines an array of actions to apply to the TSV encoded data. Directives are processed in order of appearance. Directives start with the index of the field, followed by an action followed by additional parameters if necessary. Parameters, key and action are separated by the “:” character. By default this parameter is set to an empty list.

replace

<string> <new string> Replace a given string inside the field’s value with another one.

prefix

<string> Prepend the given string to the field’s value

postfix

<string> Append the given string to the field’s value

trim

<characters> Remove the given characters from the start and end of the field’s value.

quote

Surround the field’s value with quotation marks after all directives have been processed.

time

<from fromat> <to format> Read a timestamp in the specified time.Parse-compatible format and transform it into another format compatible with time.Format.

** (unnamed) **

remove Removes the field from the result

agent

{<field>, <field>, …} Parse the field’s value as a user agent string and insert the given fields into the TSV after the given index. If no fields are given all fields are returned.

mozilla

mozilla version

platform

the platform used

os

the operating system used

localization

the language used

engine

codename of the browser engine

engine_version

version of the browser engine

browser

name of the browser

version

version of the browser

QuotedValues

(no documentation available)

Parameters (from core.SimpleFormatter)

ApplyTo

This value chooses the part of the message the formatting should be applied to. Use “” to target the message payload; other values specify the name of a metadata field to target. By default this parameter is set to “”.

SkipIfEmpty

When set to true, this formatter will not be applied to data that is empty or - in case of metadata - not existing. By default this parameter is set to false

Examples

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - format.processTSV:
      Delimiter: ","
      Directives:
        - "0:time:20060102150405:2006-01-02 15\\:04\\:05"
        - "2:remove"
        - "11:agent:os:engine:engine_version"