File¶

The File consumer reads messages from a file, looking for a customizable delimiter sequence that marks the end of a message. If the file is part of e.g. a log rotation, the consumer can be set to read from a symbolic link pointing to the current file and (optionally) be told to reopen the file by sending a SIGHUP. A symlink to a file will automatically be reopened if the underlying file is changed.

Metadata¶

NOTE: The metadata will only set if the parameter `SetMetadata` is active.

file

The file name of the consumed file (set)

dir

The directory of the consumed file (set)

Parameters¶

Enable (default: true)

Switches this plugin on or off.

File

This value is a mandatory setting and contains the name of the file to read. This field supports glob patterns. If the file pointed to is a symlink, changes to the symlink will be detected. The file will be watched for changes, so active logfiles can be scraped, too.

OffsetFilePath

This value defines a path where the individual, current file offsets are stored. The filename will the name and extension of the source file plus the extension “.offset”. If the consumer is restarted, these offset files are used to continue reading from the previous position. To disable this setting, set it to “”. By default this parameter is set to “”.

Delimiter (default: n)

This value defines the delimiter sequence to expect at the end of each message in the file. By default this parameter is set to “n”.

ObserveMode (default: poll)

This value select how the source file is observed. Available values are poll and watch. NOTE: The watch implementation uses the [fsnotify/fsnotify](https://github.com/fsnotify/fsnotify) package. If your source file is rotated (moved or removed), please verify that your file system and distribution support the RENAME and REMOVE events; the consumer’s stability depends on them. By default this parameter is set to poll.

DefaultOffset (default: newest)

This value defines the default offset from which to start reading within the file. Valid values are “oldest” and “newest”. If OffsetFile is defined and the file exists, the DefaultOffset parameter is ignored. By default this parameter is set to “newest”.

PollingDelayMs (default: 100, unit: ms)

This value defines the duration in milliseconds the consumer waits between checking the source file for new content after hitting the end of file (EOF). NOTE: This settings only takes effect if the consumer is running in poll mode! By default this parameter is set to “100”.

RetryDelaySec (default: 3, unit: s)

This value defines the duration in seconds the consumer waits between retries, e.g. after not being able to open a file. By default this parameter is set to “3”.

DirScanIntervalSec (default: 10, unit: s)

Only applies when using globs. This setting will define the interval in secnds in which the glob will be re-evaluated and new files can be scraped. By default this parameter is set to “10”.

SetMetadata (default: false)

When this value is set to “true”, the fields mentioned in the metadata section will be added to each message. Adding metadata will have a performance impact on systems with high throughput. By default this parameter is set to “false”.

BlackList

A regular expression matching file paths to NOT read. When both BlackList and WhiteList are defined, the WhiteList takes precedence. This setting is only used when glob expressions (*, ?) are present in the filename. The path checked is the one before symlink evaluation. By default this parameter is set to “”.

WhiteList

A regular expression matching file paths to read. When both BlackList and WhiteList are defined, the WhiteList takes precedence. This setting is only used when glob expressions (*, ?) are present in the filename. The path checked is the one before symlink evaluation. By default this parameter is set to “”.

Files (default: /var/log/*.log)

(no documentation available)

Parameters (from core.SimpleConsumer)¶

Streams

Defines a list of streams a consumer will send to. This parameter is mandatory. When using “*” messages will be sent only to the internal “*” stream. It will NOT send messages to all streams. By default this parameter is set to an empty list.

ShutdownTimeoutMs (default: 1000, unit: ms)

Defines the maximum time in milliseconds a consumer is allowed to take to shut down. After this timeout the consumer is always considered to have shut down. By default this parameter is set to 1000.

Modulators

Defines a list of modulators to be applied to a message before it is sent to the list of streams. If a modulator specifies a stream, the message is only sent to that specific stream. A message is saved as original after all modulators have been applied. By default this parameter is set to an empty list.

ModulatorRoutines

Defines the number of go routines reserved for modulating messages. Setting this parameter to 0 will use as many go routines as the specific consumer plugin is using for fetching data. Any other value will force the given number fo go routines to be used. By default this parameter is set to 0

ModulatorQueueSize

Defines the size of the channel used to buffer messages before they are fetched by the next free modulator go routine. If the ModulatorRoutines parameter is set to 0 this parameter is ignored. By default this parameter is set to 1024.

Examples¶

This example will read all the .log files /var/log/ into one stream and create a message for each new entry. If the file starts with sys it is ignored

FileIn:
  Type: consumer.File
  File: /var/log/*.log
  BlackList '^sys.*'
  DefaultOffset: newest
  OffsetFilePath: ""
  Delimiter: "\n"
  ObserveMode: poll
  PollingDelay: 100