File¶

The File consumer reads messages from a file, looking for a customizable delimiter sequence that marks the end of a message. If the file is part of e.g. a log rotation, the consumer can be set to read from a symbolic link pointing to the current file and (optionally) be told to reopen the file by sending a SIGHUP. A symlink to a file will automatically be reopened if the underlying file is changed.

Metadata¶

NOTE: The metadata will only set if the parameter `SetMetadata` is active.

file

The file name of the consumed file (set)

dir

The directory of the consumed file (set)

Parameters¶

Enable (default: true)

Switches this plugin on or off.

File

This value is a mandatory setting and contains the name of the file to read. The file will be read from beginning to end and the reader will stay attached until the consumer is stopped, so appends to the file will be recognized automatically.

OffsetFile

This value defines the path to a file that stores the current offset inside the source file. If the consumer is restarted, that offset is used to continue reading from the previous position. To disable this setting, set it to “”. By default this parameter is set to “”.

Delimiter (default: n)

This value defines the delimiter sequence to expect at the end of each message in the file. By default this parameter is set to “n”.

ObserveMode (default: poll)

This value select how the source file is observed. Available values are poll and watch. NOTE: The watch implementation uses the [fsnotify/fsnotify](https://github.com/fsnotify/fsnotify) package. If your source file is rotated (moved or removed), please verify that your file system and distribution support the RENAME and REMOVE events; the consumer’s stability depends on them. By default this parameter is set to poll.

DefaultOffset

This value defines the default offset from which to start reading within the file. Valid values are “oldest” and “newest”. If OffsetFile is defined and the file exists, the DefaultOffset parameter is ignored. By default this parameter is set to “newest”.

PollingDelay

This value defines the duration the consumer waits between checking the source file for new content after hitting the end of file (EOF). The value is in milliseconds (ms). NOTE: This settings only takes effect if the consumer is running in poll mode! By default this parameter is set to “100”.

SetMetadata (default: false)

When this value is set to “true”, the fields mentioned in the metadata section will be added to each message. Adding metadata will have a performance impact on systems with high throughput. By default this parameter is set to “false”.

Parameters (from core.SimpleConsumer)¶

Streams

Defines a list of streams a consumer will send to. This parameter is mandatory. When using “*” messages will be sent only to the internal “*” stream. It will NOT send messages to all streams. By default this parameter is set to an empty list.

ShutdownTimeoutMs (default: 1000, unit: ms)

Defines the maximum time in milliseconds a consumer is allowed to take to shut down. After this timeout the consumer is always considered to have shut down. By default this parameter is set to 1000.

Modulators

Defines a list of modulators to be applied to a message before it is sent to the list of streams. If a modulator specifies a stream, the message is only sent to that specific stream. A message is saved as original after all modulators have been applied. By default this parameter is set to an empty list.

ModulatorRoutines

Defines the number of go routines reserved for modulating messages. Setting this parameter to 0 will use as many go routines as the specific consumer plugin is using for fetching data. Any other value will force the given number fo go routines to be used. By default this parameter is set to 0

ModulatorQueueSize

Defines the size of the channel used to buffer messages before they are fetched by the next free modulator go routine. If the ModulatorRoutines parameter is set to 0 this parameter is ignored. By default this parameter is set to 1024.

Examples¶

This example will read the /var/log/system.log file and create a message for each new entry.

FileIn:
  Type: consumer.File
  File: /var/log/system.log
  DefaultOffset: newest
  OffsetFile: ""
  Delimiter: "\n"
  ObserveMode: poll
  PollingDelay: 100