Scribe

This producer allows sending messages to Facebook’s scribe service.

Parameters

Enable (default: true)

Switches this plugin on or off.

Address

Defines the host and port of a scrive endpoint. By default this parameter is set to “localhost:1463”.

ConnectionBufferSizeKB (default: 1024, unit: kb)

Sets the connection socket buffer size in KB. By default this parameter is set to 1024.

HeartBeatIntervalSec (default: 5, unit: sec)

Defines the interval in seconds used to query scribe for status updates. By default this parameter is set to 1.

WindowSize (default: 2048)

Defines the maximum number of messages send to scribe in one call. The WindowSize will reduce when scribe is returing “try later” to reduce load on the scribe server. It will slowly rise again for each successful write until WindowSize is reached again. By default this parameter is set to 2048.

ConnectionTimeoutSec (default: 5, unit: sec)

Defines the time in seconds after which a connection timeout is assumed. This can happen during writes or status reads. By default this parameter is set to 5.

Category

Maps a stream to a scribe category. You can define the wildcard stream (*) here, too. When set, all streams that do not have a specific mapping will go to this category (including reserved streams like _GOLLUM_). If no category mappings are set the stream name is used as category. By default this parameter is set to an empty list.

Parameters (from core.BatchedProducer)

Batch/MaxCount (default: 8192)

Defines the maximum number of messages per batch. If this limit is reached a flush is always triggered. By default this parameter is set to 8192.

Batch/FlushCount (default: 4096)

Defines the minimum number of messages required to flush a batch. If this limit is reached a flush might be triggered. By default this parameter is set to 4096.

Batch/TimeoutSec (default: 5, unit: sec)

Defines the maximum time in seconds messages can stay in the internal buffer before being flushed. By default this parameter is set to 5.

Parameters (from core.SimpleProducer)

Streams

Defines a list of streams the producer will receive from. This parameter is mandatory. Specifying “*” causes the producer to receive messages from all streams except internal internal ones (e.g. _GOLLUM_). By default this parameter is set to an empty list.

FallbackStream

Defines a stream to route messages to if delivery fails. The message is reset to its original state before being routed, i.e. all modifications done to the message after leaving the consumer are removed. Setting this paramater to “” will cause messages to be discared when delivery fails.

ShutdownTimeoutMs (default: 1000, unit: ms)

Defines the maximum time in milliseconds a producer is allowed to take to shut down. After this timeout the producer is always considered to have shut down. Decreasing this value may lead to lost messages during shutdown. Raising it may increase shutdown time.

Modulators

Defines a list of modulators to be applied to a message when it arrives at this producer. If a modulator changes the stream of a message the message is NOT routed to this stream anymore. By default this parameter is set to an empty list.

Examples

logs:
  Type: producer.Scribe"
  Stream: ["*", "_GOLLUM"]
  Address: "scribe01:1463"
  HeartBeatIntervalSec: 10
  Category:
    "access"   : "accesslogs"
    "error"    : "errorlogs"
    "_GOLLUM_" : "gollumlogs"