S3

This producer sends data to an AWS S3 Bucket.

Parameters

Enable

Enable switches the consumer on or off. By default this value is set to true.

ID

ID allows this producer to be found by other plugins by name. By default this is set to “” which does not register this producer.

Channel

Channel sets the size of the channel used to communicate messages. By default this value is set to 8192.

ChannelTimeoutMs

ChannelTimeoutMs sets a timeout in milliseconds for messages to wait if this producer’s queue is full. A timeout of -1 or lower will drop the message without notice. A timeout of 0 will block until the queue is free. This is the default. A timeout of 1 or higher will wait x milliseconds for the queues to become available again. If this does not happen, the message will be send to the retry channel.

ShutdownTimeoutMs

ShutdownTimeoutMs sets a timeout in milliseconds that will be used to detect a blocking producer during shutdown. By default this is set to 3 seconds. If processing a message takes longer to process than this duration, messages will be dropped during shutdown.

Stream

Stream contains either a single string or a list of strings defining the message channels this producer will consume. By default this is set to “*” which means “listen to all streams but the internal”.

DropToStream

DropToStream defines the stream used for messages that are dropped after a timeout (see ChannelTimeoutMs). By default this is _DROPPED_.

Formatter

Formatter sets a formatter to use. Each formatter has its own set of options which can be set here, too. By default this is set to format.Forward. Each producer decides if and when to use a Formatter.

Filter

Filter sets a filter that is applied before formatting, i.e. before a message is send to the message queue. If a producer requires filtering after formatting it has to define a separate filter as the producer decides if and where to format.

Fuse

Fuse defines the name of a fuse to burn if e.g. the producer encounters a lost connection. Each producer defines its own fuse breaking logic if necessary / applyable. Disable fuse behavior for a producer by setting an empty name or a FuseTimeoutSec <= 0. By default this is set to “”.

FuseTimeoutSec

FuseTimeoutSec defines the interval in seconds used to check if the fuse can be recovered. Note that automatic fuse recovery logic depends on each producer’s implementation. By default this setting is set to 10.

Region

Region defines the amazon region of your s3 bucket. By default this is set to “eu-west-1”.

Endpoint

Endpoint defines the amazon endpoint for your s3 bucket. By default this is set to “s3-eu-west-1.amazonaws.com”.

StorageClass

StorageClass defines the amazon s3 storage class for objects created, from http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html By default this is set to “STANDARD”.

CredentialType

CredentialType defines the credentials that are to be used when connecting to s3. This can be one of the following: environment, static, shared, none. Static enables the parameters CredentialId, CredentialToken and CredentialSecret shared enables the parameters CredentialFile and CredentialProfile. None will not use any credentials and environment will pull the credentials from environmental settings. By default this is set to none.

BatchMaxMessages

BatchMaxMessages defines the maximum number of messages to upload per batch. By default this is set to 5000.

ObjectMaxMessages

ObjectMaxMessages defines the maximum number of messages to join into an s3 object. By default this is set to 5000.

ObjectMessageDelimiter

ObjectMessageDelimiter defines the string to delimit messages within an s3 object. By default this is set to “n”.

SendTimeframeMs

SendTimeframeMs defines the timeframe in milliseconds in which a second batch send can be triggered. By default this is set to 10000, i.e. ten upload operations per second per s3 path.

BatchTimeoutSec

BatchTimeoutSec defines the number of seconds after which a batch is flushed automatically. By default this is set to 30.

TimestampWrite

TimestampWrite defines the go timestamp format that will be used in naming objects. Objects are named <s3_path><timestamp><sha1>. By default timestamp is set to “2006-01-02T15:04:05”.

PathFormatter

PathFormatter can define a formatter that extracts the path suffix for an s3 object from the object data. By default this is uses the sha1 of the object. A good formatter for this can be format.Identifier.

Compress

Compress defines whether to gzip compress the object before uploading. This adds a ”.gz” extension to objects. By default this is set to false.

LocalPath

LocalPath defines the local output directory for temporary object files. Files will be stored as “<path>/<number>”. Compressed files will have a .gz extension. State will be stored in “<path>/state”. By default this is not set, and objects will be built in memory.

UploadOnShutdown

UploadOnShutdown defines whether to upload all temporary object files on shutdown. This has no effect if LocalPath is not set. By default this is false.

FileMaxAgeSec

FileMaxAgeSec defines the maximum age of a local file before it is uploaded. This defaults to 3600 (1 hour).

FileMaxMB

FileMaxMB defines the maximum size of a local file before it is uploaded. This limit is imposed before compression occurs. This defaults to 1000 (1 GB).

StreamMapping

StreamMapping defines a translation from gollum stream to s3 bucket/path. If no mapping is given the gollum stream name is used as s3 bucket. Values are of the form bucket/path or bucket, s3:// prefix is not allowed. The full path of the object will be s3://<StreamMapping><Timestamp><PathFormat> where Timestamp is time the object is written formatted with TimestampWrite, and PathFormat is the output of PathFormatter when passed the object data.

Example

- "producer.S3":
    Enable: true
    ID: ""
    Channel: 8192
    ChannelTimeoutMs: 0
    ShutdownTimeoutMs: 3000
    Formatter: "format.Forward"
    Filter: "filter.All"
    DropToStream: "_DROPPED_"
    Fuse: ""
    FuseTimeoutSec: 5
    Stream:
        - "foo"
        - "bar"
    Region: "eu-west-1"
    Endpoint: "s3-eu-west-1.amazonaws.com"
    StorageClass: "STANDARD"
    CredentialType: "none"
    CredentialId: ""
    CredentialToken: ""
    CredentialSecret: ""
    CredentialFile: ""
    CredentialProfile: ""
    BatchMaxMessages: 5000
    ObjectMaxMessages: 5000
    ObjectMessageDelimiter: "\n"
    SendTimeframeMs: 10000
    BatchTimeoutSec: 30
    TimestampWrite: "2006-01-02T15:04:05"
    PathFormatter: ""
    Compress: false
    LocalPath: ""
    UploadOnShutdown: false
    FileMaxAgeSec: 3600
    FileMaxMB: 1000
    StreamMapping:
        "*" : "bucket/path"