AwsS3

This producer sends messages to Amazon S3.

Each “file” uses a configurable batch and sends the content by a multipart upload to s3. This principle avoids temporary storage on disk.

Please keep in mind that Amazon S3 does not support appending to existing objects. Therefore rotation is mandatory in this producer.

Parameters

Enable (default: true)

Switches this plugin on or off.

Bucket

The S3 bucket to upload to

File (default: gollum_*.log)

This value is used as a template for final file names. The string ” * ” will replaced with the active stream name. By default this parameter is set to “gollum_*.log”

Parameters (from components.AwsCredentials)

Credential/Type (default: none)

This value defines the credentials that are to be used when connecting to aws. Available values are listed below. See https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#Credentials for more information.

environment

Retrieves credentials from the environment variables of the running process

static

Retrieves credentials value for individual credential fields

shared

Retrieves credentials from the current user’s home directory

none

Use a anonymous login to aws

Credential/Id

is used for “static” type and is used as the AccessKeyID

Credential/Token

is used for “static” type and is used as the SessionToken

Credential/Secret

is used for “static” type and is used as the SecretAccessKey

Credential/File

is used for “shared” type and is used as the path to your shared Credentials file (~/.aws/credentials)

Credential/Profile (default: default)

is used for “shared” type and is used for the profile

Credential/AssumeRole

This value is used to assume an IAM role using. By default this is set to “”.

Parameters (from components.AwsMultiClient)

Region (default: us-east-1)

This value defines the used aws region. By default this is set to “us-east-1”

Endpoint

This value defines the used aws api endpoint. If no endpoint is set the client needs to set the right endpoint for the used region. By default this is set to “”.

Parameters (from components.BatchedWriterConfig)

Batch/TimeoutSec (default: 5, unit: sec)

This value defines the maximum number of seconds to wait after the last message arrived before a batch is flushed automatically. By default this parameter is set to “5”.

Batch/MaxCount (default: 8192)

This value defines the maximum number of messages that can be buffered before a flush is mandatory. If the buffer is full and a flush is still underway or cannot be triggered out of other reasons, the producer will block. By default this parameter is set to “8192”.

Batch/FlushCount (default: 4096)

This value defines the number of messages to be buffered before they are written to disk. This setting is clamped to “BatchMaxCount”. By default this parameter is set to “BatchMaxCount / 2”.

Batch/FlushTimeoutSec (default: 0, unit: sec)

This value defines the maximum number of seconds to wait before a flush is aborted during shutdown. Set this parameter to “0” which does not abort the flushing procedure. By default this parameter is set to “0”.

Parameters (from components.RotateConfig)

Rotation/Enable (default: false)

If this value is set to “true” the logs will rotate after reaching certain thresholds. By default this parameter is set to “false”.

Rotation/TimeoutMin (default: 1440, unit: min)

This value defines a timeout in minutes that will cause the logs to rotate. Can be set in parallel with RotateSizeMB. By default this parameter is set to “1440”.

Rotation/SizeMB (default: 1024, unit: mb)

This value defines the maximum file size in MB that triggers a file rotate. Files can get bigger than this size. By default this parameter is set to “1024”.

Rotation/Timestamp (default: 2006-01-02_15)

This value sets the timestamp added to the filename when file rotation is enabled. The format is based on Go’s time.Format function. By default this parameter is to to “2006-01-02_15”.

Rotation/ZeroPadding (default: 0)

This value sets the number of leading zeros when rotating files with an existing name. Setting this setting to 0 won’t add zeros, every other number defines the number of leading zeros to be used. By default this parameter is set to “0”.

Rotation/Compress (default: false)

This value defines if a rotated logfile is to be gzip compressed or not. By default this parameter is set to “false”.

Rotation/At

This value defines a specific time for rotation in hh:mm format. By default this parameter is set to “”.

Rotation/AtHour (default: -1)

(no documentation available)

Rotation/AtMin (default: -1)

(no documentation available)

Parameters (from core.SimpleProducer)

Streams

Defines a list of streams the producer will receive from. This parameter is mandatory. Specifying “*” causes the producer to receive messages from all streams except internal internal ones (e.g. _GOLLUM_). By default this parameter is set to an empty list.

FallbackStream

Defines a stream to route messages to if delivery fails. The message is reset to its original state before being routed, i.e. all modifications done to the message after leaving the consumer are removed. Setting this paramater to “” will cause messages to be discared when delivery fails.

ShutdownTimeoutMs (default: 1000, unit: ms)

Defines the maximum time in milliseconds a producer is allowed to take to shut down. After this timeout the producer is always considered to have shut down. Decreasing this value may lead to lost messages during shutdown. Raising it may increase shutdown time.

Modulators

Defines a list of modulators to be applied to a message when it arrives at this producer. If a modulator changes the stream of a message the message is NOT routed to this stream anymore. By default this parameter is set to an empty list.

Examples

This example sends all received messages from all streams to S3, creating a separate file for each stream:

S3Out:
  Type: producer.AwsS3
  Streams: "*"
  Credential:
    Type: shared
    File: /Users/<USERNAME>/.aws/credentials
    Profile: default
  Region: eu-west-1
  Bucket: gollum-s3-test
  Batch:
    TimeoutSec: 60
    MaxCount: 1000
    FlushCount: 500
    FlushTimeoutSec: 0
  Rotation:
    Timestamp: 2006-01-02T15:04:05.999999999Z07:00
    TimeoutMin: 1
    SizeMB: 20
  Modulators:
    - format.Envelope:
      Postfix: "\n"