AwsKinesis¶
This producer sends data to an AWS kinesis stream. Configuration example
Parameters¶
Enable (default: true)
Switches this plugin on or off.
StreamMapping
This value defines a translation from gollum stream names to kinesis stream names. If no mapping is given the gollum stream name is used as the kinesis stream name. By default this parameter is set to “empty”
RecordMaxMessages (default: 1)
This value defines the maximum number of messages to join into a kinesis record. By default this parameter is set to “500”.
RecordMessageDelimiter (default: n)
This value defines the delimiter string to use between messages within a kinesis record. By default this parameter is set to “n”.
SendTimeframeMs (default: 1000, unit: ms)
This value defines the timeframe in milliseconds in which a second batch send can be triggered. By default this parameter is set to “1000”.
Parameters (from components.AwsCredentials)¶
Credential/Type (default: none)
This value defines the credentials that are to be used when connecting to aws. Available values are listed below. See https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#Credentials for more information.
environment
Retrieves credentials from the environment variables of the running processstatic
Retrieves credentials value for individual credential fieldsshared
Retrieves credentials from the current user’s home directorynone
Use a anonymous login to aws
Credential/Id
is used for “static” type and is used as the AccessKeyID
Credential/Token
is used for “static” type and is used as the SessionToken
Credential/Secret
is used for “static” type and is used as the SecretAccessKey
Credential/File
is used for “shared” type and is used as the path to your shared Credentials file (~/.aws/credentials)
Credential/Profile (default: default)
is used for “shared” type and is used for the profile
Credential/AssumeRole
This value is used to assume an IAM role using. By default this is set to “”.
Parameters (from components.AwsMultiClient)¶
Region (default: us-east-1)
This value defines the used aws region. By default this is set to “us-east-1”
Endpoint
This value defines the used aws api endpoint. If no endpoint is set the client needs to set the right endpoint for the used region. By default this is set to “”.
Parameters (from core.BatchedProducer)¶
Batch/MaxCount (default: 8192)
Defines the maximum number of messages per batch. If this limit is reached a flush is always triggered. By default this parameter is set to 8192.
Batch/FlushCount (default: 4096)
Defines the minimum number of messages required to flush a batch. If this limit is reached a flush might be triggered. By default this parameter is set to 4096.
Batch/TimeoutSec (default: 5, unit: sec)
Defines the maximum time in seconds messages can stay in the internal buffer before being flushed. By default this parameter is set to 5.
Parameters (from core.SimpleProducer)¶
Streams
Defines a list of streams the producer will receive from. This parameter is mandatory. Specifying “*” causes the producer to receive messages from all streams except internal internal ones (e.g. _GOLLUM_). By default this parameter is set to an empty list.
FallbackStream
Defines a stream to route messages to if delivery fails. The message is reset to its original state before being routed, i.e. all modifications done to the message after leaving the consumer are removed. Setting this paramater to “” will cause messages to be discared when delivery fails.
ShutdownTimeoutMs (default: 1000, unit: ms)
Defines the maximum time in milliseconds a producer is allowed to take to shut down. After this timeout the producer is always considered to have shut down. Decreasing this value may lead to lost messages during shutdown. Raising it may increase shutdown time.
Modulators
Defines a list of modulators to be applied to a message when it arrives at this producer. If a modulator changes the stream of a message the message is NOT routed to this stream anymore. By default this parameter is set to an empty list.
Examples¶
This example set up a simple aws Kinesis producer:
KinesisOut:
Type: producer.AwsKinesis
Streams: "*"
StreamMapping:
"*": default
Credential:
Type: shared
File: /Users/<USERNAME>/.aws/credentials
Profile: default
Region: eu-west-1
RecordMaxMessages: 1
RecordMessageDelimiter: "\n"
SendTimeframeSec: 1