AwsKinesis

This consumer reads a message from an AWS Kinesis router.

Parameters

Enable (default: true)

Switches this plugin on or off.

KinesisStream (default: default)

This value defines the stream to read from. By default this parameter is set to “default”.

OffsetFile

This value defines a file to store the current offset per shard. To disable this parameter, set it to “”. If the parameter is set and the file is found, consuming will start after the offset stored in the file. By default this parameter is set to “”.

RecordsPerQuery (default: 100)

This value defines the number of records to pull per query. By default this parameter is set to “100”.

RecordMessageDelimiter

This value defines the string to delimit messages within a record. To disable this parameter, set it to “”. By default this parameter is set to “”.

QuerySleepTimeMs (default: 1000, unit: ms)

This value defines the number of milliseconds to sleep before trying to pull new records from a shard that did not return any records. By default this parameter is set to “1000”.

RetrySleepTimeSec

This value defines the number of seconds to wait after trying to reconnect to a shard. By default this parameter is set to “4”.

CheckNewShardsSec (default: 0, unit: sec)

This value sets a timer to update shards in Kinesis. You can set this parameter to “0” for disabling. By default this parameter is set to “0”.

DefaultOffset

This value defines the message index to start reading from. Valid values are either “newest”, “oldest”, or a number. By default this parameter is set to “newest”.

Parameters (from components.AwsCredentials)

Credential/Type (default: none)

This value defines the credentials that are to be used when connecting to aws. Available values are listed below. See https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#Credentials for more information.

environment

Retrieves credentials from the environment variables of the running process

static

Retrieves credentials value for individual credential fields

shared

Retrieves credentials from the current user’s home directory

none

Use a anonymous login to aws

Credential/Id

is used for “static” type and is used as the AccessKeyID

Credential/Token

is used for “static” type and is used as the SessionToken

Credential/Secret

is used for “static” type and is used as the SecretAccessKey

Credential/File

is used for “shared” type and is used as the path to your shared Credentials file (~/.aws/credentials)

Credential/Profile (default: default)

is used for “shared” type and is used for the profile

Credential/AssumeRole

This value is used to assume an IAM role using. By default this is set to “”.

Parameters (from components.AwsMultiClient)

Region (default: us-east-1)

This value defines the used aws region. By default this is set to “us-east-1”

Endpoint

This value defines the used aws api endpoint. If no endpoint is set the client needs to set the right endpoint for the used region. By default this is set to “”.

Parameters (from core.SimpleConsumer)

Streams

Defines a list of streams a consumer will send to. This parameter is mandatory. When using “*” messages will be sent only to the internal “*” stream. It will NOT send messages to all streams. By default this parameter is set to an empty list.

ShutdownTimeoutMs (default: 1000, unit: ms)

Defines the maximum time in milliseconds a consumer is allowed to take to shut down. After this timeout the consumer is always considered to have shut down. By default this parameter is set to 1000.

Modulators

Defines a list of modulators to be applied to a message before it is sent to the list of streams. If a modulator specifies a stream, the message is only sent to that specific stream. A message is saved as original after all modulators have been applied. By default this parameter is set to an empty list.

ModulatorRoutines

Defines the number of go routines reserved for modulating messages. Setting this parameter to 0 will use as many go routines as the specific consumer plugin is using for fetching data. Any other value will force the given number fo go routines to be used. By default this parameter is set to 0

ModulatorQueueSize

Defines the size of the channel used to buffer messages before they are fetched by the next free modulator go routine. If the ModulatorRoutines parameter is set to 0 this parameter is ignored. By default this parameter is set to 1024.

Examples

This example consumes a kinesis stream “myStream” and create messages:

KinesisIn:
  Type: consumer.AwsKinesis
  Credential:
    Type: shared
    File: /Users/<USERNAME>/.aws/credentials
    Profile: default
  Region: "eu-west-1"
  KinesisStream: myStream