AwsKinesis¶
This consumer reads a message from an AWS Kinesis router.
Parameters¶
Enable (default: true)
Switches this plugin on or off.
KinesisStream (default: default)
This value defines the stream to read from. By default this parameter is set to “default”.
OffsetFile
This value defines a file to store the current offset per shard. To disable this parameter, set it to “”. If the parameter is set and the file is found, consuming will start after the offset stored in the file. By default this parameter is set to “”.
RecordsPerQuery (default: 100)
This value defines the number of records to pull per query. By default this parameter is set to “100”.
RecordMessageDelimiter
This value defines the string to delimit messages within a record. To disable this parameter, set it to “”. By default this parameter is set to “”.
QuerySleepTimeMs (default: 1000, unit: ms)
This value defines the number of milliseconds to sleep before trying to pull new records from a shard that did not return any records. By default this parameter is set to “1000”.
RetrySleepTimeSec
This value defines the number of seconds to wait after trying to reconnect to a shard. By default this parameter is set to “4”.
CheckNewShardsSec (default: 0, unit: sec)
This value sets a timer to update shards in Kinesis. You can set this parameter to “0” for disabling. By default this parameter is set to “0”.
DefaultOffset
This value defines the message index to start reading from. Valid values are either “newest”, “oldest”, or a number. By default this parameter is set to “newest”.
Parameters (from components.AwsCredentials)¶
Credential/Type (default: none)
This value defines the credentials that are to be used when connecting to aws. Available values are listed below. See https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/#Credentials for more information.
environment
Retrieves credentials from the environment variables of the running processstatic
Retrieves credentials value for individual credential fieldsshared
Retrieves credentials from the current user’s home directorynone
Use a anonymous login to aws
Credential/Id
is used for “static” type and is used as the AccessKeyID
Credential/Token
is used for “static” type and is used as the SessionToken
Credential/Secret
is used for “static” type and is used as the SecretAccessKey
Credential/File
is used for “shared” type and is used as the path to your shared Credentials file (~/.aws/credentials)
Credential/Profile (default: default)
is used for “shared” type and is used for the profile
Credential/AssumeRole
This value is used to assume an IAM role using. By default this is set to “”.
Parameters (from components.AwsMultiClient)¶
Region (default: us-east-1)
This value defines the used aws region. By default this is set to “us-east-1”
Endpoint
This value defines the used aws api endpoint. If no endpoint is set the client needs to set the right endpoint for the used region. By default this is set to “”.
Parameters (from core.SimpleConsumer)¶
Streams
Defines a list of streams a consumer will send to. This parameter is mandatory. When using “*” messages will be sent only to the internal “*” stream. It will NOT send messages to all streams. By default this parameter is set to an empty list.
ShutdownTimeoutMs (default: 1000, unit: ms)
Defines the maximum time in milliseconds a consumer is allowed to take to shut down. After this timeout the consumer is always considered to have shut down. By default this parameter is set to 1000.
Modulators
Defines a list of modulators to be applied to a message before it is sent to the list of streams. If a modulator specifies a stream, the message is only sent to that specific stream. A message is saved as original after all modulators have been applied. By default this parameter is set to an empty list.
ModulatorRoutines
Defines the number of go routines reserved for modulating messages. Setting this parameter to 0 will use as many go routines as the specific consumer plugin is using for fetching data. Any other value will force the given number fo go routines to be used. By default this parameter is set to 0
ModulatorQueueSize
Defines the size of the channel used to buffer messages before they are fetched by the next free modulator go routine. If the ModulatorRoutines parameter is set to 0 this parameter is ignored. By default this parameter is set to 1024.
Examples¶
This example consumes a kinesis stream “myStream” and create messages:
KinesisIn:
Type: consumer.AwsKinesis
Credential:
Type: shared
File: /Users/<USERNAME>/.aws/credentials
Profile: default
Region: "eu-west-1"
KinesisStream: myStream