Plugins

The main components, consumers, routers, producers, filters and formatters are build upon a plugin architecture. This allows each component to be exchanged and configured individually with a different sets of options.

Plugin types:

• Consumers
• Producers
• Routers
• Filters
• Formatters

Aggregate plugins

To simplify complex pipeline configs you are able to aggregate plugin configurations. That means that all settings witch are defined in the aggregation scope will injected to each defined “sub-plugin”.

To define an aggregation use the keyword Aggregate as plugin type.

Parameters

Plugins

List of plugins witch will instantiate and get the aggregate settings injected.

Examples

In this example both consumers get the streams and modulator injected from the aggregation settings:

AggregatePipeline:
  Type: Aggregate
  Streams: console
  Modulators:
    - format.Envelope:
        Postfix: "\n"
  Plugins:
    consumerFoo:
      Type: consumer.File
      File: /tmp/foo.log
    consumerBar:
      Type: consumer.File
      File: /tmp/bar.log

This example shows a second use case to reuse server settings easier:

consumerConsole:
  Type: consumer.Console
  Streams: write

kafka:
  Type: Aggregate
  Servers:
    - kafka0:9092
    - kafka1:9093
    - kafka2:9094

  Plugins:
    producer:
      Type: producer.Kafka
      Streams: write
      Compression: zip
      Topics:
        write: test

    consumer:
      Type: consumer.Kafka
      Streams: read
      Topic: test
      DefaultOffset: Oldest

producerConsole:
  Type: producer.Console
  Streams: read