Kafka roundtrip¶
This example can be used for developing or testing kafka consumers and producers.
gollum config¶
With the following config gollum will create a console.consumer
with
a kafka.producer
and a kafka.consumer
with a
console.producer
. All data which write to the console will send to
kafka. The second kafka.consumer
will read all data from kafka and
send it back to your console by the console.producer
:
gollum >= v0.5.0¶
consumerConsole:
type: consumer.Console
Streams: "write"
producerKafka:
type: producer.Kafka
Streams: "write"
Compression: "zip"
Topics:
"write" : "test"
Servers:
- kafka0:9092
- kafka1:9093
- kafka2:9094
consumerKafka:
type: consumer.Kafka
Streams: "read"
Topic: "test"
DefaultOffset: "Oldest"
MaxFetchSizeByte: 100
Servers:
- kafka0:9092
- kafka1:9093
- kafka2:9094
producerConsole:
type: producer.Console
Streams: "read"
Modulators:
- format.Envelope:
Postfix: "\n"
This config example can also be found here
kafka setup for docker¶
Here you find a docker-compose setup which works for the gollum config example.
/etc/hosts entry¶
You need a valid /etc/hosts
entry to be able to use the set
hostnames:
# you can not use 127.0.0.1 or localhost here
<YOUR PUBLIC IP> kafka0 kafka1 kafka2
docker-compose file¶
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
- "2888:2888"
- "3888:3888"
kafkaone:
image: wurstmeister/kafka:0.10.0.0
ports:
- "9092:9092"
links:
- zookeeper:zookeeper
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka0
KAFKA_ZOOKEEPER_CONNECT: "zookeeper"
KAFKA_BROKER_ID: "21"
KAFKA_CREATE_TOPICS: "test:1:3,Topic2:1:1:compact"
kafkatwo:
image: wurstmeister/kafka:0.10.0.0
ports:
- "9093:9092"
links:
- zookeeper:zookeeper
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka1
KAFKA_ZOOKEEPER_CONNECT: "zookeeper"
KAFKA_BROKER_ID: "22"
KAFKA_CREATE_TOPICS: "test:1:3,Topic2:1:1:compact"
kafkathree:
image: wurstmeister/kafka:0.10.0.0
ports:
- "9094:9092"
links:
- zookeeper:zookeeper
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka2
KAFKA_ZOOKEEPER_CONNECT: "zookeeper"
KAFKA_BROKER_ID: "23"
KAFKA_CREATE_TOPICS: "test:1:3,Topic2:1:1:compact"
This docker-compose file can be run by
docker-compose -f docker-compose-kafka.yml -p kafka010 up