Write to Elasticsearch (ElasticSearch producer)

Description

This example can be used for developing or testing the ElasticSearch producer.

gollum config

With the following config gollum will create a console.consumer with a ElasticSearch.producer. All data which write to the console will send to ElasticSearch.

This payload can be used for the configured setup:

{"user" : "olivere", "message" : "It's a Raggy Waltz"}

gollum >= v0.5.0

consumerConsole:
    type: consumer.Console
    Streams: "write"

producerElastic:
    Type: producer.ElasticSearch
    Streams: write
    User: elastic
    Password: changeme
    Servers:
        - http://127.0.0.1:9200
    Retry:
        Count: 3
        TimeToWaitSec: 5
    SetGzip: true
    StreamProperties:
        write:
            Index: twitter
            DayBasedIndex: true
            Type: tweet
            Mapping:
                user: keyword
                message: text
            Settings:
                number_of_shards: 1
                number_of_replicas: 1

This config example can also be found here

ElasticSearch setup for docker

Here you find a docker-compose setup which works for the config example:

version: '2'
services:
  elasticsearch1:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.4.1
    container_name: elasticsearch1
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.4.1
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch1"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local

networks:
  esnet:

This docker-compose file can be run by:

docker-compose -f docker-compose-elastic.yml up