Performance tests

History

All tests were executed by calling time gollum -c profile.conf -ll 1.

test ver user sys cpu msg/sec
Raw pipeline 0.6.0 11,45s 3,13s 200% 1.316.153
  0.5.0 11,85s 3,69s 173% 1.116.071
  0.4.6 10,48s 3,01s 178% 1.320.132
Basic formatting 0.6.0 37,63s 4,01s 520% 1.173.945
  0.5.0 39,70s 6,09s 532% 1.163.602
  0.4.6 [1] 21,84s 5,78s 206% 746.881
8 consumers 0.6.0 325,18s 18,32s 511% 1.137.784
  0.5.0 344,33s 28,24s 673% 1.446.157
  0.4.6 319,44s 72,22s 574% 1.173.536
JSON pipeline 0.6.0 14,98s 4,77s 173% 78.377
  0.5.0 28,23s 6,33s 138% 40.033
  0.4.6 28,30s 6,30s 150% 43.400
[1]this version does not use paralell formatting

v0.6.0

JSON pipeline

Intel Core i7-7700HQ CPU @ 2.80GHz, 16 GB RAM
go1.12.3 darwin/amd64
  • 14,98s user
  • 4,77s system
  • 173% cpu
  • 11,388s total
  • 73.846 msg/sec
"Profiler":
    Type: consumer.Profiler
    Runs: 10000
    Batches: 100
    Characters: "abcdefghijklmnopqrstuvwxyz .,!;:-_"
    Message: "{\"test\":\"%64s\",\"foo\":\"%32s|%32s\",\"bar\":\"%64s\",\"thisisquitealongstring\":\"%64s\"}"
    Streams: "profile"
    KeepRunning: false
    ModulatorRoutines: 0
    Modulators:
        - format.JSON: {}
        - format.Move:
            Source: "test"
            Target: "foobar"
        - format.Delete:
            Target: "bar"
        - format.SplitToFields:
            Source: "foo"
            Delimiter: "|"
            Fields: ["foo1","foo2"]
        - format.Copy:
            Source: "thisisquitealongstring"

"Benchmark":
    Type: "producer.Benchmark"
    Streams: "profile"

v0.5.0

Raw pipeline

Intel Core i7-4770HQ CPU @ 2.20GHz, 16 GB RAM
go1.8.3 darwin/amd64
  • 11,85s user
  • 3,69s system
  • 173% cpu
  • 8,960s total
  • 1.116.071 msg/sec
"Profiler":
    Type: "consumer.Profiler"
    Runs: 100000
    Batches: 100
    Characters: "abcdefghijklmnopqrstuvwxyz .,!;:-_"
    Message: "%256s"
    Streams: "profile"
    KeepRunning: false
    ModulatorRoutines: 0

"Benchmark":
    Type: "producer.Benchmark"
    Streams: "profile"

Basic formatting

Intel Core i7-4770HQ CPU @ 2.20GHz, 16 GB RAM
go1.8.3 darwin/amd64
Please note that from this version on formatting is done in parallel.
  • 39,70s user
  • 6,09s system
  • 532% cpu
  • 8,594s total
  • 1.163.602 msg/sec
"Profiler":
    Type: "consumer.Profiler"
    Runs: 100000
    Batches: 100
    Characters: "abcdefghijklmnopqrstuvwxyz .,!;:-_"
    Message: "%256s"
    Streams: "profile"
    KeepRunning: false
    ModulatorRoutines: 4
    Modulators:
        - format.Envelope
        - format.Timestamp

"Benchmark":
    Type: "producer.Benchmark"
    Streams: "profile"

8 consumers with formatting

Intel Core i7-4770HQ CPU @ 2.20GHz, 16 GB RAM
go1.8.3 darwin/amd64
  • 344,33s user
  • 28,24s system
  • 673% cpu
  • 55,319s total
  • 1.446.157 msg/sec
 "Profiler":
    Type: Aggregate
    Runs: 100000
    Batches: 100
    Characters: "abcdefghijklmnopqrstuvwxyz .,!;:-_"
    Message: "%256s"
    Streams: "profile"
    KeepRunning: false
    ModulatorRoutines: 0
    Modulators:
        - format.Envelope
        - format.Timestamp
    Plugins:
        P01:
            Type: "consumer.Profiler"
        P02:
            Type: "consumer.Profiler"
        P03:
            Type: "consumer.Profiler"
        P04:
            Type: "consumer.Profiler"
        P05:
            Type: "consumer.Profiler"
        P06:
            Type: "consumer.Profiler"
        P07:
            Type: "consumer.Profiler"
        P08:
            Type: "consumer.Profiler"

"Benchmark":
    Type: "producer.Benchmark"
    Streams: "profile"

JSON pipeline

Intel Core i7-4770HQ CPU @ 2.20GHz, 16 GB RAM
go1.8.3 darwin/amd64
  • 28,23s user
  • 6,33s system
  • 138% cpu
  • 24,979s total
  • 40.033 msg/sec
"Profiler":
    Type: consumer.Profiler
    Runs: 10000
    Batches: 100
    Characters: "abcdefghijklmnopqrstuvwxyz .,!;:-_"
    Message: "{\"test\":\"%64s\",\"foo\":\"%32s|%32s\",\"bar\":\"%64s\",\"thisisquitealongstring\":\"%64s\"}"
    Streams: "profile"
    KeepRunning: false
    ModulatorRoutines: 0
    Modulators:
        - format.ProcessJSON:
            Directives:
                - "test:rename:foobar"
                - "bar:remove"
                - "foo:split:|:foo1:foo2"
        - format.ExtractJSON:
            Field: thisisquitealongstring

"Benchmark":
    Type: "producer.Benchmark"
    Streams: "profile"

v0.4.6

Raw pipeline

Intel Core i7-4770HQ CPU @ 2.20GHz, 16 GB RAM
go1.8.3 darwin/amd64
  • 10,48s user
  • 3,01s system
  • 178% cpu
  • 7,575s total
  • 1.320.132 msg/sec
- "consumer.Profiler":
    Runs: 100000
    Batches: 100
    Characters: "abcdefghijklmnopqrstuvwxyz .,!;:-_"
    Message: "{\"test\":\"%64s\",\"foo\":\"%32s|%32s\",\"bar\":\"%64s\",\"thisisquitealongstring\":\"%64s\"}"
    Stream: "profile"
    KeepRunning: false

- "producer.Benchmark":
    Stream: "profile"

Basic formatting

Intel Core i7-4770HQ CPU @ 2.20GHz, 16 GB RAM
go1.8.3 darwin/amd64
  • 21,84s user
  • 5,78s system
  • 206% cpu
  • 13,389s total
  • 746.881 msg/sec
- "consumer.Profiler":
    Runs: 100000
    Batches: 100
    Characters: "abcdefghijklmnopqrstuvwxyz .,!;:-_"
    Message: "%256s"
    Stream: "profile"
    KeepRunning: false

- "stream.Broadcast":
    Stream: "profile"
    Formatter: format.Timestamp
    TimestampFormatter: format.Envelope

- "producer.Benchmark":
    Stream: "profile"

8 consumers with formatting

Intel Core i7-4770HQ CPU @ 2.20GHz, 16 GB RAM
go1.8.3 darwin/amd64
  • 319,44s user
  • 72,22s system
  • 574% cpu
  • 68,17s total
  • 1.173.536 msg/sec
- "consumer.Profiler":
    Instances: 8
    Runs: 100000
    Batches: 100
    Characters: "abcdefghijklmnopqrstuvwxyz .,!;:-_"
    Message: "%256s"
    Stream: "profile"
    KeepRunning: false

- "stream.Broadcast":
    Stream: "profile"
    Formatter: format.Timestamp
    TimestampFormatter: format.Envelope

- "producer.Benchmark":
    Stream: "profile"

JSON pipeline

Intel Core i7-4770HQ CPU @ 2.20GHz, 16 GB RAM
go1.8.3 darwin/amd64
  • 28,30s user
  • 6,30s system
  • 150% cpu
  • 23,041s total
  • 43.400 msg/sec
- "consumer.Profiler":
    Runs: 10000
    Batches: 100
    Characters: "abcdefghijklmnopqrstuvwxyz .,!;:-_"
    Message: "%256s"
    Stream: "profile"
    KeepRunning: false

- "stream.Broadcast":
    Stream: "profile"
    Formatter: format.ExtractJSON
    ExtractJSONdataFormatter: format.ProcessJSON
    ProcessJSONDirectives:
        - "test:rename:foobar"
        - "bar:remove"
        - "foo:split:|:foo1:foo2"
    ExtractJSONField: thisisquitealongstring

- "producer.Benchmark":
    Stream: "profile"