"container_created_at" and "label" fields require extra handling when relaying from Docker to GELF

[avollmerhaus]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I'm trying to forward logs from my Docker installation to a Graylog server.
I pasted my vector.yaml file into the "configuration" input field down below.
Using that config, the following errors constantly pop up in Vector's logs:

 vector[19725]: 2023-11-06T15:32:32.746403Z ERROR sink{component_kind="sink" component_id=graylog component_type=socket component_name=graylog}: vector::internal_events::codecs: Failed serializing frame. error=LogEvent contains a value with an invalid type. field = "container_created_at" type = "timestamp" expected type = "string or number" error_type="encoder_failed" stage="sending" internal_log_rate_limit=true

vector[3511422]: 2023-11-10T07:49:13.240877Z ERROR sink{component_kind="sink" component_id=graylog component_type=socket}: vector::internal_events::codecs: Failed serializing frame. error=LogEvent contains a value with an invalid type. field = "label" type = "map" expected type = "string or number" error_code="encoder_serialize" error_type="encoder_failed" stage="sending" internal_log_rate_limit=true

I was able to work around this by using this VRL source:

.container_created_at = to_unix_timestamp(timestamp!(.container_created_at))
del(.label)

I'm not entirely sure what I'm loosing by deleting the "label" field.
I brought the problem up at #19072 and was advised to open a GitHub issue, so here we are :)

Configuration

sources:
  dockerd:
    type: docker_logs

sinks:
  graylog:
    type: "socket"
    inputs: ["dockerd"]
    address: "graylog.example.com:12201"
    mode: "tcp"
    encoding:
      codec: "gelf"

Version

vector 0.34.0 (x86_64-unknown-linux-gnu c909b66 2023-11-07 15:07:26.748571656)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[bruceg]

I can reproduce this with the following configuration:

sources:
  stdin:
    type: stdin
    decoding:
      codec: json
sinks:
  console:
    type: console
    inputs: [stdin]
    encoding:
      codec: gelf

With any of the following input:

{"message":"test","label":{}}
{"message":"test","label":true}

Worse yet, this malformed input causes Vector to immediately exit, which is unexpected:

2023-11-17T02:58:14.550332Z ERROR sink{component_kind="sink" component_id=console component_type=console}: vector::internal_events::codecs: Failed serializing frame. error=LogEvent contains a value with an invalid type. field = "label" type = "map" expected type = "string or number" error_code="encoder_serialize" error_type="encoder_failed" stage="sending" internal_log_rate_limit=true
2023-11-17T02:58:14.550639Z ERROR sink{component_kind="sink" component_id=console component_type=console}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Failed serializing frame." internal_log_rate_limit=true
2023-11-17T02:58:14.550853Z ERROR sink{component_kind="sink" component_id=console component_type=console}: vector::topology: An error occurred that Vector couldn't handle: the task completed with an error.

As such, the docker_logs source is just one possible trigger for this behavior and is a more generic issue with the gelf codec.

[MartinEmrich]

Similar problem here. Indeed the vector GELF sink expects the whole event already to be GELF compliant.

(#18008 (comment))

I am making progress with this remap, flattening out everything except timestamp to single-depth string key-value pairs:

            # timestamp MUST be set and be a "timestamp" type, preserve.
            old_timestamp = .timestamp
            if (is_timestamp(.observed_timestamp)) {
              old_timestamp = .observed_timestramp
              del(.observed_timestamp)
            }

            # all other stuff MUST be number or string, no complex data structures. flatten!
            stuff = flatten(., "_")
            stuff = map_keys(stuff, recursive: true) -> |key| { replace(key, ".", "_") }
            stuff = map_values(stuff, true) -> |value| { if is_timestamp(value) { value } else if is_float(value) { value } else if is_integer(value) { value } else { join(value, ", ") ?? to_string(value) ?? value } }
            . = stuff

            # restore non_string timestamp
            if (is_timestamp(old_timestamp) || is_integer(old_timestamp)) {
              .timestamp = old_timestamp
            }