Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Packets to hostNetwork pods like envoy always contain RST flag #29406

Closed
2 tasks done
TECHNOFAB11 opened this issue Nov 27, 2023 · 10 comments
Closed
2 tasks done

Packets to hostNetwork pods like envoy always contain RST flag #29406

TECHNOFAB11 opened this issue Nov 27, 2023 · 10 comments
Labels
kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

Comments

@TECHNOFAB11
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

I tried using the Gateway API but Envoy always returns

upstream connect error or disconnect/reset before headers. reset reason: connection timeout

After sniffing on the "cilium_host" network I found a lot of packets from the backend service to envoy (10.42.0.246) which never arrive and get retransmitted a bunch. I then tried to ping and curl the IP from a busybox pod once with hostNetwork: true and once with it disabled but these packets also show up in wireshark with "no response found". The envoy pod/whatever might not respond to ICMP but it should definetely accept/receive the TCP responses from backend services.
Completely disabled all IPtables rules to rule that out, the packets did still not arrive successfully.

Cilium Version

cilium-cli: 0.15.11 compiled with go1.21.3 on linux/amd64
cilium image (default): v1.14.2
cilium image (stable): v1.14.4
cilium image (running): unknown. Unable to obtain cilium version, no cilium pods found in namespace "kube-system"

Image versions

  • cilium-operator quay.io/cilium/operator-generic:v1.15.0-pre.2@sha256:e5506ddf0665307cf4012a9fbe3f1a51c722b63a12e7abd357119b7c2d1f68af: 1
  • cilium-envoy quay.io/cilium/cilium-envoy:v1.27.2-ab187719b71b513150f30209569695adf16ec869@sha256:2b590be37624547d638a578a3f31278d3be53a1a2649ba888a9f15771628521e: 1
  • hubble-relay quay.io/cilium/hubble-relay:v1.15.0-pre.2@sha256:74b20caf534472e274eb020e8ceaf4abfe8d6540c282cb648eb4e0420bccf782: 1
  • cilium quay.io/cilium/cilium:v1.15.0-pre.2@sha256:7f077c49ce091428b04de006d5f4cb01b8e32d63dc1d45fa3e372d624681e836: 1
  • hubble-ui quay.io/cilium/hubble-ui:v0.12.1@sha256:9e5f81ee747866480ea1ac4630eb6975ff9227f9782b7c93919c081c33f38267: 1
  • hubble-ui quay.io/cilium/hubble-ui-backend:v0.12.1@sha256:1f86f3400827a0451e6332262467f894eeb7caf0eb8779bd951e2caa9d027cbe: 1

Kernel Version

Linux _ 6.5.10 #1-NixOS SMP PREEMPT_DYNAMIC Thu Nov 2 08:37:00 UTC 2023 x86_64 GNU/Linux

Kubernetes Version

clientVersion:
  buildDate: "1970-01-01T01:01:01Z"
  compiler: gc
  gitCommit: bd04941a294793ec92e8703d5e5da14107902e88
  gitTreeState: clean
  gitVersion: v1.27.6+k3s1
  goVersion: go1.20.10
  major: "1"
  minor: "27"
  platform: linux/amd64
  kustomizeVersion: v5.0.1
serverVersion:
  buildDate: "1970-01-01T01:01:01Z"
  compiler: gc
  gitCommit: bd04941a294793ec92e8703d5e5da14107902e88
  gitTreeState: clean
  gitVersion: v1.27.6+k3s1
  goVersion: go1.20.10
  major: "1"
  minor: "27"
  platform: linux/amd64

Sysdump

cilium-sysdump-20231127-154254.zip

Relevant log output

Wireshark after trying to access HTTPRoute, :8080 is the backend service, 10.42.0.246:33979 is envoy:

[TCP Retransmission] 8080 → 33979 [SYN, ACK] Seq=0 Ack=1 Win=64308 Len=0 MSS=1410 SACK_PERM TSval=3338499822 TSecr=1724972011 WS=128
a bunch more retransmissions...

Anything else?

Helm Chart values:

      kubeProxyReplacement = true;
      k8sServiceHost = "_";
      k8sServicePort = 6443;
      l2announcements.enabled = true;
      ipam.operator.clusterPoolIPv4PodCIDRList = "10.42.0.0/16";
      nodePort.enabled = true;
      gatewayAPI.enabled = true;
      operator.replicas = 1;
      loadBalancer.l7.backend = "envoy";
      prometheus.enabled = true;
      envoy.enabled = true;
      hubble = {
        relay.enabled = true;
        ui.enabled = true;
        tls.auto.method = "cronJob";
      };
      clustermesh.apiserver.tls.auto.method = "cronJob";

Code of Conduct

  • I agree to follow this project's Code of Conduct
@TECHNOFAB11 TECHNOFAB11 added kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. labels Nov 27, 2023
@youngnick youngnick added the sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. label Nov 28, 2023
@TECHNOFAB11
Copy link
Author

TECHNOFAB11 commented Dec 4, 2023 •
edited
Loading

This issue also happens for me on 1.14.4 on another NixOS cluster. I guess there is a configuration issue either with Cilium or NixOS, but I was not able to find out where the packets actually get dropped. I also tried some combinations of different Cilium Helm configs, like epf masquerading etc., the only thing that changed is that Envoy was not reachable anymore outside from localhost

Netstat shows that 0 packets got dropped. It actually shows a 0 for everything except RX-OK which actually looks fine but is weird

@TECHNOFAB11
Copy link
Author

Pretty sure this has to have something to do with these logs, even though I was told the issue is apparently a different one in my previous GH issue:

[debug][filter] [cilium/conntrack.cc:178] cilium.bpf_metadata: Using conntrack map global
[debug][filter] [cilium/conntrack.cc:128] cilium.bpf_metadata: Opened IPv4 conntrack map global
[info][filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success

@TECHNOFAB11
Copy link
Author

TECHNOFAB11 commented Dec 5, 2023 •
edited
Loading

Some trace level logs from cilium-envoy:

[trace][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:146] tls inspector: recv: 74
[debug][misc] [cilium/bpf_metadata.cc:224] EGRESS POD IP: 127.0.0.1, destination IP: 127.0.0.1
[debug][filter] [cilium/conntrack.cc:178] cilium.bpf_metadata: Using conntrack map global
[filter] [cilium/conntrack.cc:197] cilium.bpf_metadata: Looking up key: 7f000001, 7f000001, 291e, a092, 6, 0
[debug][filter] [cilium/conntrack.cc:128] cilium.bpf_metadata: Opened IPv4 conntrack map global
[filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success
[trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: 7f000001, prefixlen: 32
[debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 127.0.0.1 has ID 2
[trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: 7f000001, prefixlen: 32
[debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 127.0.0.1 has ID 2
[trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: a2a005f, prefixlen: 32
[debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 10.42.0.95 has ID 8
[debug][filter] [./cilium/socket_option.h:227] Cilium SocketOption(): source_identity: 8, ingress: false, port: 10526, pod_ip: 10.42.0.95, source_addresses: /10.42.0.95:0/, mark: 80b00 (magic mark: b00, cluster: 0, ID: 8), proxy_id: 0
[trace][main] [external/envoy/source/server/overload_manager_impl.cc:568] LoadShedPoint envoy.load_shed_points.http_connection_manager_decode_headers is not found. Is it configured?
[trace][misc] [external/envoy/source/common/event/scaled_range_timer_manager_impl.cc:60] enableTimer called on 0x258ebf55f6c0 for 3600000ms, min is 3600000ms
[debug][filter] [cilium/network_filter.cc:76] cilium.network: onNewConnection
[trace][connection] [external/envoy/source/common/network/connection_impl.cc:430] [Tags: "ConnectionId":"87"] raising connection event 2
[debug][conn_handler] [external/envoy/source/extensions/listener_managers/listener_manager/active_tcp_listener.cc:157] [Tags: "ConnectionId":"87"] new connection from 127.0.0.1:41106
[trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:251] item added to deferred deletion list (size=1)
[trace][main] [external/envoy/source/common/event/dispatcher_impl.cc:125] clearing deferred deletion list (size=1)
[trace][connection] [external/envoy/source/common/network/connection_impl.cc:575] [Tags: "ConnectionId":"87"] socket event: 3
[trace][connection] [external/envoy/source/common/network/connection_impl.cc:686] [Tags: "ConnectionId":"87"] write ready
[trace][connection] [external/envoy/source/common/network/connection_impl.cc:615] [Tags: "ConnectionId":"87"] read ready. dispatch_buffered_data=0
[trace][connection] [external/envoy/source/common/network/raw_buffer_socket.cc:24] [Tags: "ConnectionId":"87"] read returns: 74
[trace][connection] [external/envoy/source/common/network/raw_buffer_socket.cc:38] [Tags: "ConnectionId":"87"] read error: Resource temporarily unavailable
[trace][filter] [cilium/network_filter.cc:202] cilium.network: onData 74 bytes, end_stream: false
[main] [external/envoy/source/server/overload_manager_impl.cc:568] LoadShedPoint envoy.load_shed_points.http1_server_abort_dispatch is not found. Is it configured?
[trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:648] [Tags: "ConnectionId":"87"] parsing 74 bytes
[trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:590] [Tags: "ConnectionId":"87"] message begin
[debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:379] [Tags: "ConnectionId":"87"] new stream
[trace][misc] [external/envoy/source/common/event/scaled_range_timer_manager_impl.cc:60] enableTimer called on 0x258ebf55f180 for 300000ms, min is 300000ms
[trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:547] [Tags: "ConnectionId":"87"] completed header: key=Host value=example.com
[trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:547] [Tags: "ConnectionId":"87"] completed header: key=User-Agent value=curl/8.4.0
[trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:841] [Tags: "ConnectionId":"87"] onHeadersCompleteImpl
[trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:547] [Tags: "ConnectionId":"87"] completed header: key=Accept value=*/*
[trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:1192] [Tags: "ConnectionId":"87"] Server: onHeadersComplete size=3
[trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:945] [Tags: "ConnectionId":"87"] message complete

The ip is from the ingress/gateway:
10.42.0.95/32 reserved:ingress

@TECHNOFAB11
Copy link
Author

hubble observe -f --identity 8 (ingress) looks fine aswell, it looks like envoy keeps retrying and nginx answers but it never arrives back at envoy somehow:

10.42.0.95:37681 (ingress) -> components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-endpoint FORWARDED (TCP Flags: SYN)
10.42.0.95:37681 (ingress) <- components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-stack FORWARDED (TCP Flags: ACK, RST)
10.42.0.95:37681 (ingress) -> components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-endpoint FORWARDED (TCP Flags: SYN)
10.42.0.95:37681 (ingress) <- components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-stack FORWARDED (TCP Flags: ACK, RST)
10.42.0.95:37681 (ingress) -> components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-endpoint FORWARDED (TCP Flags: SYN)
10.42.0.95:37681 (ingress) <- components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-stack FORWARDED (TCP Flags: ACK, RST)
10.42.0.95:37681 (ingress) -> components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-endpoint FORWARDED (TCP Flags: SYN)
10.42.0.95:37681 (ingress) <- components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-stack FORWARDED (TCP Flags: ACK, RST)
10.42.0.95:37681 (ingress) -> components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-endpoint FORWARDED (TCP Flags: SYN)
10.42.0.95:37681 (ingress) <- components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-stack FORWARDED (TCP Flags: ACK, RST)
127.0.0.1:45446 (ingress) -> 127.0.0.1:10526 (world) http-request FORWARDED (HTTP/1.1 GET http://example.com/)
127.0.0.1:45446 (ingress) <- 127.0.0.1:10526 (world) http-response FORWARDED (HTTP/1.1 503 4999ms (GET http://example.com/))

@fangpenlin
Copy link

I encountered similar problem. Also with NixOS (not sure if it's the reason). But the error message I got is a bit different

[2023-12-15 09:52:13.643][25][info][filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: No such file or directory

It seems like somehow cilium.bpf_metadata failed to lookup the item in BPF map file (/sys/fs/bpf/tc/globals/cilium_ct4_global). Here's the systrace dump:

283   accept4(137, 0x7f05b51f8578, [128], SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
283   epoll_wait(72, [{events=EPOLLIN, data={u32=154, u64=154}}], 32, 0) = 1
283   recvfrom(154, "GET / HTTP/1.1\r\nHost: localhost:"..., 65536, MSG_PEEK, NULL, NULL) = 78
283   getsockname(154, {sa_family=AF_INET, sin_port=htons(19976), sin_addr=inet_addr("127.0.0.1")}, [128 => 16]) = 0
283   bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=118, key=0x7f05b51f83c8, value=0x7f05b51f8420, flags=BPF_ANY}, 72) = -1 ENOENT (No such file or directory)
283   close(119)                        = 0
283   close(118)                        = 0

The problem is bpf map lookup element operation returns ENOENT. According to Linux doc regarding BPF operation

https://man7.org/linux/man-pages/man2/bpf.2.html

If no element is found, the operation returns -1 and sets
errno to ENOENT.

So the No such file or directory is a bit misleading here.

@fangpenlin
Copy link 

[2023-12-15 19:52:33.454][43][debug][filter] [cilium/conntrack.cc:178] cilium.bpf_metadata: Using conntrack map global
[2023-12-15 19:52:33.454][43][trace][filter] [cilium/conntrack.cc:197] cilium.bpf_metadata: Looking up key: 7f000001, 7f000001, 288d, d2ee, 6, 0
[2023-12-15 19:52:33.454][43][info][filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: No such file or directory
[2023-12-15 19:52:33.454][43][trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: 7f000001, prefixlen: 32
[2023-12-15 19:52:33.454][43][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 127.0.0.1 has ID 2
[2023-12-15 19:52:33.454][43][trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: 7f000001, prefixlen: 32
[2023-12-15 19:52:33.454][43][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 127.0.0.1 has ID 2
[2023-12-15 19:52:33.454][43][trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: a1401c6, prefixlen: 32
[2023-12-15 19:52:33.454][43][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 10.20.1.198 has ID 8
[2023-12-15 19:52:33.454][43][debug][filter] [./cilium/socket_option.h:184] Cilium SocketOption(): source_identity: 8, ingress: false, port: 10381, pod_ip: 10.20.1.198, source_addresses: /10.20.1.198:0/, mark: 80b00 (magic mark: b00, cluster: 0, ID: 8)

The odd thing is, source ip is the same as dest IP 10.20.1.198. I am using a nodeport service here. not sure if that's the reason, maybe something in between set the source ip wrong? 🤔

@fangpenlin
Copy link 

[2023-12-15 20:25:11.209][15][debug][misc] [cilium/bpf_metadata.cc:215] EGRESS POD IP: 127.0.0.1, destination IP: 127.0.0.1
[2023-12-15 20:25:11.209][15][debug][filter] [cilium/conntrack.cc:178] cilium.bpf_metadata: Using conntrack map global
[2023-12-15 20:25:11.209][15][trace][filter] [cilium/conntrack.cc:197] cilium.bpf_metadata: Looking up key: 7f000001, 7f000001, 288d, b2f6, 6, 0
[2023-12-15 20:25:11.210][15][debug][filter] [cilium/conntrack.cc:128] cilium.bpf_metadata: Opened IPv4 conntrack map global
[2023-12-15 20:25:11.210][15][info][filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: No such file or directory
[2023-12-15 20:25:11.210][15][trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: 7f000001, prefixlen: 32
[2023-12-15 20:25:11.210][15][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 127.0.0.1 has ID 2
[2023-12-15 20:25:11.210][15][trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: 7f000001, prefixlen: 32
[2023-12-15 20:25:11.210][15][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 127.0.0.1 has ID 2
[2023-12-15 20:25:11.210][15][trace][filter] [cilium/ipcache.cc:82] cilium.ipcache: Looking up key: a1401c6, prefixlen: 32
[2023-12-15 20:25:11.210][15][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 10.20.1.198 has ID 8

oh, nope, somehow it's trying to look up in the conntrack map for 127.0.0.1 to 127.0.0.1 connection. No wonder it failed. I guess this is a different bug from the original reported one. Will create a new one.

@fangpenlin fangpenlin mentioned this issue Dec 15, 2023
Closed
2 tasks
@TECHNOFAB11 TECHNOFAB11 mentioned this issue Jan 8, 2024
Open
@TECHNOFAB11
Copy link
Author

I still cannot wrap my head around why it does not work with hostNetwork pods. I installed nginx-ingress controller, envoy gateway etc. and they all work fine as they do not use the hostNetwork. Also tried all kinds of configuration combinations on multiple different servers (changed helm values, iptables/firewall, kernel modules etc.) but it just breaks everything more most of the time.
cilium monitor and hubble observe show that all the response packets contain the TCP RST flag:

10.42.0.154:39481 (ingress) <- components-system/my-nginx-646554d7fd-lwlb8:8080 (ID:59849) to-stack FORWARDED (TCP Flags: ACK, RST)

-> endpoint 3948 flow 0x6da04d17 , identity ingress->59849 state new ifindex lxc2a79ee5da833 orig-ip 10.42.0.154: 10.42.0.154:36735 -> 10.42.0.138:8080 tcp SYN
-> stack flow 0x0 , identity 59849->ingress state reply ifindex 0 orig-ip 0.0.0.0: 10.42.0.138:8080 -> 10.42.0.154:36735 tcp ACK, RST

@TECHNOFAB11 TECHNOFAB11 changed the title Packets to hostNetwork pods like envoy never arrive Packets to hostNetwork pods like envoy always contain RST flag Jan 19, 2024
@TECHNOFAB11 TECHNOFAB11 mentioned this issue Feb 5, 2024
Open
2 tasks
@github-actions GitHub Actions
Copy link

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Mar 20, 2024
zviratko referenced this issue in mhofstetter/proxy Mar 26, 2024
@mhofstetter
bpf: fix error propagation for bpf syscalls
c73651f 
Currently, interaction with BPF maps via syscalls (open, lookup) might result
in log messages of the following form, where the error detail is `success`:

```
[info][filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: Success
```

This is due to the fact that BPF maps are accessed in the starter process. Hence,
the syscalls are also executed in this separate process and the variable `errno` is never set
in the Envoy process where the log is written..

Therefore, this commit fixes the error propagation by setting the variable `errno` after
retrieving the response from the privileged client doing the call to the starter
process.

Fixes: cilium#315
Fixes: cilium#470

Signed-off-by: Marco Hofstetter <[email protected]>
@mhofstetter mhofstetter mentioned this issue Mar 27, 2024
Merged
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 3, 2024

This issue has not seen any activity since it was marked stale.
Closing.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@fangpenlin @youngnick @TECHNOFAB11