[Bug]: Docker container charts are missing for Mac docker desktop

[jaconey]

Description

I have been running Beszel on my NAS docker and everything works great! Now I tried to add my Mac mini as a new remote system, but Docker container charts are missing (everything else is fine, so SSH is working).
The Mac is running the docker desktop 4.38 app and I have symlinked the docker.sock. I've double checked that /var/run/docker.sock is mounted and accessible by the agent.

From the DEBUG log, the reason seems to be "Error getting docker stats err="context deadline exceeded (Client.Timeout or context cancellation while reading body)" I've tried running docker stats using my local user and I can see stats.

How should I debug further? Thanks

Docker version 27.5.1, build 9f9e405

Expected Behavior

Docker container charts show in the dashboard

Steps to Reproduce

Open the dashboard and open the remote system.

OS / Architecture

OS X 15.3, M2

Beszel version

0.9.1

Installation method

Docker

Configuration

services:
  beszel-agent:
    image: 'henrygd/beszel-agent'
    container_name: 'beszel-agent'
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      PORT: 45876
      KEY: 'xxxx'
    network_mode: host

Hub Logs

Agent Logs

Attaching to beszel-agent
beszel-agent  | 2025/02/19 01:38:01 DEBUG 0.9.1
beszel-agent  | 2025/02/19 01:38:01 DEBUG Not monitoring ZFS ARC err="open /proc/spl/kstat/zfs/arcstats: no such file or directory"
beszel-agent  | 2025/02/19 01:38:01 DEBUG Disk partitions=
beszel-agent  | 2025/02/19 01:38:01 INFO Detected root device name=vda1
beszel-agent  | 2025/02/19 01:38:01 INFO Detected network interface name=eth0 sent=3550991 recv=536759
beszel-agent  | 2025/02/19 01:38:01 INFO Detected network interface name=services1 sent=538408 recv=3455895
beszel-agent  | 2025/02/19 01:38:01 DEBUG GPU err="no GPU found - install nvidia-smi or rocm-smi"
beszel-agent  | 2025/02/19 01:38:01 DEBUG Getting stats
beszel-agent  | 2025/02/19 01:38:01 DEBUG Temperature sensors=[]
beszel-agent  | 2025/02/19 01:38:01 DEBUG sysinfo data="{Hostname:docker-desktop KernelVersion:6.12.5-linuxkit Cores:8 Threads:8 CpuModel: Uptime:798 Cpu:25 MemPct:13.17 DiskPct:9.57 Bandwidth:2.52 AgentVersion:0.9.1 Podman:false}"
beszel-agent  | 2025/02/19 01:38:01 DEBUG System stats data="{Stats:{Cpu:25 MaxCpu:0 Mem:3.83 MemUsed:0.5 MemPct:13.17 MemBuffCache:0.38 MemZfsArc:0 Swap:1 SwapUsed:0 DiskTotal:58.37 DiskUsed:5.3 DiskPct:9.57 DiskReadPs:7.3 DiskWritePs:0 MaxDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:1.27 NetworkRecv:1.25 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[] GPUData:map[]} Info:{Hostname:docker-desktop KernelVersion:6.12.5-linuxkit Cores:8 Threads:8 CpuModel: Uptime:798 Cpu:25 MemPct:13.17 DiskPct:9.57 Bandwidth:2.52 AgentVersion:0.9.1 Podman:false} Containers:[]}"
beszel-agent  | 2025/02/19 01:38:03 DEBUG Error getting docker stats err="context deadline exceeded (Client.Timeout or context cancellation while reading body)"
beszel-agent  | 2025/02/19 01:38:03 DEBUG Extra filesystems data=map[]
beszel-agent  | 2025/02/19 01:38:03 DEBUG Stats data="{Stats:{Cpu:25 MaxCpu:0 Mem:3.83 MemUsed:0.5 MemPct:13.17 MemBuffCache:0.38 MemZfsArc:0 Swap:1 SwapUsed:0 DiskTotal:58.37 DiskUsed:5.3 DiskPct:9.57 DiskReadPs:7.3 DiskWritePs:0 MaxDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:1.27 NetworkRecv:1.25 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[] GPUData:map[]} Info:{Hostname:docker-desktop KernelVersion:6.12.5-linuxkit Cores:8 Threads:8 CpuModel: Uptime:798 Cpu:25 MemPct:13.17 DiskPct:9.57 Bandwidth:2.52 AgentVersion:0.9.1 Podman:false} Containers:[]}"
beszel-agent  | 2025/02/19 01:38:03 INFO Starting SSH server address=:45876
beszel-agent  | 2025/02/19 01:38:13 DEBUG Getting stats
beszel-agent  | 2025/02/19 01:38:13 DEBUG Temperature sensors=[]
beszel-agent  | 2025/02/19 01:38:13 DEBUG sysinfo data="{Hostname:docker-desktop KernelVersion:6.12.5-linuxkit Cores:8 Threads:8 CpuModel: Uptime:810 Cpu:0.64 MemPct:13.33 DiskPct:9.57 Bandwidth:0.01 AgentVersion:0.9.1 Podman:false}"
beszel-agent  | 2025/02/19 01:38:13 DEBUG System stats data="{Stats:{Cpu:0.64 MaxCpu:0 Mem:3.83 MemUsed:0.51 MemPct:13.33 MemBuffCache:0.38 MemZfsArc:0 Swap:1 SwapUsed:0 DiskTotal:58.37 DiskUsed:5.3 DiskPct:9.57 DiskReadPs:0.01 DiskWritePs:0 MaxDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:0.01 NetworkRecv:0 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[] GPUData:map[]} Info:{Hostname:docker-desktop KernelVersion:6.12.5-linuxkit Cores:8 Threads:8 CpuModel: Uptime:810 Cpu:0.64 MemPct:13.33 DiskPct:9.57 Bandwidth:0.01 AgentVersion:0.9.1 Podman:false} Containers:[]}"
beszel-agent  | 2025/02/19 01:38:15 DEBUG Error getting docker stats err="context deadline exceeded (Client.Timeout or context cancellation while reading body)"
beszel-agent  | 2025/02/19 01:38:15 DEBUG Extra filesystems data=map[]

[henrygd]

Here are two bash scripts which may help figure out the problem. You can run these on the host, it doesn't need to be in a container.

First run the curl command to make sure it returns the stats properly. Change container_name in all URLs to the name of one of your containers.

curl --unix-socket /var/run/docker.sock -H "Content-Type: application/json" "http://localhost/containers/container_name/stats?stream=0&one-shot=1"

sequence.sh

Save this to sequence.sh, make it executable chmod +x ./sequence.sh, then run it with time: time ./sequence.sh

#!/bin/bash

for i in {1..20}
do
   curl -s --unix-socket /var/run/docker.sock -H "Content-Type: application/json" "http://localhost/containers/container_name/stats?stream=0&one-shot=1"
done

parallel.sh

Save this to parallel.sh, make it executable chmod +x ./parallel.sh, then run it with time: time ./parallel.sh. Change 1..10 to 1..50 to see if there's much difference in run time.

#!/bin/bash

for i in {1..10}
do
   curl -s --unix-socket /var/run/docker.sock -H "Content-Type: application/json" "http://localhost/containers/container_name/stats?stream=0&one-shot=1" &
done

wait

No rush, and I don't need the entire output. Just let me know what your times are.

[jaconey]

@henrygd Thank you for your prompt response! Very informative. I've tried all three in CLI and they all returned valid stats json. So, I think the socket is set up well and I have permission to read.

Please let me know if there's any other thing I can try. I wonder if there's something odd in the Docker Desktop app.

[jaconey]

Another data point: I just set up a Portainer agent on the same machine. I can manage all containers remotely from another server, so I think /var/run/docker.sock is set up correctly and have right permissions.

[henrygd]

Thanks, I'm not sure what would be causing this. It's likely an issue that's specific to Docker on MacOS.

Maybe try removing the :ro suffix of /var/run/docker.sock:/var/run/docker.sock:ro if your Portainer instance doesn't use it.

Also can you confirm that the /containers/json endpoint returns your containers?

curl --unix-socket /var/run/docker.sock "http://localhost/containers/json"

If you want to get into the container to test from there, you can get a shell by following the instructions here: https://beszel.dev/guide/docker-shell

[jaconey]

Very interesting. I build the shell in and run inside the container:

curl --unix-socket /var/run/docker.sock "http://localhost/containers/json"

It returns all the containers.

curl --unix-socket /var/run/docker.sock -H "Content-Type: application/json" "http://localhost/containers/container_name/stats?stream=0&one-shot=1"

also works. I guess it comes back to how Beszel query the socket.

IIUC, you're using http.Client.Get on that socket. I'm not sure what the default timeout is. Can we extend it a bit longer? It'd be great to log more detailed error if possible too.

[henrygd]

Very strange, this may be related to #513 and specifically a problem with Docker Desktop 4.38.0.

I don't use Docker Desktop so I'll try installing on Linux to see if I have the same problem.

[jaconey]

I do notice the query is quite slow (in few seconds). Even running curl so I guess the http.Client.Get timeout could help

[a-mnich]

I can reproduce the issue and also have a timeout of the docker daemon response.
I think this is related to docker/for-mac#7575

We've identified the root cause, and this will be fixed in the upcoming v4.39. Let me close this issue, but feel free to continue the conversation.

There is a workaround by changing the user-agent, but hopefully this will fix itself with the next docker desktop update for mac.

[jaconey]

Thanks for confirming @a-mnich! I'll roll back or wait for v4.39.

FWIW, although the timeout is unexpected and the root cause is a bad docker version, I'd still suggest bumping http.Client.Get timeout (or make it as a config) to avoid getting this kind of issue again.

[henrygd]

Thanks for the link to the Docker issue. I'm glad it's not on this end because I was extremely confused.

Sounds like they should have a new version out soon, but I'll add the user-agent workaround in the next release anyway.

We do have an undocumented env var DOCKER_TIMEOUT that lets you change the client timeout. You can use values like 500ms, 10s, etc.

It's undocumented because if you ever need to think about using it then something is broken.

beszel/beszel/internal/agent/docker.go

Lines 243 to 252 in 6b41a98

	// configurable timeout
	timeout := time.Millisecond * 2100
	if t, set := GetEnv("DOCKER_TIMEOUT"); set {
	timeout, err = time.ParseDuration(t)
	if err != nil {
	slog.Error(err.Error())
	os.Exit(1)
	}
	slog.Info("DOCKER_TIMEOUT", "timeout", timeout)
	}

[jaconey]

Thanks for all the help! @henrygd @a-mnich we can close it as it's not a bug in beszel