Error: SignatureDoesNotMatch only when using Ceph S3 running behind a cloudflare tunnel

[abasu0713]

Terraform Version

Terraform v1.10.4
on darwin_arm64
+ provider registry.terraform.io/cloudflare/cloudflare v4.51.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.35.1
+ provider registry.terraform.io/hashicorp/random v3.6.3

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket  = "<redacted>"
    key     = "<redacted>/terraform.tfstate"
    region  = "default"
    profile = "Terraform"
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_requesting_account_id  = true
    use_path_style              = true
    skip_s3_checksum = true
    endpoints = {
        # s3 = "https://<redacted>.arkobasu.space" <-- This doesn't work
        s3 = "http://192.168.5.81:80" <-- This works
    }
  }
  required_providers {
    random = {
      source = "hashicorp/random"
    }
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.35.1"
    }
  }
}

provider "cloudflare" {
  api_token = var.cf_api_token
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

Debug Output

2025-02-03T00:37:25.557-0600 [DEBUG] backend-s3: HTTP Request Sent: aws.region=default aws.s3.bucket=<redacted-bucket-name> rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket=<redacted-bucket-name> tf_backend.workspace-prefix=env:/ http.request.header.authorization="AWS4-HMAC-SHA256 Credential=<redacted>/20250203/default/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_content_sha256=<redacted> http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20250203T063725Z http.url="https://<redacted-record>.arkobasu.space/<redacted-bucket-name>?list-type=2&max-keys=1000&prefix=env%3A%2F" http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.10.4 (+https://www.terraform.io) m/C aws-sdk-go-v2/1.31.0 os/macos lang/go#1.23.3 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.63.0" http.request.header.amz_sdk_invocation_id=8c5bcb89-da72-4cab-8227-74a9974c963d http.request.header.accept_encoding=identity http.request.body="" http.method=GET net.peer.name=<redacted-record>.arkobasu.space
2025-02-03T00:37:25.736-0600 [DEBUG] backend-s3: HTTP Response Received: aws.region=default aws.s3.bucket=<redacted-bucket-name> rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket=<redacted-bucket-name> tf_backend.workspace-prefix=env:/ http.response.header.cf_cache_status=DYNAMIC http.response.header.x_amz_request_id=tx000004599d10cd4712119-0067a06425-1192799-default http.response.header.server=cloudflare http.duration=178 http.status_code=403 http.response.header.nel="{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}" http.response.header.alt_svc="h3=\":443\"; ma=86400" http.response_content_length=219 http.response.header.cf_ray=90c0698bee2d124e-ORD http.response.header.report_to="{\"endpoints\":[{\"url\":\"https:\/\/a.nel.cloudflare.com\/report\/v4?s=bkYoWKv7FpD9Lr8Vj8bmE8KPocsIWMjJegF5%2FeUrO8Tn5GsnSAba%2B4w9sGeZcVC4n85JSWq0EHbRCydcGWdct9N7bGDqLQ8qnQkjmhEEFDWCiCnWyYi7cPmgyOe%2Bz0BqJIxpVKiEBhSPsbWj1k%2FRtw%3D%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}" http.response.header.date="Mon, 03 Feb 2025 06:37:25 GMT"
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message></Message><RequestId>tx000004599d10cd4712119-0067a06425-1192799-default</RequestId><HostId>1192799-default-default</HostId></Error>
   http.response.header.content_type=application/xml http.response.header.accept_ranges=bytes http.response.header.x_envoy_upstream_service_time=34 http.response.header.server_timing="cfL4;desc=\"?proto=TCP&rtt=25278&min_rtt=17126&rtt_var=8980&sent=8&recv=12&lost=0&retrans=0&sent_bytes=4263&recv_bytes=2255&delivery_rate=169645&cwnd=254&unsent_bytes=0&cid=6d94aa0b9071c6de&ts=116&x=0\""
2025-02-03T00:37:25.736-0600 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 403, RequestID: tx000004599d10cd4712119-0067a06425-1192799-default, HostID: 1192799-default-default, api error SignatureDoesNotMatch: UnknownError: aws.region=default aws.s3.bucket=<redacted-bucket-name> rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket=<redacted-bucket-name> tf_backend.workspace-prefix=env:/
╷
│ Error: Failed to get existing workspaces: Unable to list objects in S3 bucket "<redacted-bucket-name>" with prefix "env:/": operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: tx000004599d10cd4712119-0067a06425-1192799-default, HostID: 1192799-default-default, api error SignatureDoesNotMatch: UnknownError
│ 
│ 

Expected Behavior

Should be able to use Ceph S3 Bucket for Backend State management

Actual Behavior

I am able to use AWS CLI and other AWS SDKs - like boto3 and dart's aws_signature_v4 without any issues. But when I use Terraform backend it throws the error. I don't have the issue when I am using the IP address based endpoint.

Steps to Reproduce

1. terraform init -migrate-state

Additional Context

It's been working great. I have 2 RGW gateways exposed to internet using Cloudflare Tunnels. I am able to use AWS CLI and SDKs (both dart and python) to interact with it using a Cloudflare DNS.. the setup is simple. I have a cloudflare tunnel running on a Kubernetes cluster (separate from the nodes actually running the RGW Gateways) -> Offloads to a Envoy Proxy -> that load balances between my RGW Gateway instances.

cat ~/.aws/config
[profile Terraform]
endpoint_url = https://<redacted-record>.arkobasu.space
region = default
output = json
alpha@Arkos-MacBook-Pro workspace 

I can confirm that Accounts and IAM API are also functional. So this profile for example is created under a new Account, by the root user of the account -> that then created this user.

I have been stuck on this for a bit. I have tried everything. The Signature validation does work when I am using something like Presign for objects and accessing them over the browser.

I would appreciate it very much if you could give me some direction.

References

No response

Generative AI / LLM assisted development?

No response

[crw]

Thanks for this report. S3 backend issues are handled by the HashiCorp AWS provider team. The S3 backend supports the AWS S3 service, compatibility with other S3 implementations is not necessarily supported although issues interacting with these services are frequently addressed.

You may also wish to use the community forum where there are more people ready to help. The GitHub issues here are monitored only by a few core maintainers. Thanks!