Default credential loader doesn't seem to follow 301s from IMDSv2 #560

jszwedko · 2022-06-22T01:20:58Z

Describe the bug

This is an indirect bug report from a user of Vector, which uses the AWS SDK.

When using the default credential chain, they see the loader reporting that it saw a 301, but it doesn't seem to automatically follow it, as I would expect.

The user is using KIAM to intercept the requests which is likely why the behavior differs from IMDSv2 generally.

See debug logs below.

Expected Behavior

The default credentials loader follows any observed 301s to fetch the IAM credentials.

Current Behavior

When a 301 is observed, it stops and bubbles up an error.

Reproduction Steps

Unfortunately I haven't observed this behavior myself, but we could ask the user for additional details if necessary.

Possible Solution

Default credential loader follows 301s from IMDSv2 API.

Additional Information/Context

User is running KIAM to intercept the metadata requests.

Vector issue: vectordotdev/vector#13211

Version

├── aws-config v0.11.0
│   ├── aws-http v0.11.0
│   │   ├── aws-smithy-http v0.41.0
│   │   │   ├── aws-smithy-eventstream v0.41.0
│   │   │   │   ├── aws-smithy-types v0.41.0
│   │   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-types v0.11.0
│   │   │   ├── aws-smithy-async v0.41.0
│   │   │   ├── aws-smithy-client v0.41.0
│   │   │   │   ├── aws-smithy-async v0.41.0 (*)
│   │   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.41.0
│   │   │   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   │   ├── aws-smithy-types v0.41.0 (*)
│   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-sdk-sso v0.11.0
│   │   ├── aws-endpoint v0.11.0
│   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   ├── aws-types v0.11.0 (*)
│   │   ├── aws-http v0.11.0 (*)
│   │   ├── aws-sig-auth v0.11.0
│   │   │   ├── aws-sigv4 v0.11.0
│   │   │   │   ├── aws-smithy-eventstream v0.41.0 (*)
│   │   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.41.0 (*)
│   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   ├── aws-types v0.11.0 (*)
│   │   ├── aws-smithy-async v0.41.0 (*)
│   │   ├── aws-smithy-client v0.41.0 (*)
│   │   ├── aws-smithy-http v0.41.0 (*)
│   │   ├── aws-smithy-http-tower v0.41.0 (*)
│   │   ├── aws-smithy-json v0.41.0
│   │   │   └── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-types v0.11.0 (*)
│   ├── aws-sdk-sts v0.11.0
│   │   ├── aws-endpoint v0.11.0 (*)
│   │   ├── aws-http v0.11.0 (*)
│   │   ├── aws-sig-auth v0.11.0 (*)
│   │   ├── aws-smithy-async v0.41.0 (*)
│   │   ├── aws-smithy-client v0.41.0 (*)
│   │   ├── aws-smithy-http v0.41.0 (*)
│   │   ├── aws-smithy-http-tower v0.41.0 (*)
│   │   ├── aws-smithy-query v0.41.0
│   │   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-smithy-xml v0.41.0
│   │   ├── aws-types v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-json v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-cloudwatch v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-query v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-smithy-xml v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-cloudwatchlogs v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-json v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-firehose v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-json v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-kinesis v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-json v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-s3 v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-sigv4 v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-eventstream v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-smithy-xml v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-sqs v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-query v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-smithy-xml v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sigv4 v0.11.0 (*)
├── aws-smithy-async v0.41.0 (*)
├── aws-smithy-client v0.41.0 (*)
├── aws-smithy-http v0.41.0 (*)
├── aws-smithy-http-tower v0.41.0 (*)
├── aws-smithy-types v0.41.0 (*)
├── aws-types v0.11.0 (*)

Environment details (OS name and version, etc.)

x86_64-unknown-linux-musl

Logs

2022-06-21T15:09:44.896310Z  INFO provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=environment variable not set
2022-06-21T15:09:44.896352Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
2022-06-21T15:09:44.896368Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2022-06-21T15:09:44.896379Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2022-06-21T15:09:44.896429Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: config file not found path=~/.aws/config
2022-06-21T15:09:44.896447Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: config file loaded path=~/.aws/config size=0
2022-06-21T15:09:44.896458Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2022-06-21T15:09:44.896477Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2022-06-21T15:09:44.896487Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2022-06-21T15:09:44.896493Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: config file loaded path=~/.aws/credentials size=0
2022-06-21T15:09:44.896511Z  INFO provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=No profiles were defined
2022-06-21T15:09:44.896529Z  INFO provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=$AWS_WEB_IDENTITY_TOKEN_FILE was not set
2022-06-21T15:09:44.896543Z  INFO provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=ECS provider not configured
2022-06-21T15:09:44.896558Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2022-06-21T15:09:44.896567Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::fs_util: loaded home directory src="HOME"
2022-06-21T15:09:44.896576Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_config_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2022-06-21T15:09:44.896585Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_config_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2022-06-21T15:09:44.896595Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_config_file: aws_config::profile::parser::source: config file not found path=~/.aws/config
2022-06-21T15:09:44.896609Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_config_file: aws_config::profile::parser::source: config file loaded path=~/.aws/config size=0
2022-06-21T15:09:44.896619Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_credentials_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2022-06-21T15:09:44.896628Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_credentials_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2022-06-21T15:09:44.896644Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_credentials_file: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2022-06-21T15:09:44.896650Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_credentials_file: aws_config::profile::parser::source: config file loaded path=~/.aws/credentials size=0
2022-06-21T15:09:44.898362Z  WARN provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider failed to provide credentials provider=Ec2InstanceMetadata error=An error occurred while loading credentials: Error response from IMDS (code: 301). Response { status: 301, version: HTTP/1.1, headers: {"content-type": "text/html; charset=utf-8", "location": "/latest/meta-data/iam/security-credentials/", "date": "Tue, 21 Jun 2022 15:09:44 GMT", "content-length": "78"}, body: SdkBody { inner: Once(Some(b"<a href=\"/latest/meta-data/iam/security-credentials/\">Moved Permanently</a>.\n\n")), retryable: true } }

The text was updated successfully, but these errors were encountered:

rcoh · 2022-06-22T17:55:47Z

generally we don't follow 301s—I'll need to consult the spec and figure out if this behavior should be honored or not. Thanks for reporting, will follow up soon

rcoh · 2022-06-22T18:14:32Z

I followed up—I suspect the issue is not the 301—Other SDKs don't follow 301s for IMDS (and if they do it's a bug / security issue).

The issue instead seems to be that we're hitting security-credentials (no trailing slash) and KIAM is (for some reason) deciding to redirect to security-credentials/.

I don't see any particular reason we couldn't hit the URL with a trailing slash from the IMDS client though

jszwedko · 2022-06-22T18:39:00Z

Thanks for the response! I can see why following 301s could be considered a security concern.

uswitch/kiam#172 seems relevant. They switched to returning a 301 from a 308 thinking that clients would handle it better.

uswitch/kiam#116 also seems relevant.

I spun up an EC2 instance to try it out and both with and without / seem to return 200s at the current time for IMDSv2 (I didn't try KIAM):

[ec2-user@ip-172-31-43-67 ~]$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v 169.254.169.254/latest/meta-data/iam/security-credentials
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/security-credentials HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Accept-Ranges: bytes
< Content-Length: 23
< Content-Type: text/plain
< Date: Wed, 22 Jun 2022 18:32:15 GMT
< Last-Modified: Wed, 22 Jun 2022 18:31:24 GMT
< Connection: close
< Server: EC2ws
<
* Closing connection 0
cloudtrail-ec2-instance

[ec2-user@ip-172-31-43-67 ~]$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v 169.254.169.254/latest/meta-data/iam/security-credentials/
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Accept-Ranges: bytes
< Content-Length: 23
< Content-Type: text/plain
< Date: Wed, 22 Jun 2022 18:32:17 GMT
< Last-Modified: Wed, 22 Jun 2022 18:31:24 GMT
< Connection: close
< Server: EC2ws
<
* Closing connection 0
cloudtrail-ec2-instance

curling the parent path,/iam, seems to return the "directory entries" with the trailing /:

$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v 169.254.169.254/latest/meta-data/iam/
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Accept-Ranges: bytes
< Content-Length: 26
< Content-Type: text/plain
< Date: Wed, 22 Jun 2022 18:34:39 GMT
< Last-Modified: Wed, 22 Jun 2022 18:31:24 GMT
< Connection: close
< Server: EC2ws
<
* Closing connection 0
info
security-credentials/

I'll leave it to you and others with more intimate knowledge of IMDSv2 but it does seem like having the client hit the endpoint with a trailing / would be appropriate.

rcoh · 2022-06-22T18:43:36Z

it seems like KIAM explicitly added this redirect for some internal reason: uswitch/kiam#121

rcoh · 2022-06-22T19:08:32Z

PR: smithy-lang/smithy-rs#1487
Unfortunately it's not going to make the release that's going out today but will go out in the next one

jszwedko · 2022-06-22T22:48:31Z

Awesome, thanks for the quick review and quick fix @rcoh !

rcoh · 2022-06-29T21:19:40Z

this shipped!

github-actions · 2022-06-29T21:20:09Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

jszwedko added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 22, 2022

jszwedko mentioned this issue Jun 22, 2022

S3 sink stopped working in 0.22 and lost events vectordotdev/vector#13211

Closed

rcoh removed the needs-triage This issue or PR still needs to be triaged. label Jun 22, 2022

rcoh self-assigned this Jun 22, 2022

rcoh added this to AWS Rust SDK Public Roadmap Jun 22, 2022

rcoh moved this to In Progress in AWS Rust SDK Public Roadmap Jun 22, 2022

rcoh moved this from In Progress to Pending Release in AWS Rust SDK Public Roadmap Jun 27, 2022

rcoh closed this as completed Jun 29, 2022

Repository owner moved this from Pending Release to Done in AWS Rust SDK Public Roadmap Jun 29, 2022

argggh mentioned this issue May 7, 2024

ImdsCredentialsProvider does not follow redirects awslabs/aws-sdk-kotlin#1303

Closed

lauzadis mentioned this issue May 30, 2024

fix: add trailing slash to endpoint and disable proxying of requests to IMDS awslabs/aws-sdk-kotlin#1320

Merged

RanVaknin mentioned this issue Nov 15, 2024

ec2-metadata-service errors in up to date AWS EKS cluster using Pod Identity aws/aws-sdk-js-v3#6667

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default credential loader doesn't seem to follow 301s from IMDSv2 #560

Default credential loader doesn't seem to follow 301s from IMDSv2 #560

jszwedko commented Jun 22, 2022 •

edited

Loading

rcoh commented Jun 22, 2022

rcoh commented Jun 22, 2022 •

edited

Loading

jszwedko commented Jun 22, 2022 •

edited

Loading

rcoh commented Jun 22, 2022

rcoh commented Jun 22, 2022

jszwedko commented Jun 22, 2022

rcoh commented Jun 29, 2022

github-actions bot commented Jun 29, 2022

Default credential loader doesn't seem to follow 301s from IMDSv2 #560

Default credential loader doesn't seem to follow 301s from IMDSv2 #560

Comments

jszwedko commented Jun 22, 2022 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

Version

Environment details (OS name and version, etc.)

Logs

rcoh commented Jun 22, 2022

rcoh commented Jun 22, 2022 • edited Loading

jszwedko commented Jun 22, 2022 • edited Loading

rcoh commented Jun 22, 2022

rcoh commented Jun 22, 2022

jszwedko commented Jun 22, 2022

rcoh commented Jun 29, 2022

github-actions bot commented Jun 29, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

jszwedko commented Jun 22, 2022 •

edited

Loading

rcoh commented Jun 22, 2022 •

edited

Loading

jszwedko commented Jun 22, 2022 •

edited

Loading