Encode URI according to RFC3986

[zirkelc]

Bug

If the path or query string of an URI contains a * character, the calculated signature on the client differs from the signature AWS calculates on the server. This results in HTTP 403 errors with The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method.

Investigation

The SignatureV4 class and getCanonicalPath method encodes the path only with default encodeURIComponent function:

smithy-typescript/packages/signature-v4/src/SignatureV4.ts

Lines 316 to 336 in 66f2b66

	if (this.uriEscapePath) {
	// Non-S3 services, we normalize the path and then double URI encode it.
	// Ref: "Remove Dot Segments" https://datatracker.ietf.org/doc/html/rfc3986#section-5.2.4
	const normalizedPathSegments = [];
	for (const pathSegment of path.split("/")) {
	if (pathSegment?.length === 0) continue;
	if (pathSegment === ".") continue;
	if (pathSegment === "..") {
	normalizedPathSegments.pop();
	} else {
	normalizedPathSegments.push(pathSegment);
	}
	}
	// Joining by single slashes to remove consecutive slashes.
	const normalizedPath = `${path?.startsWith("/") ? "/" : ""}${normalizedPathSegments.join("/")}${
	normalizedPathSegments.length > 0 && path?.endsWith("/") ? "/" : ""
	}`;
	const doubleEncoded = encodeURIComponent(normalizedPath);
	return doubleEncoded.replace(/%2F/g, "/");
	}

However, the getCanonicalQuery function encodes the query string with the escapeUri function:

smithy-typescript/packages/signature-v4/src/getCanonicalQuery.ts

Lines 9 to 37 in 66f2b66

	export const getCanonicalQuery = ({ query = {} }: HttpRequest): string => {
	const keys: Array<string> = [];
	const serialized: Record<string, string> = {};
	for (const key of Object.keys(query).sort()) {
	if (key.toLowerCase() === SIGNATURE_HEADER) {
	continue;
	}
	keys.push(key);
	const value = query[key];
	if (typeof value === "string") {
	serialized[key] = `${escapeUri(key)}=${escapeUri(value)}`;
	} else if (Array.isArray(value)) {
	serialized[key] = value
	.slice(0)
	.reduce(
	(encoded: Array<string>, value: string) => encoded.concat([`${escapeUri(key)}=${escapeUri(value)}`]),
	[]
	)
	.sort()
	.join("&");
	}
	}
	return keys
	.map((key) => serialized[key])
	.filter((serialized) => serialized) // omit any falsy values
	.join("&");
	};

This escapeUri function encodes the URI according to RFC 3986:

smithy-typescript/packages/util-uri-escape/src/escape-uri.ts

Lines 4 to 8 in 66f2b66

	export const escapeUri = (uri: string): string =>
	// AWS percent-encodes some extra non-standard characters in a URI
	encodeURIComponent(uri).replace(/[!'()*]/g, hexEncode);
	const hexEncode = (c: string) => `%${c.charCodeAt(0).toString(16).toUpperCase()}`;

The requirement for RFC 3986 encoded URIs is also described in the official docs

https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html

URI encode every byte. UriEncode() must enforce the following rules:

• URI encode every byte except the unreserved characters: 'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '_', and '~'.
• The space character is a reserved character and must be encoded as "%20" (and not as "+").
• Each URI encoded byte is formed by a '%' and the two-digit hexadecimal value of the byte.
• Letters in the hexadecimal value must be uppercase, for example "%1A".
• Encode the forward slash character, '/', everywhere except in the object key name. For example, if the object key name is photos/Jan/sample.jpg, the forward slash in the key name is not encoded.

Important
The standard UriEncode functions provided by your development platform may not work because of differences in implementation and related ambiguity in the underlying RFCs. We recommend that you write your own custom UriEncode function to ensure that your encoding will work.
To see an example of a UriEncode function in Java, see Java Utilities on the GitHub website.

The last important section refers to the SdkHttpUtils.urlEncode method from the Java SDK:

https://github.com/aws/aws-sdk-java/blob/1b3444aa78f4579c4083bd4b3858322bc343a906/aws-java-sdk-core/src/main/java/com/amazonaws/util/SdkHttpUtils.java#L66-L99

In the Java code, the * will be encoded as %2A. This encoding follow RFC3986 that specifies that the characters ! ' ( ) * must be encoded. See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent#encoding_for_rfc3986

Here is also a PR from the Amplify project that implemented RFC 3986 encoding for the query string: aws-amplify/amplify-js#2631

Conclusion

This issue can of course be mitigated if the requester encodes the URI on their own according to extendedURIEncode function. However, to be consistent with other AWS SDKs and fully compliant with the SignatureV4 spec, I think this part should be handled by the SignatureV4 class and not by the requester.

As there is also a escapeUriPath function, I would suggest adding to SignatureV4.getCanonicalPath:

https://github.com/smithy-lang/smithy-typescript/blob/66f2b660e6b1dfc40699772a51778086495fd2cf/packages/util-uri-escape/src/escape-uri-path.ts#L6C14-L6C27

[ricksterhd123]

try your hand at a PR :)

[kuhe]

which AWS service and operation did you encounter this with?

[zirkelc]

I have two people reporting this as issue when using Amazon Open Search and path contains a * character. YOu will find more details here: zirkelc/aws-signature-v4#10

I myself could reproduce this issue with AWS API Gateway.

[kuhe]

The fix PR was released. It's not enforced as the minimum version in SDK packages yet, but you can get it via the ^x.y.z range that AWS SDKs use for @smithy/* packages if you update your package lockfile.