@aws-sdk/client-s3 makes node_modules too large

[remagpie]

Describe the bug

When @aws-sdk/client-s3@3.29.0 is installed, it downloads aws-crt as a dependency which is about 200MB in size. It occupies about 60MB even when we compress it.

My team updates the lambda function by directly zipping the directory and uploading it, but we can't do it anymore because we hit the size limit of the allowed archive size.

The problem is gone after we downgrade @aws-sdk/client-s3 to 3.28.0, but we'd like to use the most recent version if possible.

Your environment

SDK version number

@aws-sdk/[email protected]

Is the issue in the browser/Node.js/ReactNative?

Node.js

Additional context

[moltar]

After bundling using esbuild and deploying to Lambda we are getting the following errors:

AWS CRT binary not present in any of the following locations:\n\t/bin/native/aws-crt-nodejs\n\t/bin/linux-x64/aws-crt-nodejs

    "stack": [
        "Error: AWS CRT binary not present in any of the following locations:",
        "\t/bin/native/aws-crt-nodejs",
        "\t/bin/linux-x64/aws-crt-nodejs",
        "    at null.node_modules/aws-crt/dist/native/binding.js (/node_modules/aws-crt/lib/native/binding.js:46:11)",
        "    at __require (/var/task/index.js:41:45)",
        "    at null.node_modules/aws-crt/dist/native/crt.js (/node_modules/aws-crt/lib/native/crt.ts:18:1)",
        "    at __require (/var/task/index.js:41:45)",
        "    at null.node_modules/aws-crt/dist/index.js (/node_modules/aws-crt/lib/index.ts:17:1)",
        "    at __require (/var/task/index.js:41:45)",
        "    at null.node_modules/@aws-sdk/signature-v4-crt/dist/cjs/CrtSignerV4.js (/node_modules/@aws-sdk/signature-v4-crt/src/CrtSignerV4.ts:22:1)",
        "    at __require (/var/task/index.js:41:45)",
        "    at null.node_modules/@aws-sdk/signature-v4-crt/dist/cjs/index.js (/node_modules/@aws-sdk/signature-v4-crt/src/index.ts:1:1)",
        "    at __require (/var/task/index.js:41:45)"
    ]

[ml1nk]

In addition since aws-crt includes a native blob which does not support alpine, it fails for example on node:16.8.0-alpine.

ERROR: Error: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /opt/main/node_modules/aws-crt/dist/bin/linux-x64/aws-crt-nodejs.node)

[webberwang]

I notice Cannot find module '/node_modules/aws-crt/dist/bin/darwin-x64/aws-crt-nodejs' from '/node_modules/aws-crt/dist/native/binding.js' in 3.29.0 (which isn't fixed even if I install the missing package), had to downgrade to 3.28.0

[matheuscanela]

The s3 adds 209 packages as it gets removed after removing the client-s3
Cannot be used with lambda, as it consumes 200+ mb

[AllanZhengYP]

Thank you for reporting this issue. We are investigating in the issue and will post update in #2750

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.