infrasys: create TUF infra in AWS using cargo make create-infra
#1723
Conversation
aashnasheth
force-pushed
the
infrasys
branch
3 times, most recently
from
6ea4f47 to
788eebc
Compare
aashnasheth
changed the title
aashnasheth
changed the title
cargo make create-infra
| .context(error::MissingConfig { | ||
| missing: "config field for a kms key", | ||
| })? | ||
| .create_kms_keys() |
There was a problem hiding this comment.
Decided to make create_kms_keys a trait of the config so users aren't surprised when config fields are changed. We also considered passing a mutable reference to config to create_kms_keys(), but thought this way was more clear.
| ); | ||
| } | ||
|
|
||
| // Set key_id using a publication key (if one is not already provided) |
There was a problem hiding this comment.
We decided to set key_id in this method as it's the only one that differentiates roles (we only want key_id to be set for publication keys, not root keys)
aashnasheth
force-pushed
the
infrasys
branch
2 times, most recently
from
79b3377 to
1504ebd
Compare
|
Pushes above add some extra tests and make minor changes to existing ones:
|
|
Push above addresses warnings raised by |
There was a problem hiding this comment.
I think we talked about this at some point and now is a good time to do it:
In addition to just setting up the Infrastructure, we should also test out whether we're able to use said infrastructure to actually update a live Bottlerocket host with a custom TUF repository. We should include that in the testing description since that's the ultimate goal.
|
Push above addresses etungsten's comments (and includes a small fix for default prefixes) |
|
Push above makes a small change to bucket policy tests |
|
Push above adds a new commit that checks if a lock file exists or not (signaling users to clean up resources and re-run infrasys accordingly) |
|
Push above adds further pubsys integration for InfraConfig::from_path_or_default usage (not just InfraConfig::from_path) |
tools/infrasys/src/error.rs
Outdated
| }, | ||
|
|
||
| #[snafu(display("Failed to get parent of path: {}", path.display()))] | ||
| Parent { path: PathBuf }, |
There was a problem hiding this comment.
At least in practice - this seems to crossover with the Path error variant above.
There was a problem hiding this comment.
I'm thinking that if there's an issue with the parent() method in specific, might be nice to know where the error is coming from instead of a general "invalid path"?
For some reason if rust's parent() method breaks, we would be able to catch it here
There was a problem hiding this comment.
I think having a separate error variant for this makes sense. The path itself isn't necessarily invalid, it's just that it doesn't have a parent (e.g. it might be /).
tools/infrasys/src/main.rs
Outdated
| } | ||
|
|
||
| async fn check_infra_lock(toml_path: &Path) -> Result<()> { | ||
| let lock_path = toml_path |
There was a problem hiding this comment.
It feels odd to me to have a function that's checking for the existence of a file, but the argument is a different file. Perhaps we can determine lock_path beforehand, and pass it to this function.
There was a problem hiding this comment.
We wanted to make sure that users weren't able to change where the lock file lived (should always live next to Infra.toml). If we pass it in via commandline, we would do this path splicing in the Makefile. We were thinking it would be easier to do via code inside main.rs?
There was a problem hiding this comment.
I still have a lot left to get through!
But the main theme of this pass is that there is some duplicate code involving the loading of either Infra.toml or Infra.lock.
It's great when files in a PR have more red than green and there is an opportunity to make that happen here in all the files that need to load either toml or lock.
aashnasheth
force-pushed
the
infrasys
branch
from
4476a6f to
a381871
Compare
|
Push above addresses initial webern and zmrow comments |
aashnasheth
force-pushed
the
infrasys
branch
from
a381871 to
b6a72eb
Compare
|
Push above addresses webern code duplication concerns with pubsys-integration code |
aashnasheth
force-pushed
the
infrasys
branch
from
b6a72eb to
6c03125
Compare
|
Push above removes |
aashnasheth
force-pushed
the
infrasys
branch
from
6c03125 to
fd13108
Compare
|
Push above adds KMSClient code to pubsys-integration commit |
aashnasheth
force-pushed
the
infrasys
branch
from
fd13108 to
a716b48
Compare
|
Push above adds webern's code to clean up tools/pubsys-setup/src/main.rs diff |
Adds binary (and supporting files) to automatically spin up TUF infra (S3 Bucket+Policy, KMS Keys, and populated root.json) across multiple regions in a single account. Edits pubsys-config (and supporting files) to accomodate new infrasys fields in Infra.toml.
aashnasheth
force-pushed
the
infrasys
branch
from
6ded81d to
ce71a76
Compare
|
Push addresses webern comments |
Issue number:
N/A
Description of changes:
create-infraandcheck-infra-lock. This PR is the implementation ofcreate-infra.main.rs, which uses helper methods from other modules (keys.rs[creating keys],s3.rs[creating s3 bucket, adding bucket policy, uploading objects],shared.rs[helper methods used in all files],errors.rs[a single error file shared by all files], androot.rs[creating root, adding keys to it, signing it]).main.rsreads from Infra.toml using pubsys-config (thus edits to a few files under pubsys-config and pubsys were required) and outputs Infra.lock in yaml (we used yaml to serialize the config because we were unable to serialize enums with structs within them in toml).Testing done:
Added a unit test for for struct -> toml -> struct -> yaml -> struct conversion (main.rs), prefix format checking (s3.rs), and bucket policy insertion (s3.rs)
Created several test Infra.toml files with various combinations of configurations and ensured that errors were thrown / the resources showed up in the correct region, with the correct settings, in my account
Example of a test where we're creating a root + signing keys
Infra.toml
cargo make create-infra output
Infra.lock
root.json
{ "signed": { "_type": "root", "spec_version": "1.0.0", "consistent_snapshot": true, "version": 1, "expires": "2022-08-23T20:13:33Z", "keys": { "fe6321b3c61d4f0cc7864e283e4dd007056d8e9ae1dd367166720225efa16681": { "keytype": "rsa", "keyval": { "public": "-----BEGIN PUBLIC KEY-----\nMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEAw+Kz32mPNDLtM4Qn+G3X\n+hf/l9h3KCFOGtBpeNxQN9NLtExA3e1QLGS6yQQNVa7ELPIvWep9hcoraxP2kv5X\nLhMdzy0cmFVFWwtlt3/Srd9paihW+cP53ODnr2/KeXUlLAco13s26dMzvxOZmwAg\nxhDzj+846n592WRWGcuGO80kWSFgfipufwld32jbUZoC9OqwvIBVn1Qk3BHm4GHI\ntG/VG65CRdHXtZ2FSC4KgeVjJons3mqGjcgXwN2dVuWmmCJgrTKvUru706KctxF3\nnvFXTMmg0MosM4woArXCHUb82lugFZeLic+egSLupivNceblubVMU2xxF3SE98hv\nQxmaWBI2sQ+4uBFvh1KkKVwAHeSfbrrU+oNYsilVXcHc+hXiyhneMxkOkt21l0iP\nP8XnOyzGLz6dEtc7aMqGlIDGPdVkQgLl/MSnRPpb641uFGENKUo8MWGCXz33G7wB\n1wtwZXJFVMdTv2juhG0Umft8lp/USuxfO7eQOe+cCmpXAgMBAAE=\n-----END PUBLIC KEY-----\n" }, "scheme": "rsassa-pss-sha256" }, "f5b8391e577e7abd0354963177ecfc2b2aff6c6aa62b59df72c37a57020acb83": { "keytype": "rsa", "keyval": { "public": "-----BEGIN PUBLIC KEY-----\nMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEAnkREp096HUOSmwgeM/K4\n4Zzky0h3rmSBXdJ+wn4q/fFjKYKkVKwxu3H2aa/GueJAb1q/SqVBHism5JNZl46E\ngTczchClMvPDQsshKH9IQ0t4bl8quaH04+zpnzWoYMQHptIv56yTTirVsgg7NBkx\noi8pXXC+fZdH1ed+qKUmr5hwz1wlZrWVwY3SCYLv7QYi/4sL8mRw8tzx7IDd4+5d\nZBXC2Qb9loQ6zjFIoxSubz5Fl1zuGk+cAG/TdOzangr0poGFwu9Xm0mh2RByuAbX\nJ61SZWRaAb9AwkTy7C2Du6hZ0ZwSfYoKcGWfnylVYLrreAxOPi3X6odNKgWt12Yq\na95+lv0n/830dg36zZ9I0eVMQZgIupM4zAUlv3QvBA83QaQnrQV5JSCYawcftah6\nbQ1BaZlAnJi9rGwZ7sGj7deJtz+BqXwSOPA3cdm9q2rinxXlLyllLlQsWcvbvVlx\nBKgi2K5gI9ZxM0T9qBvo7jy0m1GDhXve9+BrdbXmJ351AgMBAAE=\n-----END PUBLIC KEY-----\n" }, "scheme": "rsassa-pss-sha256" } }, "roles": { "root": { "keyids": [ "fe6321b3c61d4f0cc7864e283e4dd007056d8e9ae1dd367166720225efa16681" ], "threshold": 1 }, "snapshot": { "keyids": [ "f5b8391e577e7abd0354963177ecfc2b2aff6c6aa62b59df72c37a57020acb83" ], "threshold": 1 }, "targets": { "keyids": [ "f5b8391e577e7abd0354963177ecfc2b2aff6c6aa62b59df72c37a57020acb83" ], "threshold": 1 }, "timestamp": { "keyids": [ "f5b8391e577e7abd0354963177ecfc2b2aff6c6aa62b59df72c37a57020acb83" ], "threshold": 1 } } }, "signatures": [ { "keyid": "fe6321b3c61d4f0cc7864e283e4dd007056d8e9ae1dd367166720225efa16681", "sig": "813107cdf34a3181bcd6eb0f83cdce350670e26bf381f535a3e09e33d638d50c18a203f91f1c1ad2966efd689579d5c433db6093811f13d6a3a1d964ebd6839e8edf67c9b968591cd2b5a845cdb360e634cfb72328773010e0646fc1a75738cbddf008f5981e713f7364aec2b987f3125b5764317e4786843307973d6635daa4c7ea2b2a83db77663ccf6288d82f8695bd648376b2105a891d64da91af5a3f63544d4369c1a8da23f36d1039eabb2ea13abc7f6549d281c00a96db6daf32eed51180c966b0ab4251d47a1dffd57cc504b69de0c482d92d909040078e20d7ad59a27f9627aab780e3bcf3e2f5e150d5d72107d8b1fffbc7081b602588dc2608313af97d4b8d9207222c86fe9b626dd76d65ebd2c4278afe0272bb5fd84d9047356463f1463c62a813fe72641f857b970c131db9ddddc14c6d0b67aac4735a5da763ccf5450b6455f55fd4c2b6823ec19c223987a37feeac9047b2b5cd8f38c467fb6c69ed10ad78b7a02d5cca4ab4774ac58ca61bf8c2b4c5706858ef8d45f0e3" } ] }Resources show up correctly in my account (root.json populated in s3 bucket, all public access off for s3 bucket, correct bucket policy, kms keys present)
Completed E2E testing (cargo make, cargo make repo, cargo make ami, pulling updates)
Draft Review:
Here are some initial reviews that etungsten left and were addressed: aashnasheth#9
Next PR:
check-infra-lock(which will check the existence of the lock file and advise users to clean up resources if needed)Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.