terraform init to install providers the new way #24477
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
added
the
sdkv1
|
This is not yet ready to merge, because it's incomplete (it's missing some steps that |
|
This currently contains the commits from #24477, so it'll need to be rebased once that is merged. |
apparentlymart
force-pushed
the
f-init-new-installer
branch
from
cfd3d64 to
039d2c2
Compare
This was referenced Mar 28, 2020
Merged
There's still a lot of work to do here around both the UX and the follow-up steps that need to happen after installation completes, but this is enough to faciliate some initial end-to-end testing of the new-style install process.
This restores some sequential event log output similar to what we had in the previous implementation of plugin installation.
This produces a value shaped the way the provider installer expects without the need for further flattening and preprocessing.
This was previously happening during linking from another cache, but not when installing an entirely new provider.
We'll be using this for its directory hashing algorithm, as used in go.sum in Go modules, and applying it also to Terraform provider packages.
For the old-style provider cache directory model we hashed the individual executable file for each provider. That's no longer appropriate because we're giving each provider package a whole directory to itself where it can potentially have many files. This therefore introduces a new directory-oriented hashing algorithm, and it's just using the Go Modules directory hashing algorithm directly because that's already had its cross-platform quirks and other wrinkles addressed during the Go Modules release process, and is now used prolifically enough in Go codebases that breaking changes to the upstream algorithm would be very expensive to the Go ecosystem. This is also a bit of forward planning, anticipating that later we'll use hashes in a top-level lock file intended to be checked in to user version control, and then use those hashes also to verify packages _during_ installation, where we'd need to be able to hash unpacked zip files. The Go Modules hashing algorithm is already implemented to consistently hash both a zip file and an unpacked version of that zip file.
Just as with the old installer mechanism, our goal is that explicit provider installation is the only way that new provider versions can be selected. To achieve that, we conclude each call to EnsureProviderVersions by writing a selections lock file into the target directory. A later caller can then recall the selections from that file by calling SelectedPackages, which both ensures that it selects the same set of versions and also verifies that the checksums recorded by the installer still match. This new selections.json file has a different layout than our old plugins.json lock file. Not only does it use a different hashing algorithm than before, we also record explicitly which version of each provider was selected. In the old model, we'd repeat normal discovery when reloading the lock file and then fail with a confusing error message if discovery happened to select a different version, but now we'll be able to distinguish between a package that's gone missing since installation (which could previously have then selected a different available version) from a package that has been modified.
The introduction of a heirarchical addressing scheme for providers gives us an opportunity to make more explicit the special case of "built-in" providers. Thus far we've just had a special case in the "command" package that the provider named "terraform" is handled differently than all others, though there's nothing especially obvious about that in the UI. Moving forward we'll put such "built-in" providers under the special namespace terraform.io/builtin/terraform, which will be visible in the UI as being different than the other providers and we can use the namespace itself (rather than a particular name) as the trigger for our special-case behaviors around built-in plugins. We have no plans to introduce any built-in providers other than "terraform" in the foreseeable future, so any others will produce an error. This commit just establishes the addressing convention, without making use of it anywhere yet. Subsequent commits will make the provider installer and resolver codepaths aware of it, replacing existing checks for the provider just being called "terraform".
Back when we first introduced provider versioning in Terraform 0.10, we did the provider version resolution in terraform.NewContext because we weren't sure yet how exactly our versioning model was going to play out (whether different versions could be selected per provider configuration, for example) and because we were building around the limitations of our existing filesystem-based plugin discovery model. However, the new installer codepath is new able to do all of the selections up front during installation, so we don't need such a heavy inversion of control abstraction to get this done: the command package can select the exact provider versions and pass their factories directly to terraform.NewContext as a simple static map. The result of this commit is that CLI commands other than "init" are now able to consume the local cache directory and selections produced by the installation process in "terraform init", passing all of the selected providers down to the terraform.NewContext function for use in implementing the main operations. This commit is just enough to get the providers passing into the terraform.Context. There's still plenty more to do here, including to repair all of the tests this change has additionally broken.
These are cases where we were using the legacy string only to produce a message to the user or to write to the log. It's enough to make some basic Terraform commands like "terraform validate" not panic and get far enough along to see that provider startup is working.
This commit reverts an earlier change which automatically converted provider strings to legacy provider FQNs. It has become apparent that a state upgrade step will be required before upgrading to v0.13.
* configs: remove `Legacy*` Provider functions, switch to default * terraform context test updates
* terraform: add helper functions for creating test state testSetResourceInstanceCurrent and testSetResourceInstanceTainted are wrapper functions around states.Module.SetResourceInstanceCurrent() used to set a resource in state. They work with current, non-deposed resources with no dependencies. testSetResourceInstanceDeposed can be used to set a desosed resource in state. * terraform: update all tests to use modern providers and state
These are some helpers to support unit testing in other packages, allowing callers to exercise provider installation mechanisms without hitting any real upstream source or having to prepare local package directories. MockSource is a Source implementation that just scans over a provided static list of packages and returns whatever matches. FakePackageMeta is a shorthand for concisely constructing a realistic-looking but uninstallable PackageMeta, probably for use with MockSource. FakeInstallablePackageMeta is similar to FakePackageMeta but also goes to the trouble of creating a real temporary archive on local disk so that the resulting package meta is pointing to something real on disk. This makes the result more useful to the caller, but in return they get the responsibility to clean up the temporary file once the test is over. Nothing is using these yet.
This test is focused on making sure that the required_providers syntax is working, so the rewritten version does not include any special handling of pre-installed packages or "vendored" packages. Pre-installed plugins are tested in other tests such as TestInit_getUpgradePlugins.
We previously skipped this one because it wasn't strictly necessary for replicating the old "terraform init" behavior, but we do need it to work so that things like the -plugin-dir option can behave correctly. Linking packages from other cache directories and installing from unpacked directories are fundamentally the same operation because a cache directory is really just a collection of unpacked packages, so here we refactor the LinkFromOtherCache functionality to actually be in installFromLocalDir, and LinkFromOtherCache becomes a wrapper for the installFromLocalDir function that just calculates the source and target directories automatically and invalidates the metaCache.
…otGet Both of these are attempting to test -plugin-dir, which means we need some additional help to populate some suitable directories for -plugin-dir to refer to. The new installFakeProviderPackagesElsewhere helper generalizes the earlier installFakeProviderPackages to allow installing fake provider packages to an arbitrary other directory.
This one just needs a tweak for the new wording of the error message.
These tests make assertions against specific user-oriented output from the "terraform init" command, but we've intentionally changed some of these messages as part of introducing support for the decentralized provider namespace.
…heDir Due to some incomplete rework of this function in an earlier commit, the safety check for using the same directory as both the target and the cache was inverted and was raising an error _unless_ they matched, rather than _if_ they matched. This change is verified by the e2etest TestInitProviders_pluginCache, which is also updated to use the new-style cache directory layout as part of this commit.
In the new design the ProviderSource is decided by package main, not by the "command" package, and so making sure the vendor directory is included is the responsibility of that package instead. Therefore we can no longer test this at the "command" package level, but we'll retain a test for it in e2etests to record that it isn't currently working, so that we have a prompt to fix it before releasing.
* command: refactor testBackendState to write states.State testBackendState was using the older terraform.State format, which is no longer sufficient for most tests since the state upgrader does not encode provider FQNs automatically. Users will run `terraform 0.13upgrade` to update their state to include provider FQNs in resources, but tests need to use the modern state format instead of relying on the automatic upgrade. * plan tests passing * graph tests passing * json packages test update * command test updates * update show test fixtures * state show tests passing
This encapsulates the logic for selecting an implied FQN for an unqualified type name, which could either come from a local name used in a module without specifying an explicit source for it or from the prefix of a resource type on a resource that doesn't explicitly set "provider". This replaces the previous behavior of just directly calling NewDefaultProvider everywhere so that we can use a different implication for the local name "terraform", to refer to the built-in terraform provider rather than the stale one that's on registry.terraform.io for compatibility with other Terraform versions.
Built-in providers are special providers that are distributed as part of Terraform CLI itself, rather than being installed separately. They always live in the terraform.io/builtin/... namespace so it's easier to see that they are special, and currently there is only one built-in provider named "terraform". Previous commits established the addressing scheme for built-in providers. This commit makes the installer aware of them to the extent that it knows not to try to install them the usual way and it's able to report an error if the user requests a built-in provider that doesn't exist or tries to impose a particular version constraint for a built-in provider. For the moment the tests for this are the ones in the "command" package because that's where the existing testing infrastructure for this functionality lives. A later commit should add some more focused unit tests here in the internal/providercache package, too.
This was checking for a specific output error message which has changed due to our new provider installer/selection approach.
There was a remaining TODO in this package to find the true provider FQN when looking up the schema for a resource type. We now have that data available in the Provider field of configs.Resource, so we can now complete that change. The tests for this functionality actually live in the parent "command" package as part of the tests for the "terraform show" command, so this fix is verified by all of the TestShow... tests now passing except one, and that remaining one is failing for some other reason which we'll address in a later commit.
This library implements the user-specific directory layout specifications for various platforms (XDG on Unix, "Known Folders" on Windows, etc). We'll use this in a subsequent commit to add additional system-specific search directories for provider plugins, and perhaps later on also CLI configuration directories.
This restores some of the local search directories we used to include when searching for provider plugins in Terraform 0.12 and earlier. The directory structures we are expecting in these are different than before, so existing directory contents will not be compatible without restructuring, but we need to retain support for these local directories so that users can continue to sideload third-party provider plugins until the explicit, first-class provider mirrors configuration (in CLI config) is implemented, at which point users will be able to override these to whatever directories they want. This also includes some new search directories that are specific to the operating system where Terraform is running, following the documented layout conventions of that platform. In particular, this follows the XDG Base Directory specification on Unix systems, which has been a somewhat-common request to better support "sideloading" of packages via standard Linux distribution package managers and other similar mechanisms. While it isn't strictly necessary to add that now, it seems ideal to do all of the changes to our search directory layout at once so that our documentation about this can cleanly distinguish "0.12 and earlier" vs. "0.13 and later", rather than having to document a complex sequence of smaller changes. Because this behavior is a result of the integration of package main with package command, this behavior is verified using an e2etest rather than a unit test. That test, TestInitProvidersVendored, is also fixed here to create a suitable directory structure for the platform where the test is being run. This fixes TestInitProvidersVendored.
The provider fully-qualified name string used in configuration is very long, and since most providers are hosted in the public registry, most of that length is redundant. This commit adds and uses a `ForDisplay` method, which simplifies the presentation of provider FQNs. If the hostname is the default hostname, we now display only the namespace and type. This is only used in UI, but should still be unambiguous, as it matches the FQN string parsing behaviour.
The fake installable package meta used a ZIP archive which gave different checksums between macOS and Linux targets. This commit removes the target from the contents of this archive, and updates the golden hash value in the test to match. This test should now pass on both platforms.
* helper/resource: remove provider resolver test * repl tests passing * helper/resource: add some extra shimming to ShimLegacyState Some of the tests in helper/resource have to shim between legacy and modern states. Terraform expects modern states to have the Provider set for each resource (and not be a legacy provider). This PR adds a wrapper around terraform.ShimLegacyState which iterates through the resources in the state (after the shim) and adds the provider FQN.
* show text fixture update * temporarily disable providers tests
We're now longer showing the default registry hostname as part of addresses coming from that registry.
When a provider dependency is implicit rather than explicit, or otherwise when version constraints are lacking, we produce a warning recommending the addition of explicit version constraints in the configuration. This restores the warning functionality from previous Terraform versions, adapting it slightly to account for the new provider FQN syntax and to recommend using a required_providers block rather than version constraints in "provider" blocks, because the latter is no longer recommended in the documentation.
With provider dependencies now appearing inside a nested block, it seems likely that configuration examples showing dependencies out of context will sometimes mislead users into thinking that required_providers is toplevel. To give better feedback in that situation, we'll produce a specialized error in that case hinting the correct structure to the user.
apparentlymart
force-pushed
the
f-init-new-installer
branch
from
3a8c4f4 to
3e3d8f6
Compare
These will now use "default" provider addresses, rather than "legacy" ones, so that they can cooperate with the rest of Terraform that has been updated to no longer use legacy provider addresses.
|
Because we reviewed the intermediate changes above in separate PRs along the way, we decided to land this integration branch directly. Further work will occur on the |
apparentlymart
changed the title
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This integrates the new provider installer and associated machinery into the
terraform initcommand, replacing the old one.