[Enhancement]: Add region argument/parameter where appropriate

[tmccombs]

Description

When working with multiple regions, currently in most cases you have to use a separate aws provider alias for each region. While this is workable in many cases, there are other cases where it is not very workable. In the best case it means you have to duplicate your provider configuration, varying only in the region. In other cases it can seriously constrain the design of the terraform code. If resources allowed you to specify the region to use for them, through an attribute that override the default region of the provider, than it would simplify working with multiple regions without having to wait for support for more dynamic provider configuration from the terraform engine.

Example Use Cases

Here are some specific use cases where being able to create resources in multiple regions with the same provider would be beneficial:

• A module which creates a primary s3 bucket, with replicas in a number of regions, and sets up replication to those other buckets. Currently due to needing a separate provider for each region the number of regions supported would have to be a fixed number, and the resources for each bucket would need to be copied a number of times equal to the number of regions (foreach and count can't be used). (This would be possible if you had both a way to pass in a dynamic set of aliases to a module, and a way to do a for_each over such a dynamic set of providers).
• Similar to the above but for kms replica keys (and actually the above module might potentially want to create kms replicas in each region where there is a bucket).
• A module which creates a cloudfront distribution along with an ACM certificate to use with it. Since the certificate has to be in the us-east-1 region, currently you would need the caller of the module to pass in a separate provider alias for the us-east-1 region. And the module can't be sure that the caller passed in the right region.
• A module which optionally (using count) creates some resources in one or more other regions. Possibly because those resources are for backups that aren't needed in non-production cases. It's possible to do this currently, but the caller needs to pass in something for the provider for the optional region, even if it isn't used.

Affected Resource(s) and/or Data Source(s)

Many. Some of the most important ones are possibly:

• aws_s3_bucket and related
• aws_acm_certificate (in particular, you need to create certificates in the us-east-1 region to use with cloudfront)
• aws_kms_key and aws_kms_replica_key

Potential Terraform Configuration

The example from https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_replica_key

could be rewritten to something like:

resource "aws_kms_key" "primary" {
  description             = "Multi-Region primary key"
  deletion_window_in_days = 30
  multi_region            = true
  region                  = "us-east-1"
}

resource "aws_kms_replica_key" "replica" {
  description             = "Multi-Region replica key"
  deletion_window_in_days = 7
  primary_key_arn         = aws_kms_key.primary.arn
  region                  = "us-west-2"
}

Or if used in a module that accepts multiple regions something like:

resource "aws_kms_key" "primary" {
  description             = "Multi-Region primary key"
  deletion_window_in_days = 30
  multi_region            = true
  region                  = var.primary_region
}

resource "aws_kms_replica_key" "replica" {
  for_each = var.replica_regions

  description             = "Multi-Region replica key for ${each.key}"
  deletion_window_in_days = 7
  primary_key_arn         = aws_kms_key.primary.arn
  region                  = each.key
}

References

Some more specific issues:

• [Enhancement]: Add Region argument in resource aws_dx_hosted_private_virtual_interface_accepter #27681
• Add "Region" as argument reference for Data Source: aws_s3_bucket #25817
• Cannot create S3 buckets in multiple regions #8853
• Support configuring region for aws_cloudwatch_log_subscription_filter resource #24194

Would you like to implement a fix?

No response

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[sblask]

I am looking into Guardduty and need to generate 23 providers, one for each supported region to enable Guardduty, it would be much simpler to just have a for_each over the regions. Unfortunately Terraform does not allow dynamic providers...

There is also a problem with having the region on provider level, if you create resources in the wrong region and then change the region on the provider, Terraform only wants to create the resource in the new region, but doesn't want to destroy the resource in the wrong region.

[tmccombs]

If I were to create pull requests to add support for this for at least some resources, would that be likely to be accepted? (contingent on it being high enough quality of course)

[breathingdust]

Hi @tmccombs 👋 We have been chatting with @brittandeyoung about this (he has put a compelling PoC together in #31517) and we are all positive about the approach. There is some internal due diligence we need to get through in introducing what is a bit of a paradigm change in how the provider is used. We have that scheduled for next quarter after which we will update to community on how we propose to implement and introduce it to the provider, would love feedback at that point.

Appreciate your patience and input!

[cobbr2]

It's two quarters later, @breathingdust ; is there an update? I don't see one here or on #31517 . Thanks!

[take-five]

Is there any progress on this? The friction it causes in multi-region deployments is very high.
EDIT: typo

[ewbankkit]

@take-five 👋 The maintainers are actively working on an RFC to enable this functionality.
Expect more information in the next couple of months.

[skeggse]

#13992 is probably the OG issue for this request!

[ewbankkit]

Proposal

Most AWS resources are Regional – they are created and exist in a single AWS Region, and to manage these resources the Terraform AWS Provider directs API calls to endpoints in the Region. The AWS Region used to provision a resource using the provider is defined in the provider configuration used by the resource, either implicitly via environment variables or shared configuration files, or explicitly via the region argument. To manage resources in multiple Regions with a single set of Terraform modules, resources must use the provider meta-argument along with a separate provider configuration for each Region. For large configurations this adds considerable complexity – today AWS operates in 35 Regions, with 5 further Regions announced.
To address this we propose implementing per-resource Region overrides, an additional argument in every Regional resource’s schema which allows that resource to be managed in a Region other than the one defined in the provider configuration.

Let’s start with an example.
The Terraform code to create and accept a cross-Region VPC peering connection:

provider "aws" {
  region = "us-east-1"
}

provider "aws" {
  region = "us-west-2"
  alias  = "west"
}

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_vpc" "peer" {
  provider = aws.west

  cidr_block = "10.1.0.0/16"
}

# Requester's side of the connection.
resource "aws_vpc_peering_connection" "main" {
  vpc_id      = aws_vpc.main.id
  peer_vpc_id = aws_vpc.peer.id
  peer_region = "us-west-2"
  auto_accept = false
}

# Accepter's side of the connection.
resource "aws_vpc_peering_connection_accepter" "peer" {
  provider = aws.west

  vpc_peering_connection_id = aws_vpc_peering_connection.main.id
  auto_accept               = true
}

should be able to be written as:

provider "aws" {
  region = "us-east-1"
}

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_vpc" "peer" {
  region = "us-west-2"

  cidr_block = "10.1.0.0/16"
}

# Requester's side of the connection.
resource "aws_vpc_peering_connection" "main" {
  vpc_id      = aws_vpc.main.id
  peer_vpc_id = aws_vpc.peer.id
  peer_region = "us-west-2"
  auto_accept = false
}

# Accepter's side of the connection.
resource "aws_vpc_peering_connection_accepter" "peer" {
  region = "us-west-2"

  vpc_peering_connection_id = aws_vpc_peering_connection.main.id
  auto_accept               = true
}

This new Terraform code requires only a single provider configuration.

Implementation

Every Regional resource, data source and ephemeral resource will support this feature transparently – the new argument will not need to be explicitly defined in the resource’s schema and (in the majority of cases) the resource implementation will not need to be aware whether or not a resource-level Region override is in place.

Using the provider’s interceptor functionality, introduced for transparent tagging, every Regional resource will have the region argument added to its schema as a top-level attribute (an error will be raised, and the provider will fail to load, if the resource implementation defines its own similarly named attribute). The argument’s value will be validated to ensure that it is in the configured partition (AWS IAM credentials are only valid for a single partition). The injected region argument will be an Optional, ForceNew (for resources) string attribute.

Accessing The Effective Region

The same interceptor functionality used to transparently inject the new configuration block will be used to handle any configured region value for provider operations on a resource. Before the resource implementation’s handlers are called, the planned value of the attribute will be read from provided Terraform configuration (the provider's plan modification mechanism will be used to merge and configured region and the provider configuration Region) and added to the Go Context.

The provider’s global state object (colloquially known as the provider’s meta object) had a field named Region holding the provider configuration’s Region. This field was converted to a getter method in #40335 and as part of the per-resource Region override implementation the method will be modified to return the calculated Region held in the Go Context (the "in-effect Region").

AWS API Clients

The AWS API client lazy initialization mechanism will be extended to store per-Region maps of API clients and return clients for the currently in-effect Region. This ensures that all AWS API calls will be made to the correct Region.

Global Resources

Some resources (for example IAM or STS resources) are global, they exist in all of a partition’s Regions. For such resources, no region attribute will be injected.

Global Services

All resources for the following services will be considered global:

• Account Management (internal/service/account)
• CloudFront (internal/service/cloudfront and internal/service/cloudfrontkeyvaluestore)
• Global Accelerator (internal/service/globalaccelerator)
• IAM (internal/service/iam, internal/service/rolesanywhere and internal/service/sts)
• Network Manager (internal/service/networkmanager)
• Organizations (internal/service/organizations)
• Route 53 (internal/service/route53 and internal/service/route53domains)
• Route 53 ARC (internal/service/route53recoverycontrolconfig and internal/service/route53recoveryreadiness)
• Shield Advanced (internal/service/shield)
• WAF Classic (internal/service/waf)

Importing Resources

Importing existing resources into Terraform, using either the terraform import CLI command or an import block, does not access configuration and so will not be aware of any desired per-resource Region override. To allow importing resources in Regions different from that specified via provider configuration we will support a standard suffix, @<regionID>, on the import identifier. This standard suffix will be parsed and removed from the import identifier before it is passed to the resource’s import handler.

For example if the provider configuration specified the us-west-2 Region,

% terraform import aws_db_instance.default mydb-rds-instance@us-east-1

will import an RDS DB Instance in the us-east-1 Region.

Conflicting Attribute Names

As of Terraform AWS Provider v5.83.0 a small number of existing resources and data sources already have a top-level region attribute.

Data Sources

The data sources that act as per-Region lookups will remain as-is:

• aws_apprunner_hosted_zone_id
• aws_cloudfront_log_delivery_canonical_user_id
• aws_cloudtrail_service_account
• aws_elastic_beanstalk_hosted_zone
• aws_elb_hosted_zone_id
• aws_elb_service_account
• aws_lb_hosted_zone_id
• aws_redshift_service_account
• aws_sagemaker_prebuilt_ecr_image
• aws_service
• aws_service_principal

Other data sources that will remain as-is:

• aws_arn – region is a Computed-only attribute. Makes no AWS API calls

Some data sources will have a well documented change in semantics:

• aws_availability_zone – region is a Computed-only attribute, returned from an EC2 API call. Will not break existing configurations

Some data sources will have their existing region attributes deprecated at Terraform AWS Provider v6.0.0 with a new attribute as replacement:

• aws_s3_bucket
• aws_servicequotas_templates
• aws_ssmincidents_replication_set
• aws_vpc_peering_connection

Resources

Some resources will have a well documented change in semantics:

• aws_lightsail_bucket - region is a Computed-only attribute, returned from a Lightsail API call. Will not break existing configurations
• aws_s3_bucket - region is a Computed-only attribute, returned from an S3 API call. Will not break existing configurations

Some resources will have their existing region attributes deprecated at Terraform AWS Provider v6.0.0 with a new attribute as replacement:

• aws_cloudformation_stack_set_instance
• aws_config_aggregate_authorization
• aws_dx_hosted_connection
• aws_servicequotas_template
• aws_ssmincidents_replication_set

Timeline

We are now solicting feedback from the community with the goal of finalizing design by the end of January 2025. Implementation will take place in the February 2025 to April 2025 quarter with release (after a planned public beta) in Terraform AWS Provider v6.0.0.

[lorengordon]

Note that not all IAM resources are global. IAM Access Analyzer, for example, is regional.

[apparentlymart]

There is another region-related quirk with the current design of this provider: if you've previously applied a configuration containing AWS resources associated with a particular region and then you change the region argument in the provider block, the provider attempts to refresh the existing objects in the "wrong" region, concludes that they have been deleted outside of Terraform, and so proposes to create new ones in the new region while leaving the objects in the old region forgotten.

I wonder if a solution to that problem could also fit in with the changes proposed above. Here's some details on what I'm thinking, although I've not tried this out so I can't promise all of this is sound...

---

The first part I'd add is to make the PlanResourceChange phase check the region argument from the configuration value (i.e. from the desired state):

• If the value isn't null, take the value as given.
• If the value is null, substitute the current region specified in the provider configuration.

After that, compare the region in the prior state to the region in the configuration. If they are different, return a plan with the region field populated with the value from the configuration (possibly substituted from the provider configuration as above) and mark it as "requires replacement" so that Terraform Core will propose to replace the object.

Taken alone the above rule would force proposing replacement for every object that already exists, because the prior state would have a null region. To avoid that, UpgradeResourceState should replace any null region with the provider configuration's currently selected region, unless it's one of the resource types described in the proposal that already has its own region argument that requires some special treatment. That will then avoid any proposed replacement immediately after upgrading to a new version with this rule.

---

The second part is that whenever making requests for a particular object the provider should take care to always use the region associated with the resource instead of the region in the provider configuration. In particular, during ApplyResourceChange if an existing object needs to be deleted it must be deleted in the region given in its region argument, which might not match the current provider-level region and also might not match the current resource-level region.

When combined with the special planning behavior I just described, this would mean that changing the region at the provider level would behave the same as changing the region for each resource individually, and that during the apply phase the "delete" request would happen in the old region while the "create" request happens in the new region, thereby allowing Terraform to behave as if it's moving an object between regions, regardless of whether that change happened at the whole-provider level or at the per-resource level.

---

Overall I don't think what I've described here is significantly different than the proposal above. The main addition is for the provider to replace an unpopulated resource-level region argument with the provider-level region before doing any other work during the planning phase, which therefore means that each resource instance will "remember" which region it was created in regardless of whether that decision was made globally or locally.

One specific problem I've not thought very deeply about is how to handle the case where the region is changed in a way that also changes the partition. As the above proposal notes, each partition has completely separate authentication objects and so it doesn't seem like it would be possible to support a single resource instance being destroyed in a region in one partition and created in a region in a different partition in the same plan/apply round: there is no one set of credentials that could possibly work for both. Perhaps PlanResourceChange ought to present an error in that case saying that it simply isn't possible to change the partition of an object that already exists. Maybe there's another way to solve this that I'm not considering. 🤔

[ewbankkit]

@lorengordon Thanks for pointing that out. There is the same ambiguity around Route 53 -- Route 53 profiles and Route 53 resolver are Regional while vanilla Route 53 and Route 53 domains are global. I'll add the names of the service packages (directories below internal/service in the code) to the comment.

[ewbankkit]

@apparentlymart Thanks for the considered comment. We saw a previous iteration of your thoughts in #31517 (comment) and very much would like to implement that capability. The stumbling block to immediate (in the scope of the proposal for v6.0.0) implementation is the capturing of the already in-effect Region (from provider configuration) via the state upgrade mechanism -- we haven't (yet) thought of a way to force the upgrade mechanism to run (i.e. an increment of each resource's schema version number) without manually modifying the ~1500 resource implementations. We would definitely like to introduce the capability in a future version (having captured the in-effect Region once the provider has been upgraded to v6.0.0).

I have updated the proposal with details on ensuring that API calls are made to the correct Region (missed in initial copy-paste from internal design documentation).

Your point around changing a per-resource Region to one in a different partition is partly handled by implementing verification that old and new Regions are in the same partition, but we would have to think about this more in the case that we are "remembering" the prior Region. In this case I don't think we can delete the resource under the old credentials (credentials are partition-specific) as they are no longer available.

[apparentlymart]

Hi @ewbankkit!

I'd forgotten that I'd shared a similar idea before, so sorry for the redundancy but I'm pleased to see that the two times I wrote this up I wrote essentially the same thing since when I wrote that earlier comment I'd been playing with the idea more actively and so had more relevant context in my head. 😀

Terraform v0.12 and later call UpgradeResourceState unconditionally for every existing resource instance during the planning phase, regardless of whether the schema version number has changed, because that call also doubles as a way to migrate from the legacy "flatmap" serialization of state and for any JSON value transformations the SDK or framework can handle automatically without any special logic in the provider. So the schema version number is primarily a mechanism for the SDK/framework to use to decide which upgrade functions to call, and not something Terraform Core really cares about aside from saving it in the state along with the state data.

In principle then, from Terraform Core's perspective it shouldn't be necessary to increment the schema version number. Whether it's feasible to make the SDK and framework themselves run this special "global" upgrade logic without changing the schema version number is a different question of course, and not one I'm equipped to answer.

(What I've said above about UpgradeResourceState being called unconditionally does seem in conflict with the documentation you linked, but I'm pretty sure that it's the documentation that's wrong. The current Terraform Core team can hopefully confirm that assertion by referring to the relevant code.)