Init using AzureRM Storage Account appears broken with 1.11

[nathanblair]

Terraform Version

1.11.0
on darwin_arm64

Terraform Configuration Files

terraform {
  backend "azurerm" {
    storage_account_name = "REDACTED"
    container_name       = "tfstate"
    key                  = "REDACTED.json"

    use_azuread_auth = true
  }
}

Debug Output

2025-02-27T10:03:23.914-0500 [INFO]  Terraform version: 1.11.0
2025-02-27T10:03:23.914-0500 [DEBUG] using github.com/hashicorp/go-tfe v1.70.0
2025-02-27T10:03:23.914-0500 [DEBUG] using github.com/hashicorp/hcl/v2 v2.23.0
2025-02-27T10:03:23.914-0500 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2025-02-27T10:03:23.914-0500 [DEBUG] using github.com/zclconf/go-cty v1.16.0
2025-02-27T10:03:23.914-0500 [INFO]  Go runtime version: go1.23.3
...
2025-02-27T10:03:23.917-0500 [DEBUG] Configuring built-in cloud environment by name: "public"
2025-02-27T10:03:23.917-0500 [DEBUG] POST https://login.microsoftonline.com/REDACTED/oauth2/v2.0/token
2025-02-27T10:03:24.271-0500 [DEBUG] GET https://management.azure.com/subscriptions/REDACTED/resourceGroups//providers/Microsoft.Storage/storageAccounts/REDACTED?api-version=2023-01-01
Initializing modules...
2025-02-27T10:03:24.597-0500 [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json
╷
│ Error: retrieving Storage Account (Subscription: "REDACTED"
│ Resource Group Name: ""
│ Storage Account Name: "REDACTED"): unexpected status 400 (400 Bad Request) with error: InvalidApiVersionParameter: The api-version '2023-01-01' is invalid. The supported versions are '2024-11-01,2024-08-01,2024-07-01,2024-06-01-preview,2024-03-01,2023-07-01,2023-07-01-preview,2023-03-01-preview,2022-12-01,2022-11-01-preview,2022-09-01,2022-06-01,2022-05-01,2022-03-01-preview,2022-01-01,2021-04-01,2021-01-01,2020-10-01,2020-09-01,2020-08-01,2020-07-01,2020-06-01,2020-05-01,2020-01-01,2019-11-01,2019-10-01,2019-09-01,2019-08-01,2019-07-01,2019-06-01,2019-05-10,2019-05-01,2019-03-01,2018-11-01,2018-09-01,2018-08-01,2018-07-01,2018-06-01,2018-05-01,2018-02-01,2018-01-01,2017-12-01,2017-08-01,2017-06-01,2017-05-10,2017-05-01,2017-03-01,2016-09-01,2016-07-01,2016-06-01,2016-02-01,2015-11-01,2015-01-01,2014-04-01-preview,2014-04-01,2014-01-01,2013-03-01,2014-02-26,2014-04'.

Expected Behavior

Initialization proceeds successfully as it did in 1.10.x

Actual Behavior

The error above.

Steps to Reproduce

1. terraform init

Additional Context

None. Occurs on a local dev machine and using GitHub runners (that consume the setup-terraform action to install the latest version of terraform).

References

From the release notes

Updates the azure backend authentication to match the terraform-provider-azurermprovider authentication, in several ways:

github.com/hashicorp/go-azure-helpers: v0.43.0 -> v0.71.0
github.com/hashicorp/go-azure-sdk/[resource-manager/sdk]: v0.20241212.1154051. This replaces the deprecated Azure SDK used before
github.com/jackofallops/giovanni: v0.15.1 -> v0.27.0. Meanwhile, updating the azure storage API version from 2018-11-09 to 2023-11-03
Following new properties are added for the azure backend configuration:

• use_cli
• use_aks_workload_identity
• client_id_file_path
• client_certificate
• client_id_file_path
• client_secret_file_path

It looks like this was something that was changed in the 1.11 version. Though I don't imagine whatever was changed was intended to break with an API version change.

Generative AI / LLM assisted development?

???

[lewis-jackson-bots]

We're seeing this locally, but our debug log shows that the subscription is no longer being inferred the way that it used to be:

2025-02-27T14:51:25.115Z [DEBUG] GET https://management.azure.com/subscriptions//resourceGroups/REDACTED/providers/Microsoft.Storage/storageAccounts/REDACTED?api-version=2023-01-01

NOTE: the //, the subscription is now missing.

it looks like we can fix it locally by specifying subscription_id in our backend block.

[bamarch]

We encountered both issues when going from 1.10 to 1.11

First, terraform init complained that there was no subscription ID. This was solved by adding ARM_SUBSCRIPTION_ID env var, you could also probably add subscription_id to the backend config.

Second, this "invalid API version" issue.

│ Error: retrieving Storage Account (Subscription: "REDACTED"
│ Resource Group Name: ""
│ Storage Account Name: "REDACTED"): unexpected status 400 (400 Bad Request) with error: InvalidApiVersionParameter: The api-version '2023-01-01' is invalid. The supported versions are '2024-11-01,2024-08-01,2024-07-01,2024-06-01-preview,2024-03-01,2023-07-01,2023-07-01-preview,2023-03-01-preview,2022-12-01,2022-11-01-preview,2022-09-01,2022-06-01,2022-05-01,2022-03-01-preview,2022-01-01,2021-04-01,2021-01-01,2020-10-01,2020-09-01,2020-08-01,2020-07-01,2020-06-01,2020-05-01,2020-01-01,2019-11-01,2019-10-01,2019-09-01,2019-08-01,2019-07-01,2019-06-01,2019-05-10,2019-05-01,2019-03-01,2018-11-01,2018-09-01,2018-08-01,2018-07-01,2018-06-01,2018-05-01,2018-02-01,2018-01-01,2017-12-01,2017-08-01,2017-06-01,2017-05-10,2017-05-01,2017-03-01,2016-09-01,2016-07-01,2016-06-01,2016-02-01,2015-11-01,2015-01-01,2014-04-01-preview,2014-04-01,2014-01-01,2013-03-01,2014-02-26,2014-04'.

Looking closely at the error message it contains Resource Group Name: ""

The issue goes away when adding resource_group_name to backend config (see https://developer.hashicorp.com/terraform/language/backend/azurerm#example-backend-configurations)

There does not appear to be an environment variable for this, but it can be passed to terraform init via -backend-config="resource_group_name=my-resource-group-name"

These issues arose when nothing was changed other than the TF version from 1.10 to 1.11, so I think both represent breaking changes

[TyranVDM]

It also seems that now the identity used by Terraform requires read permission on the storage account resource where previously this worked with container level permission only.

[rlaveycal]

We're getting this error

Error: retrieving Storage Account (Subscription: ""
Resource Group Name: ""
Storage Account Name: "REDACTED"): unexpected status 400 (400 Bad Request) with error: InvalidSubscriptionId: The provided subscription identifier 'resourceGroups' is malformed or invalid.

We use MSI with Azure RBAC to access the storage account, so do not set subscription or resource group.

[rowanmoul]

I can also confirm that including the resource group in the backend configuration solves the issue (I already had the subscription). Given these are not required fields, there is still a bug to be addressed here (or the fields should become required).

EDIT: I spoke too soon. Adding the resource group does allow terraform init to proceed, but I get the same failure later on, halfway through terraform apply

[d-jaquay]

In case anyone else is running into these issues I'm finding that on terraform 1.10.5 the connection to the storage account was strictly over the data plane and so the storageaccount name was the only property needed. You could even use the wrong subscriptionID and the connection would still work. With terraform 1.11 the URL for the storage account is retrieved from the Azure management plane instead which means that the full resource ID is being formulated from the subscription_id and resource_group_name properties to retrieve the URL to the storage account. As stated earlier by @TyranVDM this means you also need to grant 'Microsoft.Storage/storageAccounts/read' when you could get away with just the Storage Blob Data Contributor role or terraform init fails now.

[rlaveycal]

So is the breaking change that is #36258 going to be reverted, or will the next version of TF be 2.0?

[torbendury]

In case anyone else is running into these issues I'm finding that on terraform 1.10.5 the connection to the storage account was strictly over the data plane and so the storageaccount name was the only property needed. You could even use the wrong subscriptionID and the connection would still work. With terraform 1.11 the URL for the storage account is retrieved from the Azure management plane instead which means that the full resource ID is being formulated from the subscription_id and resource_group_name properties to retrieve the URL to the storage account. As stated earlier by @TyranVDM this means you also need to grant 'Microsoft.Storage/storageAccounts/read' when you could get away with just the Storage Blob Data Contributor role or terraform init fails now.

If this is it, then fine-granular access just fell apart and I'm now forced to create Storage Accounts which serve my project teams as a backend. In the past, I was able to handle one Storage Account and grant fine granular access to the automation service principals to their containers, giving me the opportunity to also backup the whole Storage Account into whatever backup destination fitted the need.

IMHO this is a breaking change.

[magodo]

Sorry folks for the breaking change.. I've made a fix in #36623, and tested it as much as possible to ensure all the cases are covered (see the PR description).

[rmsilva1973]

@magodo Thanks a lot for the quick response! Downgraded to v1.10.5 and will try the fixed version as soon as it's released. I downloaded the source for the azurerm provider and banging my head on the wall to find out where is the code that initializes the backend. It's really surprising finding out it's on the core code. I thought the core "invoked" the provider via RPC. Is there somewhere a doc explaining how init works internally? Or looking at the core code is the way?

OTOH I've never looked at providers internally. It was a fun ride and perhaps I'll take sometime trying to tackle issues on aws or azure provider.

[black-snow]

ref. Azure/azure-cli#30953