Remove use of logging commands for staging build outputs

[mmitche]

Publishing with Publish Artifacts Only

Today, Arcade's v3 publishing task relies on Azure Pipeline's "Build Artifacts", a mutable (while the build is running) storage for build outputs. Each job publishes to a container with a well-known name. Then the "Publish to Darc" job kicks off another AzDO job that uses the Azure DevOps API to pull down artifacts from the well-known container names and pushes them to the locations associated with channels. Additionally, Arcade does so using "Azure Pipelines command comments" to upload the files.

This design has a few positive aspects:

1. There's no need to add a separate "Upload" step to jobs that publish assets. The build itself drives the publish with the "command comments".
2. The "Publish to BAR" and "Publish to Darc" jobs/steps and the actual publish pipeline don't need to know anything about which jobs were run as all artifacts from all jobs are in the well-known artifact containers.

There are a number of downsides to this approach:

1. The "Build Artifacts" storage is mutable, which means that someone can inject artifacts into the storage while the build is running.
2. Build Artifacts are slower to upload/download than Pipeline Artifacts
3. Duplicate artifacts can cause overwrites.
4. Tracing back which job produced which artifact is difficult (requires looking through all asset manifests).
5. If multiple jobs produce asset manifests with the same name, the conflict is not detected and can cause assets to be dropped (this happened in .NET 10 Preview 1 in the SDK repo).
6. We can't use 1ES templateContext to specify the outputs, so we get a less-efficient 1ES upload process for assets that we actually upload to build artifacts with tasks.
7. 1ES tooling doesn't (and can't) intercept the "command comments" to upload files, so we need some level of our own tooling to ensure that 1ES sees everything it is supposed to see.
   • There was an issue with the SBOM tooling related to this problem.

This document proposes a new design that addresses these downsides.

Proposal

1. Use the feature of the PushToBuildStorage task to push to a location on disk to publish assets to disk instead of using "command comments" to publish assets to Build Artifacts.

• We can use the same layout that individual verticals in the VMR publish its artifacts in.

2. Use PublishPipelineArtifacts or 1ES templateContext (when in a 1ES official pipeline) to upload the outputs of a given job to a specific pipeline artifact for that job.
3. Update the "Publish Assets" job to take a list of pipeline artifacts to publish.
4. When pushing to BAR, we will always produce a "merged asset manifest", even if there's only one asset manifest, and save it to a well known Pipeline Artifact.
5. Each asset in the asset manifest will contain the name of the Pipeline Artifact that it is stored in.
6. The publishing pipeline will pull down the merged manifest and use it to determine where to download assets from.

Bonus Aspects of this Design

This design could also help us with solving #15317.

We would be able to use this design to help remove some of the custom tooling we have in the VMR around publishing assets for a given vertical and (if we change how we handle the join-verticals job) publishing assets for the VMR as a whole.