Rustwide fails due to symlink resolution with new rustup

[syphar]

Verification

• I searched for recent similar issues at https://github.com/rust-lang/rustup/issues?q=is%3Aissue+is%3Aopen%2Cclosed and found no duplicates.
• I am on the latest version of Rustup according to https://github.com/rust-lang/rustup/tags and am still able to reproduce my issue.

Problem

coming from this report: rust-lang/rustwide#94 , currently docs.rs builds are broken because of this issue.

There seems to be something wrong with the symlinks that are now generated instead of hardlinks for the binaries (#4023).

We are using fs::canonicalize before passing it to the binaries:
https://github.com/rust-lang/rustwide/blob/744375a9050e0584124ea11adaeab41781fa8a50/src/utils.rs#L111-L112

Steps

What information would you need to be able to track this down?

we can reproduce it when just trying to run any docs.rs build,

the two errors I saw where both just showing that the call does to rustup install instead of cargo install (or other calls to cargo are done on rustup)

Possible Solution(s)

No response

Notes

No response

Rustup version

1.28.0

Installed toolchains

it happens with just one toolchain installed, tested with `stable-aarch64-apple-darwin` or `stable-x86_64-unknown-linux-gnu`

OS version

Ubuntu 18.04.2 LTS on the server, or newer ubuntu in docker images

[syphar]

cc @skius @Skgland

[djc]

Would be nice to get a bit closer to the root cause of this issue, do you think you can do that? Do you have a reproduction scenario that doesn't involve rustwide/docs.rs?

[ChrisDenton]

Quick workarounds I can think of:

• Maybe std::path::absolute would help?
• Or else just canonicalise the parent path than rejoin the file name?

[syphar]

Would be nice to get a bit closer to the root cause of this issue, do you think you can do that? Do you have a reproduction scenario that doesn't involve rustwide/docs.rs?

not yet, below is something that works with rustwide only.

I don't know rustwide internals / the build process very well, so a more low level example will take more time.

I'll probably first try to find a way to downgrade rustup for rustwide.

the rustwide / rustup example

The rustwide docs-builder example lets us see the error directly.

It can be made a little faster by choosing a smaller docker base image:

[...]
    // Create a new workspace in .workspaces/docs-builder
    let workspace =
        WorkspaceBuilder::new(Path::new(".workspaces/docs-builder"), "rustwide-examples")
            // this is new
            .sandbox_image(SandboxImage::remote(
                "ghcr.io/rust-lang/crates-build-env/linux-micro",
            )?)
            .init()?;
[...]

[syphar]

Quick workarounds I can think of:

* Maybe [`std::path::absolute`](https://doc.rust-lang.org/stable/std/path/fn.absolute.html) would help?

* Or else just `canonicalise` the parent path than rejoin the file name?

You mean in our rustwide code? Or rustup?

if rustwide, I first need to figure out why we actually canonicalize the paths before calling, that needs digging into old crater code before rustwide was extracted

[ChrisDenton]

On the rustwide side.

I don't think canonicalizing the filename can ever work with symlinked proxies because rustup shims rely on the link name to know where to direct the command to whereas canonicalizing a symlink removes that information by design. Canonicalizing the parent directory should always be fine though.

On the rustup side we could remove symlink support and only hard link (we've had this symlink behaviour for a long time but hard linking used to take precedence). Or else there might be something we can do to fix whatever problem necessitated canonicalizing.

[syphar]

Or else there might be something we can do to fix whatever problem necessitated canonicalizing.

I'm still trying to figure that out. It was added in rust-lang/crater@2c3e40c, but no specific explaination as to why.

Canonicalizing the parent directory should always be fine though.

I will see if I can figure out something here. ( hoping it's the only issue )

[Skgland]

I don't think canonicalizing the filename can ever work with symlinked proxies because rustup shims rely on the link name to know where to direct the command to whereas canonicalizing a symlink removes that information by design. Canonicalizing the parent directory should always be fine though.

Explicitly setting arg0 could work but that's only available on Unix and not on Windows.
Not canonicalizing is probably easier.

[ChrisDenton]

Oh of course. It can be done on Windows too but we haven't added it to the standard library yet. We really should. And possibly make a cross-platform arg0 function. But I realise that doesn't help in this moment.

[syphar]

so, path::absolute doesn't work. My current hunch is that this is related to how rustwide maps the toolchain into the docker container, and it somehow breaks when we try to use symlinks

[syphar]

Canonicalizing the parent directory should always be fine though.

this also doesn't work. Not sure why yet

[ChrisDenton]

My current hunch is that this is related to how rustwide maps the toolchain into the docker container, and it somehow breaks when we try to use symlinks

Ah, could this be #4222? Which is definitely something we should fix.

[syphar]

My current hunch is that this is related to how rustwide maps the toolchain into the docker container, and it somehow breaks when we try to use symlinks

Ah, could this be #4222? Which is definitely something we should fix.

could be. From what I remember, the toolchain installation is done outside of docker on the host, and then the directory is mapped into the docker container.

[syphar]

an additional thought:

I'm not 100% sure how docker maps filesystems, but I could imagine that even when we would switch to relative symlinks, this might not work, if the relative links points outside of the docker-mapped fileystem.

[syphar]

I'm really happy to find a way to make this work with rustwide, but I would really like to do this properly, without having the pressure of docs.rs builds not working.

I'll probably try first if I can downgrade again, our first try didn't work.

[syphar]

I got docs.rs build again by forcing rustwide to use 1.27.1 and blocking the self update.

I'll probably wait for #4222 / #4226 to be released and try again. If that doesn't work I have to dig a little deeper into how rustup works with the links/ shims, and also how exactly the toolchain is set up & linked into the rustwide docker containers.

[ChrisDenton]

Ok, new release should be out now. Sorry for the delay, unfortunately the release process is not anywhere near as simple as pressing a button.

[syphar]

Ok, new release should be out now. Sorry for the delay, unfortunately the release process is not anywhere near as simple as pressing a button.

so #4226 is in that release?

[syphar]

sorry, saw it in the release notes.

Will look if it works now later

[ceronman]

There is a similar problem with the change to symlinks instead of hard links in the new version of rustup.
If you are on WSL, Windows can't follow symlinks. Due to this, RustRover users on windows can't select a WSL based toolchain, because both cargo and rustc are now symlinks to rustup, but windows can't follow those links: RUST-17235 Unable to add Rust toolchain on WSL

[ChrisDenton]

I don't know their code but judging from the error "can't fine Cargo" I guess it's testing existence of cargo by attempting to follow the symlink? Ideally they would only test if the symlink itself exists (in Rust that can be done with std::fs::symlink_metadata and in Java you'd use the NOFOLLOW_LINKS link option).

[ceronman]

@ChrisDenton If this was Unix that's possible, but on Windows unfortunately it's not possible to follow symlinks on a linux filesystem, see my link above. So Java at least is unable to follow the link, or even consider the file as existing.

[ChrisDenton]

Right yes, what I was trying to say is that there doesn't appear to be a reason to follow the link. They are presumably only testing for existence so they don't need to follow symlinks.