Skip to content

Conversation

@abdurriq
Copy link

@abdurriq abdurriq commented Dec 4, 2025

Should fix #55. This change does away with the aliases approach and instead installs shims which call the ado-auth-helper and pass the result to the actual executable. To hijack calls to the actual executables, they need to be installed further up in PATH (as high as possible), to ensure that they have the highest priority when the command is run. This should be a more robust approach than aliases, which are tricky to forcibly source in non-interactive shells.

The resolve-shim.sh helper script finds the actual executable by using which to enumerate all executables, and then using the 2nd (not 1st, because 1st is the shim script).

TODO:

  • README needs to be updated (apparently auto-generated?)
  • Test changes

@@ -0,0 +1,6 @@
#!/bin/bash
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These all (apart from dotnet + nuget) could be auto-generated, since they are identical except in naming.

@abdurriq abdurriq force-pushed the azure-artifacts-shims-fix branch from d59af58 to 2e02c21 Compare December 4, 2025 19:47
Comment on lines +2 to +3
[[ ${RESOLVE_SHIMS_IMPORTED} == "true" ]] && return
RESOLVE_SHIMS_IMPORTED=true
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Useful to avoid the function being redefined according to Copilot. Probably unnecessary.

…ents

These environments do not respect the BASH_ENV variable so installing these alias hooks does not work
@abdurriq abdurriq force-pushed the azure-artifacts-shims-fix branch from 2e02c21 to c298b99 Compare December 4, 2025 19:55
@markphip
Copy link
Contributor

markphip commented Dec 4, 2025

This is similar to #83

Can we think about compatibilty for all the people using :latest? Even if this works perfectly/better, it would still break anyone that was using targetfiles with :latest.

I still also wonder how reliable PATH is and whether since this is a devcontainer if there was anything else we could do to help the shims be first in PATH?

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

@microsoft-github-policy-service agree company="Microsoft"

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

Can we think about compatibilty for all the people using :latest? Even if this works perfectly/better, it would still break anyone that was using targetfiles with :latest.

Ah good point; I expect we need to maintain backward compatibility for these then?

I still also wonder how reliable PATH is and whether since this is a devcontainer if there was anything else we could do to help the shims be first in PATH?

Yeah, this is a bit tricky, which is why I preferred to delegate it to the consumer (by providing a suitable shimDirectory that is high-enough priority in PATH). I noticed that @delilahw explicitly prepended the folder containing the scripts to PATH, which I will copy. However, I'm not sure if this is a sure-fire enough way, considering other features may interfere with this, but I guess it should be robust enough for most use-cases.

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

I guess latest could be updated in a transparent way too, where targetFiles is either ignored, or respected in that PATH is extended for them. I'll try to see if I can figure out how to do that in a neat way.

@markphip
Copy link
Contributor

markphip commented Dec 5, 2025

I guess latest could be updated in a transparent way too, where targetFiles is either ignored, or respected in that PATH is extended for them. I'll try to see if I can figure out how to do that in a neat way.

My thought is that we leave the targetFiles parameter in the config so that if someone has that specified in their devcontainer.json the build does not fail. For the implementation we could just create the file and the content could just have some comments like:

# File created for backwards compatibility only
# use version 2 if the functionality is needed
# "features": {
#     "ghcr.io/microsoft/codespace-features/artifacts-helper:2": {}
# }

There are some people that source this script so I do think we need to put a script in place to not break that functionality.

@markphip
Copy link
Contributor

markphip commented Dec 5, 2025

I still also wonder how reliable PATH is and whether since this is a devcontainer if there was anything else we could do to help the shims be first in PATH?

Yeah, this is a bit tricky, which is why I preferred to delegate it to the consumer (by providing a suitable shimDirectory that is high-enough priority in PATH).

I do like your approach. I was just kind of thinking for documentation purposes if there were any other suggestions we could make about things the customer could do with their devcontainer.json configuration. I was not sure if the system PATH could be configured via remoteEnv or some other setting.

It would be good to make a few test Codespaces using the universal, dotnet and node images and just compare the PATH to see if there is a consistent location we could use. What folder does dotnet, node, yarn typically get installed and where is that on default PATH etc.

@markphip
Copy link
Contributor

markphip commented Dec 5, 2025

I did a code search and only found one repo using targetFiles. I created a PR to pin them to version 2. Let's just drop the option as that will be cleaner for the future.

cc: @scaryrawr

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

I did a code search and only found one repo using targetFiles. I created a PR to pin them to version 2. Let's just drop the option as that will be cleaner for the future..

Oh nice, thank you very much! In that case, I'll drop that option + implementation, and keep only the new implementation. I will check the PATH env variable in the different images as you suggested to see if there is a consistent location. I was thinking $HOME/.local/bin which seems quite high on PATH, at least in the image we are using, but I'll verify that further (also, it won't likely work for scripts run as another user / root; not sure how crucial that is).

@markphip
Copy link
Contributor

markphip commented Dec 5, 2025

#55 discusses some of the reasons people were trying targetFiles. The scenario I want to see work is when VS Code extensions, such as the C# tooling, run these commands from the extension. I am hoping that the shim approach will finally make that work.

BTW, could we put some checks and a wait/retry in the shim script? When a Codespace first starts there is a delay until the ADO extension activates. If, as an example, the C# extension runs dotnet restore before this happens even if the shim is in place the authentication will not work.

I have used a script like this in my postStartCommand to handle this:

#!/bin/bash
set -e

echo "::step::Waiting for AzDO Authentication Helper..."

# Wait up to 3 minutes for the ado-auth-helper to be installed
MAX_WAIT=180
ELAPSED=0

while [ $ELAPSED -lt $MAX_WAIT ]; do
    if [ -f "${HOME}/ado-auth-helper" ]; then
        echo "::step::Running dotnet restore..."
        dotnet restore
        echo "::step::✓ Restore completed successfully"
        exit 0
    fi
    sleep 2
    ELAPSED=$((ELAPSED + 2))
    
    # Progress indicator every 20 seconds
    if [ $((ELAPSED % 20)) -eq 0 ]; then
        echo "  Still waiting... (${ELAPSED}s elapsed)"
    fi
done

# Timeout reached
echo "::error::AzDO Authentication Helper not found after ${MAX_WAIT} seconds"
echo "Expected location: ${HOME}/ado-auth-helper"
echo "Restore cannot proceed without authentication"
exit 1

And we do something similar in the external-repository extension to make sure the ADO extension is available before trying to access git.

We ought to be able to build that in here as well so that when the shim is invoked it checks to see if the environment is ready yet before moving on. I would guess that could be added to your auth-ado.sh script to make obtaining the token more robust and handle this startup scenario better.

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

#55 discusses some of the reasons people were trying targetFiles. The scenario I want to see work is when VS Code extensions, such as the C# tooling, run these commands from the extension. I am hoping that the shim approach will finally make that work.

This is the same motivation for me!

BTW, could we put some checks and a wait/retry in the shim script? When a Codespace first starts there is a delay until the ADO extension activates. If, as an example, the C# extension runs dotnet restore before this happens even if the shim is in place the authentication will not work.

Got it, will add this too.

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

It would be good to make a few test Codespaces using the universal, dotnet and node images and just compare the PATH to see if there is a consistent location we could use. What folder does dotnet, node, yarn typically get installed and where is that on default PATH etc.

There doesn't seem to be any consistency between them unfortunately.

universal:latest

docker run -it mcr.microsoft.com/devcontainers/universal:2 bash -c "echo \$PATH && which dotnet node yarn"

PATH

/home/codespace/.dotnet
/home/codespace/nvm/current/bin
/home/codespace/.php/current/bin
/home/codespace/.python/current/bin
/home/codespace/java/current/bin
/home/codespace/.ruby/current/bin
/home/codespace/.local/bin
/usr/local/python/current/bin
/usr/local/py-utils/bin
/usr/local/jupyter
/usr/local/oryx
/usr/local/go/bin
/go/bin
/usr/local/sdkman/bin
/usr/local/sdkman/candidates/java/current/bin
/usr/local/sdkman/candidates/gradle/current/bin
/usr/local/sdkman/candidates/maven/current/bin
/usr/local/sdkman/candidates/ant/current/bin
/usr/local/rvm/gems/default/bin
/usr/local/rvm/gems/default@global/bin
/usr/local/rvm/rubies/default/bin
/usr/local/share/rbenv/bin
/usr/local/php/current/bin
/opt/conda/bin
/usr/local/nvs
/usr/local/share/nvm/current/bin
/usr/local/hugo/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/share/dotnet
~/.dotnet/tools

Binaries

  • /home/codespace/.dotnet/dotnet
  • /home/codespace/nvm/current/bin/node
  • /usr/bin/yarn

dotnet

docker run -it mcr.microsoft.com/devcontainers/dotnet bash -c "echo \$PATH && which dotnet"

PATH

/usr/local/share/nvm/current/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/home/vscode/.dotnet
/home/vscode/.dotnet/tools

Binaries

  • /usr/bin/dotnet

javascript-node

docker run -it mcr.microsoft.com/devcontainers/javascript-node bash -c "echo \$PATH && which node yarn"

PATH

/usr/local/share/nvm/current/bin
/usr/local/share/npm-global/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin

Binaries

  • /usr/local/bin/node
  • /usr/local/bin/yarn

@markphip
Copy link
Contributor

markphip commented Dec 5, 2025

That is what I was afraid of. I think we also have to consider what PATH does VS Code extensions have and also before we use an existing folder we need to make sure that the tools themselves do not install in thos folders.

What if we made something like /usr/bin our default since it is not likely any of the tools would try to install in that folder and then maybe we can do something in our install.sh to try to move it to the top of PATH? Clearly these features are figuring out ways to add things to PATH so there must be a technique we can find that kind of works.

I would also suggest we just create our own folder in ~/.artifacts-helper and add it to front of PATH or something like that. But I worry that this folder will not be in PATH for VS Code extensions.

@markphip
Copy link
Contributor

markphip commented Dec 5, 2025

I assume you do not have any way to test this?

I propose you add a new feature folder called artifacts-testing and copy in whatever is needed for this to run. Then we can release that and use it to test and get this worked out before pushing version 3 of this feature.

We can keep artifacts-testing around for future testing. Maybe put something in the NOTES.md that it is for testing only and not to rely on it for production Codespaces?

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

I assume you do not have any way to test this?

I was going to try to setup a Docker container and some how mock calls to verify that the correct binaries are called (e.g. have a mocked dotnet file in ~/.dotnet/dotnet and assert that the shim calls it), but I'm not sure how this fits into the current testing framework.

If not, the alternative you mentioned would be more straightforward.

@markphip
Copy link
Contributor

markphip commented Dec 5, 2025

Well by test I meant actually use it and work out what is needed for it to solve the problems we want to solve.

But for the automated tests, I think a valid test would be to use a base image that has dotnet and/or node installed and just successfully run dotnet --version

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

That is what I was afraid of. I think we also have to consider what PATH does VS Code extensions have and also before we use an existing folder we need to make sure that the tools themselves do not install in thos folders.

My understanding regarding this is that the Docker ENV should be respected by all tools, including VS Code (not quite sure how Docker bakes it into the container, though). Assuming that is the case, as long as we can prepend our shims path late enough that nothing else interferes, it should be OK.

maybe we can do something in our install.sh to try to move it to the top of PATH?

Yeah, this is the tricky bit though as I've not figured out which file is used to determine PATH for all shells, including non-interactive. I thought it was /etc/environment but apparently that is a PAM feature, so not 100% sure if it's applicable.

I would also suggest we just create our own folder in ~/.artifacts-helper and add it to front of PATH or something like that. But I worry that this folder will not be in PATH for VS Code extensions.

I think the location should be flexible, as long as we can correctly modify PATH.

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

Well by test I meant actually use it and work out what is needed for it to solve the problems we want to solve.

Ah, right. I may be able to do this by building a custom image and then using it in my Codespace which is currently broken (aliases don't work in VS Code, so C# tests fail).

But for the automated tests, I think a valid test would be to use a base image that has dotnet and/or node installed and just successfully run dotnet --version

Yup, that makes sense! Some edge-cases (e.g. ensuring the exact base binary is the one called, not another further down in PATH, etc.) could be added, but probably overcomplicated for our purposes.

@abdurriq
Copy link
Author

abdurriq commented Dec 5, 2025

Ah, right. I may be able to do this by building a custom image and then using it in my Codespace which is currently broken (aliases don't work in VS Code, so C# tests fail).

Ran into a hurdle where the change seems to work (with the feature bundled in the repo's .devcontainers), but then the Codespace fails to build successfully with an apt not found issue (apparently somehow using Alpine Linux). I'll try to have another stab at this next week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

artifacts-helper alias does not work in shell scripts

3 participants