Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an article about faster CI #23

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

MahdiBM
Copy link

@MahdiBM MahdiBM commented Jan 4, 2025

In this article, you'll walk through optimizing Vapor's Penny Bot CI times to go from 10 minutes in tests and 14 minutes 30 seconds in deployments, down to less than 4 minutes for tests CI, and 3 minutes for deployments. The bigger your project is, the bigger the gap will be.

Checklist

  • Finish writing the article.
  • More images.
  • Investigate using code snippets? Do they work with yaml? (nope, thought so, wouldn't make sense)
  • Show Diff in code changes so readers can see what changed since last step?
  • Re-read the article a few times, iron out errors, finalize the main content.
  • Add RunsOn section(s)


So in the "Cache .build" step, you're caching your `.build` directory with the same key as the one in the "Restore .build" step.

With the `if` condition, you're avoiding spending time in the cache step when you already found an **exact match** for the cache key in the "Restore .build" step. This is because `actions/cache` rejects cache entries with duplicate names, and does not have an update mechanism.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is really needed? Couldn't you just use actions/cache@v4 in the first step and depend on the default behaviour? I believe it works like you would expect (i.e. save the new .build if an exact match hasn't been found)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is really needed?

What exactly are you referring to by "this"?

By "first step" you mean "Restore .build"? what would it help to use "actions/cache" or "actions/cache/restore"? I assumed those do the same thing, while one being more explicit.


![Delete Cache In GitHub UI](delete-caches-in-github-ui.png)

## RunsOn machines - runson/cache (If sponsored)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

v2.6.3 now ships with magic caching, might be worth looking!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've already looked at it, I'm not sure it changes things too much 🙂

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, i think this thing means we won't need to use runs-on/cache anymore? just set up the magic cache?
That's slightly nicer than using runs-on/cache, and I'll happily take it.

While there are a lot of similarities in implementing caching in tests CI and deployment CI, there are still some differences.
The project build in the deployment CI usually happens in a Dockerfile. While people have found ways around this, it's not easy to use `actions/cache` for files and folders that are in a Dockerfile.

To make your Dockerfile cache-friendly, you need to pull out the build step out of the Dockerfile and move it to the deployment CI GitHub Actions file.
Copy link

@crohr crohr Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also could use buildx gha caching to cache Docker layers? Although cannot reuse the same .build cache as for tests so might be why you're not mentioning it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't used Docker layer caching too much so let me know if I'm wrong.

With .build caching, we're not totally caching everything that a new build would need.
So a new build will use some amount of artifacts from .build for e.g. files and modules that are already built.
But swift will rebuild stuff that do need a rebuild, and the final executable will be a slightly different executable than the previous cache/build.

In Docker, if you need to update a layer, you simply need to do it totally from the beginning.

We probably could use Docker layer caching here for something other than the .build directory, but that will only save like 10s, which is nice, but not that noticeable compared to the other stuff. It might also require some changes in the referenced Dockerfiles which makes things harder to adopt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants