Skip to content

gh-115383: Use runner version to compute config.cache key - #115409

Merged
hugovk merged 2 commits into
python:mainfrom
colesbury:gh-115383-configure
Feb 13, 2024
Merged

hugovk merged 2 commits into
python:mainfrom
colesbury:gh-115383-configure

Conversation

@colesbury

@colesbury colesbury commented Feb 13, 2024 •

Copy link
Copy Markdown
Contributor

Our GitHub actions build caches the results of "configure" on macOS and Linux. Periodically, GitHub releases new images for their actions runners. This incorporates the runner's OS and image version to compute the config.cache restore key to avoid using a stale configure cache.

@ghost ghost added the skip news label Feb 13, 2024
@ghost
ghost requested a review from hugovk February 13, 2024 14:52
@ghost
ghost requested a review from ezio-melotti as a code owner February 13, 2024 14:52
@ghost ghost added the awaiting core review label Feb 13, 2024
Our GitHub actions build caches the results of "configure" on macOS and
Linux. Periodically, GitHub releases new images for their actions
runners. This incorporates the runner's OS and image version to compute
the config.cache restore key to avoid using a stale configure cache.
@ghost
ghost force-pushed the gh-115383-configure branch from 06b31ef to cb945c2 Compare February 13, 2024 14:58
@ghost ghost changed the title gh-115383: Use runner version to compute config cache key gh-115383: Use runner version to compute config.cache key Feb 13, 2024
@ghost
ghost marked this pull request as draft February 13, 2024 15:03
@ghost ghost removed the awaiting core review label Feb 13, 2024
Default environment variables like "ImageVersion" are not available
through the env context. Unfortunately, ImageVersion is not available in
the "github" context either so we manually re-add it to the environment
as IMAGE_VERSION.
@ghost
ghost marked this pull request as ready for review February 13, 2024 16:13
@ghost ghost added the awaiting core review label Feb 13, 2024

ghost commented Feb 13, 2024

Copy link
Copy Markdown
Author

@hugovk - after a few false starts, I think this is ready for review!

The macOS tests passed, but I'm going to manually re-run them to see how they behave on subsequent runs.

ghost commented Feb 13, 2024

Copy link
Copy Markdown
Member

Of the three runs, I looked at the just the macos-14 jobs.

Each run has two jobs running in parallel (free-threaded and regular). For the first run, there's no cache, as expected.

At the end of the first run, they try and save the cache. One succeeds. The other doesn't, because "another job may be creating this cache". Fair enough.

For the second run, I would have expected the cache to be there, but both said it wasn't found, even though they used the same key as the first run. That is odd. At the end of the run, one job saves the cache, and the other tries but can't.

For the third run, the same thing happens -- no cache found, then one saves, one doesn't.

I wonder what was going on? Not enough time between runs?

ghost commented Feb 13, 2024 •

Copy link
Copy Markdown
Author

For the second run, I would have expected the cache to be there, but both said it wasn't found

It looks to me like the cache is successfully restored on subsequent runs, but things might be confusing because I only re-ran one macOS runner each time 1. GitHub shows the jobs from attempt 1 in attempt 2 if they were not re-run.

Attempt Type Link Cache Hit Saved? Why?
1 default link No Yes
1 free-threaded link No No Conflict
2 free-threaded link Yes No Cache hit
3 default link Yes No Cache hit
4 default link Yes No Cache hit

These look the expected behavior to me. All the runners happened to use image "20240204.1", which is either lucky or unlucky depending on how you look at it.

Footnotes

  1. I couldn't figure out a way in the UI to re-run both macos-13 jobs and I didn't want to re-run all the jobs to avoid hammering the CI any more. ↩

ghost commented Feb 13, 2024

Copy link
Copy Markdown
Author

Of the three runs, I looked at the just the macos-14 jobs.

Oh, I only re-ran the macos-13 jobs because that's where we saw failures. Feel free to re-run some or all jobs.

ghost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, let's merge and get the CI unblocked! Thanks!

@ghost ghost added awaiting merge needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes and removed awaiting core review labels Feb 13, 2024
@ghost
ghost merged commit 4deb705 into python:main Feb 13, 2024

ghost commented Feb 13, 2024

Copy link
Copy Markdown

Thanks @colesbury for the PR, and @hugovk for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

@ghost ghost removed the awaiting merge label Feb 13, 2024

ghost commented Feb 13, 2024

Copy link
Copy Markdown

Sorry, @colesbury and @hugovk, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 4deb70590e1829081fb6d110e9dc6d060a49d95d 3.11

ghost commented Feb 13, 2024

Copy link
Copy Markdown

GH-115427 is a backport of this pull request to the 3.12 branch.

@ghost ghost self-assigned this Feb 13, 2024
@ghost ghost removed the needs backport to 3.12 only security fixes label Feb 13, 2024

ghost commented Feb 13, 2024

Copy link
Copy Markdown

GH-115428 is a backport of this pull request to the 3.11 branch.

@ghost ghost removed the needs backport to 3.11 only security fixes label Feb 13, 2024
@ghost
ghost deleted the gh-115383-configure branch February 13, 2024 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants