Skip to content

Self hosted runners running out of disk space #244

Open

Description

@mdboom

The pystats self-hosted runner ran out of disk space and therefore stopped fulfilling requests. While this issue is most acute on that runner, since it's a VM with only 30GB of disk space, these issues probably apply equally to all self-hosted runners, so we should have a solution that applies to both.

Here are some sources of disk space usage that should be addressed:

  1. The .git directory for the benchmarking repo. We keep the repo size down by periodically manually deleting old history. However, since the checkout is retained on the runners, this doesn't actually remove the old content. This was initially 12GB, and after running git gc it went to 4GB. This can be automated.

  2. There are multiple versions of the dependencies for Github Actions installed, left behind when GHA self-updates. These have the names externalsX.YY and are roughly .5GB each. There is an issue for this that claims a fix has been merged, but nonetheless these dirs seem to remain.

  3. The pip cache is about 1GB. Unfortunately, I don't think it's possible to control the amount of space used for the pip cache. The best we can probably do is periodically purge it.

Activity

  1. self-assigned this
    on Aug 1, 2024
  2. mdboom commented on Sep 6, 2024

    @mdboom
    ContributorAuthor

    Step 1. has been implemented.

    Step 2. requires an upstream fix from Github, IMHO.

    Step 3. is still TBD.

  3. chris-eibl commented on Dec 30, 2024

    @chris-eibl
    Member

    Regarding step3: until limiting the cache pypa/pip#3138 is possible, pip cache purge or pip cache remove <pattern> seem to be the only options.

    Options --editable and --no-cache-dir are sometimes mentioned to workaround too big caches, but I think not applicable for pyperformance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions