GitHub Actions runner benchmarks: 12 open-source builds on Sengi and GitHub’s runner
How long do open-source builds take on Sengi and on GitHub’s runner, and what do they cost?
Measured November 17, 2026 on 12 open-source projects, each built at a pinned commit with its own CI commands, on Sengi runners with the same 4 vCPU as GitHub’s runner: building all 12 once cost $0.094 on Sengi Basic 4 vCPU against $0.624 on GitHub-hosted 4 vCPU, and Sengi Basic 4 vCPU cost less per build on 12 of 12. Sengi Basic 4 vCPU took less time than GitHub’s runner on 9 of 12 projects and more time on 3 (Codex CLI, Mastodon and Guava). Sengi Premium 4 vCPU took less time than GitHub’s runner on 11 of 11 projects; Mastodon had too few runs to compare because 1 of 3 runs on Sengi Premium 4 vCPU failed. Times are median job seconds over at least 3 runs; these results cover these builds only. 3 jobs failed, all on Sengi runners, and none is counted in any median. Mastodon on Sengi Basic 2 vCPU and Mastodon on Sengi Premium 4 vCPU: Sengi's build cache served a truncated layer, a short read ending in unexpected EOF, so the Docker image build failed. TypeScript on Sengi Basic 2 vCPU: the compiler test suite failed on mocha test timeouts.
Measured November 17, 2026, last checked November 17, 2026. Every number on this page is in one JSON file and a CSV file, and every median links to its run log in the public repository sengirunners/benchmarks.
Results at 4 vCPU: GitHub’s runner, Sengi Basic 4 vCPU and Sengi Premium 4 vCPU
One row per project and runner. Time is the median job run time in seconds, from the job’s start to its end, over the runs counted; min and max are the fastest and slowest of those runs. Cost per build is the median of the runs’ billed minutes, each job rounded up to a whole minute, at the runner’s list price. A cell with fewer than 3 runs prints “not enough runs”.
| Project | Commit | Runner | Runs | Median time (s) | Min–max (s) | Cost per build (USD) | Time against GitHub | Cost against GitHub |
|---|---|---|---|---|---|---|---|---|
| Black | 87928e6 | GitHub-hosted 4 vCPU | 4 | 99 | 96–133 | $0.024 | baseline | baseline |
| Black | 87928e6 | Sengi Basic 4 vCPU | 3 | 69 | 67–90 | $0.004 | 30% less time | 83% less cost |
| Black | 87928e6 | Sengi Premium 4 vCPU | 3 | 69 | 53–82 | $0.012 | 30% less time | 50% less cost |
| GitHub CLI | fc4b137 | GitHub-hosted 4 vCPU | 4 | 184 | 170–198 | $0.042 | baseline | baseline |
| GitHub CLI | fc4b137 | Sengi Basic 4 vCPU | 3 | 163 | 161–185 | $0.006 | 11% less time | 86% less cost |
| GitHub CLI | fc4b137 | Sengi Premium 4 vCPU | 3 | 161 | 153–167 | $0.018 | 12% less time | 57% less cost |
| Codex CLI | 01fc69f | GitHub-hosted 4 vCPU | 4 | 368 | 347–415 | $0.084 | baseline | baseline |
| Codex CLI | 01fc69f | Sengi Basic 4 vCPU | 3 | 381 | 369–391 | $0.014 | 4% more time | 83% less cost |
| Codex CLI | 01fc69f | Sengi Premium 4 vCPU | 3 | 286 | 279–313 | $0.030 | 22% less time | 64% less cost |
| Mastodon | 3987b9f | GitHub-hosted 4 vCPU | 4 | 153 | 124–200 | $0.036 | baseline | baseline |
| Mastodon | 3987b9f | Sengi Basic 4 vCPU | 3 | 168 | 115–190 | $0.006 | 10% more time | 83% less cost |
| Mastodon | 3987b9f | Sengi Premium 4 vCPU | 2 (1 failed) | not enough runs | – | – | – | – |
| Flask | 22d9247 | GitHub-hosted 4 vCPU | 4 | 22 | 18–29 | $0.012 | baseline | baseline |
| Flask | 22d9247 | Sengi Basic 4 vCPU | 3 | 14 | 13–16 | $0.002 | 35% less time | 83% less cost |
| Flask | 22d9247 | Sengi Premium 4 vCPU | 3 | 14 | 13–16 | $0.006 | 35% less time | 50% less cost |
| Guava | 666338c | GitHub-hosted 4 vCPU | 4 | 735 | 730–740 | $0.156 | baseline | baseline |
| Guava | 666338c | Sengi Basic 4 vCPU | 3 | 759 | 754–772 | $0.026 | 3% more time | 83% less cost |
| Guava | 666338c | Sengi Premium 4 vCPU | 3 | 650 | 632–680 | $0.066 | 12% less time | 58% less cost |
| AI SDK | 45e1fbc | GitHub-hosted 4 vCPU | 4 | 184 | 181–190 | $0.048 | baseline | baseline |
| AI SDK | 45e1fbc | Sengi Basic 4 vCPU | 3 | 164 | 160–173 | $0.006 | 11% less time | 88% less cost |
| AI SDK | 45e1fbc | Sengi Premium 4 vCPU | 3 | 123 | 115–125 | $0.018 | 33% less time | 63% less cost |
| Pydantic | 001dea0 | GitHub-hosted 4 vCPU | 4 | 121 | 113–133 | $0.030 | baseline | baseline |
| Pydantic | 001dea0 | Sengi Basic 4 vCPU | 3 | 106 | 104–112 | $0.004 | 12% less time | 87% less cost |
| Pydantic | 001dea0 | Sengi Premium 4 vCPU | 3 | 89 | 83–99 | $0.012 | 26% less time | 60% less cost |
| TypeScript | 050880c | GitHub-hosted 4 vCPU | 4 | 409 | 372–543 | $0.090 | baseline | baseline |
| TypeScript | 050880c | Sengi Basic 4 vCPU | 3 | 394 | 367–425 | $0.014 | 4% less time | 84% less cost |
| TypeScript | 050880c | Sengi Premium 4 vCPU | 3 | 310 | 308–334 | $0.036 | 24% less time | 60% less cost |
| Vite | 39ddf7c | GitHub-hosted 4 vCPU | 4 | 259 | 254–266 | $0.060 | baseline | baseline |
| Vite | 39ddf7c | Sengi Basic 4 vCPU | 3 | 183 | 173–209 | $0.008 | 29% less time | 87% less cost |
| Vite | 39ddf7c | Sengi Premium 4 vCPU | 3 | 190 | 162–240 | $0.024 | 26% less time | 60% less cost |
| Vue | 5be58b4 | GitHub-hosted 4 vCPU | 4 | 62 | 48–68 | $0.018 | baseline | baseline |
| Vue | 5be58b4 | Sengi Basic 4 vCPU | 3 | 48 | 47–50 | $0.002 | 23% less time | 89% less cost |
| Vue | 5be58b4 | Sengi Premium 4 vCPU | 3 | 41 | 37–42 | $0.006 | 34% less time | 67% less cost |
| fd | 4f81778 | GitHub-hosted 4 vCPU | 4 | 70 | 63–72 | $0.024 | baseline | baseline |
| fd | 4f81778 | Sengi Basic 4 vCPU | 3 | 48 | 46–53 | $0.002 | 31% less time | 92% less cost |
| fd | 4f81778 | Sengi Premium 4 vCPU | 3 | 44 | 41–46 | $0.006 | 37% less time | 75% less cost |
Sengi Basic 4 vCPU took less time than GitHub’s runner on 9 of 12 projects and more time on 3 (Codex CLI, Mastodon and Guava); building all 12 once cost $0.094 on Sengi Basic 4 vCPU against $0.624 on GitHub-hosted 4 vCPU.
Sengi Premium 4 vCPU took less time than GitHub’s runner on 11 of 11 projects; Mastodon had too few runs to compare because 1 of 3 runs on Sengi Premium 4 vCPU failed; building all 11 once cost $0.234 on Sengi Premium 4 vCPU against $0.588 on GitHub-hosted 4 vCPU.
Median job time at 4 vCPU, by project
Not like-for-like: Sengi Basic 2 vCPU and Sengi Premium 2 vCPU
Sengi Basic 2 vCPU has half the cores of GitHub’s 4 vCPU runner. Sengi Premium 2 vCPU has half the cores of GitHub’s 4 vCPU runner. These Sengi runners were run on the same projects; they are not a like-for-like comparison, and no GitHub runner of their size was measured. The last two columns still set each against GitHub-hosted 4 vCPU.
| Project | Commit | Runner | Runs | Median time (s) | Min–max (s) | Cost per build (USD) | Time against GitHub | Cost against GitHub |
|---|---|---|---|---|---|---|---|---|
| Black | 87928e6 | Sengi Basic 2 vCPU | 4 | 117 | 108–123 | $0.002 | 18% more time | 92% less cost |
| Black | 87928e6 | Sengi Premium 2 vCPU | 4 | 77 | 73–91 | $0.006 | 22% less time | 75% less cost |
| GitHub CLI | fc4b137 | Sengi Basic 2 vCPU | 4 | 349 | 340–455 | $0.006 | 90% more time | 86% less cost |
| GitHub CLI | fc4b137 | Sengi Premium 2 vCPU | 4 | 311 | 305–329 | $0.018 | 69% more time | 57% less cost |
| Codex CLI | 01fc69f | Sengi Basic 2 vCPU | 4 | 764 | 604–2167 | $0.014 | 108% more time | 84% less cost |
| Codex CLI | 01fc69f | Sengi Premium 2 vCPU | 4 | 661 | 540–777 | $0.035 | 80% more time | 59% less cost |
| Mastodon | 3987b9f | Sengi Basic 2 vCPU | 3 (1 failed) | 208 | 138–1643 | $0.004 | 36% more time | 89% less cost |
| Mastodon | 3987b9f | Sengi Premium 2 vCPU | 4 | 129 | 127–159 | $0.009 | 16% less time | 75% less cost |
| Flask | 22d9247 | Sengi Basic 2 vCPU | 4 | 19 | 16–21 | $0.001 | 14% less time | 92% less cost |
| Flask | 22d9247 | Sengi Premium 2 vCPU | 4 | 13 | 9–15 | $0.003 | 42% less time | 75% less cost |
| Guava | 666338c | Sengi Basic 2 vCPU | 4 | 971 | 900–2185 | $0.017 | 32% more time | 89% less cost |
| Guava | 666338c | Sengi Premium 2 vCPU | 4 | 813 | 695–941 | $0.042 | 11% more time | 73% less cost |
| AI SDK | 45e1fbc | Sengi Basic 2 vCPU | 4 | 289 | 227–333 | $0.006 | 57% more time | 89% less cost |
| AI SDK | 45e1fbc | Sengi Premium 2 vCPU | 4 | 210 | 184–304 | $0.012 | 14% more time | 75% less cost |
| Pydantic | 001dea0 | Sengi Basic 2 vCPU | 4 | 133 | 118–150 | $0.003 | 10% more time | 90% less cost |
| Pydantic | 001dea0 | Sengi Premium 2 vCPU | 4 | 107 | 94–117 | $0.006 | 12% less time | 80% less cost |
| TypeScript | 050880c | Sengi Basic 2 vCPU | 3 (1 failed) | 583 | 494–701 | $0.010 | 43% more time | 89% less cost |
| TypeScript | 050880c | Sengi Premium 2 vCPU | 4 | 565 | 412–679 | $0.030 | 38% more time | 67% less cost |
| Vite | 39ddf7c | Sengi Basic 2 vCPU | 4 | 459 | 444–500 | $0.008 | 78% more time | 87% less cost |
| Vite | 39ddf7c | Sengi Premium 2 vCPU | 4 | 401 | 368–586 | $0.023 | 55% more time | 63% less cost |
| Vue | 5be58b4 | Sengi Basic 2 vCPU | 4 | 89 | 77–128 | $0.002 | 43% more time | 89% less cost |
| Vue | 5be58b4 | Sengi Premium 2 vCPU | 4 | 68 | 63–74 | $0.006 | 9% more time | 67% less cost |
| fd | 4f81778 | Sengi Basic 2 vCPU | 4 | 70 | 59–100 | $0.002 | the same time | 92% less cost |
| fd | 4f81778 | Sengi Premium 2 vCPU | 4 | 56 | 49–62 | $0.003 | 21% less time | 88% less cost |
Where did Sengi take more time?
Every result where a Sengi runner’s median was above GitHub’s is printed in the tables above and listed here.
At 4 vCPU, the same size as GitHub’s runner
- Codex CLI, Sengi Basic 4 vCPU: 381 s against 368 s on GitHub’s runner, 4% more time, at $0.014 a build against $0.084.
- Mastodon, Sengi Basic 4 vCPU: 168 s against 153 s on GitHub’s runner, 10% more time, at $0.006 a build against $0.036.
- Guava, Sengi Basic 4 vCPU: 759 s against 735 s on GitHub’s runner, 3% more time, at $0.026 a build against $0.156.
Not like-for-like: smaller or larger Sengi runners against GitHub’s 4 vCPU runner
- Black, Sengi Basic 2 vCPU: 117 s against 99 s on GitHub’s runner, 18% more time, at $0.002 a build against $0.024.
- GitHub CLI, Sengi Basic 2 vCPU: 349 s against 184 s on GitHub’s runner, 90% more time, at $0.006 a build against $0.042.
- Codex CLI, Sengi Basic 2 vCPU: 764 s against 368 s on GitHub’s runner, 108% more time, at $0.014 a build against $0.084.
- Mastodon, Sengi Basic 2 vCPU: 208 s against 153 s on GitHub’s runner, 36% more time, at $0.004 a build against $0.036.
- Guava, Sengi Basic 2 vCPU: 971 s against 735 s on GitHub’s runner, 32% more time, at $0.017 a build against $0.156.
- AI SDK, Sengi Basic 2 vCPU: 289 s against 184 s on GitHub’s runner, 57% more time, at $0.006 a build against $0.048.
- Pydantic, Sengi Basic 2 vCPU: 133 s against 121 s on GitHub’s runner, 10% more time, at $0.003 a build against $0.030.
- TypeScript, Sengi Basic 2 vCPU: 583 s against 409 s on GitHub’s runner, 43% more time, at $0.010 a build against $0.090.
- Vite, Sengi Basic 2 vCPU: 459 s against 259 s on GitHub’s runner, 78% more time, at $0.008 a build against $0.060.
- Vue, Sengi Basic 2 vCPU: 89 s against 62 s on GitHub’s runner, 43% more time, at $0.002 a build against $0.018.
- GitHub CLI, Sengi Premium 2 vCPU: 311 s against 184 s on GitHub’s runner, 69% more time, at $0.018 a build against $0.042.
- Codex CLI, Sengi Premium 2 vCPU: 661 s against 368 s on GitHub’s runner, 80% more time, at $0.035 a build against $0.084.
- Guava, Sengi Premium 2 vCPU: 813 s against 735 s on GitHub’s runner, 11% more time, at $0.042 a build against $0.156.
- AI SDK, Sengi Premium 2 vCPU: 210 s against 184 s on GitHub’s runner, 14% more time, at $0.012 a build against $0.048.
- TypeScript, Sengi Premium 2 vCPU: 565 s against 409 s on GitHub’s runner, 38% more time, at $0.030 a build against $0.090.
- Vite, Sengi Premium 2 vCPU: 401 s against 259 s on GitHub’s runner, 55% more time, at $0.023 a build against $0.060.
- Vue, Sengi Premium 2 vCPU: 68 s against 62 s on GitHub’s runner, 9% more time, at $0.006 a build against $0.018.
How was it measured?
- Time: Job run time, start to end, from the GitHub API. Queue time is kept apart.
- Runs: Median of the measured runs, with n, min and max; a median needs n >= 3.
- Cost: Billed minutes (each job rounded up to a whole minute) times the runner's list price per minute.
- GitHub cost: GitHub does not bill public repositories for this runner; its cost is what a private repository pays at GitHub's listed rate for the measured size.
- Order: One runner class at a time; at most four jobs at once within a run. One warm-up run per class first fills dependency caches.
- Selection: The project list is the whole workload list of an existing open-source CI benchmark suite, adopted before running it here. Each project is pinned to a release commit.
- Exclusions: Every run is listed. A job is left out of the medians only for the reason given next to it.
- Dates: 2026-11-17 to 2026-11-17.
- Runner size: the vCPU count printed for each runner was read with
nprocinside the job, not taken from a label. - Prices: GitHub-hosted 4 vCPU $0.012, Sengi Basic 2 vCPU $0.001, Sengi Premium 2 vCPU $0.003, Sengi Basic 4 vCPU $0.002 and Sengi Premium 4 vCPU $0.006 a minute. GitHub does not bill public repositories for its standard runner, which is 4 vCPU there. The GitHub column is priced at GitHub’s listed rate for its 4 vCPU Linux larger runner (sold to Team and Enterprise Cloud organizations), $0.012 a minute: what a private repository pays for these minutes beyond its plan’s included usage.
- Queue time is recorded for every run in the data files and is not part of the times above.
Which projects, at which commits?
The project list is the workload list of an existing open-source CI benchmark suite. We adopted the whole list before running it here and did not choose the projects. Each is pinned to a release commit. Every project we ran is shown, including the ones where Sengi took more time. A project is left out only when its build fails on GitHub’s runner as well, and this page lists it with the reason.
| Project | Repository | Language | What the job runs | Commit |
|---|---|---|---|---|
| Black | psf/black | Python | Python 3.13 test suite (tox) | 87928e6d6761a4a6d22250e1fee5601b3998086e |
| GitHub CLI | cli/cli | Go | Unit and integration tests with the race detector, then a build | fc4b137cdef0a6bd28fd461b7cf9c84a5812a8cd |
| Codex CLI | openai/codex | Rust | cargo clippy over the Rust workspace, tests included | 01fc69f4026735edfdf6789820549727a4867b11 |
| Mastodon | mastodon/mastodon | Docker | Docker image build (linux/amd64) after a one-file change, layer cache on | 3987b9fd624010eeeaeb7cad96606b05aa813f6a |
| Flask | pallets/flask | Python | Python 3.13 test suite (tox) | 22d924701a6ae2e4cd01e9a15bbaf3946094af65 |
| Guava | google/guava | Java | Maven install and verify on JDK 25 | 666338cc49d5686c07e3e4d17f0aa1cec25a2be6 |
| AI SDK | vercel/ai | TypeScript | Turborepo build of three Next.js examples | 45e1fbc3ea10d22f852e12b1c1851e289c85e835 |
| Pydantic | pydantic/pydantic | Python | pytest suite on Python 3.13 | 001dea020e0809844e5b17666432c9135a976f46 |
| TypeScript | microsoft/TypeScript | TypeScript | Compiler test suite on Node 24 | 050880ce59e30b356b686bd3144efe24f875ebc8 |
| Vite | vitejs/vite | TypeScript | Build, unit tests and Playwright end-to-end tests on Node 24 | 39ddf7ccf7e7469ff6a3ba37bca38c32ea804d6e |
| Vue | vuejs/core | TypeScript | Unit tests (Vitest) | 5be58b4c475c1d14b4abacbfeda610394a0ee4e5 |
| fd | sharkdp/fd | Rust | cargo build --release, then cargo test | 4f81778774463bf414a184cbe6d5219ad2229646 |
The benchmark workflows are at commit e73c896795525a61b47ce9f44657a1a045b0a98b of sengirunners/benchmarks. Project names identify the code that was built; none of the projects is affiliated with Sengi.
Is there a page for each project?
Yes. Each project has its own page with the same numbers for that project alone: its results, where Sengi took more time, any failed job and its run logs.
- Black CI build time and cost
- GitHub CLI CI build time and cost
- Codex CLI CI build time and cost
- Mastodon CI build time and cost
- Flask CI build time and cost
- Guava CI build time and cost
- AI SDK CI build time and cost
- Pydantic CI build time and cost
- TypeScript CI build time and cost
- Vite CI build time and cost
- Vue CI build time and cost
- fd CI build time and cost
What was left out?
3 jobs failed, all on Sengi runners, and none is counted in any median. Mastodon on Sengi Basic 2 vCPU and Mastodon on Sengi Premium 4 vCPU: Sengi's build cache served a truncated layer, a short read ending in unexpected EOF, so the Docker image build failed. TypeScript on Sengi Basic 2 vCPU: the compiler test suite failed on mocha test timeouts. Each failed job is also counted in its row’s failures.
- Run 44030937414: warm-up run: fills each runner's dependency caches first; not measured
- Run 44033028003: warm-up run: fills each runner's dependency caches first; not measured
- Run 44037191239: warm-up run: fills each runner's dependency caches first; not measured
- Run 44039576472: warm-up run: fills each runner's dependency caches first; not measured
- Run 44041540026: warm-up run: fills each runner's dependency caches first; not measured
- Job 133910810755: the compiler test suite failed on mocha test timeouts
- Job 134032483827: Sengi's build cache served a truncated layer, a short read ending in unexpected EOF, so the Docker image build failed
- Job 134048517537: Sengi's build cache served a truncated layer, a short read ending in unexpected EOF, so the Docker image build failed
Where is the data?
/data/github-actions-benchmarks.json: runners, projects and commits, the summary rows, exclusions and every run with its log URL./data/github-actions-benchmarks.csv: one line per project and runner, with the project commit, the cost per build and the median run’s log URL.
Prices per minute for every runner are on Sengi pricing and in the runner comparison.
Frequently asked questions
Do Sengi runners finish open-source builds in less time than GitHub’s runner?
On these builds, at these commits: Sengi Basic 4 vCPU took less time than GitHub’s runner on 9 of 12 projects and more time on 3 (Codex CLI, Mastodon and Guava); Sengi Premium 4 vCPU took less time than GitHub’s runner on 11 of 11 projects; Mastodon had too few runs to compare because 1 of 3 runs on Sengi Premium 4 vCPU failed. Times are median job seconds over at least 3 runs at 4 vCPU, measured November 17, 2026.
What does a build cost on Sengi Basic compared with GitHub’s runner?
Measured November 17, 2026, building all 12 once cost $0.094 on Sengi Basic 4 vCPU against $0.624 on GitHub-hosted 4 vCPU. Cost is billed minutes, each job rounded up to a whole minute, at the list price: $0.002 a minute for Sengi Basic 4 vCPU and $0.012 for a 4 vCPU GitHub runner.
Why does the comparison use 4 vCPU?
The benchmark runs in a public repository so that every run log is public, and GitHub’s ubuntu-24.04 runner measured 4 vCPU there (nproc inside each job). The Sengi runners with the same vCPU count are compared with it directly; Sengi Basic 2 vCPU has half the cores of GitHub’s 4 vCPU runner, and Sengi Premium 2 vCPU has half the cores of GitHub’s 4 vCPU runner, so they are shown separately and not as a like-for-like comparison.
Why is GitHub’s runner priced if public repositories are not billed for it?
GitHub does not bill public repositories for its standard runner, which is 4 vCPU there. The GitHub column is priced at GitHub’s listed rate for its 4 vCPU Linux larger runner (sold to Team and Enterprise Cloud organizations), $0.012 a minute: what a private repository pays for these minutes beyond its plan’s included usage.
How were the projects chosen?
The project list is the workload list of an existing open-source CI benchmark suite, adopted whole before any of it was run here. Each project is pinned to a release commit. Every project run is shown, including the ones where Sengi took more time.
Where Sengi took more time, is that shown?
Yes. At 4 vCPU: Codex CLI on Sengi Basic 4 vCPU, 381 s against 368 s; Mastodon on Sengi Basic 4 vCPU, 168 s against 153 s; Guava on Sengi Basic 4 vCPU, 759 s against 735 s. 17 more results on smaller or larger Sengi runners, not like-for-like with GitHub’s 4 vCPU runner, took more time than it; all are listed on this page.
Can I check the numbers?
Yes. Every printed median links to the log of the run nearest that median in the public repository sengirunners/benchmarks, at commit e73c896. Every run, failed ones included, is in the JSON file on this page, and the CSV file has one line per project and runner.
Keep your workflow. Cut your runner rate.
Sengi Basic starts at $0.001 a minute for 2 vCPU. Choose 4 or 8 vCPU for $0.002 or $0.004 a minute. Premium stays optional at $0.003, $0.006 and $0.012.
Start without a card ↗︎