Self-hosted CI runners can give teams more control over build capacity, network access, tooling, and costs, but they also become part of the production security surface. This checklist provides a repeatable maintenance workflow: inventory every runner, limit its permissions, isolate untrusted jobs, clean workspaces and caches, monitor failure signals, and retire machines safely. Use it during monthly or quarterly reviews, before onboarding repositories, and whenever a runner image, CI provider, or deployment architecture changes.
Overview
A self-hosted CI runner is a machine, virtual machine, container, or other execution environment that receives jobs from a CI/CD platform and runs build, test, packaging, or deployment commands. A GitHub Actions self-hosted runner is one example, but the same operational concerns apply to other self-hosted build agents.
The central distinction is between the convenience of a persistent runner and the isolation of an ephemeral runner. A persistent runner keeps its operating system, tools, workspaces, and caches between jobs. That can reduce setup time, but it also increases the chance that one job affects another. An ephemeral runner is created for a limited job or workload and then discarded or reset. It generally offers a cleaner boundary, although it requires reliable image creation, provisioning, cleanup, and capacity management.
| Architecture | Useful when | Primary maintenance concern |
|---|---|---|
| Persistent host | Trusted repositories need stable tools or specialized hardware | Workspace contamination, drift, and long-lived credentials |
| Ephemeral virtual machine | Jobs need stronger separation or varied operating-system images | Image patching, provisioning reliability, and startup time |
| Ephemeral container | Builds can run with a repeatable container image | Container escape risk, host access, and image provenance |
| Autoscaled pool | Demand varies across projects or release periods | Capacity limits, scale-down cleanup, and orphaned resources |
No executor is automatically safe. A secure build runner depends on the job trust model, operating-system controls, network policy, secrets configuration, and the permissions granted by the CI platform. Treat runner administration as an ongoing operating process rather than a one-time installation task.
What to track
1. Maintain a runner inventory
Keep one authoritative record for every runner or runner pool. At minimum, record:
- Runner name, unique identifier, owner, environment, and physical or cloud location.
- Operating system, architecture, image or snapshot version, kernel or base-image revision, and installed toolchain.
- Executor type, labels or capabilities, assigned repositories, and permitted workflows.
- Network zones, outbound destinations, inbound exposure, and administrative access paths.
- Registration date, last successful job, last patch date, image rebuild date, and planned replacement date.
- Whether the runner is persistent, ephemeral, autoscaled, dedicated, or shared.
Inventory drift is itself a useful signal. A runner that cannot be mapped to an owner or repository should be disabled for investigation rather than left in service.
2. Review permissions and trust boundaries
Use the smallest practical scope for runner registration, repository access, cloud roles, package publishing, and deployment credentials. Separate runners by trust level when jobs come from different sources. In particular, do not place unreviewed pull-request code and privileged deployment jobs on the same persistent host.
Review these questions for each runner:
- Can a job access the host filesystem, container socket, metadata service, or administrator interface?
- Can the runner reach databases, production services, internal source repositories, or signing systems?
- Are credentials short-lived and scoped to one workflow, environment, or deployment action?
- Can contributors modify workflow files that execute on this runner?
- Does the CI platform still list the runner as active, authorized, and assigned to the intended repositories?
For related controls around environment variables and credentials, use the CI/CD Environment and Secrets Management Guide for Small Teams.
3. Check isolation, workspaces, and caches
A persistent runner should never assume that a previous job left the workspace in a safe state. At the end of every job, remove generated files, temporary credentials, package artifacts, test databases, and source directories that are not required by an explicitly controlled cache.
Audit caches for both security and correctness. Cache only dependency data or build outputs that are safe to reuse, use keys that include relevant operating-system and toolchain inputs, and prevent untrusted jobs from writing data consumed by privileged workflows. When in doubt, invalidate the cache and rebuild it from a trusted job.
Containers can improve repeatability, but a container is not a complete isolation strategy if the job has access to a host-level container socket, broad Linux capabilities, sensitive mounted paths, or unrestricted network access. Document the boundary you are relying on and test it as part of runner maintenance.
4. Track patching and configuration drift
Record the last successful operating-system patch, runner agent update, base-image rebuild, toolchain update, and security configuration review. Define an owner and a target remediation window for missing updates. If a runner cannot be patched without manual intervention, treat that dependency as an operational risk and plan replacement or reconstruction.
Prefer immutable images or scripted provisioning where practical. Store runner configuration as code, keep image build definitions under review, and make it possible to recreate a runner without copying unknown state from an old machine.
5. Monitor health and capacity
Monitor more than whether a runner is online. Useful signals include queued-job duration, job-start failures, execution time by runner, disk and memory pressure, CPU saturation, network errors, repeated tool installation failures, unexpected reboots, registration changes, and orphaned autoscaling resources.
Compare these signals with pipeline results. A rise in failed builds may come from an application change, a dependency update, a corrupted cache, a runner image drift, or a capacity bottleneck. The CI/CD Pipeline Troubleshooting guide can complement this runner-focused review.
Cadence and checkpoints
A practical schedule separates frequent checks from deeper reviews:
| Cadence | Checkpoint | Expected action |
|---|---|---|
| Every job or daily | Workspace cleanup, job result, disk pressure, and registration status | Remove residue, investigate abnormal failures, and quarantine unhealthy runners |
| Monthly | Inventory, permissions, labels, network access, secrets, and cache behavior | Reconcile records, remove unnecessary access, and rotate or invalidate unsafe state |
| Quarterly | Operating-system support, runner agent behavior, isolation design, image age, and capacity | Rebuild images, test replacement, review architecture, and update documentation |
| Before a release | Deployment credentials, production reachability, rollback path, and available capacity | Confirm only intended runners can deploy and that a replacement path exists |
| After an incident | Logs, workspace state, credentials, network traffic, and job provenance | Preserve evidence, revoke exposed access, rebuild affected runners, and document lessons |
Use a short maintenance record for each review: reviewer, date, runners checked, exceptions, remediation owner, deadline, and evidence. This turns a checklist into an auditable process without requiring a large operations team.
How to interpret changes
Trend changes rather than reacting to one isolated job. A gradual increase in queue time can indicate insufficient capacity or an incorrect runner label. A sudden increase in build duration may indicate image drift, disk pressure, slower dependency downloads, or a cache problem. Repeated failures on one host suggest local corruption or hardware instability; the safest first response is often to drain, inspect, and replace the runner instead of repeatedly retrying jobs.
Different failure patterns require different responses:
- Only untrusted or pull-request jobs fail: review checkout behavior, dependency access, network restrictions, and cache permissions.
- Only deployment jobs fail: verify environment credentials, runner labels, approval controls, and access to the target environment.
- Several repositories fail after an image change: compare tool versions, system libraries, environment variables, and executor configuration with the previous image.
- Jobs pass after retries: investigate capacity, network reliability, resource exhaustion, and cleanup rather than treating retries as the fix.
- An unknown runner appears: disable or isolate it, review registration credentials and audit logs, and confirm repository assignments before restoring service.
When a runner may have processed sensitive code or credentials, assume its workspace and local secrets could be exposed. Revoke relevant credentials, preserve appropriate logs, rebuild from a trusted image, and document the decision. Avoid returning a potentially compromised persistent host to service merely because a health check passes.
When to revisit
Run the full checklist monthly for small, changing environments and at least quarterly for stable environments. Revisit it immediately after a CI provider release that changes executor behavior, an operating-system or base-image change, a new repository onboarding, a security audit, a capacity increase, an unexplained build failure, or any suspected credential exposure.
Use this decommissioning runbook when a runner is replaced:
- Drain the runner and prevent new jobs from being assigned to it.
- Confirm that queued workflows have another approved execution path.
- Remove repository assignments, labels, registration tokens, service accounts, and deployment permissions.
- Revoke or rotate credentials that were present on the host, including cloud, package, signing, and SSH access.
- Preserve required logs and inventory evidence, then remove workspaces, caches, snapshots, disks, and autoscaling records according to your retention requirements.
- Update monitoring, documentation, infrastructure code, and ownership records.
- Run a test build on the replacement and verify that it has only the intended network and repository access.
Finally, link runner maintenance to the wider delivery workflow. Check environment differences with the Cloud Deployment Environment Parity Checklist, review repository controls with the Open-Source Repository Security Checklist, and include runner recovery in your repository backup and disaster recovery plan. A runner is ready for continued use when its owner, permissions, image, isolation boundary, monitoring, and replacement path are all documented and testable.