A failed CI/CD pipeline is easier to fix when you diagnose it in a consistent order. This guide provides a practical workflow for separating source, dependency, test, runner, secret, artifact, container, and deployment problems—then tracking recurring signals so the next failure is faster to resolve.
Overview
CI/CD pipeline troubleshooting should begin with evidence, not assumptions. A red pipeline may indicate a real code defect, but it can also result from an expired credential, a changed dependency, an unavailable service, a mismatched runtime, or a deployment that completed but failed its health check.
Start by identifying the first failing step. Later failures are often consequences rather than causes. Record the commit, branch or tag, workflow run, runner type, target environment, and approximate failure time. Compare the failed run with the most recent successful run. This creates a small boundary around the change that needs investigation.
Use this decision sequence:
- Did the job start? If not, inspect workflow triggers, branch filters, runner capacity, permissions, and configuration syntax.
- Did the source checkout succeed? Check repository access, ref names, submodules, and checkout credentials.
- Did dependency installation succeed? Check lockfiles, package registries, network access, cache state, and runtime versions.
- Did compilation or tests fail? Reproduce the exact command locally or in a matching container, then isolate the first failing test or compiler message.
- Did packaging or artifact upload fail? Verify output paths, file permissions, artifact size limits, and retention settings.
- Did deployment fail or did the deployed service fail? Separate authentication and infrastructure errors from application startup, routing, configuration, and health-check failures.
Keep troubleshooting changes separate from product changes where possible. A temporary diagnostic step can be useful, but it should be removed or documented after the incident. Teams using repository hosting with CI/CD can also keep a short troubleshooting runbook beside the pipeline definition so maintainers do not have to reconstruct the process during an outage.
What to track
A useful pipeline record tracks recurring variables, not just whether a run passed. The following checklist gives maintainers and engineering teams enough context to identify patterns.
Source and workflow configuration
- Commit, pull request, tag, or release identifier.
- Workflow file version and the job or stage that failed.
- Changed build scripts, lockfiles, container files, deployment manifests, and environment configuration.
- Trigger type, branch rules, matrix parameters, and any conditional steps.
A workflow that never starts may have a trigger or permissions problem rather than a build problem. A workflow that starts only for some branches may have an overly narrow path or branch filter.
Dependencies and runtime
For dependency resolution failures, capture the package manager, runtime version, lockfile status, registry URL, and exact error. Do not immediately delete the lockfile or disable verification. First determine whether the lockfile is stale, the requested package is unavailable, credentials are missing, or the runner cannot reach the registry.
Cache behavior deserves its own observation. A stale cache can preserve a broken dependency or hide a reproducibility problem. Compare a cached run with a clean run, and use a cache key that changes when relevant lockfiles or runtime versions change. For a deeper maintenance process, see the open-source dependency update workflow.
Tests and build output
Record the first failing test, test selection, exit code, compiler or bundler version, and whether the failure is deterministic. Distinguish assertion failures from timeouts, port conflicts, missing fixtures, filesystem assumptions, and tests that depend on ordering or system time.
When a build fails, preserve the command that ran and the relevant output. A shortened log may hide the initial error. Redact tokens, passwords, private keys, and sensitive configuration before sharing logs in an issue or chat channel.
Secrets, permissions, and artifacts
Secrets failures commonly appear during registry login, cloud authentication, signing, database migration, or deployment. Check whether the secret exists in the current repository, branch, environment, or approval context. Verify its name and scope without printing its value. Review permissions for the job identity and confirm that the requested action is allowed in the target environment.
For artifact failures, verify that the expected files were actually produced. A successful compile does not guarantee that the packaging step wrote to the directory used by the upload step. Check relative paths, working directories, executable permissions, archive contents, and artifact naming.
Containers and deployment health
For a container build, separate image creation from image execution. Inspect the build context, base image, dependency installation, exposed port, entrypoint, user permissions, and runtime environment. Run the image locally with the same command and required non-secret variables when practical. Teams deploying open-source Docker applications can use the production readiness guide for Docker apps as a complementary checklist.
For deployment errors, record the target, image or artifact version, rollout status, startup logs, readiness result, and application health response. A deployment command can succeed while the new process fails to start. Treat health-check failures as a separate diagnostic branch, not as proof that the build was invalid.
Cadence and checkpoints
Review pipeline reliability on a regular schedule rather than only after a serious incident. A monthly or quarterly checkpoint is usually enough for a small team, provided someone owns the review and records decisions.
At each checkpoint, inspect:
- Recent failed runs and the first failing stage.
- Repeated failure categories, such as flaky tests, expired secrets, dependency resolution, or runner capacity.
- Median and worst-case build duration compared with the team’s expected feedback window.
- Jobs that require manual reruns, approvals, or undocumented local fixes.
- Cache hit behavior, artifact retention, log accessibility, and cleanup practices.
- Changes to runtimes, base images, package registries, cloud targets, deployment manifests, and runner images.
After a significant infrastructure or workflow change, perform an earlier checkpoint. Examples include introducing a new deployment target, moving from a hosted runner to a self-hosted runner, changing a container base image, rotating credentials, adding a test matrix, or adopting Kubernetes. A Kubernetes upgrade runbook can help structure checks when the deployment platform changes.
Test rollback separately from ordinary deployment. Define which prior artifact or image is considered safe, who can initiate the rollback, and how the team confirms that traffic has returned to the previous version. A rollback point that exists only in theory is not a dependable recovery option.
How to interpret changes
Look for relationships between pipeline variables instead of treating every failed run as unique. A sharp increase in dependency failures after a lockfile or registry change suggests a resolution or access issue. Failures that occur only on one operating system, runtime version, or matrix entry suggest an environment mismatch. Intermittent test failures with different error locations suggest flakiness, shared state, timing, or resource contention.
Longer build times can result from a larger test suite, a cache miss, a slower runner, network retries, or serialization introduced by a new job dependency. Compare step durations before changing the pipeline. Removing tests or making jobs artificially parallel may conceal the cause and reduce confidence.
Deployment failures should be classified by phase:
- Before rollout: authentication, permissions, artifact retrieval, manifest validation, or infrastructure configuration.
- During rollout: scheduling, capacity, image retrieval, migrations, startup, or readiness.
- After rollout: routing, application errors, configuration drift, database compatibility, or external service behavior.
For every recurring failure, write a short post-incident note containing the symptom, root cause or current hypothesis, evidence, safe workaround, permanent fix, and owner. If the cause is not known, say so. A documented unknown is more useful than an incorrect explanation that gets repeated in future incidents.
When to revisit
Revisit this troubleshooting guide during the team’s monthly or quarterly pipeline review, and immediately after a new failure pattern appears. Update the runbook when provider terminology, runner capabilities, workflow syntax, authentication methods, or deployment tooling changes. Command examples should be checked whenever the project changes its package manager, runtime, container tooling, or cloud target.
Use this final action list when a pipeline fails:
- Open the run and identify the first failed step.
- Capture the commit, environment, runner, command, exit code, and relevant log lines.
- Compare the failure with the latest successful run.
- Reproduce the command with matching versions and inputs, without exposing secrets.
- Classify the issue as source, dependency, test, permission, artifact, container, deployment, or health-check related.
- Apply the smallest safe fix and rerun the affected scope.
- Verify the deployed service, not only the deployment command.
- Record the cause, rollback point, and preventive action.
Teams can pair this process with CI/CD environment and secrets management guidance and reusable pipeline templates for open-source projects. The goal is not to eliminate every failure. It is to make failures observable, bounded, recoverable, and less likely to recur.