A reliable cloud release process needs more than a deploy button. This runbook explains when to use blue-green or canary deployment, which signals to monitor, how to decide whether to promote or roll back, and how to verify recovery after a traffic switch.
Overview
Blue-green and canary deployments are two ways to reduce the risk of releasing a cloud application. Both introduce a new version alongside the existing one, but they control production traffic differently.
In a blue-green deployment, the current version serves traffic in one environment and the candidate version runs in a separate environment. After validation, a load balancer, gateway, service router, or deployment controller switches traffic from the old environment to the new one. Rollback can be relatively direct: route traffic back to the previous environment while it remains available.
In a canary deployment, the new version receives a limited share of production traffic first. The team observes application health and user-impact signals before increasing the percentage. Canary traffic may be assigned by percentage, customer group, region, header, cookie, or another routing rule supported by the platform.
| Consideration | Blue-green | Canary |
|---|---|---|
| Traffic exposure | Usually switches most or all traffic at once | Increases gradually through defined stages |
| Rollback | Often a fast traffic reversal if the old environment remains ready | Usually stops promotion, reduces canary traffic, then restores the stable route |
| Infrastructure cost | Requires capacity for two environments during the transition | Requires duplicate capacity for at least the canary slice |
| Best fit | Predictable applications with a clear traffic switch and strong environment parity | High-risk changes where limited user exposure provides useful evidence |
| Main risk | A broad issue may affect many users immediately after promotion | Low-volume or uneven traffic may hide problems during the canary |
Neither strategy makes an incompatible database change safe by itself. Both depend on accurate health checks, observable application behavior, compatible data changes, and a team that knows who can stop or reverse the release. Treat this document as a reusable cloud deployment runbook and adapt the commands and terminology to your cloud hosting or deployment platform.
What to track
1. Release and routing state
Record the release identifier, image digest or artifact version, commit, configuration revision, environment name, and deployment time. Confirm which environment is stable, which is the candidate, and where traffic is currently routed. Do not rely on a service name alone if a platform can point that name to several revisions.
For a canary, write down each promotion stage before deployment begins. For example, the plan might move from internal testers to a small traffic slice, then to a larger slice, and finally to full traffic. The exact percentages matter less than having explicit gates and an owner for each decision.
2. Health checks and readiness
Separate checks that answer different questions:
- Process or liveness: Is the instance running, and can it respond?
- Readiness: Can it serve the required requests, reach necessary dependencies, and accept traffic?
- Functional: Can a representative user journey complete successfully?
- Dependency: Are databases, queues, object storage, identity services, and external APIs behaving within expected limits?
A health endpoint should not report success merely because the process is alive. At the same time, making every dependency a hard readiness requirement can prevent recovery when an optional service is unavailable. Define the checks according to the application’s actual failure modes, and test them before a production release.
3. User-impact signals
Monitor error rate, latency by percentile, request volume, timeouts, saturation, and availability for both the stable and candidate versions. Compare equivalent routes and traffic groups rather than looking only at aggregate application metrics. A small canary can look healthy in aggregate while one important endpoint fails consistently.
Also track business or workflow signals appropriate to the application, such as completed transactions, login success, job completion, message processing, or checkout abandonment. These signals may reveal a broken user journey that infrastructure metrics do not show.
4. Session, cache, and state behavior
Before switching traffic, determine where sessions and short-lived state live. If sessions are stored only in process memory, a user moved between environments may be logged out or lose an in-progress action. Prefer a shared, compatible session store or use routing behavior that accounts for the application’s session design. Treat sticky sessions as a compatibility aid, not as a substitute for understanding state.
Check cache key compatibility, cache invalidation behavior, background workers, scheduled jobs, and outbound message handling. During a blue-green transition, both versions may briefly run at the same time. During a canary, they may process related requests concurrently.
5. Database and API compatibility
Use an expand-and-contract approach for schema changes where possible: add compatible fields or tables first, deploy code that can work with both old and new shapes, migrate data, and remove obsolete structures only after the previous version is no longer needed. Avoid coupling a traffic switch to an irreversible migration.
Apply the same discipline to events, APIs, queues, and serialized objects. The stable and candidate versions should be able to coexist for the duration of the rollback window. Review the cloud deployment environment parity checklist before release if configuration or infrastructure differences could affect the result.
Promotion checklist
- Artifact, commit, configuration, and migration status are recorded.
- Candidate health and functional checks pass.
- Database changes are backward compatible with the stable version.
- Traffic rules, weights, and rollback actions are tested or documented.
- Dashboards and alerts cover both stable and candidate revisions.
- Session, cache, worker, and scheduled-job behavior is understood.
- Rollback authority and approval owners are available during the window.
- A rollback deadline is defined before stale environments are removed.
Cadence and checkpoints
Use checkpoints to turn a deployment into a controlled sequence rather than one large decision. The following pattern works for either strategy, with the exposure step adjusted to match the release design.
- Preflight: Confirm the artifact, configuration, secrets, infrastructure changes, data migration plan, dashboards, and routing configuration.
- Deploy candidate: Start the new environment or revision without exposing it broadly. Wait for readiness and run smoke tests.
- Observe: Allow enough time to exercise normal startup, dependency calls, background work, and representative requests. A fixed short wait is not sufficient if important jobs run less frequently.
- Limited exposure: For canary releases, send the first planned traffic slice. For blue-green releases, use controlled validation or internal access before the switch.
- Gate: Compare candidate metrics with the stable baseline. Require an explicit owner to approve promotion, pause it, or roll back.
- Increase or switch: Move to the next canary stage or switch blue-green traffic. Keep the previous version available for the agreed rollback window.
- Verify: Recheck user journeys, error budgets or service objectives used by the team, background processing, logs, and data consistency.
Approval gates should be based on observable conditions, not optimism. Define thresholds before the release, including what constitutes a rollback even if the service remains technically available. If a platform’s load balancer or deployment service uses different terms for revisions, target groups, backends, routes, or weights, map those terms into the runbook and review the mapping after platform changes.
Rollback decision tree
- Is there a confirmed severe user or data impact? Stop promotion and roll back immediately. Preserve evidence and notify the incident owner.
- Is the candidate failing readiness or functional checks? Keep traffic on the stable version, remove candidate exposure, and investigate before retrying.
- Are error rate, latency, saturation, or business signals materially worse than the baseline? Pause at the current stage. If the trend is worsening or the agreed threshold is exceeded, roll back.
- Is the signal ambiguous? Hold the current traffic level, validate the metric, check sampling and routing, and set a short decision deadline. Do not increase exposure while measurement is uncertain.
- Is the issue caused by an irreversible data change? Do not assume traffic reversal will repair it. Follow the data recovery plan and involve the database owner.
Rollback procedure
- Announce the rollback and assign one person to operate the route change.
- Stop canary promotion or freeze the blue-green switch.
- Restore traffic to the stable revision using the documented routing control.
- Confirm that new requests reach the intended revision and that old candidate traffic drains as expected.
- Disable candidate background workers or scheduled jobs if duplicate processing is possible.
- Preserve logs, metrics, deployment events, and configuration state before making unrelated changes.
- Run post-rollback functional checks and monitor until the service returns to its normal baseline.
How to interpret changes
Compare the candidate with a relevant baseline: the same routes on the stable version, a similar traffic period, and the same dependency conditions. A rising error rate may result from a new application defect, a changed route, a dependency outage, or an observability problem. Check deployment events and infrastructure changes before assigning a cause.
Look for correlated changes across signals. Higher latency with increased CPU may indicate resource pressure; normal CPU with higher database wait time points elsewhere. A healthy endpoint combined with failed user journeys suggests that shallow health checks are missing an important dependency or workflow.
Do not overreact to a single noisy data point, but do not average away a severe failure affecting a small customer group. Segment metrics by revision, route, region, tenant class, device type, or other meaningful dimensions. For open-source applications and teams operating their own infrastructure, keep dashboards and alert definitions in version control where practical, and review them alongside application changes.
After a rollback, classify the result: successful traffic restoration, partial restoration, failed rollback, or rollback blocked by data compatibility. A failed rollback is a deployment-system concern as well as an application concern. Review image availability, routing permissions, secret versions, capacity, and environment parity. The CI/CD pipeline troubleshooting guide can help separate build, deployment, and runtime failures.
Keep stale environments only while they provide a defined recovery or investigation benefit. Before deleting one, verify that the replacement is stable, logs and artifacts are retained, database compatibility is no longer needed, and any scheduled work has been transferred safely. Record the deletion decision so an incident review can distinguish an intentional cleanup from an accidental loss of rollback capacity.
When to revisit
Review this runbook before every high-risk production release, database migration, traffic-routing change, major dependency upgrade, or incident exercise. Revisit it immediately after a failed canary, an unexpected rollback, a change to the cloud load balancer, or a deployment-platform release that changes health checks, routing weights, revision management, or approval behavior.
On a monthly or quarterly cadence, sample recent deployments and check whether the documented procedure matches what operators actually did. Update:
- Platform terminology, routing controls, and rollback commands.
- Health-check paths, readiness dependencies, and synthetic tests.
- Baseline metrics, alert thresholds, and business-impact signals.
- Database compatibility and migration assumptions.
- Session, cache, worker, and scheduled-job handling.
- Approval owners, escalation paths, and rollback deadlines.
- Retention rules for artifacts, logs, configurations, and stale environments.
Before the next release, copy the runbook into the change record and fill in the release-specific values: candidate version, traffic stages, success thresholds, rollback target, owners, and verification steps. A deployment strategy is only useful when the team can execute it under pressure. Practice the route reversal, confirm access to the required controls, and update the document whenever the real system changes.