> ## Documentation Index > Fetch the complete documentation index at: https://ara-90a60a07.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Recurring checks > Cadence, ownership, cost, and incident flow for Ara's standing production and staging verification. # Recurring checks Ara separates fast outage detection from deep product verification. GitHub scheduled workflows are best-effort, so they do not own HTTP, DNS, TLS, or multi-region availability. Checkly runs those probes every 15 minutes; Ara evaluates telemetry continuously. GitHub owns exact product transactions and slower control-plane, runtime, security, backup, and release checks. The typed inventory in `scripts/recurring-checks-manifest.ts` is the canonical machine-readable list. Its test fails when a scheduled workflow is added, removed, or retimed without updating the inventory. ## Lanes | Lane | Executor | Cadence | Coverage | Cost policy | | --------------------- | -------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | | Telemetry | Ara telemetry | Continuous | Errors, failure rates, freshness, queues, breakers, and deadman alerts | Query-only | | External availability | Checkly | 15 minutes | HTTP, DNS, and TLS in parallel from North America, Europe, and Asia-Pacific | No inference or sandbox | | Exact production path | GitHub Actions | 15 minutes | Authenticated API plus exact session path; Hands and connector scenarios become due every six hours | Bounded real inference | | Control plane | GitHub Actions | Hourly | Reliability rates, staging deploy drift, inference authorization, and model catalogs | Free or read-only | | Deep runtime | GitHub Actions | Every 6 hours | Staging session, SSE, compaction, confinement, Main toolbelt, and Hands | Bounded real inference/sandbox | | Daily | GitHub Actions | Daily | Release, backup, field budget, security drift, API fuzzing, plugin OAuth, site, runtime, PR review, and model-runtime evaluation | One independent verdict per concern | | Weekly | GitHub Actions | Weekly | CodeQL, full-tree gitleaks, and cross-repository skill drift | Backstop; code-change checks remain event-driven | Localization generation is not scheduled; deterministic localization validation remains in CI. Globalping's consumer/datacenter diagnostic remains manual for incident investigation; it is not a paging scheduler. ## Production and staging Use the same scenario contract where comparison matters, but not the same verdict: * Staging failures block promotion and stay scoped to the synthetic `ara-staging` workspace and staging telemetry stream. * Production failures alert operators and feed rollback evidence, using only the dedicated `canary-box` workspace. * Public HTTP, DNS, and TLS probes are organization-free. * Never point a recurring check at Acme, a personal workspace, or a customer workspace. The exact production path grades its session scenarios through the same named Ara evaluation contract used by the runtime evaluation suite. A harness-contract mismatch fails the canary and its regression test instead of being mistaken for a product-path result. CLI release discovery validates the complete GitHub release inventory first. If GitHub's unauthenticated API is rate limited from the web edge, it validates the public latest-release manifest asset instead; the embedded known-good version is only the final recovery floor when both live paths are unavailable. ## Failure flow 1. Open the failing job and identify whether the failure is product, provider, test harness, or scheduler delivery. 2. Correlate the exact environment, run/session ID, deployment SHA, and telemetry event. Do not combine staging and production evidence. 3. Preserve real failures. Do not add retries or loosen assertions merely to turn the workflow green. 4. For a production regression, follow the emergency rollback recommendation flow before changing traffic. 5. For a broken harness, disable only the known-broken schedule, keep deterministic coverage, and record the restoration condition. Transient Device startup failures return the exact lease-owned run to the pending queue and append a recovery breadcrumb. A late event append may advance that run's event sequence, so recovery retries the sequence handoff without overwriting the competing event or stranding the run in `running`. ## Change flow Every recurring-check change must include all of the following in one PR: 1. Update the workflow or Checkly construct. 2. Update `scripts/recurring-checks-manifest.ts` with cadence, environment, cost, importance, and proof. 3. Update this page when the lane or incident flow changes. 4. Run `bun test scripts/recurring-checks-manifest.test.ts` and the closest script tests. 5. For Checkly changes, let `.github/workflows/deploy-checkly.yml` deploy after merge and verify the deployment run before considering the monitor live. 6. For secret-bearing probes, keep Infisical as the source of truth and admit reviewed default-branch code only. Daily promotion is special. It runs at 9:22 AM in `America/Los_Angeles`, away from GitHub's top-of-hour scheduler congestion and without paired UTC schedules for daylight saving time. The hourly control-plane lane checks between 10:00 AM and 4:59 PM Pacific for a successful or active daily controller, including the 4:17 PM pass needed to recover a controller cancelled after the previous hourly check. If the scheduled opportunity is missing or only cancelled, it dispatches the same trusted controller exactly once; a genuine failed run remains blocked for investigation instead of being blindly retried. Before opening the promotion PR, the controller actively dispatches or observes the protected `main` to `staging-dev` sync and fails closed on a real merge conflict. The sync lane uses a short-lived, repository-scoped Ara GitHub App token only to author its branch and pull request; the promotion lane uses the same scoped identity only to author its pull request. The built-in workflow token retains labels and orchestration, and Mergify remains the sole merge controller. Human approval for changes to `AGENTS.md` or `DEVELOPMENT.md` is enforced only when a pull request enters `staging-dev`. Once that exact revision has passed staging admission, the scheduled `staging-dev` to `main` promotion does not request the approval again. Production promotion remains autonomous and is governed by its exact-head staging, migration, deployment, telemetry, and production receipt checks. Deployment observers use independent staging and production concurrency lanes. Each admitted deployment-status event records a lightweight receipt before the longer release-verdict window; a newer same-environment deployment may supersede that long observation, but staging activity cannot cancel production evidence and the hourly reconciliation remains the backstop for same-SHA or manual redeployments. For a migration-bearing release, the controller requires an idempotent staging-migration run for the exact promotion head. It reuses an active or successful run, re-runs a failed receipt, and dispatches a missing one before it arms protected auto-merge. A later workflow-only or documentation commit therefore cannot accidentally erase the migration proof. The controller first waits for the exact staging web candidate and the required API revision to be live, then refreshes `run-staging-gate` on the standing promotion PR. This ordering avoids a CI/deploy cycle in which the API waits for the very gate that waits for the API. The gate is produced for the exact current staging head; a previous green check never authorizes a newer head, and a candidate API may be an ancestor only when intervening commits do not touch the API deployment surface. The gate polls through any remaining cutover propagation instead of treating its first stale revision as terminal. If `staging-dev` advances during migration proof, staging admission, arming, or the protected merge wait, the controller stops waiting on the superseded deployment, disarms any stale auto-merge and proof label, and proves that one successor controller is active or discoverable. That run records a terminal `superseded` receipt without opening a blocked-release incident; failure to establish the successor remains a real failure. Every admitted run records one terminal release-train receipt (`released`, `noop`, `held`, `superseded`, or `failure`), and a telemetry no-data monitor detects a missing receipt across the daily boundary. Keeping the controller check on `main` also prevents the staging deployment platform's CI wait from depending on the controller that is waiting for that deployment. Live API evidence is correlated to the current deployment identity, not merely a recent commit with the same SHA. The telemetry veto starts at that exact deployment's `deploy.live` timestamp, followed by a short ingestion settle period, and both error lanes require the exact candidate commit. Errors caused by the candidate therefore remain fail-closed while errors from an older concurrently active deployment cannot poison the next release. Individual failed staging agent runs are handled by their measured wave threshold instead of making ordinary test activity a release veto. After protected auto-merge succeeds, the same release run checks out the merged commit and waits for the exact production web bundle and API deployment verdict; a merge alone is not reported as a successful production release. Background provider work must not keep a database transaction open across network I/O. In particular, personal-memory sync and verified erasure use durable leases, short snapshot/finalization transactions, and post-provider fencing. This keeps the release telemetry veto sensitive to genuine transaction timeouts without mistaking an intentionally long provider operation for valid database work.