> ## Documentation Index > Fetch the complete documentation index at: https://help.stralos.app/llms.txt > Use this file to discover all available pages before exploring further. # Deployment > Deployment model for Stralos services Stralos service repositories define per-service Terraform stacks for AWS Lambda behind regional REST API Gateway v1. ## Shared API topology The Activities, Assistant, Documents, Estimate, Global, Rates, and Scheduler stacks currently define: * regional REST API Gateway v1 with stage name `main`; * Lambda proxy integrations; * structured API access logging and X-Ray where configured; * optional association with the environment's regional Web ACL; * a stable `region_key`, initially `au-nz`, used in names, SSM paths, and state keys; and * S3-native Terraform state locking. The provider region is derived from the stack's region-key map. Backend keys follow `//terraform.tfstate` and require Terraform 1.10 or later because the backend uses `use_lockfile = true`. ## Canonical deployment path GitHub Actions is the canonical deployment path. Service workflows call the reusable Terraform workflow in `stralos-core`, then run the configured migration and Terraform stages. Dev deployments build from the triggering source revision; release promotions use the immutable artifact path described below. The configured progression is dev, test, then prod. A job that targets a GitHub Environment is approval-gated only when that environment and its protection rules exist. Confirm those rules in GitHub before relying on an approval gate. ### Relational API immutable release path Activities, Documents, Estimate, Global, Rates, Scheduler, and Assistant publish immutable source-SHA artifacts on the guarded dev path. Test and production are coordinated by `stralos-core`'s `release-test.yml` and `release-prod.yml`. Member `prepare-release.yml` and release-event promotion paths are retired. | Trigger | Target | Behaviour | | - | - | - | | Pull request | dev | Validation and plan; no apply | | Push to `main` | dev | Publish and apply after pre-flight guards | | `deploy-dev.yml` dispatch | dev | Publish and apply from the asserted commit | | Central test release | test | Stage a candidate, then activate it through the central barriers | | Central production release | prod | Promote the recorded test candidate without rebuilding, then activate | The central release separates staging from activation: 1. **Publish dev** — the member verifies its source ref and publishes the immutable artifact set before applying dev. 2. **CREATE** — core resolves the source, freezes its promotion identity, and dispatches the member's `release-create.yml` at the annotated release tag. API CREATE uses `terraform.yml` in stage mode. It publishes immutable versions without moving aliases or running migrations. 3. **ACTIVATE** — core runs the service migration through the configured runner, then moves the recorded aliases and checks the edge. The frontend flip waits for the API activation barrier. A bare merge to `main` does not touch test or production. Do not dispatch retired workflows to bypass a failed central release. See [Release model](https://help.stralos.app/internal/release-model.md) for source selection, barriers, and frontend activation. Migrating APIs declare a `MIGRATION_MANIFEST.json`; non-migrating services omit migration inputs from CREATE. The member's workflow supplies its manifest and migration hash paths so staging and activation use the same migration identity. Activities and Scheduler request owner access. Rates requests owner and platform access. Global requests owner and platform access. Documents requests owner and platform access. Estimate requests owner, platform, and global-owner access. The pinned database-bootstrap runner resolves connections per manifest entry, exposes them only inside that entry's subprocess, and executes the entry's declared command. It defaults older manifests without a `database_access` field to owner access. ### Migration execution For a service with a `MIGRATION_MANIFEST.json`, the dev full-mode Terraform path runs migrations before apply. On the central test and production path, CREATE stages without migrations; ACTIVATE runs them before moving aliases. There is no content-hash skip gate; the migration tool itself applies only pending entries. The workflow records the migration-content hash as evidence so environments can be compared, but the hash does not decide whether a run occurs. Protocol 2 executes the manifest inside the environment's VPC CodeBuild project. It reads role-specific connection URLs from SSM by convention and does not run Terraform. The deployed source tree is used for dev; promotions use the frozen migration source. A run records started, per-entry, and completed evidence in the migration ledger, so an interrupted release cannot be mistaken for a completed one. Write outages are armed by cutover markers, not by the manifest's migration class. The shared workflow pinned by these services reads both `drizzle/CUTOVER_REQUIRED` and versioned markers such as `drizzle/0036_CUTOVER_REQUIRED`. Non-comment lines name migration files. The workflow subtracts migrations completed by another release in the same environment. Completion recorded for the same release does not disarm a retry of that release. Missing or unreadable ledger evidence retains outage arming. When armed, deployment stops and drains writers before migration, then restores their previous concurrency after successful deployment. Markers can remain in source: a later release can exclude migrations whose completion was recorded for another release in that environment. Marker detection fails closed when it cannot establish completion. The shared database-bootstrap Terraform stack is provisioned separately through `stralos-core`'s `db-bootstrap.yml`. A push to its `main` path applies dev; other environment applies are explicit, plan-reviewed dispatches. Ordinary service migration runs do not apply or rotate that shared database infrastructure. ### Automatic and deliberate dev deploy Activities, Assistant, Documents, Estimate, Global, Rates, Scheduler, and Admin API run `deploy-dev.yml` on pushes to `main`. The push path uses the same immutable publish-then-apply chain as manual deployment, with additional pre-flight guards: * A superseded push is refused when its SHA no longer matches the current `main` tip. * `scripts/auto-deploy-guard.mjs` checks every commit through the pushed SHA for cutover marker changes. * Adding, modifying, removing, or renaming a cutover-marker path anywhere in that range refuses automatic publication. An add-then-remove sequence is still caught even when the net diff is empty. * Unreadable or invalid live SHAs, incomplete commit enumeration, diverged or behind history, and API failures refuse the automatic path. The guard behaves slightly differently for the Admin API compared with the other services: * **Live commit source** — The other services read the commit currently served from their `/version` endpoint. The Admin API has no `/version` route, so its guard reads the head SHA of the newest successful `deploy-dev.yml` run on `main` from the GitHub Actions API. * **Marker presence arms immediately** — For the other services a versioned marker in the tree does not block a push once its migration is recorded in the ledger. The Admin API cutover marker (`drizzle/CUTOVER_REQUIRED`) gates every writer on every deploy while it is present, so the guard refuses any push that finds the bare marker at the pushed commit. A marker merged but not yet deleted blocks all subsequent pushes until it is deleted. * **Range rule** — Both the Admin API guard and the other services refuse a push when any commit in the range added, changed, removed, or renamed a cutover-marker path. A delete is caught for the same reason as an add: lifting a writer outage unattended is as deliberate an act as taking one. A refused push has not published or applied that candidate. For a range touching a cutover marker, review the intended outage and use a deliberate dispatch: ```bash theme={null} # Deploy the current main commit gh workflow run deploy-dev.yml # Assert the main commit you expect to deploy gh workflow run deploy-dev.yml -f ref= ``` The `ref` input is an assertion, not a historical commit selector. Dispatch from `main`; the workflow verifies the input against its run commit and chains immutable publication before apply. A failed or skipped publish fails completion. Each repository serialises dev deployment with `cancel-in-progress: false`. This prevents overlapping applies but does not guarantee dispatch order; GitHub retains at most one pending run in a concurrency group. ### Migration class and writer outage All seven relational services keep `MIGRATION_MANIFEST.json` at `additive-online`. The class is validated release metadata; it does not arm a write outage. A release that needs writers stopped must commit a supported cutover marker at the deployed SHA and name the affected migration file. The dev apply, dev plan, test, and prod call sites in every service pass `stop_writers_function` and `cutover_concurrency_ignored_by_terraform: true`. Each API Lambda also ignores Terraform changes to `reserved_concurrent_executions`. This standing wiring arms an outage when marker reconciliation cannot prove every named migration was completed by another release in that environment. See [API writer cutover](https://help.stralos.app/internal/runbooks.md#api-writer-cutover) for the current marker and recovery procedure. ### Destructive migration guard Every pull request in these services runs `scripts/check-destructive-migrations.mjs`. It refuses a changed `drizzle/*.sql` file containing `DROP TABLE`, `DROP COLUMN`, `ALTER COLUMN … TYPE`, `SET NOT NULL`, `RENAME COLUMN`, or `RENAME TO` unless a cutover marker names that exact file. All seven pull-request guards recognise the unversioned marker. Rates also accepts versioned markers with a matching migration prefix; the other six pull-request guards still require the unversioned form. This differs from the automatic-deploy guards and pinned apply workflow, which recognise both forms. `DROP CONSTRAINT`, `DROP INDEX`, `DROP POLICY`, and `DROP NOT NULL` do not trigger this bridge guard. Run the same check locally against the current base: ```bash theme={null} node scripts/check-destructive-migrations.mjs --base "$(git merge-base origin/main HEAD)" ``` ### Release by alias (Estimate, Rates, and Assistant) The Estimate, Rates, and Assistant API stacks define a `live` Lambda alias. API Gateway integrates with the alias's invoke ARN, not the bare function's. The bare function always serves `$LATEST`, so without the alias a version change would require a full Terraform apply and rollback would be impossible without one. With the alias in place, a release is a single `update-alias` call that points `live` at an already-published version. A rollback is the same operation in reverse, at the same cost — no build, no apply, no new bytes. Each stack's `release-alias.yml` workflow performs the move: | Input | Values | | - | - | | `target` | `dev` or `test` (Estimate and Rates); `dev` only (Assistant) | | `version` | Published Lambda version number, `previous` (rollback to the highest published version below current), or `current` (report only — no move) | The workflow resolves what is currently served, validates the requested version is published, conditionally moves the alias with an optimistic-concurrency revision guard, then proves the move through the edge URL before recording a job summary. `$LATEST` is refused as a target. The alias exists specifically so that `$LATEST` is never what API Gateway serves. **Terraform re-points the alias on the next apply.** The alias is re-pointed to the version Terraform just published whenever a Terraform apply runs. `release-alias.yml` is for the window between applies — moving the alias between two already-published versions without triggering a full deployment cycle. The API Gateway invoke permission is qualified to the alias (`qualifier = aws_lambda_alias.live.name`). A permission on the bare function does not cover `function:live`; without the qualifier, API Gateway's invoke returns 403 on the first deploy after the integration moves. The permission uses `statement_id_prefix` (not `statement_id`) with `create_before_destroy = true` so that the new grant exists before the old one is removed and there is no 403 window during the transition. An alias move changes only the code version. Database migrations, environment variables, IAM policies, and API Gateway or stage configuration still require a Terraform apply. ### Assistant API The Assistant service uses the same guarded dev path as the other relational APIs — a push to `main` applies dev through `deploy-dev.yml`. For test and prod, releases are orchestrated centrally by `stralos-core` (`release-test.yml` / `release-prod.yml`) dispatching `release-create.yml` in this repository; the per-repo `prepare-release.yml` freeze and `terraform-deploy.yml` promote steps are retired. The Assistant carries its own `MIGRATION_MANIFEST.json` with one `additive-online` entry for the `assistant` service. The service deploys on the `nodejs24.x` managed Lambda runtime. It uses the response-streaming Lambda invocation path for SSE turns. Its Global, Rates, Estimate, and Scheduler URLs all resolve through the regional application origin supplied by `regional_app_base_url`, rather than through separate peer-service SSM lookups. Assistant uses the shared [Lambda alias release mechanism](#release-by-alias-estimate-rates-and-assistant), with additional identity checks described below. Terraform manages three runtime settings under `////services/assistant/`: `ai-provider`, `bedrock-model-id`, and the `enabled` kill switch. The Lambda role can invoke the configured cross-region Bedrock inference profile and its foundation models in `ap-southeast-2` and `ap-southeast-4`. #### Alarm topic All CloudWatch alarms route to the SNS topic identified by `alarm_sns_topic_arn`. The variable defaults to the dev-account topic; a non-dev apply must override it with that environment's own topic ARN. Two validations enforce a correct value at plan time: * **ARN format** — the value must be a full SNS topic ARN matching `arn:aws:sns:::`. A subscription ARN or any other non-topic ARN silently never delivers notifications and is rejected. * **Same-account guard (fail-closed)** — the account segment of the topic ARN must match the account the stack is applying into. A mismatch most commonly means the dev default was left in place for a test or prod apply. Because the dev topic's resource policy grants publish only to the dev account, a cross-account topic ARN would cause alarms to silently never fire. The plan fails on a mismatch rather than applying and leaving alarms non-functional. #### Alias workflow and proof `release-alias.yml` moves the LIVE Lambda alias by command, without a Terraform apply or a new build. For the Assistant service, `release-alias.yml` targets `dev` only; test and prod rollback runs through `stralos-core`'s `rollback-api.yml`, which moves every alias in the stack and writes the release record. Dispatch `release-alias.yml` with a target environment and a version — a published Lambda version number, `previous` (the highest published version below the current live version, which is what a rollback means), or `current` (report only, no move). The assistant is not on the regional CloudFront distribution; `https://au-nz..stralos.app/api/assistant/*` falls through to the SPA origin and returns 200 vacuously. The workflow resolves the service's own API Gateway stage URL from the SSM parameter Terraform writes at `/stralos//au-nz/services/assistant/api-url`, and probes `GET /api/assistant/version` after moving the alias. The proof requires the response to carry the exact `version` and `commit` values stamped onto the target Lambda version. `lambdaVersion` in the response body is compared against the actual alias target to confirm the pointer moved to the right version, not merely any live version. Before moving the alias, the workflow reads `SERVICE_VERSION` and `SERVICE_COMMIT` from the target version's immutable Lambda configuration (`get-function-configuration --qualifier `). A target that predates the M4 deploy-identity stamp (values of `None` or `unknown`) is refused before the alias is touched. This means rolling back to a version published before the first M4 apply is not supported via alias move; redeploy via `Terraform deploy` at the desired commit instead. The next Terraform apply after any alias move re-points the alias to the version it publishes. The alias move command is for the space between applies. #### Deploy-identity stamp Terraform stamps `SERVICE_VERSION` and `SERVICE_COMMIT` onto the API Lambda at apply time. `GET /api/assistant/version` returns these values alongside `lambdaVersion` — the Lambda version number the runtime is actually executing (from `AWS_LAMBDA_FUNCTION_VERSION`). The route is auth-exempt and sets `Cache-Control: no-store`. It is an infra probe, not a documented API route, and is absent from the OpenAPI descriptors. `lambdaVersion` is assistant-specific. Two consecutive applies of the same main commit publish two Lambda versions carrying the same `SERVICE_VERSION` and `SERVICE_COMMIT`; `lambdaVersion` is the only value that differs between them, which is why the alias-move proof uses it to confirm the pointer moved. #### Deterministic deployment package `scripts/build-lambda.sh` produces a byte-identical ZIP on every build of the same source. The script stages the three output files (`lambda.mjs`, `lambda.mjs.map`, `routeCatalog.json`) in a temporary directory, normalises all `mtime` values to a fixed timestamp, and feeds an explicitly sorted file list to `zip -X` (which drops extended timestamp and uid/gid fields). The destination ZIP is removed before each run so that `zip` cannot append to an existing archive. Two builds of the same source produce the same bytes, which is required for the R5 promotion model where the `lambdaArtifact.digest` in the promotion manifest must remain stable across rebuilds. #### Post-release smoke `smoke.yml` is a manual `workflow_dispatch` that configures the optional Claude Code runner and smokes the AI gateway after a release reaches test or prod. Dispatch it with `environment=test` or `environment=prod` after every release that reaches either environment: ```bash theme={null} gh workflow run smoke.yml -f environment=test ``` Dev smoke runs automatically inside `deploy-dev.yml`'s `configure-claude-code` job after every dev deploy and does not need a separate dispatch. The central release does not yet run the smoke automatically; a post-activate hook is a planned follow-up. ### Documents API runtime and provider requirements The Documents service runs on the `nodejs24.x` Lambda runtime. The Terraform AWS provider must be `~> 6.19` or later; earlier 5.x or pre-6.19 provider versions cannot express the `nodejs24.x` runtime string and will fail to plan. The repository's `.node-version` file pins Node.js 24 for CI and local builds. #### Deploy-time environment variables Terraform stamps these variables into the Lambda on every apply: | Variable | Terraform variable | Purpose | | - | - | - | | `SERVICE_VERSION` | `var.service_version` | Semver reported by `GET /api/documents/version`. Carries the frozen promotion manifest's `appVersion` from release-model M1 onward; defaults to `0.0.0-unprovisioned` until wired. | | `SERVICE_COMMIT` | `var.service_commit` | 40-character Git commit SHA reported as `commit` in the `GET /api/documents/version` response. Stamped on every apply so dev builds — which carry no semver — have an identity to verify against, and so the edge-served Lambda can be tied to its source commit. Defaults to `"unknown"` when not supplied. | `stralos-core`'s `terraform.yml` exports `TF_VAR_service_commit` for both dev deploys and promotions, so the field is populated automatically in the standard CI path. #### Bundle budget `esbuild.config.mjs` limits the request-path bundle, `dist/lambda.mjs`, to 15 MiB. `pnpm run build:no-cache` fails when the bundle exceeds that budget, so the Lambda packaging build stops before creating its ZIP. Use narrow SDK entry points instead of package-root imports when a dependency exposes generated clients; worker bundles follow the same import discipline but do not have this hard limit. #### Live alias release model Every Lambda in the Documents stack is published on every Terraform apply (`publish = true`) and invoked through a `live` alias, never the bare function. The bare function always serves `$LATEST`, so without an alias a version published ahead of activation would go live the moment the apply landed and a rollback would require a full apply. The API Lambda (`api`) carries its own `live` alias in `api.tf`. All 26 non-API functions — the 9 stage workers, 9 DLQ consumers, sweep, sync dispatch, sync worker, sync DLQ consumer, drive fetch worker, drive fetch DLQ consumer, estate events, and retention — are aliased by a single `for_each` resource (`aws_lambda_alias.function_live` over `local.live_aliased_functions` in `release.tf`). Every SQS event-source mapping and every EventBridge schedule invokes the alias; none targets the bare function. The `release_functions` Terraform output lists all 27 aliased functions (API plus the 26 above), each with its `function_name`, the version the apply published, and the alias names that activation moves. The `enable_on_activate` output is empty: no triggers are created disabled awaiting activation. A Terraform test in `terraform/tests/test_release_by_alias.py` enforces these constraints statically against every `.tf` file: every function must publish a version, every function must be in `local.live_aliased_functions` (or be `api`), every SQS mapping and EventBridge Lambda target must invoke an alias, and both aliases must carry `ignore_changes = [function_version]` so a stage-mode plan never moves them. * **Stage** — Terraform publishes immutable versions without moving existing aliases. Activation uses `release_functions` to move them. * **Release without deploy** — release tooling calls `update-alias` to move `live` to any earlier published version. No Terraform apply is required and no new Lambda version is created. * **Rollback** — point `live` back to the previous version number. The cost and mechanics are identical to a forward release. The API Gateway invoke permission is scoped to the `live` alias qualifier. A permission on the bare function does not cover `function:live`, so Terraform uses `statement_id_prefix` (not a fixed `statement_id`) and `create_before_destroy = true` on the permission resource. This ensures the new grant exists before the old one is removed, so there is no window in which API Gateway holds no permission to invoke the function. Each EventBridge schedule grant is likewise qualified to the alias and uses `statement_id_prefix` with `create_before_destroy`. #### Billing access enforcement The Documents API includes a per-environment flag that gates whether the billing entitlements guard is active. | Terraform variable | Default | Purpose | | - | - | - | | `enable_billing_enforcement` | `false` | When `true`, the Lambda receives a non-empty `BILLING_ENTITLEMENTS_TABLE` and the billing access guard checks that table on every request. When `false`, the table name is an empty string and the guard is inactive. | Set `enable_billing_enforcement = true` in the environment's `.tfvars` file only after `stralos-api-billing` is configured and the DynamoDB billing entitlements table for that environment has been provisioned. Enabling the flag before the table exists causes every API request to fail at the entitlements check. Set the flag independently in each environment only after its billing service and entitlements table are ready. ### Manual helper limitation Service `scripts/deploy.sh` helpers rebuild the on-disk Lambda package and run Terraform apply against an already prepared working directory. They do not select or initialise the correct remote backend, and they do not replace the migration orchestration in the reusable workflow. Use them only as a reviewed recovery path with the target backend and migration state already verified. ## Shell publication The shell releases through separate produce and activate steps. Each repository publishes a commit- or version-addressed candidate. `stralos-core` gates and writes the remote composition, while the shell owns the final `index.html` activation. ### Trigger matrix | Event | Job | Deploys | | - | - | - | | Pull request | `build` (sanity only) | No | | Push to `main` | `publish.yml` sanity build and `deploy-dev.yml` producer | Produces the by-SHA dev bundle; does not activate | | `deploy-dev.yml` dispatch | `verify-ref` → reusable build → produce → `deployed` | Produces the asserted ref; does not activate | | Manual `workflow_dispatch` via `activate-dev.yml` | `activate-dev` | Yes — activates the published dev bundle | | Pre-release published (normally an `rc/*` tag), or `deploy-test.yml` dispatch | `deploy-test.yml` | Produces the by-version test bundle; does not activate | | `gate-test.yml`, then `activate-test.yml` dispatch | Core manifest gate, then shell activation | Yes — activates test | | Full release (normally a `v*` tag), or `deploy-prod.yml` dispatch | `deploy-prod.yml` | Produces the by-version prod bundle only when `PROD_DEPLOY_ENABLED=true`; does not activate | | `gate-prod.yml`, then `activate-prod.yml` dispatch | Core manifest gate, then shell activation | Yes — activates prod | An `rc/X.Y.Z` pre-release supplies version `X.Y.Z` to test, while a `vX.Y.Z` full release supplies `X.Y.Z` to prod. These prefixes are operator conventions, not workflow guards: both parsers remove an optional `rc/` and then an optional `v`, and accept the result when it is stable SemVer. The prod producer is dormant until the repository variable `PROD_DEPLOY_ENABLED` is `true`. ### Dev path Every push to `main` enters `deploy-dev.yml`. `verify-ref` proves the requested ref is the run commit, the reusable `build` job produces one handoff, and `publish-immutable` writes the deterministic release archive. The subsequent `dev` job writes the parked `shell//` serving bundle without changing the root `index.html`. Finally, the fail-closed `deployed` job requires all four preceding jobs to have succeeded, preventing a skipped producer from appearing as a successful run. Producing alone does not activate the shell in dev; it makes the build available for activation. To activate the published dev bundle in the dev environment, dispatch `activate-dev.yml`: ```sh theme={null} # Activate a specific published SHA gh workflow run activate-dev.yml -f sha= ``` The required `sha` input must be exactly 40 lowercase hexadecimal characters. The workflow takes the composition lease, runs the shared compatibility gate against the live remote set, verifies the parked `shell//index.html`, writes root `index.html` and the candidate's `built-against.json` when present through the fenced writer, and invalidates CloudFront. A push to `main` runs the build and `publish-immutable` jobs but not activation. To roll back, first produce the older ref with `deploy-dev.yml` if its immutable shell bundle is absent, then dispatch `activate-dev.yml` with that commit's SHA. The older commit does not need to be pushed to `main`. The shared concurrency group (`group: publish-shell-dev`, `cancel-in-progress: false`) serialises activation with the producer's dev serving job so they cannot write the shell bucket concurrently. The producer declares it at job level; the activation workflow declares it at workflow level. #### Composition lease Three workflows across two repositories can change what dev serves: `activate-dev.yml` in `stralos-shell`, and `gate-dev.yml` and `reconcile-dev.yml` in `stralos-core`. A GitHub concurrency group cannot serialise across repositories, so all three take one S3 composition lease before reading or writing the root index. The lease lives in `REMOTES_BUCKET`. When `activate-dev.yml` runs from a manual dispatch, it takes the lease itself. If another writer already holds the lease, the run exits as skipped and stays green. A held lease is a queueing outcome, not an incident. When `stralos-core`'s `reconcile-dev.yml` dispatches this workflow it passes a `correlation` input (`-`). The activation run adopts the reconciler's lease by presenting that exact correlation. Adoption requires the current unconsumed correlation and consumes that correlation. A later write must still match the lease's current correlation. An adopted run does not release the lease; the reconciler still has to confirm the root index moved. Both root pointers carry the fence and lease owner under which they were written. The fenced writer checks the current lease and correlation before each write, uses ETag-conditional updates, and refuses a higher stored fence or an equal fence with a different owner. An adopted run shares the reconciler's owner and fence; the current-correlation check constrains its writes. The two root pointers are still separate writes, not an atomic transaction. ### Build workspace The production build materialises a one-member pnpm workspace containing only `stralos-shell`. It installs `@stralos-co-nz/ui` and `@stralos-co-nz/domain` from the registry at the exact package pins and lockfile versions committed by the shell. No feature source is compiled into the host, and the build records `composed_with` as an empty object. Owner repositories are checked out with `WORKSPACE_CHECKOUT_TOKEN` only in the isolated integration-fixture job. That job validates source compatibility but contributes no files to the production artifact. Contract-v2 production rejects UI and Domain SHA overrides. ### Test and prod path Test and prod use the same ordered, two-writer activation: 1. Produce the shell and chosen remotes at the target version. Producing writes a version-addressed serving bundle and parked `index.html`; it does not intentionally change the live root pointer. Treat a version as single-use: the test produce claims the version with a write-once `manifest.sha256.json` before the first object write, and any subsequent produce that diverges byte-for-byte from that claim is refused before uploading anything. Prod does not build: `deploy-prod.yml` promotes the bytes test served for that version, verified against test's claim and completion marker, and copies them write-once. See [Prod](https://help.stralos.app/internal/release-model.md#prod). 2. Dispatch `stralos-core`'s `gate-test.yml` or `gate-prod.yml` with the candidate shell version and remote-version set. The gate checks each remote's built-against evidence against the candidate shell and, on a `GO` verdict, dispatches the single-writer assembler for `/remotes-manifest.json`. 3. Wait for the dispatched `remotes-manifest.yml` run to complete successfully. 4. Dispatch the shell's matching `activate-test.yml` or `activate-prod.yml`. It gates the candidate shell against the manifest, promotes the parked document to root `index.html`, and invalidates CloudFront. Run the core gate before shell activation. The manifest and root document are separate mutable objects, so this sequence is not an atomic cross-object transaction. After the assembler writes `/remotes-manifest.json`, browsers using the current root document may observe the new remote set before the shell swap. Keep the interval short and choose a remote set compatible with both the current and candidate hosts when uninterrupted operation is required. Treat the gate-to-activation period as an exclusive operator transition: do not run another core gate or manifest assembly for that environment until shell activation completes. Shell activation checks the live manifest before copying the root document, but the check is not bound to a manifest version and the core and shell workflows do not share a cross-repository lock. Roll back by repeating the gate and activation with versions that were already produced. ### Re-run behaviour Do not reuse a shell release identifier. The dev archive write fails when its key already exists. The test produce claims the version before writing any objects: `manifest.sha256.json` is written with `--if-none-match '*'` so only one run can create it, and every subsequent run must reproduce the identical byte set or be refused. A mismatch produces the `occupied-by-different-bytes` signal, which is unrepairable by retry: cut a new version. A transient S3 failure is distinct from a mismatch and is retryable. A prod promote re-verifies the test claim before writing, and prod writes are write-once, so a re-run can only complete the same bytes. Re-running activation also rewrites the live root objects and invalidates CloudFront; it is not a no-op. ### Environment variables Each publish job declares a GitHub Environment, so an environment-scoped value shadows a repository-level variable of the same name. Configure the applicable values separately for dev, test, and prod. | Variable | Purpose | | - | - | | `AWS_SHELL_ROLE_ARN` | OIDC role assumed by produce and activation jobs. | | `AWS_REGION` | AWS region; defaults to `ap-southeast-2` when unset. | | `SHELL_BUCKET` | Serving bucket for the shell's `index.html` and static chunks. | | `REMOTES_BUCKET` | Remotes bucket; used by `activate-dev.yml` to take the composition lease and to verify the current correlation before writing root pointers. Required for dev activation only (test and prod activation carries no lease). | | `CLOUDFRONT_DISTRIBUTION_ID` | CloudFront distribution invalidated after activation. | | `ARTIFACTS_BUCKET` | Authoritative artifact store used by shell production. | | `EXPECTED_AWS_ACCOUNT_ID` | Twelve-digit account ID checked before any serving-bucket write. | The workflows fail closed when required variables are unavailable or the authenticated account does not match `EXPECTED_AWS_ACCOUNT_ID`. See [Release model](https://help.stralos.app/internal/release-model.md) for the produce, compatibility-gate, and activation contracts. ## Admin console static site release path `stralos-admin-web` is a standalone Vite SPA, separate from the shell Module Federation graph. Dev publishes an immutable bundle. Test and production participate in the central release CREATE and ACTIVATE lifecycle, targeting S3 and CloudFront. ### Trigger matrix | Event | Target | Deploys | | - | - | - | | Pull request | — | No (build and verify only) | | Push to `main` | dev | Yes — `deploy-dev.yml` triggers automatically; immutable bundle published and served | | `deploy-dev.yml` manual dispatch | dev | Yes — dispatch `ref=` to assert an exact commit | | Central `release-test.yml` | test | Stages the dev bundle during CREATE; publishes the staged index during ACTIVATE | | Central `release-prod.yml` | prod | Stages the tested release bytes under the final version; publishes the staged index during ACTIVATE | ### Dev path `deploy-dev.yml` triggers on every push to `main`. It calls `deploy.yml` through `workflow_call` for the build, then publishes the immutable source-SHA-keyed artifact and runs `scripts/publish-admin-web.sh` to sync the bundle to the dev serving bucket with appropriate cache headers, invalidate CloudFront, and verify the SPA and same-origin `/health` route. A merge to `main` deploys admin-web to dev automatically, consistent with the shell and feature remotes. `deploy.yml` itself still builds and verifies the production bundle but does not publish or serve anything. It exists as the reusable build target that `deploy-dev.yml` calls. The central release stages the existing dev artifact. The `ref` input on a manual dispatch of `deploy-dev.yml` is an assertion rather than a selector: the run fails unless it resolves to the current `main` tip, so what dev serves is always a commit on `main`. The production bundle is environment-neutral: the client defaults to same-origin `/api/admin`. No environment hostname is embedded in the bundle bytes; the same artifact is promoted to test and prod. **Serving layout.** The bundle is built with a relative Vite base (`VITE_APP_BASE=./`), making it location-independent. `scripts/publish-admin-web.sh` rewrites `index.html`'s own `./` asset references to the absolute `//` prefix using `scripts/rebase-index.mjs`, then publishes: * The full bundle (excluding `index.html`) under the immutable `s3://…//` prefix, served with `Cache-Control: public, max-age=31536000, immutable`. * The rebased `index.html` at the origin root, served with `Cache-Control: no-cache`. Old `//` prefixes remain available. The publish script can repoint the root index to existing bytes, but `deploy-dev.yml` guards its dispatch against current main; it is not an arbitrary older-commit rollback entry point. ### Release path Start test and production releases from `stralos-core` using `release-test.yml` and `release-prod.yml`. Do not dispatch the member's `release-create.yml` independently: it stages bytes but cannot activate the site. 1. **Produce in dev** — `deploy-dev.yml` builds the environment-neutral bundle, records its source-SHA-keyed artifact in `stralos-artifacts-dev`, and serves it in dev. The immutable artifact must exist before CREATE. 2. **CREATE** — the central orchestrator dispatches admin-web's `release-create.yml` at the release tag with the selected SHA, version, environment, and correlation ID. It calls the core `static-site-release-create.yml`, verifies the frozen dev artifact, and stages its assets and rebased index under `//`. The index remains parked at `//index.html`; `stage-complete.json` records staging completion. CREATE leaves the root serving index unchanged and emits a `release-create-result` artifact for the coordinator. 3. **ACTIVATE** — after the central readiness gates pass, the coordinator's frontend activation copies the parked index to the root with an `If-Match` guard. Test stages `X.Y.Z-rc.N`; production stages the corresponding final `X.Y.Z` from the same frozen bytes. Neither rebuilds the bundle. A merge to admin-web main updates dev only. Use the central release status and recovery procedures in [Release model](https://help.stralos.app/internal/release-model.md) for test and production; the removed `prepare-release.yml` and `promote-static-site.yml` paths are not supported. ### Environment variables Each deploy and promotion job declares a GitHub Environment, so environment-scoped variables shadow repository-level defaults. | Variable | Purpose | | - | - | | `AWS_ADMIN_WEB_ROLE_ARN` | OIDC role assumed by the publish or promote job. | | `AWS_REGION` | AWS region; must be `ap-southeast-2`. | | `ADMIN_WEB_BUCKET` | Serving bucket for the admin SPA. | | `CLOUDFRONT_DISTRIBUTION_ID` | CloudFront distribution invalidated after publication or promotion. | | `ARTIFACTS_BUCKET` | Immutable artifact store; `stralos-artifacts-dev` for all environments. | | `APP_ORIGIN` | Origin probed after publication or promotion to verify the deployed SPA and `/health` route. | | `EXPECTED_AWS_ACCOUNT_ID` | 12-digit AWS account ID used to confirm the authenticated promotion account. | ## Multi-account deployment Core and service Terraform support environment-specific AWS account IDs. When configured, Terraform validates the active account and can assume a deploy role for the target account. The `terraform_deploy_role_name` variable is optional (defaults to `""`). When it is empty, the AWS provider's `assume_role` block is omitted entirely via a Terraform `dynamic` block. Passing an empty string to a static `assume_role` block produces a provider validation error in recent AWS provider versions; the dynamic form avoids this so that local applies and CI runs without a deploy role work without modification. Set `terraform_deploy_role_name` only when deploying to a target account that requires cross-account role assumption. ## Drive-sync deploy prerequisites Before the first apply that enables cloud-drive sync, and before switching `sync_provider_mode` to `live` in Terraform: 1. **Egress** — worker Lambdas run in private subnets. Confirm outbound reachability to `googleapis.com`, `graph.microsoft.com`, and `login.microsoftonline.com` via NAT or equivalent in `stralos-core`. 2. **SSM credentials** — provision the Google service-account key, Microsoft Graph client secret, and client ID out of band. The sync worker and API Lambda roles should be the only readers. Never commit their values. 3. **Auth gate** — unconditional authentication must be active before a sync-capable deployment is reachable beyond the team. 4. **Tenant-isolation gate** — do not onboard another organisation to live drive sync until the organisation-security milestones and their deployment tests pass. Terraform defaults `sync_provider_mode` to `live`. Set it to `fixture` explicitly when deploying infrastructure without provider calls. ## Global API live alias Every triggered Lambda in the Global API stack publishes an immutable numbered version on every apply (`publish = true`) and is invoked through a `live` alias, never through the bare function. This applies to the API Gateway–fronted function and to every scheduled background function. | Function | Alias resource | Trigger | | - | - | - | | `api` | `live` | API Gateway | | `event_flush` | `event_flush_live` (`live`) | EventBridge schedule | | `directory_flush` | `directory_flush_live` (`live`) | EventBridge schedules | | `report_flush` | `report_flush_live` (`live`) | EventBridge schedules (count-gated) | | `sweep` | `sweep_live` (`live`) | EventBridge schedule | | `retention` | `retention_live` (`live`) | EventBridge schedule | Invoking the bare function's ARN always serves `$LATEST`. The alias breaks that coupling: a release is an `update-alias` operation that moves the pointer to an already-published version, and a rollback is the same operation in reverse — no build, no apply, no new bytes. Every alias requires its own invoke grant. An `aws_lambda_permission` on the bare function does not cover the `function:live` qualifier. Without a permission qualified to the alias, the trigger's invoke is refused on the first deploy after the integration moves. Each Terraform stack attaches the permission directly to the alias using `statement_id_prefix` with `create_before_destroy`, so the Terraform plan shows a replace (`+/-`) rather than a destroy-then-create. Aliases ignore Terraform changes to `function_version`. A stage-mode apply publishes versions without moving the aliases. Activation, or the core full-mode post-apply flip, moves them using the `release_functions` output. The `mint` Lambda uses a different scheme: it carries one `mint-for-` alias per authorised caller rather than a single `live` alias. Those alias ARNs are stable across releases, so each caller's IAM grant keeps working without change. The `release_functions` Terraform output lists every aliased function with its most recently published version number and the alias names activation must move. Release tooling reads this output after an apply to flip each alias in one coordinated step. A Terraform test in `terraform/tests/test_release_by_alias.py` enforces these constraints statically against every `.tf` file in the repository. It asserts that every `aws_lambda_function` resource has a corresponding alias entry, that every function publishes a version, that API Gateway integrations resolve through the alias invoke ARN rather than the bare function ARN, and that every trigger invoke permission carries a qualifier equal to the alias name. A new triggered function must be added to the test's alias map or the test fails. ## Global API idle connection management The Global API Lambda releases idle Postgres connections at the end of every invocation. A frozen Lambda container never runs pg's idle timer, so without this step each container holds its sockets open until the container is reaped. The release runs last after the handler, closing only the connections that were idle on entry so that concurrent checkouts are unaffected. Keep `DB_KEEP_IDLE_CONNECTIONS` unset or `false` on the Lambda, including when RDS Proxy is in front of the database. Tenant `set_config` calls pin a client session, so retaining idle clients would retain pinned database connections and recreate connection exhaustion behind the proxy. ## Global API billing enforcement The Global API Terraform stack supports opt-in subscription access enforcement via the `enable_billing_enforcement` variable (default: `false`). When enabled, the Lambda reads from a DynamoDB entitlements table and the `registerBillingAccessGuard` plugin rejects user write operations for organisations without an active subscription. Set `enable_billing_enforcement = true` in the environment's `.tfvars` file only after Stripe billing is configured for that environment. Dev, test, and prod all enable it; dev uses Stripe test mode and prod uses live Stripe. Do not enable enforcement in an environment where billing is not yet provisioned; doing so will block all organisation write operations. When enforcement is on, Terraform creates an IAM policy granting the Lambda `dynamodb:GetItem` on the billing entitlements table (`--billing`), scoped to keys matching the `ORG#*` and `LINK#*` prefixes. `ORG#` holds a company's billing entitlement. `LINK#` holds contractor-company link state, which the shared billing guard reads on every request from a contractor person (a token with a `via` claim). Without `LINK#*`, that read is denied and every contractor request returns `503 billing_unavailable`. The other product APIs that enforce billing grant the same two prefixes. The `BILLING_ENTITLEMENTS_TABLE` environment variable is wired to the Lambda automatically; it is an empty string when enforcement is off, which disables the guard. ## Global API org self-service relay The Global API relays `/api/global/org/*` user-management requests to `stralos-admin-api`'s `/api/admin/org/*`. Those admin-api methods are `AWS_IAM` with a resource-policy `Deny` for any other caller, and a browser cannot SigV4-sign, so the Global API Lambda signs each forwarded request with its own role. | Variable | Type | Purpose | | - | - | - | | `admin_org_url` | `string` | Base URL of `stralos-admin-api`'s API Gateway stage, for example `https://.execute-api..amazonaws.com/main`. It must be the `execute-api` host, not `admin.`, because SigV4 derives its credential scope from the hostname; a plan-time check enforces this. Empty (the default) leaves the relay unconfigured and the org routes return `503`. | | `admin_org_execute_api_arns` | `list(string)` | `execute-api:Invoke` ARNs for admin-api's org methods. Required whenever `admin_org_url` is set. Include both the bare `.../api/admin/org` form and the `.../api/admin/org/*` form, mirroring admin-api's own `local.org_path_arns`. | Setting `admin_org_url` wires `ADMIN_ORG_URL` on the Lambda and attaches the `execute-api:Invoke` policy. The grant is two-sided: admin-api's `org_admin_caller_role_arns` must also name this service's Lambda execution role. Either side alone produces a `403` at API Gateway with no application log on either side. See [Organisation self-service Terraform variables](https://help.stralos.app/internal/admin-console.md#organisation-self-service-terraform-variables) for the admin-api side. ## Global API observability `stralos-api-global` supports opt-in PostHog error tracking and structured HTTP logs. Both signals are disabled by default. Configure the project token through the inline Terraform value when supplied; otherwise Global falls back to the documented SSM parameter. See [Observability](https://help.stralos.app/internal/observability.md) for configuration variables and redaction behaviour. ## Authentication deployment requirements `stralos-api-global` needs EdDSA signing-key material to publish JWKS and issue user or service tokens. When `auth_signing_key_ssm_parameter` is configured, Terraform manages the encrypted SSM SecureString parameter and wires `AUTH_SIGNING_KEY_SSM_PARAMETER` on the Lambda. The post-apply signing-key finaliser generates the Ed25519 JWK and replaces the bootstrap marker. Run the finaliser before enabling an issuer; without key material, JWKS, OTP verification, refresh, and service-token minting remain unavailable. Where the SecureString already exists before this stack owns it (for example, created by `stralos-core` account-baseline), set `adopt_existing_auth_signing_key = true` so Terraform imports the parameter object without replacing its key material. Admin event ingest needs `admin_event_jwks_url`, `admin_event_issuer` (`stralos-admin-`), and `admin_event_audience` (`stralos-runtime-`) to verify the admin event flusher's envelopes. Without them the ingest route returns `503` and new workspaces stay `provisioning`. `POST /api/global/auth/oidc/exchange` needs the OIDC identity-stack SSM parameter names (`oidc_issuer_ssm_parameter`, `oidc_client_id_ssm_parameter`, `oidc_client_secret_ssm_parameter`, `oidc_domain_ssm_parameter`) and `oidc_redirect_uri`; without them it returns `503`. Global, Estimate, Activities, Documents, Rates, Scheduler, and Assistant need `AUTH_JWKS_URL` pointed at the Global API's public JWKS. Their legacy HS256 verification paths are retired. The variable is required in every environment: each service's `buildApp` throws before registering any routes when it is unset or blank, with this message: ```text theme={null} AUTH_JWKS_URL is not configured: set it to api-global's /api/global/.well-known/jwks.json ``` Terraform supplies it from `var.auth_jwks_url` in each environment's `tfvars`; `stralos-local` sets it locally. ## Global SES configuration The Global Terraform source defines SES domain identity, DKIM, bounce/complaint alarms, and gated OTP email delivery. Domain validation pins the permitted domain for each environment and requires the configured From address to belong to that domain; it does not require one exact mailbox. Provision the emitted DNS verification records and confirm SES account readiness before enabling delivery. Do not enable generated-OTP exposure in a shared environment. ## Database connection roles Runtime code supports owner, tenant-runtime, and platform database connections. Local development may fall back to the owner connection when the specialised URLs are absent. Provide dedicated owner, tenant-runtime, and platform connection parameters to services that use the three-role database model. Do not rely on the local owner-connection fallback for tenant request traffic. The database-connected API stacks set `NODE_EXTRA_CA_CERTS` to the AWS Lambda runtime's `/var/runtime/ca-cert.pem` bundle. Operators do not provision this file. Rates sources its owner, tenant-runtime, and platform connection parameter names from the `db-bootstrap` `service_connection_parameters` output. Terraform writes them to `DATABASE_URL_SSM_PARAMETER`, `RUNTIME_DATABASE_URL_SSM_PARAMETER`, and `PLATFORM_DATABASE_URL_SSM_PARAMETER`; an explicit variable can override each default. Use these role-specific TLS URLs rather than the retired shared RDS connection string. The Rates organisation backfill that ran before the constraining RLS migration was cross-database, and its `scripts/backfill-organisation.mjs` has been deleted. Cross-database migrations are not allowed; a new populated Rates database that needs organisation provenance needs a reviewed migration plan, not an ad-hoc command during apply. ## SSM parameter caching Rates, Documents, Estimate, Global, and Scheduler delegate service-discovery SSM reads to the process-wide `ssmParameters` singleton exported by `@stralos-co-nz/api-common/aws/ssm`. Cache identity includes the parameter name, decryption mode, and TTL. Successful reads use a five-minute TTL by default and identical concurrent misses coalesce; errors, empty values, and Terraform placeholders are not retained as usable configuration. ### Cold-start preload Every API Lambda (Activities, Assistant, Billing, Documents, Estimate, Global, Rates, and Scheduler) loads the SSM parameters it reads once at cold start, before it serves any request. Each service lists them in `src/config/ssm-preload.ts`, and `lambda.ts` awaits `preloadApiSsmParameters()` during INIT. The preload uses `ssmParameters.preloadFromEnv()`: batched `GetParameters` calls of up to ten names, with jittered backoff on throttling during INIT only. A variable that is not set is skipped, which is how local development runs with direct URL variables. Each entry is required unless it is marked `required: false`: * A required parameter that cannot be loaded rejects the preload and fails the Lambda INIT, so the environment never accepts traffic and Lambda retries the cold start. * An optional (fail-soft) parameter that cannot be loaded is logged and the feature that reads it degrades. For example, an unreadable `POSTHOG_PROJECT_TOKEN_SSM_PARAMETER` disables telemetry, an unreadable Global `GOOGLE_MAPS_SERVER_API_KEY_SSM_PARAMETER` makes the routing route answer `502`, and an unreadable Rates `ESTIMATE_API_URL_SSM_PARAMETER` makes re-price delivery answer that it is not configured. Rates' Google provider parameters fail only an import that uses that provider. After the preload, SSM-backed readers answer from the in-memory cache. When a cached value expires, the previous value is served while one deduplicated refresh runs in the background (stale-while-revalidate); a failed refresh keeps the stale value. The one blocking re-read follows `invalidate()`. The Rates database handle uses it after a credential rotation, so its next read of the database URL goes back to SSM. Each API's Lambda role grants `ssm:GetParameters` for the preload alongside `ssm:GetParameter`. When you add a `*_SSM_PARAMETER` read, add it to `src/config/ssm-preload.ts`; `test/ssmPreload.unit.test.ts` fails until you do. ### Peer API URL resolution Peer URL resolvers use the same precedence: a direct URL variable, then the SSM parameter named by its companion `*_SSM_PARAMETER` variable. Terraform supplies parameter names—not hard-coded gateway URLs—to service Lambdas. No resolver has a localhost fallback. When neither variable is set, the resolver throws a configuration error at call time rather than calling a hard-coded port. | Caller | Peer | Direct variable | SSM parameter variable | | - | - | - | - | | Activities | Global | `STRALOS_GLOBAL_URL` | `STRALOS_GLOBAL_URL_SSM_PARAMETER` | | Activities | Rates | `RATES_API_URL` | `RATES_API_URL_SSM_PARAMETER` | | Scheduler | Global | `GLOBAL_API_URL` | `GLOBAL_API_URL_SSM_PARAMETER` | | Scheduler | Rates | `RATES_API_URL` | `RATES_API_URL_SSM_PARAMETER` | | Rates | Global | `STRALOS_GLOBAL_URL` | `STRALOS_GLOBAL_URL_SSM_PARAMETER` | | Rates | Estimate | `ESTIMATE_API_URL` | `ESTIMATE_API_URL_SSM_PARAMETER` | | Rates | Activities | `ACTIVITIES_API_URL` | `ACTIVITIES_API_URL_SSM_PARAMETER` | | Rates | Rate extract | `RATE_EXTRACT_URL` | `RATE_EXTRACT_URL_SSM_PARAMETER` | | Estimate | Global | `STRALOS_GLOBAL_URL` | `STRALOS_GLOBAL_URL_SSM_PARAMETER` | | Estimate | Rates | `RATES_API_URL` | `RATES_API_URL_SSM_PARAMETER` | | Estimate | Documents | `DOCUMENTS_API_URL` | `DOCUMENTS_API_URL_SSM_PARAMETER` | | Estimate | Activities | `ACTIVITIES_API_URL` | `ACTIVITIES_API_URL_SSM_PARAMETER` | | Documents | Global | `GLOBAL_API_URL` | `GLOBAL_API_URL_SSM_PARAMETER` | | Documents | Estimate | `ESTIMATE_API_URL` | `ESTIMATE_API_URL_SSM_PARAMETER` | | Documents | Rates | `RATES_API_URL` | `RATES_API_URL_SSM_PARAMETER` | | Global | Estimate | `ESTIMATE_API_URL` | `ESTIMATE_API_URL_SSM_PARAMETER` | | Global | Documents | `DOCUMENTS_API_URL` | `DOCUMENTS_API_URL_SSM_PARAMETER` | | Global | Rates | `RATES_API_URL` | `RATES_API_URL_SSM_PARAMETER` | The error names both variables, for example: ```text theme={null} Activities API URL is not configured: set ACTIVITIES_API_URL or ACTIVITIES_API_URL_SSM_PARAMETER ``` Each deployed Lambda environment sets the `*_SSM_PARAMETER` variables through Terraform, and `stralos-local` sets the direct URL variables for every service. When running a service outside `stralos-local`, set one of the two variables for each peer it calls. A missing URL is a configuration error, not a peer outage, so most callers resolve it outside the `try` around the peer request and do not report it as `502` or `503`. Some advisory reads still catch it and degrade. Activities' price-range lookup logs a warning once and leaves lines unranged. A few Estimate display reads (organisation default hours, export labels, project name, and library activity name search) fall back to their defaults. The Assistant service uses its own `STRALOS_*_API_URL` variables with no SSM companion; see [Assistant service](https://help.stralos.app/internal/assistant-service.md#openapi-operation-discovery). Terraform derives peer parameter paths as `////services//api-url`. Use the corresponding Terraform override only when a peer uses a non-standard path. ## Global API service-discovery parameter The Global API Terraform stack publishes the regional API Gateway stage URL to an SSM String parameter at deploy time. Peer services that call Global at runtime read this parameter through the shared five-minute SSM cache rather than embedding a hard-coded URL. The default parameter path is: ```text theme={null} ////services/global/api-url ``` The path can be overridden via the `api_url_ssm_parameter` Terraform variable. Terraform owns and updates the value whenever the API Gateway stage URL changes; the parameter is tagged with the standard name prefix for the stack. The Terraform output `api_url_ssm_parameter` exposes the resolved parameter name so that peer stacks can consume it by reference. The output `api_url` exposes the raw stage invoke URL directly. ## Directory outbox Global source code atomically records a lower-cased user email and a `create` directory-outbox row. Its dev Terraform stack provisions the DynamoDB writer permissions, directory flusher Lambda, one-minute flush schedule, and daily reconciliation schedule. ## Assistant monitoring Assistant alarms cover Lambda errors and throttles plus API Gateway 5xx responses. See [Assistant service](https://help.stralos.app/internal/assistant-service.md) for the thresholds, topic wiring, and coverage-check boundary. ## Rates API deployment Rates follows the shared automatic-dev, migration-marker, and destructive-guard controls described in [Relational API immutable release path](#relational-api-immutable-release-path). Its alias-release workflow joins the same dev concurrency group as the apply, so an alias move queues behind an in-progress deployment. ### Lambda alias model Every Lambda in the Rates stack publishes an immutable numbered version on every apply and is served exclusively through a `live` alias — never the bare function. This covers all five functions: | Function | Alias resource | | - | - | | `api` (HTTP entrypoint) | `live` | | `rate_import_worker` | `rate_import_worker_live` | | `events` (rate-category consumer) | `events_live` | | `event_flush` | `event_flush_live` | | `retention` | `retention_live` | API Gateway integrates with the `api` function through its `live` alias. The three SQS event-source mappings (rate-import work queue, rate-import dead-letter queue, and rate-category queue) and the two EventBridge schedules (event-flush and retention) all target their respective `live` aliases. Every trigger invoke grant carries the alias qualifier; a permission on the bare function does not cover `function:live` and the trigger would be refused on the first deploy after the alias is added. Every alias sets `lifecycle { ignore_changes = [function_version] }`. A stage-mode plan therefore never moves an alias; only the post-apply alias flip (on full-mode core runs) or a manual `release-alias.yml` dispatch moves it. The stack's `release_functions` Terraform output lists every aliased function with its most-recently published version and the alias names activation must move. `stralos-core`'s full-mode `terraform.yml` reads this output after the apply and flips each listed alias. The `enable_on_activate` output lists any triggers that are created disabled and must be enabled after the alias flip. For the current stack this list is empty: the SQS mappings and EventBridge schedules already exist and stay enabled; they are only retargeted from the bare function ARN to the alias ARN during the initial migration. A Terraform policy test in `terraform/tests/test_release_by_alias.py` enforces these constraints statically against every `.tf` file: every function must have a corresponding alias, every alias must be named `live`, every API Gateway integration must target the alias invoke ARN, every EventBridge target must invoke an alias ARN, every invoke permission must carry the alias qualifier, and the `release_functions` output must name every aliased function. CI runs this test on every pull request. ### Lambda connection pool management Each Lambda container opens up to three Postgres connections per pool (owner, runtime, and platform). This cap prevents a burst of parallel containers from exhausting the database's connection slots — without a bound, ten idle sockets per pool across a fleet of containers can fill the available slots entirely and return 503 to every subsequent request. At the end of each invocation the handler closes every idle connection. A frozen Lambda container never runs pg's idle timer, so connections held by an idle container count against the database's slot ceiling until the container is reaped. Releasing on the way out makes held connections proportional to in-flight requests rather than to the count of containers that have ever warmed up. Keep `DB_KEEP_IDLE_CONNECTIONS` unset or `false`, including behind RDS Proxy. Tenant `set_config` calls pin a client session, so retaining idle clients would retain pinned database connections and recreate connection exhaustion behind the proxy. ### Pricing-rule sweep `rules:check` remains the second command in `MIGRATION_MANIFEST.json` and therefore runs on every deployment. The former nightly `schedule` arm is retired: there is currently no between-deploy sweep for pricing-rule drift caused only by data changes. `RULES_SAFETY_MANIFEST.json` remains in the repository but no workflow currently consumes it. ## Rates Lambda binary responses The Rates Lambda adapter declares `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` in its `binaryMimeTypes` list. API Gateway v1 proxy integrations require an explicit binary media type registration to base64-encode binary response bodies; without it, the XLSX workbook is transmitted as text and arrives at the client corrupted. The XLSX media type is the only binary response the Rates API produces. All other responses are `application/json`. ## Rates API monitoring Rates alarms cover the request Lambda's errors and throttles plus API Gateway 5xx responses. Alarm and OK transitions publish to the shared topic. The topic variable must be a full, same-account SNS topic ARN; each non-dev environment supplies its own value. ## Estimate monitoring Estimate Terraform defines CloudWatch alarms for its two Lambda functions: the HTTP entrypoint (`api`) and the lifecycle pending-transition sweeper (`sweep`). Each function carries an absolute **Errors** alarm and a **Throttles** alarm, both using `treat_missing_data = notBreaching`. The HTTP entrypoint tolerates up to four errors per five-minute period before paging; every other function (including the sweeper) pages on the first error. The entrypoint also carries a guarded **error-rate** alarm using metric math (`IF(inv >= 10, 100 * err / inv, 0)`) that trips at 20% and prevents divide-by-zero false alerts on idle periods or very low traffic. The sweeper additionally carries a **dead-man's-switch** alarm on `AWS/Events SuccessfulInvocationAttempts` (scoped to its EventBridge rule) with `treat_missing_data = breaching`. A 30-minute window covers up to two missed ticks of the `rate(10 minutes)` schedule before paging; this catches both a stopped schedule and a firing schedule whose Lambda invoke is consistently failing. An **API Gateway 5xx** alarm on `AWS/ApiGateway 5XXError` closes the gap where the application returns an HTTP 5xx as a successful Lambda invocation (the Lambda Errors metric stays zero). Its threshold is one 5xx per five-minute period. All alarms fire the shared alarm SNS topic managed by `stralos-admin-api`. A Terraform `check` compares the alarm map with the Estimate Lambdas already live in AWS and reports a non-blocking plan warning for a gap. A function introduced by the current apply is visible only on a later plan, so this check is drift detection rather than a deployment gate. ## Documents monitoring Documents alarms cover the API, pipeline and sync workers, DLQ consumers, scheduled sweep, API Gateway 5xx responses, and account-wide cost guards. See [Stage observability](https://help.stralos.app/internal/document-ai-pipeline.md#stage-observability) for the canonical thresholds and coverage boundaries. ## Admin API release model A merge to `main` deploys dev automatically. The `deploy-dev.yml` push arm applies the same publish-then-apply chain as a manual dispatch; see [Automatic and deliberate dev deploy](#automatic-and-deliberate-dev-deploy) for the guard that prevents a push from taking or lifting the writer cutover unattended. The admin API's main Lambda integrations use a `live` alias rather than `$LATEST`; the optional dedicated signup integration targets its separate signup Lambda. Publishing the main API artifact with `update-function-code --publish` creates an immutable numbered version without moving the alias. `scripts/finalize-deploy.sh` owns the release step. After promoting the API bytes it calls `scripts/promote-alias.sh`, which moves `live` to the version just published and verifies both the alias target and that version's `CodeSha256`. A failure stops the deployment before the `artifacts_promoted` marker is recorded. Terraform cannot select the promoted artifact: `aws_lambda_function.api` ignores `source_code_hash`, so a Terraform-published version would snapshot whatever code was already on the function. The finaliser is the component that knows the correct target version. The alias update uses the current revision ID for optimistic concurrency and refuses `$LATEST`. Main-API invoke permissions are qualified to `live` and use create-before-destroy lifecycle behaviour, preventing a 403 window when the grant changes. See [Admin console — Lambda alias release](https://help.stralos.app/internal/admin-console.md#lambda-alias-release) for the deployment-rail proof. ### Central release integration (self-activating member) `stralos-admin-api` participates in the `stralos-core` central release as a **self-activating member**: it owns both the CREATE and ACTIVATE phases rather than delegating them to the core orchestrator. The central release dispatches `release-create.yml` (CREATE phase) and `release-activate.yml` (ACTIVATE phase) at the release tag. These entry-point workflows delegate immediately to `release-create-admin.yml` and `release-activate-admin.yml`. #### CREATE phase (`release-create-admin.yml`) CREATE does everything that can happen ahead of the release without changing what users are served: 1. **Freeze** — test freezes the promotion manifest for the candidate Lambda artifact (`X.Y.Z-rc.N`). Prod re-derives the same manifest from `source_version` (which is content-addressed), refusing a manifest it had to create. This means prod only promotes what test froze. 2. **Stage** — calls `stralos-core`'s `terraform.yml` in `mode: stage`. The apply ships the new ZIP to the `migrate` and `retention` functions and applies config and infrastructure changes, but moves no alias. `api:live` and `retention:live` keep serving the old code throughout CREATE. Infrastructure that must not land before its code keeps the legacy `terraform-deploy.yml` path. 3. **Record** — writes the activation payload (the `migrate` function's published version and `CodeSha256`, and the staged `retention` version). Also prunes old versions that the last activation left behind, since ACTIVATE defers pruning to the next CREATE. `dev` is accepted for a dry run: no freeze occurs, and the stage builds the ZIP from the checkout exactly as `deploy-dev.yml` does. The `CREATE` and `ACTIVATE` concurrency groups (`admin-terraform-` for test and prod, `deploy-dev` for dev) prevent any CREATE apply or ACTIVATE from interleaving with a `terraform-deploy.yml` apply or a dev deployment. The `terraform/CONTRACT_OK` file lists the resource replacements the stage plan policy permits. The API Gateway deployment is create-before-destroy and may be replaced when routes change; `terraform_data` guards have no runtime effect. All other deletes or replacements are refused by the plan policy. #### ACTIVATE phase (`release-activate-admin.yml`) ACTIVATE runs as a single job, minimising the number of runner queues the release pays for: 1. **Verify** — confirms the environment still holds exactly what CREATE applied: the `migrate` function version and its `CodeSha256`, and the staged `retention` version. A CREATE or deployment that applied since would have replaced these values, and the verification refuses to proceed. 2. **Quiesce** — captures every writer's current concurrency state, then gates writers when `drizzle/CUTOVER_REQUIRED` asks for it, using `scripts/quiesce-deploy.sh` as every deployment path does. 3. **Finalise** — migrates through the `migrate` function, promotes the migrated ZIP onto the writers, moves `api:live`, reopens writers, and health-checks, using `scripts/finalize-deploy.sh`. The finaliser reads Terraform stack outputs from the state object via `TF_OUTPUTS_FILE` (a single S3 read) instead of running `terraform init` on the critical path. Version pruning is deferred to the next CREATE (`FINALIZE_PRUNE=skip`). 4. **Move `retention:live`** — points `retention:live` to the version CREATE staged. ACTIVATE runs on the Lambda-compute runner when CREATE recorded `cutover_required: false` (approximately 17-second queue time). When a cutover is armed, it uses the EC2 runner for its writer-drain headroom. On failure after writer state has been captured, ACTIVATE runs `cutover-recover.yml`. Before the migration boundary the recovery reopens the old bytes; after it the recovery resumes finalisation. Either way the release stops at `failed` status for triage. #### `scripts/release-state.sh` `release-state.sh outputs ` reads the Terraform stack's outputs directly from its S3 state object. It provides the `TF_OUTPUTS_FILE` that ACTIVATE's finaliser reads in place of `terraform output`, eliminating a `terraform init` and provider download from the critical path. The script reads the backend bucket and key from `terraform/backend-.hcl` so the two cannot disagree. Only the `.outputs` section is written to disk; the rest of the state object is never retained. ## Admin API monitoring The admin Terraform defines CloudWatch coverage for the admin Lambda plane and API Gateway. Alarms publish to `${name_prefix}-alarms`, for example `stralos-dev-admin-alarms`. Terraform creates no subscription by default. See [Admin console monitoring](https://help.stralos.app/internal/admin-console.md#monitoring) for thresholds and the secret-sync dead-man's switch. ## Scheduler API monitoring Scheduler alarms cover Lambda errors and throttles plus API Gateway 5xx responses. Alarm and OK transitions publish to the shared topic. The topic must be a full, same-account SNS topic ARN, and each non-dev environment supplies its own value. The live-inventory coverage check is non-blocking drift detection; a same-apply function addition becomes visible on a later plan. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.