Shared API topology
The Activities, Assistant, Documents, Estimate, Global, Rates, and Scheduler stacks currently define:- regional REST API Gateway v1 with stage name
main; - Lambda proxy integrations;
- structured API access logging and X-Ray where configured;
- optional association with the environment’s regional Web ACL;
- a stable
region_key, initiallyau-nz, used in names, SSM paths, and state keys; and - S3-native Terraform state locking.
<region_key>/<service>/terraform.tfstate and require Terraform 1.10 or later because the backend uses use_lockfile = true.
Canonical deployment path
GitHub Actions is the canonical deployment path. Service workflows call the reusable Terraform workflow instralos-core, then run the configured migration and Terraform stages. Dev deployments build from the triggering source revision; release promotions use the immutable artifact path described below.
The configured progression is dev, test, then prod. A job that targets a GitHub Environment is approval-gated only when that environment and its protection rules exist. Confirm those rules in GitHub before relying on an approval gate.
Relational API immutable release path
Activities, Documents, Estimate, Global, Rates, Scheduler, and Assistant publish immutable source-SHA artifacts on the guarded dev path. Test and production are coordinated bystralos-core’s release-test.yml and release-prod.yml. Member prepare-release.yml and release-event promotion paths are retired.
The central release separates staging from activation:
- Publish dev — the member verifies its source ref and publishes the immutable artifact set before applying dev.
- CREATE — core resolves the source, freezes its promotion identity, and dispatches the member’s
release-create.ymlat the annotated release tag. API CREATE usesterraform.ymlin stage mode. It publishes immutable versions without moving aliases or running migrations. - ACTIVATE — core runs the service migration through the configured runner, then moves the recorded aliases and checks the edge. The frontend flip waits for the API activation barrier.
main does not touch test or production. Do not dispatch retired workflows to bypass a failed central release. See Release model for source selection, barriers, and frontend activation.
Migrating APIs declare a MIGRATION_MANIFEST.json; non-migrating services omit migration inputs from CREATE. The member’s workflow supplies its manifest and migration hash paths so staging and activation use the same migration identity.
Activities and Scheduler request owner access. Rates requests owner and platform access. Global requests owner and platform access. Documents requests owner and platform access. Estimate requests owner, platform, and global-owner access. The pinned database-bootstrap runner resolves connections per manifest entry, exposes them only inside that entry’s subprocess, and executes the entry’s declared command. It defaults older manifests without a database_access field to owner access.
Migration execution
For a service with aMIGRATION_MANIFEST.json, the dev full-mode Terraform path runs migrations before apply. On the central test and production path, CREATE stages without migrations; ACTIVATE runs them before moving aliases. There is no content-hash skip gate; the migration tool itself applies only pending entries. The workflow records the migration-content hash as evidence so environments can be compared, but the hash does not decide whether a run occurs.
Protocol 2 executes the manifest inside the environment’s VPC CodeBuild project. It reads role-specific connection URLs from SSM by convention and does not run Terraform. The deployed source tree is used for dev; promotions use the frozen migration source. A run records started, per-entry, and completed evidence in the migration ledger, so an interrupted release cannot be mistaken for a completed one.
Write outages are armed by cutover markers, not by the manifest’s migration class. The shared workflow pinned by these services reads both drizzle/CUTOVER_REQUIRED and versioned markers such as drizzle/0036_CUTOVER_REQUIRED. Non-comment lines name migration files. The workflow subtracts migrations completed by another release in the same environment. Completion recorded for the same release does not disarm a retry of that release. Missing or unreadable ledger evidence retains outage arming.
When armed, deployment stops and drains writers before migration, then restores their previous concurrency after successful deployment. Markers can remain in source: a later release can exclude migrations whose completion was recorded for another release in that environment. Marker detection fails closed when it cannot establish completion.
The shared database-bootstrap Terraform stack is provisioned separately through stralos-core’s db-bootstrap.yml. A push to its main path applies dev; other environment applies are explicit, plan-reviewed dispatches. Ordinary service migration runs do not apply or rotate that shared database infrastructure.
Automatic and deliberate dev deploy
Activities, Assistant, Documents, Estimate, Global, Rates, Scheduler, and Admin API rundeploy-dev.yml on pushes to main. The push path uses the same immutable publish-then-apply chain as manual deployment, with additional pre-flight guards:
- A superseded push is refused when its SHA no longer matches the current
maintip. scripts/auto-deploy-guard.mjschecks every commit through the pushed SHA for cutover marker changes.- Adding, modifying, removing, or renaming a cutover-marker path anywhere in that range refuses automatic publication. An add-then-remove sequence is still caught even when the net diff is empty.
- Unreadable or invalid live SHAs, incomplete commit enumeration, diverged or behind history, and API failures refuse the automatic path.
- Live commit source — The other services read the commit currently served from their
/versionendpoint. The Admin API has no/versionroute, so its guard reads the head SHA of the newest successfuldeploy-dev.ymlrun onmainfrom the GitHub Actions API. - Marker presence arms immediately — For the other services a versioned marker in the tree does not block a push once its migration is recorded in the ledger. The Admin API cutover marker (
drizzle/CUTOVER_REQUIRED) gates every writer on every deploy while it is present, so the guard refuses any push that finds the bare marker at the pushed commit. A marker merged but not yet deleted blocks all subsequent pushes until it is deleted. - Range rule — Both the Admin API guard and the other services refuse a push when any commit in the range added, changed, removed, or renamed a cutover-marker path. A delete is caught for the same reason as an add: lifting a writer outage unattended is as deliberate an act as taking one.
ref input is an assertion, not a historical commit selector. Dispatch from main; the workflow verifies the input against its run commit and chains immutable publication before apply. A failed or skipped publish fails completion.
Each repository serialises dev deployment with cancel-in-progress: false. This prevents overlapping applies but does not guarantee dispatch order; GitHub retains at most one pending run in a concurrency group.
Migration class and writer outage
All seven relational services keepMIGRATION_MANIFEST.json at additive-online. The class is validated release metadata; it does not arm a write outage. A release that needs writers stopped must commit a supported cutover marker at the deployed SHA and name the affected migration file.
The dev apply, dev plan, test, and prod call sites in every service pass stop_writers_function and cutover_concurrency_ignored_by_terraform: true. Each API Lambda also ignores Terraform changes to reserved_concurrent_executions. This standing wiring arms an outage when marker reconciliation cannot prove every named migration was completed by another release in that environment. See API writer cutover for the current marker and recovery procedure.
Destructive migration guard
Every pull request in these services runsscripts/check-destructive-migrations.mjs. It refuses a changed drizzle/*.sql file containing DROP TABLE, DROP COLUMN, ALTER COLUMN … TYPE, SET NOT NULL, RENAME COLUMN, or RENAME TO unless a cutover marker names that exact file. All seven pull-request guards recognise the unversioned marker. Rates also accepts versioned markers with a matching migration prefix; the other six pull-request guards still require the unversioned form. This differs from the automatic-deploy guards and pinned apply workflow, which recognise both forms. DROP CONSTRAINT, DROP INDEX, DROP POLICY, and DROP NOT NULL do not trigger this bridge guard.
Run the same check locally against the current base:
Release by alias (Estimate, Rates, and Assistant)
The Estimate, Rates, and Assistant API stacks define alive Lambda alias. API Gateway integrates with the alias’s invoke ARN, not the bare function’s. The bare function always serves $LATEST, so without the alias a version change would require a full Terraform apply and rollback would be impossible without one.
With the alias in place, a release is a single update-alias call that points live at an already-published version. A rollback is the same operation in reverse, at the same cost — no build, no apply, no new bytes.
Each stack’s release-alias.yml workflow performs the move:
The workflow resolves what is currently served, validates the requested version is published, conditionally moves the alias with an optimistic-concurrency revision guard, then proves the move through the edge URL before recording a job summary.
$LATEST is refused as a target. The alias exists specifically so that $LATEST is never what API Gateway serves.
Terraform re-points the alias on the next apply. The alias is re-pointed to the version Terraform just published whenever a Terraform apply runs. release-alias.yml is for the window between applies — moving the alias between two already-published versions without triggering a full deployment cycle.
The API Gateway invoke permission is qualified to the alias (qualifier = aws_lambda_alias.live.name). A permission on the bare function does not cover function:live; without the qualifier, API Gateway’s invoke returns 403 on the first deploy after the integration moves. The permission uses statement_id_prefix (not statement_id) with create_before_destroy = true so that the new grant exists before the old one is removed and there is no 403 window during the transition.
An alias move changes only the code version. Database migrations, environment variables, IAM policies, and API Gateway or stage configuration still require a Terraform apply.
Assistant API
The Assistant service uses the same guarded dev path as the other relational APIs — a push tomain applies dev through deploy-dev.yml. For test and prod, releases are orchestrated centrally by stralos-core (release-test.yml / release-prod.yml) dispatching release-create.yml in this repository; the per-repo prepare-release.yml freeze and terraform-deploy.yml promote steps are retired. The Assistant carries its own MIGRATION_MANIFEST.json with one additive-online entry for the assistant service.
The service deploys on the nodejs24.x managed Lambda runtime. It uses the response-streaming Lambda invocation path for SSE turns. Its Global, Rates, Estimate, and Scheduler URLs all resolve through the regional application origin supplied by regional_app_base_url, rather than through separate peer-service SSM lookups.
Assistant uses the shared Lambda alias release mechanism, with additional identity checks described below.
Terraform manages three runtime settings under /<project>/<environment>/<region_key>/services/assistant/: ai-provider, bedrock-model-id, and the enabled kill switch. The Lambda role can invoke the configured cross-region Bedrock inference profile and its foundation models in ap-southeast-2 and ap-southeast-4.
Alarm topic
All CloudWatch alarms route to the SNS topic identified byalarm_sns_topic_arn. The variable defaults to the dev-account topic; a non-dev apply must override it with that environment’s own topic ARN.
Two validations enforce a correct value at plan time:
- ARN format — the value must be a full SNS topic ARN matching
arn:aws:sns:<region>:<account-id>:<topic-name>. A subscription ARN or any other non-topic ARN silently never delivers notifications and is rejected. - Same-account guard (fail-closed) — the account segment of the topic ARN must match the account the stack is applying into. A mismatch most commonly means the dev default was left in place for a test or prod apply. Because the dev topic’s resource policy grants publish only to the dev account, a cross-account topic ARN would cause alarms to silently never fire. The plan fails on a mismatch rather than applying and leaving alarms non-functional.
Alias workflow and proof
release-alias.yml moves the LIVE Lambda alias by command, without a Terraform apply or a new build. For the Assistant service, release-alias.yml targets dev only; test and prod rollback runs through stralos-core’s rollback-api.yml, which moves every alias in the stack and writes the release record. Dispatch release-alias.yml with a target environment and a version — a published Lambda version number, previous (the highest published version below the current live version, which is what a rollback means), or current (report only, no move).
The assistant is not on the regional CloudFront distribution; https://au-nz.<env>.stralos.app/api/assistant/* falls through to the SPA origin and returns 200 vacuously. The workflow resolves the service’s own API Gateway stage URL from the SSM parameter Terraform writes at /stralos/<env>/au-nz/services/assistant/api-url, and probes GET <url>/api/assistant/version after moving the alias. The proof requires the response to carry the exact version and commit values stamped onto the target Lambda version. lambdaVersion in the response body is compared against the actual alias target to confirm the pointer moved to the right version, not merely any live version.
Before moving the alias, the workflow reads SERVICE_VERSION and SERVICE_COMMIT from the target version’s immutable Lambda configuration (get-function-configuration --qualifier <version>). A target that predates the M4 deploy-identity stamp (values of None or unknown) is refused before the alias is touched. This means rolling back to a version published before the first M4 apply is not supported via alias move; redeploy via Terraform deploy at the desired commit instead.
The next Terraform apply after any alias move re-points the alias to the version it publishes. The alias move command is for the space between applies.
Deploy-identity stamp
Terraform stampsSERVICE_VERSION and SERVICE_COMMIT onto the API Lambda at apply time. GET /api/assistant/version returns these values alongside lambdaVersion — the Lambda version number the runtime is actually executing (from AWS_LAMBDA_FUNCTION_VERSION). The route is auth-exempt and sets Cache-Control: no-store. It is an infra probe, not a documented API route, and is absent from the OpenAPI descriptors.
lambdaVersion is assistant-specific. Two consecutive applies of the same main commit publish two Lambda versions carrying the same SERVICE_VERSION and SERVICE_COMMIT; lambdaVersion is the only value that differs between them, which is why the alias-move proof uses it to confirm the pointer moved.
Deterministic deployment package
scripts/build-lambda.sh produces a byte-identical ZIP on every build of the same source. The script stages the three output files (lambda.mjs, lambda.mjs.map, routeCatalog.json) in a temporary directory, normalises all mtime values to a fixed timestamp, and feeds an explicitly sorted file list to zip -X (which drops extended timestamp and uid/gid fields). The destination ZIP is removed before each run so that zip cannot append to an existing archive. Two builds of the same source produce the same bytes, which is required for the R5 promotion model where the lambdaArtifact.digest in the promotion manifest must remain stable across rebuilds.
Post-release smoke
smoke.yml is a manual workflow_dispatch that configures the optional Claude Code runner and smokes the AI gateway after a release reaches test or prod. Dispatch it with environment=test or environment=prod after every release that reaches either environment:
deploy-dev.yml’s configure-claude-code job after every dev deploy and does not need a separate dispatch. The central release does not yet run the smoke automatically; a post-activate hook is a planned follow-up.
Documents API runtime and provider requirements
The Documents service runs on thenodejs24.x Lambda runtime. The Terraform AWS provider must be ~> 6.19 or later; earlier 5.x or pre-6.19 provider versions cannot express the nodejs24.x runtime string and will fail to plan. The repository’s .node-version file pins Node.js 24 for CI and local builds.
Deploy-time environment variables
Terraform stamps these variables into the Lambda on every apply:stralos-core’s terraform.yml exports TF_VAR_service_commit for both dev deploys and promotions, so the field is populated automatically in the standard CI path.
Bundle budget
esbuild.config.mjs limits the request-path bundle, dist/lambda.mjs, to 15 MiB. pnpm run build:no-cache fails when the bundle exceeds that budget, so the Lambda packaging build stops before creating its ZIP. Use narrow SDK entry points instead of package-root imports when a dependency exposes generated clients; worker bundles follow the same import discipline but do not have this hard limit.
Live alias release model
Every Lambda in the Documents stack is published on every Terraform apply (publish = true) and invoked through a live alias, never the bare function. The bare function always serves $LATEST, so without an alias a version published ahead of activation would go live the moment the apply landed and a rollback would require a full apply.
The API Lambda (api) carries its own live alias in api.tf. All 26 non-API functions — the 9 stage workers, 9 DLQ consumers, sweep, sync dispatch, sync worker, sync DLQ consumer, drive fetch worker, drive fetch DLQ consumer, estate events, and retention — are aliased by a single for_each resource (aws_lambda_alias.function_live over local.live_aliased_functions in release.tf). Every SQS event-source mapping and every EventBridge schedule invokes the alias; none targets the bare function.
The release_functions Terraform output lists all 27 aliased functions (API plus the 26 above), each with its function_name, the version the apply published, and the alias names that activation moves. The enable_on_activate output is empty: no triggers are created disabled awaiting activation.
A Terraform test in terraform/tests/test_release_by_alias.py enforces these constraints statically against every .tf file: every function must publish a version, every function must be in local.live_aliased_functions (or be api), every SQS mapping and EventBridge Lambda target must invoke an alias, and both aliases must carry ignore_changes = [function_version] so a stage-mode plan never moves them.
- Stage — Terraform publishes immutable versions without moving existing aliases. Activation uses
release_functionsto move them. - Release without deploy — release tooling calls
update-aliasto moveliveto any earlier published version. No Terraform apply is required and no new Lambda version is created. - Rollback — point
liveback to the previous version number. The cost and mechanics are identical to a forward release.
live alias qualifier. A permission on the bare function does not cover function:live, so Terraform uses statement_id_prefix (not a fixed statement_id) and create_before_destroy = true on the permission resource. This ensures the new grant exists before the old one is removed, so there is no window in which API Gateway holds no permission to invoke the function. Each EventBridge schedule grant is likewise qualified to the alias and uses statement_id_prefix with create_before_destroy.
Billing access enforcement
The Documents API includes a per-environment flag that gates whether the billing entitlements guard is active.
Set
enable_billing_enforcement = true in the environment’s .tfvars file only after stralos-api-billing is configured and the DynamoDB billing entitlements table for that environment has been provisioned. Enabling the flag before the table exists causes every API request to fail at the entitlements check.
Set the flag independently in each environment only after its billing service and entitlements table are ready.
Manual helper limitation
Servicescripts/deploy.sh helpers rebuild the on-disk Lambda package and run Terraform apply against an already prepared working directory. They do not select or initialise the correct remote backend, and they do not replace the migration orchestration in the reusable workflow. Use them only as a reviewed recovery path with the target backend and migration state already verified.
Shell publication
The shell releases through separate produce and activate steps. Each repository publishes a commit- or version-addressed candidate.stralos-core gates and writes the remote composition, while the shell owns the final index.html activation.
Trigger matrix
An
rc/X.Y.Z pre-release supplies version X.Y.Z to test, while a vX.Y.Z full release supplies X.Y.Z to prod. These prefixes are operator conventions, not workflow guards: both parsers remove an optional rc/ and then an optional v, and accept the result when it is stable SemVer. The prod producer is dormant until the repository variable PROD_DEPLOY_ENABLED is true.
Dev path
Every push tomain enters deploy-dev.yml. verify-ref proves the requested ref is the run commit, the reusable build job produces one handoff, and publish-immutable writes the deterministic release archive. The subsequent dev job writes the parked shell/<sha>/ serving bundle without changing the root index.html. Finally, the fail-closed deployed job requires all four preceding jobs to have succeeded, preventing a skipped producer from appearing as a successful run. Producing alone does not activate the shell in dev; it makes the build available for activation.
To activate the published dev bundle in the dev environment, dispatch activate-dev.yml:
sha input must be exactly 40 lowercase hexadecimal characters. The workflow takes the composition lease, runs the shared compatibility gate against the live remote set, verifies the parked shell/<sha>/index.html, writes root index.html and the candidate’s built-against.json when present through the fenced writer, and invalidates CloudFront.
A push to main runs the build and publish-immutable jobs but not activation. To roll back, first produce the older ref with deploy-dev.yml if its immutable shell bundle is absent, then dispatch activate-dev.yml with that commit’s SHA. The older commit does not need to be pushed to main.
The shared concurrency group (group: publish-shell-dev, cancel-in-progress: false) serialises activation with the producer’s dev serving job so they cannot write the shell bucket concurrently. The producer declares it at job level; the activation workflow declares it at workflow level.
Composition lease
Three workflows across two repositories can change what dev serves:activate-dev.yml in stralos-shell, and gate-dev.yml and reconcile-dev.yml in stralos-core. A GitHub concurrency group cannot serialise across repositories, so all three take one S3 composition lease before reading or writing the root index. The lease lives in REMOTES_BUCKET.
When activate-dev.yml runs from a manual dispatch, it takes the lease itself. If another writer already holds the lease, the run exits as skipped and stays green. A held lease is a queueing outcome, not an incident.
When stralos-core’s reconcile-dev.yml dispatches this workflow it passes a correlation input (<lease-fence>-<uuid>). The activation run adopts the reconciler’s lease by presenting that exact correlation. Adoption requires the current unconsumed correlation and consumes that correlation. A later write must still match the lease’s current correlation.
An adopted run does not release the lease; the reconciler still has to confirm the root index moved.
Both root pointers carry the fence and lease owner under which they were written. The fenced writer checks the current lease and correlation before each write, uses ETag-conditional updates, and refuses a higher stored fence or an equal fence with a different owner. An adopted run shares the reconciler’s owner and fence; the current-correlation check constrains its writes. The two root pointers are still separate writes, not an atomic transaction.
Build workspace
The production build materialises a one-member pnpm workspace containing onlystralos-shell. It installs @stralos-co-nz/ui and @stralos-co-nz/domain from the registry at the exact package pins and lockfile versions committed by the shell. No feature source is compiled into the host, and the build records composed_with as an empty object.
Owner repositories are checked out with WORKSPACE_CHECKOUT_TOKEN only in the isolated integration-fixture job. That job validates source compatibility but contributes no files to the production artifact. Contract-v2 production rejects UI and Domain SHA overrides.
Test and prod path
Test and prod use the same ordered, two-writer activation:- Produce the shell and chosen remotes at the target version. Producing writes a version-addressed serving bundle and parked
index.html; it does not intentionally change the live root pointer. Treat a version as single-use: the test produce claims the version with a write-oncemanifest.sha256.jsonbefore the first object write, and any subsequent produce that diverges byte-for-byte from that claim is refused before uploading anything. Prod does not build:deploy-prod.ymlpromotes the bytes test served for that version, verified against test’s claim and completion marker, and copies them write-once. See Prod. - Dispatch
stralos-core’sgate-test.ymlorgate-prod.ymlwith the candidate shell version and remote-version set. The gate checks each remote’s built-against evidence against the candidate shell and, on aGOverdict, dispatches the single-writer assembler for/remotes-manifest.json. - Wait for the dispatched
remotes-manifest.ymlrun to complete successfully. - Dispatch the shell’s matching
activate-test.ymloractivate-prod.yml. It gates the candidate shell against the manifest, promotes the parked document to rootindex.html, and invalidates CloudFront.
/remotes-manifest.json, browsers using the current root document may observe the new remote set before the shell swap. Keep the interval short and choose a remote set compatible with both the current and candidate hosts when uninterrupted operation is required.
Treat the gate-to-activation period as an exclusive operator transition: do not run another core gate or manifest assembly for that environment until shell activation completes. Shell activation checks the live manifest before copying the root document, but the check is not bound to a manifest version and the core and shell workflows do not share a cross-repository lock. Roll back by repeating the gate and activation with versions that were already produced.
Re-run behaviour
Do not reuse a shell release identifier. The dev archive write fails when its key already exists. The test produce claims the version before writing any objects:manifest.sha256.json is written with --if-none-match '*' so only one run can create it, and every subsequent run must reproduce the identical byte set or be refused. A mismatch produces the occupied-by-different-bytes signal, which is unrepairable by retry: cut a new version. A transient S3 failure is distinct from a mismatch and is retryable. A prod promote re-verifies the test claim before writing, and prod writes are write-once, so a re-run can only complete the same bytes. Re-running activation also rewrites the live root objects and invalidates CloudFront; it is not a no-op.
Environment variables
Each publish job declares a GitHub Environment, so an environment-scoped value shadows a repository-level variable of the same name. Configure the applicable values separately for dev, test, and prod.
The workflows fail closed when required variables are unavailable or the authenticated account does not match
EXPECTED_AWS_ACCOUNT_ID.
See Release model for the produce, compatibility-gate, and activation contracts.
Admin console static site release path
stralos-admin-web is a standalone Vite SPA, separate from the shell Module Federation graph. Dev publishes an immutable bundle. Test and production participate in the central release CREATE and ACTIVATE lifecycle, targeting S3 and CloudFront.
Trigger matrix
Dev path
deploy-dev.yml triggers on every push to main. It calls deploy.yml through workflow_call for the build, then publishes the immutable source-SHA-keyed artifact and runs scripts/publish-admin-web.sh to sync the bundle to the dev serving bucket with appropriate cache headers, invalidate CloudFront, and verify the SPA and same-origin /health route. A merge to main deploys admin-web to dev automatically, consistent with the shell and feature remotes.
deploy.yml itself still builds and verifies the production bundle but does not publish or serve anything. It exists as the reusable build target that deploy-dev.yml calls. The central release stages the existing dev artifact.
The ref input on a manual dispatch of deploy-dev.yml is an assertion rather than a selector: the run fails unless it resolves to the current main tip, so what dev serves is always a commit on main.
The production bundle is environment-neutral: the client defaults to same-origin /api/admin. No environment hostname is embedded in the bundle bytes; the same artifact is promoted to test and prod.
Serving layout. The bundle is built with a relative Vite base (VITE_APP_BASE=./), making it location-independent. scripts/publish-admin-web.sh rewrites index.html’s own ./ asset references to the absolute /<sha>/ prefix using scripts/rebase-index.mjs, then publishes:
- The full bundle (excluding
index.html) under the immutables3://…/<sha>/prefix, served withCache-Control: public, max-age=31536000, immutable. - The rebased
index.htmlat the origin root, served withCache-Control: no-cache.
/<sha>/ prefixes remain available. The publish script can repoint the root index to existing bytes, but deploy-dev.yml guards its dispatch against current main; it is not an arbitrary older-commit rollback entry point.
Release path
Start test and production releases fromstralos-core using release-test.yml and release-prod.yml. Do not dispatch the member’s release-create.yml independently: it stages bytes but cannot activate the site.
- Produce in dev —
deploy-dev.ymlbuilds the environment-neutral bundle, records its source-SHA-keyed artifact instralos-artifacts-dev, and serves it in dev. The immutable artifact must exist before CREATE. - CREATE — the central orchestrator dispatches admin-web’s
release-create.ymlat the release tag with the selected SHA, version, environment, and correlation ID. It calls the corestatic-site-release-create.yml, verifies the frozen dev artifact, and stages its assets and rebased index under/<version>/. The index remains parked at/<version>/index.html;stage-complete.jsonrecords staging completion. CREATE leaves the root serving index unchanged and emits arelease-create-resultartifact for the coordinator. - ACTIVATE — after the central readiness gates pass, the coordinator’s frontend activation copies the parked index to the root with an
If-Matchguard. Test stagesX.Y.Z-rc.N; production stages the corresponding finalX.Y.Zfrom the same frozen bytes. Neither rebuilds the bundle.
prepare-release.yml and promote-static-site.yml paths are not supported.
Environment variables
Each deploy and promotion job declares a GitHub Environment, so environment-scoped variables shadow repository-level defaults.Multi-account deployment
Core and service Terraform support environment-specific AWS account IDs. When configured, Terraform validates the active account and can assume a deploy role for the target account. Theterraform_deploy_role_name variable is optional (defaults to ""). When it is empty, the AWS provider’s assume_role block is omitted entirely via a Terraform dynamic block. Passing an empty string to a static assume_role block produces a provider validation error in recent AWS provider versions; the dynamic form avoids this so that local applies and CI runs without a deploy role work without modification. Set terraform_deploy_role_name only when deploying to a target account that requires cross-account role assumption.
Drive-sync deploy prerequisites
Before the first apply that enables cloud-drive sync, and before switchingsync_provider_mode to live in Terraform:
- Egress — worker Lambdas run in private subnets. Confirm outbound reachability to
googleapis.com,graph.microsoft.com, andlogin.microsoftonline.comvia NAT or equivalent instralos-core. - SSM credentials — provision the Google service-account key, Microsoft Graph client secret, and client ID out of band. The sync worker and API Lambda roles should be the only readers. Never commit their values.
- Auth gate — unconditional authentication must be active before a sync-capable deployment is reachable beyond the team.
- Tenant-isolation gate — do not onboard another organisation to live drive sync until the organisation-security milestones and their deployment tests pass.
sync_provider_mode to live. Set it to fixture explicitly when deploying infrastructure without provider calls.
Global API live alias
Every triggered Lambda in the Global API stack publishes an immutable numbered version on every apply (publish = true) and is invoked through a live alias, never through the bare function. This applies to the API Gateway–fronted function and to every scheduled background function.
Invoking the bare function’s ARN always serves
$LATEST. The alias breaks that coupling: a release is an update-alias operation that moves the pointer to an already-published version, and a rollback is the same operation in reverse — no build, no apply, no new bytes.
Every alias requires its own invoke grant. An aws_lambda_permission on the bare function does not cover the function:live qualifier. Without a permission qualified to the alias, the trigger’s invoke is refused on the first deploy after the integration moves. Each Terraform stack attaches the permission directly to the alias using statement_id_prefix with create_before_destroy, so the Terraform plan shows a replace (+/-) rather than a destroy-then-create.
Aliases ignore Terraform changes to function_version. A stage-mode apply publishes versions without moving the aliases. Activation, or the core full-mode post-apply flip, moves them using the release_functions output.
The mint Lambda uses a different scheme: it carries one mint-for-<caller> alias per authorised caller rather than a single live alias. Those alias ARNs are stable across releases, so each caller’s IAM grant keeps working without change.
The release_functions Terraform output lists every aliased function with its most recently published version number and the alias names activation must move. Release tooling reads this output after an apply to flip each alias in one coordinated step.
A Terraform test in terraform/tests/test_release_by_alias.py enforces these constraints statically against every .tf file in the repository. It asserts that every aws_lambda_function resource has a corresponding alias entry, that every function publishes a version, that API Gateway integrations resolve through the alias invoke ARN rather than the bare function ARN, and that every trigger invoke permission carries a qualifier equal to the alias name. A new triggered function must be added to the test’s alias map or the test fails.
Global API idle connection management
The Global API Lambda releases idle Postgres connections at the end of every invocation. A frozen Lambda container never runs pg’s idle timer, so without this step each container holds its sockets open until the container is reaped. The release runs last after the handler, closing only the connections that were idle on entry so that concurrent checkouts are unaffected. KeepDB_KEEP_IDLE_CONNECTIONS unset or false on the Lambda, including when RDS Proxy is in front of the database. Tenant set_config calls pin a client session, so retaining idle clients would retain pinned database connections and recreate connection exhaustion behind the proxy.
Global API billing enforcement
The Global API Terraform stack supports opt-in subscription access enforcement via theenable_billing_enforcement variable (default: false). When enabled, the Lambda reads from a DynamoDB entitlements table and the registerBillingAccessGuard plugin rejects user write operations for organisations without an active subscription.
Set enable_billing_enforcement = true in the environment’s .tfvars file only after Stripe billing is configured for that environment. Dev, test, and prod all enable it; dev uses Stripe test mode and prod uses live Stripe. Do not enable enforcement in an environment where billing is not yet provisioned; doing so will block all organisation write operations.
When enforcement is on, Terraform creates an IAM policy granting the Lambda dynamodb:GetItem on the billing entitlements table (<project>-<environment>-billing), scoped to keys matching the ORG#* and LINK#* prefixes. ORG#<orgId> holds a company’s billing entitlement. LINK#<linkId> holds contractor-company link state, which the shared billing guard reads on every request from a contractor person (a token with a via claim). Without LINK#*, that read is denied and every contractor request returns 503 billing_unavailable. The other product APIs that enforce billing grant the same two prefixes. The BILLING_ENTITLEMENTS_TABLE environment variable is wired to the Lambda automatically; it is an empty string when enforcement is off, which disables the guard.
Global API org self-service relay
The Global API relays/api/global/org/* user-management requests to stralos-admin-api’s /api/admin/org/*. Those admin-api methods are AWS_IAM with a resource-policy Deny for any other caller, and a browser cannot SigV4-sign, so the Global API Lambda signs each forwarded request with its own role.
Setting
admin_org_url wires ADMIN_ORG_URL on the Lambda and attaches the execute-api:Invoke policy. The grant is two-sided: admin-api’s org_admin_caller_role_arns must also name this service’s Lambda execution role. Either side alone produces a 403 at API Gateway with no application log on either side. See Organisation self-service Terraform variables for the admin-api side.
Global API observability
stralos-api-global supports opt-in PostHog error tracking and structured HTTP logs. Both signals are disabled by default. Configure the project token through the inline Terraform value when supplied; otherwise Global falls back to the documented SSM parameter. See Observability for configuration variables and redaction behaviour.
Authentication deployment requirements
stralos-api-global needs EdDSA signing-key material to publish JWKS and issue
user or service tokens. When auth_signing_key_ssm_parameter is configured,
Terraform manages the encrypted SSM SecureString parameter and wires
AUTH_SIGNING_KEY_SSM_PARAMETER on the Lambda. The post-apply signing-key
finaliser generates the Ed25519 JWK and replaces the bootstrap marker. Run the
finaliser before enabling an issuer; without key material, JWKS, OTP
verification, refresh, and service-token minting remain unavailable.
Where the SecureString already exists before this stack owns it (for example,
created by stralos-core account-baseline), set adopt_existing_auth_signing_key = true
so Terraform imports the parameter object without replacing its key material.
Admin event ingest needs admin_event_jwks_url, admin_event_issuer
(stralos-admin-<env>), and admin_event_audience (stralos-runtime-<env>) to verify
the admin event flusher’s envelopes. Without them the ingest route returns 503 and new
workspaces stay provisioning. POST /api/global/auth/oidc/exchange needs the OIDC
identity-stack SSM parameter names (oidc_issuer_ssm_parameter,
oidc_client_id_ssm_parameter, oidc_client_secret_ssm_parameter,
oidc_domain_ssm_parameter) and oidc_redirect_uri; without them it returns 503.
Global, Estimate, Activities, Documents, Rates, Scheduler, and Assistant need
AUTH_JWKS_URL pointed at the Global API’s public JWKS. Their legacy HS256
verification paths are retired. The variable is required in every environment:
each service’s buildApp throws before registering any routes when it is unset
or blank, with this message:
var.auth_jwks_url in each environment’s tfvars;
stralos-local sets it locally.
Global SES configuration
The Global Terraform source defines SES domain identity, DKIM, bounce/complaint alarms, and gated OTP email delivery. Domain validation pins the permitted domain for each environment and requires the configured From address to belong to that domain; it does not require one exact mailbox. Provision the emitted DNS verification records and confirm SES account readiness before enabling delivery. Do not enable generated-OTP exposure in a shared environment.Database connection roles
Runtime code supports owner, tenant-runtime, and platform database connections. Local development may fall back to the owner connection when the specialised URLs are absent. Provide dedicated owner, tenant-runtime, and platform connection parameters to services that use the three-role database model. Do not rely on the local owner-connection fallback for tenant request traffic. The database-connected API stacks setNODE_EXTRA_CA_CERTS to the AWS Lambda runtime’s /var/runtime/ca-cert.pem bundle. Operators do not provision this file.
Rates sources its owner, tenant-runtime, and platform connection parameter names from the db-bootstrap service_connection_parameters output. Terraform writes them to DATABASE_URL_SSM_PARAMETER, RUNTIME_DATABASE_URL_SSM_PARAMETER, and PLATFORM_DATABASE_URL_SSM_PARAMETER; an explicit variable can override each default. Use these role-specific TLS URLs rather than the retired shared RDS connection string.
The Rates organisation backfill that ran before the constraining RLS migration was cross-database, and its scripts/backfill-organisation.mjs has been deleted. Cross-database migrations are not allowed; a new populated Rates database that needs organisation provenance needs a reviewed migration plan, not an ad-hoc command during apply.
SSM parameter caching
Rates, Documents, Estimate, Global, and Scheduler delegate service-discovery SSM reads to the process-widessmParameters singleton exported by
@stralos-co-nz/api-common/aws/ssm. Cache identity includes the parameter name,
decryption mode, and TTL. Successful reads use a five-minute TTL by default and
identical concurrent misses coalesce; errors, empty values, and Terraform
placeholders are not retained as usable configuration.
Cold-start preload
Every API Lambda (Activities, Assistant, Billing, Documents, Estimate, Global, Rates, and Scheduler) loads the SSM parameters it reads once at cold start, before it serves any request. Each service lists them insrc/config/ssm-preload.ts, and lambda.ts awaits
preloadApiSsmParameters() during INIT. The preload uses
ssmParameters.preloadFromEnv(): batched GetParameters calls of up to ten names, with
jittered backoff on throttling during INIT only. A variable that is not set is skipped,
which is how local development runs with direct URL variables.
Each entry is required unless it is marked required: false:
- A required parameter that cannot be loaded rejects the preload and fails the Lambda INIT, so the environment never accepts traffic and Lambda retries the cold start.
- An optional (fail-soft) parameter that cannot be loaded is logged and the feature that
reads it degrades. For example, an unreadable
POSTHOG_PROJECT_TOKEN_SSM_PARAMETERdisables telemetry, an unreadable GlobalGOOGLE_MAPS_SERVER_API_KEY_SSM_PARAMETERmakes the routing route answer502, and an unreadable RatesESTIMATE_API_URL_SSM_PARAMETERmakes re-price delivery answer that it is not configured. Rates’ Google provider parameters fail only an import that uses that provider.
invalidate(). The Rates database handle uses it after a
credential rotation, so its next read of the database URL goes back to SSM.
Each API’s Lambda role grants ssm:GetParameters for the preload alongside
ssm:GetParameter. When you add a *_SSM_PARAMETER read, add it to
src/config/ssm-preload.ts; test/ssmPreload.unit.test.ts fails until you do.
Peer API URL resolution
Peer URL resolvers use the same precedence: a direct URL variable, then the SSM parameter named by its companion*_SSM_PARAMETER variable. Terraform supplies
parameter names—not hard-coded gateway URLs—to service Lambdas.
No resolver has a localhost fallback. When neither variable is set, the
resolver throws a configuration error at call time rather than calling a
hard-coded port.
The error names both variables, for example:
*_SSM_PARAMETER variables through
Terraform, and stralos-local sets the direct URL variables for every service.
When running a service outside stralos-local, set one of the two variables
for each peer it calls.
A missing URL is a configuration error, not a peer outage, so most callers
resolve it outside the try around the peer request and do not report it as
502 or 503. Some advisory reads still catch it and degrade. Activities’
price-range lookup logs a warning once and leaves lines unranged. A few Estimate
display reads (organisation default hours, export labels, project name, and
library activity name search) fall back to their defaults. The Assistant
service uses its own STRALOS_*_API_URL variables with no SSM companion; see
Assistant service.
Terraform derives peer parameter paths as
/<project>/<environment>/<region_key>/services/<service>/api-url. Use the
corresponding Terraform override only when a peer uses a non-standard path.
Global API service-discovery parameter
The Global API Terraform stack publishes the regional API Gateway stage URL to an SSM String parameter at deploy time. Peer services that call Global at runtime read this parameter through the shared five-minute SSM cache rather than embedding a hard-coded URL. The default parameter path is:api_url_ssm_parameter Terraform variable. Terraform owns and updates the value whenever the API Gateway stage URL changes; the parameter is tagged with the standard name prefix for the stack.
The Terraform output api_url_ssm_parameter exposes the resolved parameter name so that peer stacks can consume it by reference. The output api_url exposes the raw stage invoke URL directly.
Directory outbox
Global source code atomically records a lower-cased user email and acreate
directory-outbox row. Its dev Terraform stack provisions the DynamoDB writer
permissions, directory flusher Lambda, one-minute flush schedule, and daily
reconciliation schedule.
Assistant monitoring
Assistant alarms cover Lambda errors and throttles plus API Gateway 5xx responses. See Assistant service for the thresholds, topic wiring, and coverage-check boundary.Rates API deployment
Rates follows the shared automatic-dev, migration-marker, and destructive-guard controls described in Relational API immutable release path. Its alias-release workflow joins the same dev concurrency group as the apply, so an alias move queues behind an in-progress deployment.Lambda alias model
Every Lambda in the Rates stack publishes an immutable numbered version on every apply and is served exclusively through alive alias — never the bare function. This covers all five functions:
API Gateway integrates with the
api function through its live alias. The three SQS event-source mappings (rate-import work queue, rate-import dead-letter queue, and rate-category queue) and the two EventBridge schedules (event-flush and retention) all target their respective live aliases. Every trigger invoke grant carries the alias qualifier; a permission on the bare function does not cover function:live and the trigger would be refused on the first deploy after the alias is added.
Every alias sets lifecycle { ignore_changes = [function_version] }. A stage-mode plan therefore never moves an alias; only the post-apply alias flip (on full-mode core runs) or a manual release-alias.yml dispatch moves it.
The stack’s release_functions Terraform output lists every aliased function with its most-recently published version and the alias names activation must move. stralos-core’s full-mode terraform.yml reads this output after the apply and flips each listed alias.
The enable_on_activate output lists any triggers that are created disabled and must be enabled after the alias flip. For the current stack this list is empty: the SQS mappings and EventBridge schedules already exist and stay enabled; they are only retargeted from the bare function ARN to the alias ARN during the initial migration.
A Terraform policy test in terraform/tests/test_release_by_alias.py enforces these constraints statically against every .tf file: every function must have a corresponding alias, every alias must be named live, every API Gateway integration must target the alias invoke ARN, every EventBridge target must invoke an alias ARN, every invoke permission must carry the alias qualifier, and the release_functions output must name every aliased function. CI runs this test on every pull request.
Lambda connection pool management
Each Lambda container opens up to three Postgres connections per pool (owner, runtime, and platform). This cap prevents a burst of parallel containers from exhausting the database’s connection slots — without a bound, ten idle sockets per pool across a fleet of containers can fill the available slots entirely and return 503 to every subsequent request. At the end of each invocation the handler closes every idle connection. A frozen Lambda container never runs pg’s idle timer, so connections held by an idle container count against the database’s slot ceiling until the container is reaped. Releasing on the way out makes held connections proportional to in-flight requests rather than to the count of containers that have ever warmed up. KeepDB_KEEP_IDLE_CONNECTIONS unset or false, including behind RDS Proxy. Tenant set_config calls pin a client session, so retaining idle clients would retain pinned database connections and recreate connection exhaustion behind the proxy.
Pricing-rule sweep
rules:check remains the second command in MIGRATION_MANIFEST.json and therefore runs on every deployment. The former nightly schedule arm is retired: there is currently no between-deploy sweep for pricing-rule drift caused only by data changes. RULES_SAFETY_MANIFEST.json remains in the repository but no workflow currently consumes it.
Rates Lambda binary responses
The Rates Lambda adapter declaresapplication/vnd.openxmlformats-officedocument.spreadsheetml.sheet in its binaryMimeTypes list. API Gateway v1 proxy integrations require an explicit binary media type registration to base64-encode binary response bodies; without it, the XLSX workbook is transmitted as text and arrives at the client corrupted.
The XLSX media type is the only binary response the Rates API produces. All other responses are application/json.
Rates API monitoring
Rates alarms cover the request Lambda’s errors and throttles plus API Gateway 5xx responses. Alarm and OK transitions publish to the shared topic. The topic variable must be a full, same-account SNS topic ARN; each non-dev environment supplies its own value.Estimate monitoring
Estimate Terraform defines CloudWatch alarms for its two Lambda functions: the HTTP entrypoint (api) and the lifecycle pending-transition sweeper (sweep).
Each function carries an absolute Errors alarm and a Throttles alarm, both using treat_missing_data = notBreaching. The HTTP entrypoint tolerates up to four errors per five-minute period before paging; every other function (including the sweeper) pages on the first error. The entrypoint also carries a guarded error-rate alarm using metric math (IF(inv >= 10, 100 * err / inv, 0)) that trips at 20% and prevents divide-by-zero false alerts on idle periods or very low traffic.
The sweeper additionally carries a dead-man’s-switch alarm on AWS/Events SuccessfulInvocationAttempts (scoped to its EventBridge rule) with treat_missing_data = breaching. A 30-minute window covers up to two missed ticks of the rate(10 minutes) schedule before paging; this catches both a stopped schedule and a firing schedule whose Lambda invoke is consistently failing.
An API Gateway 5xx alarm on AWS/ApiGateway 5XXError closes the gap where the application returns an HTTP 5xx as a successful Lambda invocation (the Lambda Errors metric stays zero). Its threshold is one 5xx per five-minute period.
All alarms fire the shared alarm SNS topic managed by stralos-admin-api. A Terraform
check compares the alarm map with the Estimate Lambdas already live in AWS and reports
a non-blocking plan warning for a gap. A function introduced by the current apply is
visible only on a later plan, so this check is drift detection rather than a deployment
gate.
Documents monitoring
Documents alarms cover the API, pipeline and sync workers, DLQ consumers, scheduled sweep, API Gateway 5xx responses, and account-wide cost guards. See Stage observability for the canonical thresholds and coverage boundaries.Admin API release model
A merge tomain deploys dev automatically. The deploy-dev.yml push arm applies the same publish-then-apply chain as a manual dispatch; see Automatic and deliberate dev deploy for the guard that prevents a push from taking or lifting the writer cutover unattended.
The admin API’s main Lambda integrations use a live alias rather than $LATEST; the optional dedicated signup integration targets its separate signup Lambda. Publishing the main API artifact with update-function-code --publish creates an immutable numbered version without moving the alias.
scripts/finalize-deploy.sh owns the release step. After promoting the API bytes it calls scripts/promote-alias.sh, which moves live to the version just published and verifies both the alias target and that version’s CodeSha256. A failure stops the deployment before the artifacts_promoted marker is recorded.
Terraform cannot select the promoted artifact: aws_lambda_function.api ignores source_code_hash, so a Terraform-published version would snapshot whatever code was already on the function. The finaliser is the component that knows the correct target version.
The alias update uses the current revision ID for optimistic concurrency and refuses $LATEST. Main-API invoke permissions are qualified to live and use create-before-destroy lifecycle behaviour, preventing a 403 window when the grant changes. See Admin console — Lambda alias release for the deployment-rail proof.
Central release integration (self-activating member)
stralos-admin-api participates in the stralos-core central release as a self-activating member: it owns both the CREATE and ACTIVATE phases rather than delegating them to the core orchestrator.
The central release dispatches release-create.yml (CREATE phase) and release-activate.yml (ACTIVATE phase) at the release tag. These entry-point workflows delegate immediately to release-create-admin.yml and release-activate-admin.yml.
CREATE phase (release-create-admin.yml)
CREATE does everything that can happen ahead of the release without changing what users are served:
- Freeze — test freezes the promotion manifest for the candidate Lambda artifact (
X.Y.Z-rc.N). Prod re-derives the same manifest fromsource_version(which is content-addressed), refusing a manifest it had to create. This means prod only promotes what test froze. - Stage — calls
stralos-core’sterraform.ymlinmode: stage. The apply ships the new ZIP to themigrateandretentionfunctions and applies config and infrastructure changes, but moves no alias.api:liveandretention:livekeep serving the old code throughout CREATE. Infrastructure that must not land before its code keeps the legacyterraform-deploy.ymlpath. - Record — writes the activation payload (the
migratefunction’s published version andCodeSha256, and the stagedretentionversion). Also prunes old versions that the last activation left behind, since ACTIVATE defers pruning to the next CREATE.
dev is accepted for a dry run: no freeze occurs, and the stage builds the ZIP from the checkout exactly as deploy-dev.yml does.
The CREATE and ACTIVATE concurrency groups (admin-terraform-<env> for test and prod, deploy-dev for dev) prevent any CREATE apply or ACTIVATE from interleaving with a terraform-deploy.yml apply or a dev deployment.
The terraform/CONTRACT_OK file lists the resource replacements the stage plan policy permits. The API Gateway deployment is create-before-destroy and may be replaced when routes change; terraform_data guards have no runtime effect. All other deletes or replacements are refused by the plan policy.
ACTIVATE phase (release-activate-admin.yml)
ACTIVATE runs as a single job, minimising the number of runner queues the release pays for:
- Verify — confirms the environment still holds exactly what CREATE applied: the
migratefunction version and itsCodeSha256, and the stagedretentionversion. A CREATE or deployment that applied since would have replaced these values, and the verification refuses to proceed. - Quiesce — captures every writer’s current concurrency state, then gates writers when
drizzle/CUTOVER_REQUIREDasks for it, usingscripts/quiesce-deploy.shas every deployment path does. - Finalise — migrates through the
migratefunction, promotes the migrated ZIP onto the writers, movesapi:live, reopens writers, and health-checks, usingscripts/finalize-deploy.sh. The finaliser reads Terraform stack outputs from the state object viaTF_OUTPUTS_FILE(a single S3 read) instead of runningterraform initon the critical path. Version pruning is deferred to the next CREATE (FINALIZE_PRUNE=skip). - Move
retention:live— pointsretention:liveto the version CREATE staged.
cutover_required: false (approximately 17-second queue time). When a cutover is armed, it uses the EC2 runner for its writer-drain headroom.
On failure after writer state has been captured, ACTIVATE runs cutover-recover.yml. Before the migration boundary the recovery reopens the old bytes; after it the recovery resumes finalisation. Either way the release stops at failed status for triage.
scripts/release-state.sh
release-state.sh outputs <env> <destination.json> reads the Terraform stack’s outputs directly from its S3 state object. It provides the TF_OUTPUTS_FILE that ACTIVATE’s finaliser reads in place of terraform output, eliminating a terraform init and provider download from the critical path.
The script reads the backend bucket and key from terraform/backend-<env>.hcl so the two cannot disagree. Only the .outputs section is written to disk; the rest of the state object is never retained.
Admin API monitoring
The admin Terraform defines CloudWatch coverage for the admin Lambda plane and API Gateway. Alarms publish to${name_prefix}-alarms, for example
stralos-dev-admin-alarms. Terraform creates no subscription by default. See
Admin console monitoring for thresholds and the
secret-sync dead-man’s switch.