Skip to main content
stralos-admin-api is the management-plane API for organisations, users, memberships, platform configuration, and new-organisation seed templates. It is a standalone Fastify service with its own Postgres store.

​
Capabilities

​
Worker schedules

The Terraform stack deploys separate Lambda entry points for the API, database migration, event worker, and operator operations. API Gateway exposes only the operator API. The operator function requires explicit IAM invocation and has no public route or recurring trigger. The Cognito and invite schedules are separate EventBridge rules rather than modes inside the flush Lambda. This keeps the outbox flusher from being blocked by a Cognito outage, and prevents the pool reconciler from running at flush frequency.

​
OpenAPI contract

The admin API publishes a machine-readable OpenAPI 3.1 document at GET /api/admin/openapi.json. This endpoint is unauthenticated (no operator session required), built once at startup, and returns byte-stable bytes per deploy. It is a documentation contract only — it is explicitly excluded from the copilot tool substrate because the described operator surface accepts operator sessions, while the copilot acts as a tenant user. The separate organisation self-service routes are not part of this document. The document covers 39 operations across nine resource groups: Success response bodies are currently phased (x-stralos-response-status: undocumented); the operation IDs and request schemas are stable and machine-readable. The security scheme is operatorAuth (HTTP bearer JWT); every protected operation returns a 401 when a valid operator session token is absent. There is no per-route 403 — any active operator may call any operation. The document version (info.version) is 1.0.0. This is the descriptor-contract version, managed independently of the service or deploy version; it advances only on an intended contract change.

​
Authentication boundary

The admin API separates these authenticated channels:
  • Operators — Stralos employees. Sign-in uses Google OAuth2 (authorisation code + PKCE, server-side redemption) restricted to the stralos.co.nz Google Workspace domain. The service verifies the JWKS signature, RS256 only, the allow-listed issuer, the exact audience, the ceremony nonce, email_verified, and the hd Workspace claim alongside the address domain. A successful ceremony returns a short-lived operator session JWT signed by the admin service’s own keypair (audience stralos-admin-<env>); a Google ID token is never accepted as a session. This preserves logout, immediate operator disable, and revocation — none of which a foreign token can express. MFA is now an organisational control: the service delegates second-factor enforcement to Google Workspace 2SV policy and cannot verify which factors were used. In dev, test, and prod, google_auto_provision_operators is enabled: any verified stralos.co.nz Workspace account is seeded as an operator row on first sign-in, audited as operator.provisioned. The Workspace domain gate is the bounding control. The bootstrap CLI remains available for environments where auto-provisioning is off. While the console remains open, the browser renews its bearer token via POST /api/admin/auth/refresh without repeating the Google ceremony. Each renewal extends the session by 30 minutes, clamped to 12 hours from the original sign-in.
  • Runtime report flusher — a machine principal. Reports use a dedicated JWT type (stralos-report+jwt) verified against a per-region JWKS.
  • Organisation members and administrators — tenant end-users. These callers present an ordinary api-global access token (typ: "user", audience stralos-api) verified by auth/orgAdminToken.ts. The user-management routes (/api/admin/org/users) additionally re-derive administrator authority from the admin store via auth/orgAdmin.ts. The organisation switcher route (GET /api/admin/org/organisations) makes no role check beyond a valid token; any active member may call it.
  • Release journey gate — a machine principal used by the promotion pipeline (PRD 03). It presents a service token issued by api-global (typ: "service", audience stralos-admin-api, scope admin:probe-read) and is verified against a per-region JWKS configured by ADMIN_SERVICE_TOKEN_SOURCES. This principal can reach only the service probe route. It reads no tenant row, no operator row, and no catalogue item.
  • Login relays — api-login sends signed, body-bound commands for self-service signup completion, membership acceptance, and session revocation. The three command families share a signing key but use distinct token types and audiences; no family’s captured command is valid as another.
The login route, health check, event JWKS endpoint, and GET /api/admin/openapi.json are unauthenticated. Operator routes require a live operator session. Organisation self-service routes (/api/admin/org/*) are exempt from the operator hook and require a valid org user token instead. The service probe routes (/api/admin/service/*) are exempt from the operator hook and require a valid service token instead. The org self-service surface is not covered by the openapi.json descriptor contract; it uses a separate authentication path and is not part of the copilot tool substrate. POST /api/admin/memberships/accept uses the relay-authenticated signed-command channel. It rejects an operator session token and is excluded from the OpenAPI descriptor contract. See Relay-authenticated invitation acceptance below.

​
Operator session renewal

POST /api/admin/auth/refresh extends a live operator session and returns a fresh bearer token without another Google sign-in. An operator who keeps the console open is renewed automatically rather than being redirected to Google every 30 minutes.

​
How it works

The route requires a valid operator bearer token. The deny-list, expiry, and operator-active checks all run in the auth pre-handler before the route body executes. Inside the route body, the service:
  1. Re-reads the operator row and confirms the operator is still active. A disable that races this route does not result in a freshly extended session.
  2. Calls renewSession, which extends operator_sessions.expires_at in a conditional UPDATE. The WHERE clause repeats the unrevoked and unexpired predicates, so a logout that races the renewal wins — the revoked row is not resurrected.
  3. Mints a new bearer token with a TTL equal to the time remaining on the renewed session.

​
Session lifetime ceiling

The token TTL is 30 minutes. A session’s maximum lifetime from the original Google sign-in is 12 hours (OPERATOR_SESSION_MAX_LIFETIME_MS). Each renewal is clamped to that ceiling rather than extended past it. Once the ceiling is reached, the next renewal attempt returns 401 and the operator must sign in again via Google.

​
Renewal request and response

The route is not auth-exempt. It is excluded from the OpenAPI descriptor contract, which covers only protected operator operations that accept a live session.

​
Managed records

The admin store currently owns:
  • organisations and their home region (including signup_discount_code if a discount code was submitted at signup, and country_code if a country was supplied at signup or updated via the org profile endpoint);
  • global user identities (including Cognito provisioning state and invite tracking);
  • organisation memberships and roles (including invitation generation);
  • the admin-side platform configuration contract;
  • platform list definitions and legacy new-organisation seed vocabulary;
  • rate-category seed templates (keyed by business type, with ordered allowed units);
  • disposal substance seed templates (keyed by business type; seeded with 21 workbook defaults per type by migration 0040, replaced with 12 revised defaults by migration 0046 where unedited);
  • per-organisation seed snapshots (immutable, created at org-create time);
  • operator credentials and sessions;
  • audit records;
  • provisioning-event outbox and aggregate-version records;
  • Cognito operation queue (cognito_operations); and
  • legal acceptance records (legal_acceptances) — an append-only history of each user’s Terms and Privacy acceptances, keyed by user and cascaded on user erasure.
Revoking a membership sets its admin record to disabled; it does not delete the row. Disabling a user or changing an email updates the admin store and enqueues the corresponding event. Enable event synchronisation and its recurring workers separately for each environment that should deliver these changes. default_markup_percent and pricing_average_speed_kmh are retired admin configuration keys. Attempts to set them return 422 with code config_key_removed. Default markup is organisation-owned in the Global API.

​
Platform catalogue

The admin global_lists store separates code-coupled platform values from legacy new-organisation seed vocabulary and signup configuration: Migration 0005_seed_platform_lists_retire_markup.sql seeds the canonical project statuses, activity statuses and basis values, and work-item statuses with stable IDs. Existing rows are preserved by ON CONFLICT DO NOTHING.

​
Delete behaviour

DELETE /global-lists/:id soft-deletes the item. The row is stamped with deleted_at and deleted_by and kept for history. It is hidden from list results and from the signup business-type check. A repeat DELETE on an already-deleted item returns 204 with no change and no second audit record. Any other write (PUT) to a soft-deleted item returns 409 with code deleted. Unique constraints are enforced among live rows only. A value that was previously deleted can be re-created under the same listKey and scope.

​
Global list item fields

The runtime Global API owns the immutable unit_catalogue. It is deliberately outside the admin settings and event-projection contracts; do not add unit catalogue rows to admin list or seed-template payloads. There are currently no writable app_config keys. Unknown keys return 422 unknown_config_key; the two retired pricing keys return 422 config_key_removed.

​
Rate category templates

Rate category templates define the default rate categories a new organisation starts with. Each template belongs to a business type and carries the units each category may be priced in. Names are unique per business type — a global unique would let exactly one type own a name such as “Internal labour”.

​
Template fields

parentGroup classifies the template within six taxonomy nodes: The create body is strict — unknown fields are refused with 400. This prevents silent acceptance of misspelled field names such as allowedUnits or unitCodes. parentGroup is required on create; omitting it returns 400. On update it is optional (omit means keep the current value). allowedUnitCodes must contain at least one code. The API validates the array for shape (non-empty string, unique, minimum one entry) but not for membership against the unit catalogue — the catalogue lives in api-global’s tenant database and is not reachable from the admin service. The admin console picker reads its options from UNIT_CATALOGUE in @stralos-co-nz/ui (27 units); a code not in that list but valid on the wire will be accepted and passed through to the tenant ingest, which silently drops codes it does not recognise. isCartageVehicle is invalid on any parent group other than plant_vehicles. The API returns 422 cartage_requires_plant_vehicles when the combination is submitted. The admin console enforces this client-side to provide an immediate error rather than waiting for a server rejection.

​
Delete behaviour

DELETE /rate-category-templates/:id soft-deletes the template. The row is stamped with deleted_at and deleted_by and kept for history. It is hidden from list results and excluded from the new-organisation seed snapshot. A repeat DELETE on an already-deleted template returns 204 with no change and no second audit record. Any other write (PUT) to a soft-deleted template returns 409 with code deleted. Unique constraints are enforced among live rows only. A name that was previously deleted can be re-created under the same businessType. Organisations already seeded before a template was deleted are unaffected. Their categories came from a frozen org_seed_snapshots copy taken at create time.

​
Sub-category templates (retired)

The authored sub-category level (rate_sub_category_templates) is retired. The tenant flattened its category tree (api-global 0036), so the authored third level existed only to be promoted back to one. Existing frozen org_seed_snapshots rows can still carry subCategories entries, and tenant ingest continues to promote them. The routes, the General sub-category auto-creation, and the cascade from category reparenting are no longer in effect.

​
Seeding behaviour

Rate category templates do not emit standalone provisioning events. They are captured in the seed snapshot at organisation-create time, while new snapshots emit an empty vocab array. Immutable older snapshots may still replay vocabulary. Changes do not affect organisations created before the edit. Migration 0029_disposal_units_each_lot.sql changes every disposal template whose stored array is still exactly ["cubic_metre", "tonne"] to ["cubic_metre", "tonne", "each", "lot"]. Templates with any operator-edited unit list are preserved, and cubic metre remains the first/default unit. Migration 0030_landscaping_aggregate.sql adds Aggregate to the landscaping starter set under quarry_materials with cubic metre and tonne. Because snapshots are immutable, both changes affect future organisations only.

​
Disposal substance templates

Disposal substance templates define the default disposal substances a new organisation starts with. Each template belongs to a business type (the same key used by rate-category templates) and carries the density and compaction values used in disposal calculations. Migration 0040 seeded 21 workbook defaults for every live business type. Migration 0046 replaces them with 12 revised defaults, but only for a business type whose live rows are still exactly the 21 that 0040 seeded (same names, values, reinstatement flag and sort order, nothing added or removed). Any operator-edited set is kept. The old rows are soft-deleted (deleted_by null) before the new ones are inserted, and the migration is re-runnable. Existing organisations and frozen seed snapshots are not changed. When a new business type is added via POST /global-lists, the 12 current defaults from src/lib/disposalSubstanceDefaults.ts are inserted in the same transaction so no type is ever left without a starter set.

​
Template fields

The create body is strict — unknown fields are refused with 400.

​
Delete behaviour

DELETE /disposal-substance-templates/:id soft-deletes the template. The row is stamped with deleted_at and deleted_by and kept for history. It is hidden from list results and excluded from new-organisation seed snapshots. A repeat DELETE on an already-deleted template returns 204 with no change and no second audit record. Any other write (PUT) to a soft-deleted template returns 409 with code deleted. Unique name constraints are enforced among live rows only.

​
Seeding behaviour

Disposal substance templates do not emit standalone provisioning events. They are frozen into the seed snapshot at organisation-create time alongside rate categories. Changes affect future organisations only — organisations already created own their substances in the tenant and are never modified from here. Existing organisations created before migration 0040 received their disposal substances from the tenant’s own built-in defaults. A null disposalSubstances field in a seed snapshot has the same effect: the tenant falls back to its own defaults.

​
Deploy-time tenant bootstrap (db:init)

The db:init bootstrap provisions the founding tenant for a brand-new environment: one organisation, one administrator, one membership, and the Cognito intent that will give that administrator an account. You can run it locally with pnpm db:init; deployments invoke the same operation through the migration Lambda. It is idempotent, so a second run is a no-op that exits 0.

​
Configuration

All three naming variables must be set together or omitted together. Omitting them is a valid state (the command skips gracefully and exits 0). Setting only some of them is a hard error, because that is the shape a templating mistake produces. Terraform maps these environment variables from db_init_organisation_name, db_init_admin_email, db_init_admin_name, db_init_organisation_timezone, db_init_home_region_key, and db_init_business_type. The first three must be set together or all omitted. Plan-time guards also reject empty values, malformed email addresses, and a home-region key that is absent from the configured region registry. Only the migration Lambda receives the resulting ADMIN_INIT_* environment variables. There is no password variable and none should be added. The administrator’s credential is the temporary password delivered by the Cognito invite email after the sweeper creates the account. A plaintext password in configuration would sit at rest in SSM and be silently ignored wherever Cognito owns the credential.

​
What it creates

In a single transaction with a transaction-scoped advisory lock (key 5150120002):
  1. Checks whether a user with ADMIN_INIT_ADMIN_EMAIL already exists. If one is found, the run returns already_initialised and exits 0 without writing anything.
  2. Calls createTenantInTx — the same function used by self-service signup — to insert the organisation, the admin user (provisioning='pending'), an active administrator membership, the seed snapshot, audit records, and outbox events in one atomic write.
  3. Enqueues a create row in cognito_operations in the same transaction.
Because the Cognito intent and the user row commit together, there is no state in which a user exists that nothing will ever provision.

​
Local versus deployed behaviour

The Postgres writes are identical in both environments. The difference is who else is running:
  • Deployed — the events worker’s Cognito sweeper picks up the create operation, calls AdminCreateUser, marks the user provisioned, and only then is the invite eligible to be released.
  • Local — there is no sweeper. The operation sits pending indefinitely and nothing calls Cognito. The regional runtime’s OTP sign-in authenticates from Postgres, so the administrator can log in regardless.
The absence of the sweeper is the whole difference. No if (cognito) branch exists in the bootstrap, deliberately: two bootstrap paths could drift.

​
Exit codes and output

​
Deployment rail

After every Terraform apply, scripts/finalize-deploy.sh checks db_init_configured. When it is true, the script invokes the immutable migration Lambda version with {"mode":"db-init"}. Both the normal migration mode and db-init run migrations first, so the bootstrap cannot reach an unmigrated schema. The invocation occurs before writers are restored. The finaliser requires a structured created or already_initialised response; a Lambda error, an older artifact that does not support the mode, or an unexpected skipped response fails closed and leaves writers disabled. Because the schema has already changed at this point, recovery resumes the finaliser and safely repeats the idempotent operation. Idempotency is keyed on the administrator’s email address, not the organisation name. A name is configuration and gets edited; keying on it would create a second tenant on the next deploy instead of recognising the first.

​
Lambda alias release

API Gateway integrations for the main admin API Lambda target aws_lambda_alias.live, not $LATEST. The optional dedicated signup method targets aws_lambda_function.signup instead. Publishing a new main API artifact with update-function-code --publish cuts a numbered version but does not move the alias, so the main API edge keeps serving the previous deployment’s bytes until the alias is explicitly updated. scripts/promote-alias.sh moves the live alias onto the version the finaliser just published and verifies the move via two independent read-backs: the alias must resolve to that version, and get-function-configuration --qualifier live must carry this run’s CodeSha256. The finaliser calls promote-alias.sh immediately after promoting the API function code and before recording the artifacts_promoted progress marker. A failure here fails the deployment closed. Terraform cannot perform this step. aws_lambda_function.api carries ignore_changes = [source_code_hash], so any version Terraform publishes snapshots the code already on the function, not the newly promoted artifact. Only the finaliser, which owns the promotion, knows the correct target version. scripts/promote-alias.sh refuses $LATEST as a target and uses the alias revision ID for optimistic concurrency, so a concurrent release-alias.yml rollback cannot be silently overwritten by a deploy. assert_promoted_writers, which runs after writers are restored, verifies the alias qualifier as well as $LATEST to confirm the edge is serving the promoted artifact.

​
Organisation lifecycle

An organisation is created with status provisioning. It is not usable until the provisioning saga (see below) confirms that both required runtime projections have landed and flips the org to active. The full lifecycle is: Status is not freely settable via the API. Activation and failure transitions are owned exclusively by the provisioning saga.

​
Organisation slug

Every organisation has a slug: a globally unique, URL-safe identifier used as the tenant key in login.<env>/t/<slug>. The slug is set at create time and cannot be changed afterwards. POST /organisations accepts an optional slug field. When the caller omits it, the API derives a slug from the organisation name. If derivation fails (the name reduces to a reserved word or fewer than two characters), the API returns 422 org_slug_underivable and the operator must supply an explicit slug. A slug that is already taken returns 409 org_slug_taken. The slug is carried on every organisation.* event payload so the runtime can resolve a tenant to its home region. See Organisation slug for the full format rules, reserved words, and event-delivery behaviour.

​
Self-service signup

The login site exposes /signup. The browser first submits only the prospective administrator’s email address to stralos-api-login. The start response is deliberately uniform: it does not reveal whether the address is already registered or whether a message was sent. The signup form also fetches the available business types via GET /signup/business-types. This endpoint is unauthenticated and read-only. It returns only values from global_lists(scope='signup_vocab', listKey='business_types') — operator-editable entries — and returns 503 when signup is not configured. Values in this list are public by construction; do not add anything here that an anonymous caller should not see. After the user supplies the emailed code, their name, the organisation name, their type of business, and a password, api-login relays the completion request to admin-api. Admin verifies the code and creates these records in one transaction:
  • the organisation in provisioning status, with the required businessType recorded, signupDiscountCode set when a discountCode was submitted, and countryCode set when a countryCode was supplied;
  • an active, provisioned user;
  • an active administrator membership;
  • a legal_acceptances row recording the accepted Terms and Privacy versions;
  • the seed snapshot (categories selected by businessType), audit records, and required outbox events; and
  • the directory replacement for the user’s region set.
The completion request requires three fields for Terms and Privacy acceptance: These fields are required on every completion request — they cannot be omitted or defaulted. The legal_acceptances table is append-only: a later re-acceptance of a newer version adds a new row, so the table is the full history of what each user agreed to and when. The acceptance row is written in the same transaction as the user; there is no user without an acceptance record, and no acceptance record without a user. The ON DELETE CASCADE constraint on user_id ensures that user erasure (the retention Lambda) removes both the user row and the acceptance history together. Apply migration 0039_legal_acceptances.sql before promoting code that writes the new columns. The optional discountCode field on the completion request accepts a code in the format [A-Za-z0-9-]+ (1–100 characters). Admin stores it on the organisation as signup_discount_code but does not validate it against the billing service at signup time. The billing triage read returns it as signupDiscountCode so operators can see what code the organisation signed up with. The completion request requires a live businessType. A missing type returns 422 business_type_required; an unknown type returns 422 unknown_business_type. The signup page presents its completion state after this transaction and directs the administrator to sign in; signup itself does not issue a browser session. Independently, the normal provisioning saga activates the organisation after its runtime projections are confirmed.

​
Password requirements

The completion request enforces the Cognito user pool’s full password policy before the one-time code is consumed or a Cognito account is created. A password that does not satisfy every rule returns 400 with per-rule field errors. The special-character set is the 32 printable ASCII punctuation characters Cognito’s pool policy declares as satisfying RequireSymbols. Space is deliberately excluded: the pool permits a space inside a password but does not document it as satisfying the symbols requirement. The same rule is duplicated in stralos-api-login and stralos-login-web; change all three together.

​
Email alias recovery

The Cognito pool declares AliasAttributes: ["email"], making the email address a second unique namespace alongside the UUID username. A signup attempt that fails after creating a Cognito account but before completing the database transaction leaves an orphaned account holding the email alias. Previously, retries received AliasExistsException from AdminCreateUser and returned 503 permanently. The completion path now resolves this: when AliasExistsException is raised, admin-api calls AdminGetUser to locate the alias holder and adopts it if all four conditions hold:
  1. The email attribute on the Cognito account matches the address being signed up.
  2. There is no users row for the address or for the holder’s username.
  3. A signup_intents row records that this flow originally minted that username for this address, with a status of completing, expired, or failed.
  4. No enabled tenant was ever created from that intent.
Any condition that does not hold returns 409. A claimed alias cannot be cleared by retrying — the adoption path recovers only orphans this flow demonstrably created, and nothing is ever deleted. See Get started with Stralos for the user-facing steps and Admin provisioning — signup channel for the server-to-server boundary.

​
Seed template snapshot

When an organisation is created, the API atomically:
  1. Validates that a live businessType was supplied. A missing value returns 422 business_type_required; an unknown value returns 422 unknown_business_type. This applies to POST /organisations, self-service signup completion, and db:init.
  2. Inserts the organisation row with status provisioning.
  3. Reads the rate_category_templates rows and the disposal_substance_templates rows for the organisation’s businessType and freezes them into an immutable org_seed_snapshots row (the templateSnapshotId).
  4. Enqueues both organisation.created and seed_template.provisioned outbox rows in the same transaction.
Because the snapshot is written at create time, later edits to rate-category templates cannot mutate a pending provisioning event. Reconciliation re-arms the existing outbox row, so redelivery reuses the frozen payload and never rereads live templates. The seed_template.provisioned payload carries: Permanent backwards compatibility. Every org_seed_snapshots row already written is immutable and may carry subCategories and vocab. The payload type, the emission schema, and the tenant consumer must all keep accepting both fields forever. What changed is only what a new snapshot contains. parentGroup, subCategories, canHaveTravel, and canHaveHaulage are permanently optional in the schema — a field added to a contract over immutable stored payloads is optional forever. parentGroup on each legacy sub-category item is likewise permanently optional. Snapshots that carry no sub-category parentGroup remain valid; the consumer falls back to the enclosing category’s parentGroup. The seed_template aggregate uses the org id as its aggregateId, giving the runtime a clean per-org high-water key. The seed_template aggregate maps onto the organisation advisory lock so delivery and reconciliation for both aggregates serialise on a single lock.

​
Provisioning saga

The provisioning saga is a scheduled sweep (not a request-path operation) that drives orgs through provisioning → active (or provisioning → failed). It is invoked as:
  • EventBridge: every minute.
  • Lambda: pass { mode: "saga" } to the flush Lambda handler.
  • CLI: pnpm flush -- --saga.
For each org in provisioning or failed status, the saga evaluates two gates inside a serialised transaction: Gate 1 — outbox delivery: both required outbox rows (organisation.created and seed_template.provisioned) are processed (the flusher’s terminal state, meaning the runtime ingest returned 2xx). Gate 2 — replica confirmation: the runtime replica read independently confirms that both projections are applied at or above the required aggregate version. When both gates pass, the saga:
  1. Updates the org status to active.
  2. Writes an audit record (organisation.activated).
  3. Enqueues organisation.updated { status: active } via the normal allocator path (version-ordered after organisation.created, so the flusher delivers them in order).
When Gate 1 finds a conflicted required row (definitive projection failure), the saga marks the org failed, writes an audit record, enqueues organisation.updated { status: failed } for replica consistency, and fires an operator alert once (on the provisioning → failed edge only). A failed org is recoverable. Reconciliation re-arms the conflicted row; once redelivery converges and the replica confirms, the saga activates the org. Recovery is always forward — no revert.

​
Concurrency

Each org evaluation holds a FOR UPDATE row lock with a status re-check, so two concurrent saga runs cannot double-flip. The saga uses the normal allocator path and does not hold the flusher/reconciler per-aggregate advisory lock.

​
Event delivery

Applicable provisioning-aggregate mutations enqueue versioned events in the same transaction as the admin-store change. Rate-category templates do not emit standalone events; they are captured in the seed snapshot at organisation-create time. New snapshots carry no org_seed_template vocabulary, although immutable older snapshots may still replay it (see Rate category templates). Authentication, session, and operator changes do not enqueue provisioning events. The flusher signs event envelopes with the event keypair, delivers pending rows, and records retry, processed, or conflict outcomes. The dev EventBridge rule invokes it every minute. Each outbox row carries a target_region_key. The flusher, reconciler, and predecessor-blocking query all filter on this column, so every region has its own independent delivery stream for the same aggregate. The delivery address for each event type is: The flusher enforces two delivery-ordering dependencies:
  • Per-aggregate predecessor blocking: a row is not delivered while any lower-version row for the same aggregate and target region is non-processed.
  • Cross-aggregate seed dependency: a seed_template.provisioned row is not delivered until its org’s organisation.created row is processed.
The full event type catalogue is: Every user event (user.created, user.updated, user.disabled, user.email_changed) carries a sessionGeneration field (M4c): a non-negative integer that the runtime applies as a monotone greatest(existing, incoming). The session generation is the retained regional sign-out-everywhere counter; incrementing it invalidates regional Global API sessions in every region the event reaches. It does not invalidate the separate central login-service session store. Legacy outbox rows may carry no sessionGeneration; the runtime treats an absent value as a no-op for the generation column.

​
Project lead departure

When an operator disables a user or revokes a membership, the admin API contacts the affected runtime region to verify that the user is not the last project lead on any active project before the disable or revoke takes effect. This check prevents a project from being left without a lead. The check runs inside the same database transaction as the admin-store write. Before the transaction commits, the API:
  1. Enqueues the outbox event (user.disabled or membership.revoked) as normal.
  2. Records a durable project_lead_departure_intents row for every affected region. This row is written independently of the identity transaction so that recovery can proceed even if the canonical transaction is rolled back.
  3. Sends a signed projectLeadReservation=prepare request to the runtime ingest URL for each target region. The runtime checks whether the user holds any last-lead position and returns 409 if so.
If any region responds 409, the route returns 409 with code project_last_lead and the message “Appoint a replacement before deactivating the last project lead.” The admin-store transaction is rolled back and no event is enqueued.

​
Transport configuration

The departure check uses the same RUNTIME_INGEST_URLS map as ordinary event delivery. It does not use RUNTIME_REPLICA_URLS. A deployment that configures only RUNTIME_INGEST_URLS (the API ingest map) without RUNTIME_REPLICA_URLS (the worker replica map) will have departure checks work correctly. Using the replica URL for this call was a bug; the runtime ingest endpoint is the correct target for the reservation protocol. The RUNTIME_INGEST_URLS variable must be set for departure checks to function. A missing or empty map causes the route to return 503 with “Project leadership authority is not configured.”

​
Durable recovery

The project_lead_departure_intents table provides crash recovery for the two-phase departure protocol: The recovery pass (recoverProjectLeadDepartures) runs under the same per-user derivation lock as the canonical outbox writers. When the paired outbox row has reached processed, the intent is considered complete and deleted. When the outbox row is absent (the canonical transaction rolled back), the recovery sends a projectLeadReservation=cancel request to release any runtime reservation and then deletes the intent. The recovery pass is invoked by the event flusher Lambda on every scheduled tick. No separate EventBridge rule is required.

​
Capabilities table entry

​
Runtime report channel

The admin API accepts signed runtime→admin reports at POST /api/admin/reports/runtime. Reports are sent by stralos-api-global to relay state transitions that originated in a region’s runtime rather than in an operator action. They use a dedicated JWT type (stralos-report+jwt) and are verified against a per-region JWKS distinct from the admin signing key. The route requires IAM authorisation and SigV4 plus an EdDSA JWS signed by the runtime keypair. A distinct issuer and audience, the protected-header type, the environment claim, and signing-key separation prevent a user access token from being accepted as a report. The channel requires report_flusher_role_arn and a per-region report_source_map containing { issuer, jwksUrl }. The route is unavailable when the channel is not configured. The user.session_revocation_requested report carries only { userId }. It holds no ordering value and no generation: two distinct requests both increment the counter (the counter is monotone, so two increments are correct and safe), and redeliveries are caught by the runtime_report_seen primary key, which replays the recorded outcome rather than incrementing again. A report is a command rather than a direct regional write. The admin API validates it against current admin state and applies accepted changes through its normal versioned outbox path.

​
Report outcomes and rejection codes

Membership-acceptance reports return applied, noop, or rejected. A rejection includes one of the following codes:

​
Membership payload fields

All membership events carry invitationGeneration in addition to the core fields. This is the current incarnation counter for the membership’s invited state: it starts at 1 and increments each time a membership transitions back to invited (for example, a revoke followed by a re-grant). The membership.status_changed event emitted on an acceptance carries the generation that was current when the user consented. A report whose generation does not exactly match the current value is rejected as stale_invitation.

​
Relay-authenticated invitation acceptance

POST /api/admin/memberships/accept is the IDP-M7 acceptance path. The central login service (stralos-api-login) captures the user’s consent during sign-in and relays it here as a signed JWS command. Admin performs the invited → active transition in a transaction and delivers membership.status_changed through the normal outbox path. This path exists because the login service can write nothing to the admin database under M7: its memberships grant is SELECT on enumerated columns only. The consent therefore travels as a relay command, using the same pattern as self-service signup.

​
Transport

The route uses a dedicated relay command model, separate from both the operator session and the runtime report channel: A separate typ and audience from signup’s command family are deliberate. Both families share a signing key, so audience and typ together are the entire separation: a captured signup command must not function as an acceptance command, and vice versa. The request body is bound by hash in the JWS for the same reason signup’s is. Without hash binding, a captured token would authenticate the relay but not the specific request body — it could be replayed against a different (userId, organisationId) pair, accepting an invitation the user never clicked.

​
Invitation acceptance request and response

​
Relationship to the runtime report path

The pre-M7 acceptance path is membership.acceptance_reported on the runtime report channel: the runtime flips its own replica, reports the fact, and admin applies a CAS. The M7 path is the reverse: consent arrives centrally first and the region learns through the normal admin → region projection. The two paths must not both be live for one membership. A regional accept_membership_invite CAS would conflict with the projection that the central acceptance already authorised. The store module records this ownership constraint. The membership.status_changed event emitted on a successful M7 acceptance carries the same MembershipPayload shape as other membership events, including invitationGeneration.

​
Audit

The acceptance is audited with actor user-consent:<userId>@<organisationId> and operator_id = NULL. The subject of this action is a person, not an operator.

​
Relay-authenticated session revocation (PRD 05 §8.4)

POST /api/admin/sessions/revoke-all is the sign-out-everywhere path for central browser sessions. The central login service cannot write to the admin database under D41’s grant — its role has SELECT on users.session_generation and nothing more — so a user’s sign-out-everywhere request travels here as a signed JWS command. This is the third relay family, after signup and membership acceptance.

​
Why a relay rather than a report

stralos-api-global reaches the session-revocation write through its runtime-report channel: it enqueues a user.session_revocation_requested report into its regional runtime database, and the flusher delivers it to POST /api/admin/reports/runtime. stralos-api-login has no regional runtime database — it is the login host — so the report channel is not available to it.

​
Transport and command authentication

The typ, audience, and command-type enum are the entire separation between the three relay families: all are signed with the same key. A captured command of any family must not be usable as another. The test suite proves a command from one family is refused by every other. The request body is bound by SHA-256 hash inside the JWS. Without this binding, a captured token would authenticate the relay but not the specific request body. An attacker holding their own relay command could replay it against a different userId and sign out anyone in the estate. Body-hash binding is what makes the PRD 05 D14 guarantee (“can revoke none but the CALLER’s own sessions”) hold across the relay hop.

​
Session revocation request and response

The userId field is the only field. The login service resolves it from a validated refresh token; it is not a caller-supplied target. There is deliberately no operator-override field: an operator who needs to end a user’s sessions uses POST /users/:id/disable, which is audited against the operator’s own identity. The response is 200 rather than 202 because the write completes before the response. After that success, the login service returns 202 {} to the browser because regional revocation still converges asynchronously. An unavailable or refused relay returns a dependency error instead.

​
Shared transaction

The revocation transaction is store/sessionRevocation.ts — the same implementation the runtime-report path uses via applySessionRevocation. Both paths increment the same monotone counter, write the same audit record, and enqueue user.updated to the same sticky set of regions. Two implementations would risk one forgetting the user.updated enqueue, which is the only mechanism by which regions learn the new generation. On the relay path the user.updated event carries no additional region in the fan-out set, because the login host has no regional session state. The sticky set (the union of every region the user has ever been projected into) is sufficient: a user who holds a live central session has completed the sign-in ceremony and therefore has at least one active membership region.

​
Idempotency

The revocation command has a commandId (jti) and a 120-second expiry but no ledger. A replay is harmless because the counter is monotone — a second increment is a strictly stricter session cutoff and can never resurrect a session. This reasoning is the same one applied to two distinct simultaneous POST /api/login/auth/sessions/revoke-all requests, which the report channel also counts twice intentionally.

​
Audit

The revocation is audited with actor relay:<relayKey> and operator_id = NULL. The audit action is user.sessions_revoked, which is distinct from the report channel’s report.applied.user.session_revocation_requested, so the audit log unambiguously identifies which channel performed each write.

​
Organisation billing triage

The billing triage surface lets operators inspect and refresh an organisation’s billing state from Stripe through the regional billing service.

​
Routes

Both routes require an active operator session. They return 503 with code: "billing_unavailable" when BILLING_ADMIN_URLS is not configured for the environment.

​
Billing triage response fields

The provider.mode field distinguishes a Stripe test-mode environment from a live one. A "test" value means the organisation’s billing data lives in Stripe test mode; credentials and events from that environment do not affect real payment instruments.

​
Refresh

POST /api/admin/organisations/:id/billing/refresh forces a re-read from Stripe via the regional billing service. Admin audits intent before the request and the outcome after. A failed request is audited as organisation.billing.refresh_failed. The read and refresh paths share the same signed command channel (stralos-billing-admin-command+jwt, audience stralos-billing-admin-<env>, expiry 30 seconds).

​
Configuration

Set billing_admin_urls in Terraform (type map(string), region key → HTTPS endpoint). The value is encoded as BILLING_ADMIN_URLS on the API Lambda. An empty map disables both billing triage routes.

​
Discount codes

The discount code surface lets operators manage customer discount codes through the regional billing service.

​
Routes

All routes require an active operator session. They return 503 with code: "discounts_unavailable" when BILLING_ADMIN_URLS is not configured.

​
List

GET /api/admin/discount-codes returns a paginated list of discount codes. Use the optional startingAfter query parameter (format promo_[A-Za-z0-9]+) for cursor-based pagination.

​
Create

POST /api/admin/discount-codes creates a new discount code. The request body is validated before the billing service is contacted, so a malformed request returns 400 without incurring a provider round-trip. Either percentOff or amountOff must be set, but not both. A "repeating" duration requires durationInMonths; other durations must omit it. A fixed amount requires currency; a percentage must omit it.

​
Activate or deactivate

POST /api/admin/discount-codes/:id/active activates or deactivates a code without affecting existing redemptions. The path parameter must match promo_[A-Za-z0-9]+.

​
Audit trail

Every create and set-active operation is audited twice: once as discount.<operation>.requested before the billing service call, and once as either discount.<operation>.completed or discount.<operation>.failed after. List operations are not audited. This two-phase record ensures a process failure between the request and the provider call leaves a traceable outcome in the audit log. A provider error returns 503 with code: "discount_request_failed". A provider validation error (for example, a duplicate code) returns 400 with the same code. The signed command uses typ: stralos-billing-discount-command+jwt, audience stralos-billing-discounts-<env>, and a 30-second expiry.

​
Contractor companies

A contractor company works inside other companies, its client companies, through a company link. A contractor company on the contractor plan can create a company for a client and place its own people there. Any company can send a contractor company a link by email. stralos-admin-api is the system of record for links and contractor memberships. The tenant-facing routes are reached through the Global API relay; see Contractor companies in the Global API reference.

​
Who can act

Each route re-derives authority from the admin store:
  • Contractor admin: an admin of the token’s company in their own right (the membership has no via_link_id), whose company is on the contractor plan. Manages client companies and the contractor’s people in them.
  • Company admin: an admin of the token’s company in their own right. Manages the contractors linked to that company.
A contractor person who is an admin inside a client company acts for their contractor company there, not for the client company, so the company-admin routes refuse them with 403 company_admin_required. A missing contractor plan returns 403 contractor_plan_required. There is at most one live link between two companies. Links never cross regions: a contractor can only be linked to companies homed in its own region.

​
Creating a company for a client

POST /api/admin/org/client-companies takes the company’s name, country, business type, time zone, optional numbering prefix and starting number (still named jobNumberPrefix and startingJobNumber on the wire; api-global applies them to the new company’s quote numbering), and the contractor’s people to place in it with their roles. The creator is always included as an admin. The new company is homed in the contractor’s region and records createdByOrganisationId. Its link is created active, with origin: created and managedBy: contractor. A created company’s name only has to be unique among the same contractor’s companies; a clash returns 409 client_company_name_taken. After the transaction commits, admin sends api-billing a cover command, so the company is covered by the contractor’s billing, and a link_state command. The response reports billingSync: "pending" if either command could not be sent; adding people later re-sends the cover.

​
Inviting a contractor

POST /api/admin/org/contractors/invites takes an email address. The address must belong to a user who is not disabled and whose default company is an active contractor company in the same region; otherwise the route returns 422 contractor_not_found or 422 contractor_region_mismatch. A live link between the two companies returns 409 contractor_already_linked, unless it is an expired invitation, which is ended and replaced. The new link is pending, invited and company-managed, and the invitation email is sent to that address. It expires after 14 days. Anyone signed in can read an invitation with GET /api/admin/org/contractor-invites/:token, which reports whether they can accept it and, if not, why (not_admin, not_contractor, wrong_company or expired). POST /api/admin/org/contractor-invites/:token/accept requires an admin, in their own right, of the invited contractor company while it is on the contractor plan. Accepting activates the link, gives the accepting admin an admin membership in the client company through the link, and sends a link_state command.

​
Contractor people and seats

Contractor people are memberships with via_link_id set to their link. A contractor admin adds one of their own active people to a client company with POST /api/admin/org/client-companies/:linkId/people and removes them with POST /api/admin/org/client-companies/:linkId/people/:userId/remove. People are added as active members without an invitation email; they receive an “added” notification instead. Contractor people never take one of the client company’s seats. stralos-api-login mints a contractor person’s token only while their link is active, and stamps it with the via claim (org, link, name). Migration 0042 grants the login role column-scoped SELECT on the link columns it needs, and nothing else on that table.

​
Ending, closing and taking over

Either side can end a live link: the contractor with POST /api/admin/org/client-companies/:linkId/end, the company with POST /api/admin/org/contractors/:linkId/end. Ending a link disables every membership that came through it and sends a link_state command. Project memberships are kept. A free company the contractor created and still manages cannot be unlinked (409 link_managed_by_contractor_close_instead). The contractor closes it instead with POST /api/admin/org/client-companies/:linkId/close, which disables the company and ends the link. Closing any other link returns 409 link_not_closable. POST /api/admin/org/client-companies/end-all ends every live link the contractor has, one transaction per link, and is used when a contractor company changes back to standard. A link that cannot be ended, for example because removing a person would leave a project without a lead, is reported in the response and does not stop the others. A company’s own admin takes over a created company with POST /api/admin/org/contractors/:linkId/take-over. The company must already have its own billing (409 company_needs_billing_first otherwise). The link becomes company-managed and records handedOverAt; the contractor’s people stay until the link is ended.

​
Billing commands

Both are signed commands (typ: stralos-billing-contractor-command+jwt, audience stralos-billing-admin-<env>, 30-second expiry) sent to the company’s home region through the BILLING_ADMIN_URLS endpoints.

​
Organisation payload

organisation.created and organisation.updated carry createdByOrganisationId: the contractor company that created the company, or null. The field is optional on parse, because outbox rows written before migration 0041 do not carry it.

​
Routes

All routes are under /api/admin/org and use the org self-service authentication (auth/orgAdminToken.ts and auth/orgAdmin.ts, or auth/orgUser.ts for the invitation routes). Operators can read a company’s links in both directions with GET /api/admin/organisations/:id/contractor-links; the admin console shows them in the billing triage panel.

​
Reconciliation

Reconciliation re-arms outbox rows the runtime missed and detects state drift between the admin store and the runtime replica. It runs as a scheduled pass (Lambda { mode: "reconcile" } or pnpm flush -- --reconcile) and only re-arms rows to pending; the flush schedule delivers them. M4 adds a conflicted-settle step: a conflicted row whose aggregate version is at or below the replica’s applied high-water mark is settled to processed rather than re-armed, preventing it from blocking successors and from stranding the provisioning saga in failed indefinitely.

​
User identity provisioning

IDP-M2b adds a Cognito provisioning layer between the admin-store user write and the invite email. Every user-mutating route enqueues an intent row in cognito_operations on the same database transaction as the users row change. The sweeper Lambda ({ mode: "cognito" }) executes those intents out of band — never on the request path.

​
User provisioning states

The provisioning column is set to provisioned only after AdminCreateUser succeeds. Invite dispatch gates on provisioning = 'provisioned', so no invite can point at a login that does not yet exist.

​
Cognito operation queue

Each operation reads the live users row at execution time (except link_provider, which has its payload fixed at enqueue). A crash between the Cognito call and the post-commit write leaves the operation pending for the next tick; UsernameExistsException on re-drive is treated as success. Operations move from pending to applied. Only create and link_provider can enter a terminal failed state; disable, enable, delete, and email_change retry until they converge or an operator resolves the underlying problem.

​
Cognito pool reconciliation

The daily cognito-reconcile pass ({ mode: "cognito-reconcile" }) diffs the Cognito pool against the admin store and repairs discrepancies by enqueueing sweeper operations. It owns the cognito_operations table and the users.provisioning column only — it never touches users.status, memberships.status, or the admin outbox. Pool accounts whose Username is not a UUID and is not a linked provider shadow are reported as foreign accounts and never touched.

​
Invite email dispatch

IDP-M2b delivers invite emails through Amazon SES v2 from a dedicated mail subdomain (id.<envdomain>), keeping mail identity ownership separate from the application apex.

​
IAM requirements

When invite mail is enabled, three roles need ses:SendEmail:
  • signup sends verification codes during self-service signup;
  • event_flusher sends the scheduled first invitation;
  • the shared lambda role serves POST /api/admin/users/:id/resend-invite.
All three roles use the same enumerated identity set. ses:FromAddress fixes the sender mailbox, while the resource list contains the sender identity and configuration set. SES sandbox accounts also authorise sends against the recipient identity. The Terraform source therefore includes both domain and address-pattern identity ARNs for stralos.co.nz in the dev and test environment allowlist. staging and prod receive the sender identity only. This is an environment allowlist in ses.tf, not a .tfvars setting. SES production access is granted per account. After an environment leaves the sandbox, remove that environment from ses_sandbox_recipient_environments; do not remove another environment until its own account has production access. The first-send path records its claim before calling SES. If delivery fails after that point, the dispatcher raises an alert and leaves the row claimed, and an operator must use the resend action. Keep the shared send policy attached to all three roles so verification, first-send, and recovery paths remain aligned.

​
Dispatch behaviour

The scheduled dispatcher ({ mode: "invites" }) runs two passes in sequence on every tick. Account invite pass — sends each eligible user’s first invite exactly once (at-most-once guarantee). Eligibility requires:
  • provisioning = 'provisioned' — the Cognito account exists;
  • status is not disabled; and
  • users.invite_sent_at is null — not yet attempted.
The claim commits before the SES call. A send that fails after the claim leaves invite_sent_at set with a null invite_message_id (attempted, outcome unknown). The dispatcher logs an error and raises an alert; an operator resolves it via the resend endpoint. Membership notification pass — sends a separate notification email to users who are invited into an additional organisation. An existing user invited into a further organisation previously received no signal: the Invitations section appears only in the login chooser during a sign-in ceremony, and sessions live 30 days, so an invitation could sit invisible for weeks. This pass is distinct from the account invite pass. Eligibility requires:
  • memberships.status = 'invited';
  • memberships.notified_at is null — not yet attempted;
  • the user is not disabled; and
  • the organisation is active — mail only when acceptance can succeed.
The “additional” condition is enforced by requiring a strictly-older sibling membership row for the same user (any status, including historical). This keeps the two passes disjoint: a brand-new user’s first membership is handled by the account invite pass, which already deep-links that invitation in the account email. The same at-most-once discipline applies: memberships.notified_at is set before the SES call and never cleared automatically. A post-claim SES failure logs an error and leaves the row claimed (notified_at set, notified_message_id null) for an operator to resolve. It is never re-sent automatically. One exception: if eligibility fails after the claim but before sending (for example, the organisation was disabled mid-tick), the claim is released — the outcome is definitively known (nothing was sent), so releasing is safe. Existing invited membership rows without a notified_at value become eligible for one notification each on the first post-deploy tick. This is the intended behaviour: they are invitations that nobody has been told about. The email tells the recipient that a named organisation has invited them and directs them to sign in and accept via the organisation chooser.

​
Provisioning and invite fields

users table memberships table (membership notification pass)

​
Operator resend

POST /api/admin/users/:id/resend-invite allows an operator to re-send an invite via SES. The route requires provisioning = 'provisioned'; it returns 409 with code user_not_provisioned when the Cognito account has not yet been created. Returns 503 when invite email is not configured (mail_hosted_zone_id not set). This path runs on the api Lambda, which does not hold Cognito admin grants and cannot mint a temporary password. The SES email therefore carries no credential and states this plainly. An operator who needs the recipient to receive a working temporary password should use the organisation-admin resend path instead.

​
Organisation-admin resend

POST /api/admin/org/users/:userId/resend-invite is the organisation-admin resend path. It enqueues a resend_invite Cognito operation in the same transaction as the request. The sweeper executes that operation by calling AdminCreateUser with MessageAction: RESEND, which mints a fresh temporary password and lets Cognito send the invitation email. The sweeper records invite_sent_at inside the same transaction. inviteMessageId is always null on this path because Cognito does not return a message ID; the audit row carries channel: "cognito-resend" to distinguish it from an SES send with an unknown outcome. Previously, the sweeper’s resend_invite execution also triggered a second SES email from the admin API, resulting in two invitation emails per resend — the first being Cognito’s stock template naming the internal UUID as the username. The Cognito pool now carries an InviteMessageTemplate (configured in stralos-core) that names the email address and includes the temporary password token; the admin API no longer sends a second email on this path.

​
Sends with an unknown outcome

There is no operator worklist query. findUnconfirmedInvites and findUnconfirmedMembershipNotifications had no route, CLI or console view calling them, and were removed in stralos-admin-api PR 186. An operator finds these rows by their state instead:
  • Account invites: users rows with invite_sent_at set and invite_message_id null, provisioning = 'provisioned', a status other than disabled, and no signup_intent_id (self-service signups never get an invite). A Cognito resend also leaves the message id null, so check the audit row’s channel as described under the users table above. Resolve each one with the operator resend endpoint.
  • Membership notifications: memberships rows with notified_at set, notified_message_id null and status = 'invited'. There is no automated retry.
Invite links deep-link to the regional shell login URL for the user’s organisation. The ?invite=<membershipId> parameter is a selector only — it names which invitation the user clicked, but grants no authority on its own. The exchange re-validates the membership against the authenticated user before accepting it. For federated sign-in (Google, Microsoft Entra), the pre-signup Lambda reads a DynamoDB lookup table to match an inbound provider identity to a native user UUID. IDP-M2b transfers the writer responsibility from elsewhere to the link_provider Cognito operation. The admin-api writes the lookup row as part of the link_provider sweeper operation. A queued link_provider with no IDENTITY_LINKS_TABLE configured fails loudly rather than silently skipping, because a missing row means a federated user cannot sign in at all. Identity keys are built from immutable provider identifiers only: A link_provider operation is idempotent: if the same user already holds the key, the write returns already-linked. An attempt to claim a key held by a different user is a definitive fault that alerts an operator and does not retry.

​
Invitation generation

The memberships.invitation_generation column (a bigint, default 1) is a stale-acceptance guard. When the runtime reports a membership acceptance, the report envelope carries the invitationGeneration value the runtime projected. The report channel rejects the acceptance with stale_invitation if the value does not match the current admin store value. The generation advances only when a revoked membership is re-invited (status transitions back to invited). It does not advance on acceptance, revocation, or role change. The migration 0007_idp_m2a_report_channel.sql backfills invitationGeneration = 1 into legacy outbox rows so delivery can parse the field.

​
Directory backfill

M3b introduces the directory backfill: a one-shot CLI mode that emits a directory.regions_replaced event for every user who does not yet have one, giving the runtime a directory anchor for each account. Run it before relying on directory routing for accounts created before M3b. Run it with:
The backfill is resumable. Each invocation processes up to the configured deadline and prints its runId; pass that id on the next invocation to continue from the cursor rather than restarting. A clean run requires failed = 0 for the whole runId, the cursor to have reached the end of the users table, and every emitted replacement to be confirmed at the directory anchor via the runtime replica. The DIRECTORY_WRITER_REGION_KEY environment variable must be set to the region key of the runtime that holds the directory write path. It must name a region present in both RUNTIME_INGEST_URLS and RUNTIME_REPLICA_URLS. There is no default; omitting it causes the process to fail at startup.

​
SSM parameter management

The seven admin SSM parameters are Terraform-managed resources defined in terraform/ssm-parameters.tf. Terraform owns the parameter objects; the deployment finaliser or an out-of-band operator step writes the application and Google values after apply. The signup-code pepper keeps its Terraform-generated random value and is readable only by the signup role. The Signup role column indicates whether the signup Lambda’s IAM grant includes the parameter. The Google credentials are excluded: the signup function has no operator routes and the ADMIN_GOOGLE_* environment variables are stripped from it entirely. The google_client_id parameter is a public identifier and remains type = "String". Keeping the declared type aligned with adopted parameters also prevents Terraform from rewriting an existing value during adoption. The database path is fixed by the stack. The signing-key paths may be overridden with event_signing_key_ssm_parameter, operator_signing_key_ssm_parameter, and totp_encryption_key_ssm_parameter. The Google credential paths may be overridden with google_client_id_ssm_parameter and google_client_secret_ssm_parameter. The totp_encryption_key parameter is retained from the previous email + TOTP scheme. It is still provisioned by Terraform but is no longer used by the sign-in path.

​
Cold-start preload

src/lambda.ts awaits preloadApiSsmParameters() (src/config/ssm-preload.ts) before it serves any request. The preload uses the shared @stralos-co-nz/api-common parameter cache: batched GetParameters calls of up to ten names, with jittered retry on throttling. After the preload, SSM-backed reads answer from memory, and an expired value is still served while the cache refreshes it in the background, so no request waits on Parameter Store. Two functions run lambda.ts, api and signup, with different environment variables. The preload skips any variable a function is not given. A new *_SSM_PARAMETER read on the request path must be added to the preload list; test/ssmPreload.unit.test.ts fails until it is. Both the shared Lambda role (ReadAdminKeyMaterial) and the signup role (ReadAdminSignupMaterial) grant ssm:GetParameter and ssm:GetParameters for the batched preload.

​
Google sign-in Terraform variables

Enabling Google sign-in requires four steps in order:
  1. Create the OAuth client in Google Cloud.
  2. Store the client ID and secret in SSM at the expected paths.
  3. Set enable_google_sign_in = true, google_hosted_domain, and (if not using the default) google_redirect_uri in the environment’s .tfvars.
  4. Apply Terraform, then sign in.
While enable_google_sign_in = false, the sign-in routes answer 503. With the password path removed, this means the admin console for that environment is unreachable until the flag is enabled.

​
New environment provisioning

For a fresh environment, Terraform creates all seven parameter objects with a cryptographically random terraform-bootstrap-pending:<random> placeholder using a write-only (value_wo) block. The placeholder is ephemeral — it is never persisted to plan state or Terraform state. value_wo_version = 0 combined with ignore_changes = [value_wo_version] ensures that subsequent applies do not overwrite the values written by the finaliser. For signup_code_pepper, that random initial value is the retained secret rather than a placeholder awaiting the deployment finaliser. The prevent_destroy = true lifecycle guard prevents accidental destruction of parameters that hold live key material. The four application SecureStrings (database_url, event_signing_key, operator_signing_key, totp_encryption_key) are populated by the deployment finaliser after apply. The two Google sign-in credentials (google_client_id, google_client_secret) have no finaliser; write their real values out of band:
Supply the secret command from a protected 0600 JSON input file containing the parameter name, value, SecureString type, and Overwrite: true; remove that file through the approved secrets-handling workflow afterwards. Do not put the client secret directly in shell history or process arguments. Until the real values are written, Google sign-in cannot complete successfully.

​
Adopting pre-existing parameters

Where parameters were provisioned before Terraform ownership, set adopt_existing_ssm_parameters = true in the environment’s .tfvars file. This activates import blocks for every parameter flagged adopt = true in the managed-parameters map. Adoption brings those parameters under Terraform management without replacing their values. The ignore_changes rule on value_wo_version ensures that the import apply cannot re-seed a parameter that already holds live key material. The adopt flag is set only on parameters that genuinely pre-exist in a given environment. Parameters created fresh by this apply do not carry adopt = true; including them in an import block would fail the plan because Terraform cannot import a resource that does not yet exist. Leave adopt_existing_ssm_parameters at its default false when Terraform should create fresh parameters.

​
IDP-M2b Terraform variables

The following optional variables enable Cognito provisioning, invite email, the federated identity-link table, and the runtime→admin report channel. Every variable defaults to null or false; an unset value omits the corresponding feature entirely rather than half-configuring it. The cognito and invites schedules are each gated on their respective variables: an unset cognito_user_pool_id means no cognito EventBridge rule is created, so there is no periodic failure from a missing pool ID. Invite mail sends from no-reply@<invite mail domain>. Terraform creates the identity on a dedicated mail subdomain rather than the environment apex, so each environment owns one identity and its DNS records in a single state. It also creates the <prefix>-invites configuration set (TLS required, reputation metrics on) and passes its name to the Lambdas as SES_CONFIGURATION_SET; every send names it. Prod enables ses_feedback_notifications_enabled, because its SES production access relies on bounces and complaints reaching the alerts topic. The account-level SES suppression list (bounces and complaints) is enabled out of band in the account; Terraform does not manage it. Set region_login_urls whenever mail_hosted_zone_id is set. Enabling invite mail adds INVITE_FROM_ADDRESS to the API and events functions. The API resolves the regional URLs at startup for operator resend, and the events worker resolves them when it dispatches invitations. If neither REGION_LOGIN_URLS nor the single-region INVITE_LOGIN_URL fallback is present, resend configuration cannot initialise and invite dispatch fails. Keep region_login_urls and mail_hosted_zone_id together in each environment’s configuration.

​
Organisation self-service Terraform variables

The following variables enable the organisation self-service surface (/api/admin/org/*). Both must be set together. Setting only one half leaves the surface in an unsafe or non-functional state, so they are deployed in two separate Terraform applies in a fixed order. The map key in user_token_source_map is compared for exact equality against organisations.home_region_key. Spelling it differently from the stored region key (for example aunz instead of au-nz) causes every organisation to receive 403 with no application log entry that names the mismatch. ADMIN_USER_TOKEN_SOURCES is set on the API Lambda only, not on the shared Lambda environment. Placing it in the shared environment causes an init failure and a 502 on every route across all five Lambda functions, including /health, whenever the value is absent or malformed. The issuer attribute is optional. Omit it to use the default stralos-api-global. Terraform materialises an unset optional attribute as null, so the Lambda environment encoding explicitly strips the null key before calling jsonencode; the consumer’s schema accepts an absent issuer but rejects null.

​
Phased deployment

These two variables must be applied in two separate Terraform applies:
  1. Phase 1 — set org_admin_caller_role_arns only. This creates the AWS_IAM method chain on /api/admin/org/* and the matching resource-policy Deny. The path is now reachable only by the named roles and still returns 503 because the verifier is off.
  2. Phase 2 — set user_token_source_map. This enables ADMIN_USER_TOKEN_SOURCES on the API Lambda and the org self-service routes start answering.
Applying both variables in the same Terraform run is unsafe. Terraform may update the Lambda environment before the new API Gateway deployment and stage go live. In that window the verifier is on while the path is still served by the root {proxy+} catch-all (no IAM requirement), which places the org self-service surface on the open internet with only the user token as a gate. Phase 1 must be confirmed live in each environment before Phase 2 is applied to that environment.

​
Release journey probe Terraform variable

service_token_source_map enables the GET /api/admin/service/probe endpoint used by the PRD 03 promotion pipeline to verify a promoted environment answers an authenticated admin request. The map shape mirrors user_token_source_map. The issuer attribute is optional; omit it to use the default stralos-api-global. Terraform strips null issuer values before encoding to avoid an empty-string init failure. ADMIN_SERVICE_TOKEN_SOURCES is a separate environment variable from ADMIN_USER_TOKEN_SOURCES. They are two independent trust decisions. Setting one does not enable the other.

​
Service probe endpoint

GET /api/admin/service/probe?nonce=<hex> requires a bearer service token with scope admin:probe-read. The response body contains only constants and the caller’s own nonce: The nonce echo is load-bearing. Without it a 200 response could be satisfied by a cached response or an error page that returned the right shape. The endpoint reads no tenant row, no operator row, and no catalogue item. The route returns 503 when ADMIN_SERVICE_TOKEN_SOURCES is not configured, 401 when no bearer token is presented or verification fails, 403 when the token does not carry admin:probe-read, and 503 when the api-global JWKS cannot be fetched (distinguished from a bad token to avoid misleading the promotion gate’s negative control).

​
Deployment environments

The Terraform deploy workflow plans dev on pull requests. Dev applies run through the separate manual Deploy dev workflow. Test and production apply through a validated release promotion or through an explicit manual dispatch carrying the frozen promotion digest. Run the workflow with target=test to deploy the test stack. The repository also contains staging.tfvars with https://admin.staging.stralos.app and event sync disabled, but staging is not a selectable workflow target. For a first production apply, do not create the Google SSM parameters by hand: Terraform creates them with bootstrap placeholders. Register https://admin.stralos.app/auth/callback on the shared Google OAuth client, then replace both placeholders with the real client ID and secret as described in New environment provisioning. Until they contain real values, Google sign-in cannot complete successfully. The production configuration auto-provisions a verified stralos.co.nz Workspace account on first sign-in, so no operator bootstrap is required for this path. When enable_event_sync = false, Terraform omits the event flush, reconciliation, provisioning saga, Cognito sweep, and invite EventBridge schedules and triggers. The API, database-migration Lambda, and events Lambda still deploy; the events Lambda remains present but unscheduled.

​
Writer cutover step

The deploy workflow includes an optional writer-cutover step that transfers the directory-writer role to the current environment. The step is opt-in: it does not run on a standard push to main or a normal dispatch. Enable it explicitly when you intend to perform the cutover as part of a deploy. When the step is enabled and the cutover fails for any reason — including an unreadable or absent marker — the workflow fails closed. The deploy does not continue past a failed cutover. Confirm the marker is readable and the expected writer state is in place before enabling the step.

​
Infrastructure protection and backups

The production Terraform configuration associates the admin CloudFront distribution (admin.stralos.app) with the shared edge WAF Web ACL. The regional WAF association remains on the API Gateway stage behind it. Edge filtering runs before a request reaches that origin. The admin RDS instance’s automated backup retention is configured to 30 days in production and 7 days in dev and test. These database backups cover the admin store and its supporting tables.

​
Monitoring

The admin plane carries CloudWatch alarms covering all admin Lambda functions and the API Gateway entrypoint. Alarms fire to an SNS topic (${name_prefix}-alarms). The topic requires a subscription to deliver notifications; creating that subscription is an operational follow-up and is not provisioned by Terraform.

​
Lambda alarms

Every admin Lambda has two alarms:
  • Errors — trips when the function throws (crash, timeout, OOM, or init failure). The threshold is 1 for rarely-invoked functions (migrate, operator-ops, secret-sync, and signup when enabled) and slightly higher for the traffic-bearing ones (api: 3, events: 5) to account for self-healing retries without masking sustained failures.
  • Throttles — trips on the first throttle in a 5-minute window.
The two high-throughput functions (api and events) also carry an error-rate alarm. It evaluates 100 * errors / invocations only when there are at least 10 invocations in the period, which prevents a low-denominator false alarm from a quiet window. The threshold is 20%. This alarm catches a proportional burst (such as a credential-desync outage) even when the absolute Errors threshold has not been reached. A Terraform check block compares the alarm map with the Lambda functions already live in AWS and reports a non-blocking plan warning for any gap. A function introduced by the same apply is visible only on a later plan, so this is drift detection rather than a deployment gate.

​
Secret-sync dead-man’s switch

The secret-sync function carries an additional sweep-stalled alarm. It monitors AWS/Events SuccessfulInvocationAttempts on the sweep EventBridge rule (not the Lambda invocation count), so it fires for both a dead schedule and a schedule that fires but fails to deliver. The alarm uses a 1-hour period, evaluation_periods = 3, and treat_missing_data = breaching. If the default hourly schedule is changed, the period must be updated to match.

​
API Gateway 5xx alarm

The api Lambda handles failures internally and returns HTTP 5xx responses rather than throwing. Those responses count as successful Lambda invocations, so the Lambda Errors alarm stays at zero during an application or dependency outage. The API Gateway 5XXError metric captures them instead. The ${name_prefix}-apigw-5xx alarm trips on the first 5xx in a 5-minute window (threshold = 1, treat_missing_data = notBreaching). It complements the Lambda Errors alarm: together they cover both function-level failures and application-level HTTP failures for the API entrypoint.

​
Safe error handler

Route handlers map known errors to typed responses. Any error that handleRouteError does not map is rethrown; previously Fastify’s default handler answered with error.message, which let verbatim Postgres text reach the response body. The safe error handler (registerSafeErrorHandler from api-common) intercepts every rethrown error and answers with fixed copy plus a correlation reference:
  • A database-saturation error (Postgres code 53300) returns 503 with code: "service_unavailable" and Retry-After: 5. The driver text never appears in the response; the full error is written to the log under the same reference.
  • Any other unhandled error returns 500 with code: "internal_error" and a reference. The describer logs only name, code, message, and stack — not any additional properties the error carries. This prevents token-claim errors from leaking the decoded JWT payload into structured logs.
The reference in both responses is the same value the caller receives, so an operator can locate the log entry without the error text being visible to the caller.

​
API Lambda Postgres pool

The API Lambda’s Postgres pool is capped to prevent connection exhaustion on the shared admin database. Local development, the CLI tools, and the one-shot Lambda functions (migration, operator-ops) keep the pg defaults (max: 10, no connection timeout).

​
Related page

See Admin console frontend for the operator SPA.

​
Lambda activation

The api and retention functions expose named live aliases. API Gateway targets the API alias, and the weekly retention schedule targets the retention alias. The dedicated signup integration targets its separate function. Terraform lists the aliased functions in release_functions and ignores alias function_version changes. In a full deployment, the core post-apply step moves the listed aliases to the versions Terraform published. The API function ignores source_code_hash, so that initial version can hold the previous code: its finaliser subsequently promotes the migrated API artifact to the alias. Retention code is shipped by Terraform, so its post-apply alias move activates that version. Do not assume the post-apply API alias move alone activates the new API artifact.