Async Runtime

Novantra uses the async runtime for background work, route rate limits, worker coordination, and broker wakeups. The database outbox remains the durable source of truth: a business change writes its normal records and an outbox row in the same transaction. Cache and broker providers help the worker fleet react faster and coordinate safely, but they do not replace the database record.

What It Handles

Broker messages carry only pointers such as deployment scope, target id, outbox event id, and event type. They must not contain document text, prompts, outputs, credentials, storage keys, database URLs, vectors, embeddings, or raw provider payloads.

Supported Profiles

Single-DB pilot is valid for staging or early production if it is clearly treated as non-HA. Do not use it as evidence for active-active, 99.99%, or enterprise HA commitments.

Deployment profiles encode this distinction. A single-db-pilot profile can run the async runtime, but Novantra will not treat it as HA evidence. HA evidence requires a profile with HA database posture, shared cache, broker, PITR, restore testing, and connection pooling. Active-active or 99.99 evidence is not supported by current profiles, which do not promote active-active database posture.

Cache And Rate Limits

Production product deployments require a Redis-compatible shared cache for route limits, locks, and approved hot-path caches. Local development and tests may use in-memory or database fallback.

Cloud, Sovereign, Control Plane, Customer Portal, and Public Access runtimes read the shared route-limit cache from:

ASYNC_RUNTIME_REDIS_URL=rediss://redis.example.invalid
ASYNC_RUNTIME_REDIS_NAMESPACE=novantra:cloud:route-limits

Use rediss:// in production profiles. Cloud, Sovereign, Control Plane, and Public Access production runtimes fail closed when the shared route-limit store is required but not configured. Customer Portal can use the provider database-backed route-limit window store as an explicit fallback profile.

Cloud and Sovereign /api/v1 token-auth, per-service-account, and per-organization API rate limits also use this shared async runtime store when Redis is configured. The public API keeps the same X-RateLimit-* and Retry-After headers; database-backed API rate-limit windows remain the module-owned fallback path.

The shared cache is only for operational metadata:

• opaque namespace keys and hashes;
• counters and fixed-window rate-limit state;
• short-lived capability or posture refs;
• worker coordination state.

It is not a place to store customer-authored content, secrets, document text, prompts, outputs, embeddings, vectors, database connection strings, or storage paths.

Broker Wakeups

RabbitMQ is the supported self-hosted broker adapter. DB polling remains available as fallback and as the local/dev default.

Use a broker when you need lower latency or more worker capacity. A worker receiving a broker message must still claim the canonical outbox row from the owning database before it does work. If a broker message is duplicated, delayed, or lost, the database outbox and fallback polling path preserve correctness.

Durable Subscriptions

Modules can declare durable subscriptions through the deployment manifest. A durable subscription is a cursor-backed worker path for replayable projections or safe side effects, such as dashboard counters, risk badges, connector health summaries, or realtime signals.

The subscription handler receives the same pointer-safe outbox event that lives in the owning database. The cursor advances only after the handler finishes successfully. If a handler fails, Novantra records a sanitized reason code and leaves the cursor unadvanced so the work can be retried without losing the event.

The async runtime owns dispatch, cursor movement, worker job wrapping, and sanitized failure posture. It does not own the business projection or result. Handlers must call the owning module’s command, projection, or action seam instead of writing another module’s records directly.

Each durable subscription must declare an owner matrix before it can run. The matrix names the source event owner, source data owner, work or projection owner, status owner, result owner, evidence owner, security-event owner, and any downstream handoff owners such as connectors, content extraction, AI, SIEM, reporting, or assurance. This keeps the runtime generic: it can coordinate the work, but the module that owns the business result remains responsible for the data, evidence, and user-facing status.

Current product subscriptions include connector-owned health aggregation, connector-owned AI-preparation job status updates, and a compliance-posture projection for connector health. In each case the event payload remains pointer-only: the connector health record, connector job, and connector instance stay with Enterprise Connectors, while Compliance Posture records only a sanitized signal against the connector instance.

Worker Configuration

The jobs worker defaults to DB polling:

pnpm jobs:worker -- --deployment=onprem --scope=install

To use RabbitMQ for outbox wakeups, provide the connection URL through an environment variable managed by your runtime or secret manager:

ASYNC_RUNTIME_RABBITMQ_URL=amqps://broker.example.invalid pnpm jobs:worker -- --deployment=onprem --scope=install --transport=rabbitmq

You can point the worker at a different environment variable name:

NOVANTRA_QUEUE_URL=amqps://broker.example.invalid pnpm jobs:worker -- --deployment=onprem --scope=install --transport=rabbitmq --rabbitmqUrlEnv=NOVANTRA_QUEUE_URL

Do not persist broker URLs or passwords in runtime binding records. Store credentials in the deployment secret manager and expose them only to the worker process.

Operation Status Handoff

Routes that submit long-running work return the normal Novantra response envelope and, where the owning module exposes status, an asyncOperation object. The object contains only sanitized metadata:

• operation id;
• owner scope;
• resource type and resource id;
• stage and module status;
• status URL;
• optional progress counts;
• optional retry timing or sanitized reason code.

When the work is queued or running, the route returns 202 Accepted and sets Location to the status URL. When the work completed during the request, the route may keep its normal 200 OK or 201 Created response and still include asyncOperation so clients can link to the run record.

The status URL points back to the owning module. Storage migrations, key-management operations, backup/restore operations, release-gate checks, AI runs, and content-extraction runs keep their durable status in their own tables. The async runtime does not create a separate business-status database.

Enterprise Connector bulk Prepare selected for AI uses this pattern. When eligible source refs create connector-owned jobs, the route returns 202 Accepted, sets Location to the connector job status URL, and includes asyncOperation metadata with source/handoff counts, retry count, next retry time, and sanitized failure code where present. The connector job only proves source-owner handoff posture; extraction, indexing, embeddings, vectors, retrieval, and AI answers remain with their owning modules.

Isolation

Tenant and organization databases remain the durable isolation boundary. Provider or install databases may hold only sanitized worker coordination, target discovery, aggregate lag, and health posture. Shared Redis or RabbitMQ infrastructure is acceptable only when namespaces, vhosts, queues, and permissions prevent cross-tenant access and messages remain pointer-only.

Regulated or dedicated-cell profiles may require physically separate cache or broker infrastructure. Treat that as a deployment profile decision, not a product code fork.

Health And Failure Modes

Readiness should distinguish between:

• DB polling available: workers can still process durable outbox rows.
• Cache unavailable: production route limits and locks should fail closed when the cache is required.
• Broker unavailable: workers fall back to DB polling if the business mutation committed.
• Worker degraded: heartbeat, failure count, dead-letter count, or lag is outside policy.

Operator and readiness surfaces should consume the sanitized async runtime health summary. It reports only status, reason codes, timestamps, and aggregate counts for bindings, workers, targets, dispatch receipts, and subscription cursors. It does not expose raw provider diagnostics, queue names, credentials, tenant payloads, prompts, source text, vectors, embeddings, or database URLs.

Provider health checks return sanitized status and reason codes. They should not include raw provider errors, URLs, credentials, stack traces, or payloads.

Runtime Probes

Web and worker runtimes should report liveness and readiness with the shared async runtime probe contract:

• liveness means the process is alive and should not check databases, cache, brokers, or tenant state;
• readiness means the process can accept its assigned work and required dependencies are reachable;
• draining readiness returns not-ready so orchestration can stop assigning new work while liveness remains alive;
• degraded async health stays visible through reason codes, but does not automatically fail readiness unless the dependency is unavailable or misconfigured.

Worker probes should include only sanitized worker identity, worker kind, target ref, build version, dependency keys, status, and reason codes.

Diagnostic Retention

Async runtime diagnostics follow the manifest-declared async runtime diagnostics retention rule. Cleanup is same-scope:

• provider and install cleanup can prune terminal dispatch receipts, stale worker heartbeats, disabled target projections, released or expired shard leases, old provider validation messages, and old cursor diagnostic snapshots;
• organization cleanup can prune only org-local terminal dispatch receipts and cursor diagnostic snapshots;
• cleanup must not delete canonical outbox_events, active subscription cursors, module-owned operation records, audit records, security events, or evidence records.

Security And Logging

Logs, SIEM exports, audit metadata, and health records should include:

• provider kind;
• capability state;
• worker id;
• queue or namespace posture as a sanitized ref;
• aggregate counts and lag;
• reason code.

They should not include:

• Redis or AMQP URLs;
• passwords, bearer tokens, or signing secrets;
• raw queue/topic names containing customer identifiers;
• document text, source text, prompt/output content, vectors, embeddings, or provider payloads.

Related

• Health & Monitoring
• Backup & Restore
• Cloud Regions & Residency
• Self Managed Secret Keys