Workflows & runtime · 1.3.4

Durable workflows, typed LLM tools, vector search, realtime, queues, tenants.

Carrier^™’s runtime surface in 1.3.4: workflow, llm client, rag, queue, tenant, flag, sla, invariant, realtime stream / subscription / watch, native PDF, native OTLP telemetry, the typed api_explorer, and in-language test blocks.

Workflows →LLM clients docs/workflows.md

Workflows & sagas

Durable background processes. Each run stores input, checkpoints every successful step, retries failed steps up to the declared limit, and runs compensation blocks in reverse when a later step fails.

Run state

carrier_workflow_state

Task queue

carrier_workflow_tasks

Status enum

WorkflowStatus · 9 variants

Modes

linear · saga · graph

src/workflows/checkout.carrier

workflow · step · compensate

workflow CheckoutSaga {
  input: CheckoutRequest
  output: CheckoutResult
  timeout_seconds: 300
  max_retries: 2
 
  step reserve_inventory -> ReservationSummary
  retry {
    max_retries: 3
    delay_seconds: 10
  } {
    return Inventory.reserve(input.sku, input.quantity)
  }
 
  compensate reserve_inventory {
    Inventory.release(reserve_inventory.reservation_id)
  }
 
  step charge_card -> ChargeSummary {
    if input.simulate_failure {
      fail("checkout_failed", "card authorization declined")
    }
    return Payments.charge(input.payment_method, input.amount_cents)
  }
 
  compensate charge_card {
    Payments.refund(charge_card.charge_id)
  }
 
  step fulfill -> CheckoutResult after charge_card {
    return Fulfillment.ship(
      reservation: reserve_inventory.reservation_id,
      charge: charge_card.charge_id
    )
  }
 
  return fulfill
}

Per-step retry

Each step can override the workflow-wide retry budget via retry { max_retries · delay_seconds }. Linear attempt counters reset after each successful step.

Durable leases

timeout_seconds marks leased runs as timed_out. Compensation progress survives crashes — completed compensations never re-run.

Operator recovery

WorkflowName.retry_compensation(run_id) resets compensation_failed runs back to compensating — no hand-edits to system tables.

Parallel graph execution

Branches run in parallel when workers are free. Explicit after-dependencies plus when-guards keep branching deterministic and compile-time checked.

src/workflows/greeting_graph.carrier

after · when · parallel

workflow GreetingGraph {
  input: GreetingGraphInput
  output: GreetingWorkflowOutput
 
  step prepare -> GreetingWorkflowOutput {
    return { status: "prepared", message: "Hello " + input.name }
  }
 
  step auto_approve -> GreetingWorkflowOutput
    after prepare when !input.manual_review
  {
    return { status: "auto_approved", message: prepare.message }
  }
 
  step manual_review -> GreetingWorkflowOutput
    after prepare when input.manual_review
  {
    return { status: "manually_reviewed", message: prepare.message + " (reviewed)" }
  }
 
  step finish -> GreetingWorkflowOutput after auto_approve, manual_review {
    return {
      status: "finished"
      message: input.manual_review ? manual_review.message : auto_approve.message
    }
  }
 
  return finish
}

How it runs

Graph workflows enqueue one row in carrier_workflow_tasks per runnable step. Independent branches execute in parallel across available workflow workers.

Guard semantics

A when guard that evaluates false marks the step as skipped and still unblocks dependents once their other dependencies are terminal.

SLA declarations

Source-controlled service-level contracts attached to a workflow. Carrier validates the workflow + step references and emits the contract under .carrier/manifest.json slas + OpenAPI x-carrier-slas. Runtime SLA instance storage and breach scheduling are out of scope for this slice — the contract is the artifact.

src/slas/sepsis.carrier

attainment · escalation

sla SepsisBundle {
  workflow:    AdmitSuspectedSepsis
  starts_when: step_started("RecordVitals")
  ends_when:   step_completed("CompleteBundle")
  attainment:  95%
  deadline:    4_hours
  warning_at:  80%
  breach_at:   100%
  scope:       tenant
  measure_by:  calendar_month
  escalation:  notify(role: "charge_nurse") on breach_imminent
  compliance_evidence: attach_to_audit
  report:      include_in("ClinicalQualityDashboard")
  exclude_when: field("comfort_care_only") == true
}

Start / end

starts_when + ends_when accept step_started, step_completed, event(...), and status(...).

Windows

measure_by: calendar_month or rolling_7_days. attainment, deadline, warning_at, breach_at.

Escalation + reports

notify(...) on breach_imminent | breached, compliance_evidence: attach_to_audit, report: include_in(...).

Invariants & verification

A focused first slice of compile-time invariant verification. Carrier prefers a real z3 binary on PATH; if absent, a deterministic bounded search keeps verification usable in minimal CI environments. Results land in .carrier/verification.json and become CI gates via carrier check --verify --fail-on-unknown.

src/invariants.carrier

invariant · verification · assume

verification {
  solver:             z3
  mode:               bounded
  max_records:        1
  max_workflow_depth: 10
  timeout:            30_seconds
}
 
invariant InventoryNeverNegative {
  subject:     InventoryItem
  must_always: quantity_on_hand >= 0
}
 
invariant LabResultRequiresReviewer {
  subject:    LabResult
  transition: status -> "reviewed"
  requires:   reviewed_by != ""
}
 
invariant NoProviderOverlap {
  subject:    Appointment
  must_never:
    exists other Appointment where
      other.provider_id == provider_id &&
      overlaps(other.start_slot, other.end_slot, start_slot, end_slot)
}
 
assume StripeDoesNotOverRefund {
  external:   stripe.refund
  guarantees: refund.amount <= charge.remaining_refundable_amount
}

Forms

must_always, must_never, transition style with transition: field -> value + requires, and bounded existentials with exists Model where ... and overlaps(...).

Trusted assumptions

assume Name { external · guarantees } pins the upstream contract Carrier is allowed to rely on during verification.

Solver fallback

CARRIER_Z3_PATH or z3 on PATH is preferred. Without a solver, the scalar subset falls back to a bounded deterministic search.

Tank hardening · service.runtime

A first-class runtime block under service that bounds depth across requests, actions, routes, workflows, expressions, and JSON bodies. The runtime captures panics, emits structured CARRIER_DEPTH_LIMIT_EXCEEDED errors, and exposes /ops/runtime + /ops/flight (Carrier Flight Recorder).

src/main.carrier

runtime · panic_policy · ops_routes

service App {
  openapi { title: "Tank-Hardened API"  version: "1.0.0" }
 
  runtime {
    request_depth_limit:    32
    action_depth_limit:     16
    workflow_depth_limit:   16
    expression_depth_limit: 64
    json_body_depth_limit:  16
    schema_expansion_depth: 16
    body_size_limit:        2_megabytes
    artifact_size_limit:    50_megabytes
    request_timeout_ms:     5_000
 
    panic_policy: catch
    evidence:     emit_when_explicitly_enabled
    ops_routes:   local_only
  }
}

zsh

tune · check --verify · harden --chaos --run --sandbox-db

# 1. local sizing advisor
$ carrier tune --profile production
# writes .carrier/tune.json + .carrier/tune.runtime.carrier (copy-ready)
 
# 2. budgets + verification + invariant proofs + tests
$ carrier check --verify --profile production --fail-on-unknown
 
# 3. attack harness — health/auth/depth/oversize/concurrency probes
$ carrier harden --chaos --run --sandbox-db
# spawns the binary on loopback, mints tenant A/B JWTs,
# probes isolation, drops the owned database after probes
 
# 4. live evidence + Flight Recorder
$ curl -s http://127.0.0.1:4000/ops/flight?request_id=... | jq

Depth + size limits

Per-service caps for request_depth_limit, action_depth_limit, workflow_depth_limit, json_body_depth_limit, body_size_limit, and artifact_size_limit.

Flight Recorder

In-process, bounded ring buffer of probe events. Inspect with /ops/flight?request_id=<id> when runtime.ops_routes isn’t off.

Sandbox-DB live probes

harden --run --sandbox-db creates a unique carrier_harden_* Postgres database, runs the live probe set against the generated child service, then drops the owned database.

LLM clients & routing

llm client declarations expose a typed tool surface. Tool schemas come from Carrier types; execution reapplies auth and policy; every call is audited. 1.0 adds per-tenant USD budgets, downgrade-on-overage, and an explicit llm client routed wrapper for primary/fallback dispatch.

src/support/agents.carrier

llm client routed · budget · downgrade

llm client SupportAgent {
  provider:       "openai"
  wire_format:    "openai"
  model:          env("LLM_MODEL", "gpt-4.1-mini")
  api_key:        env("LLM_API_KEY", "")
  max_tokens:     600
  max_turns:      8
  temperature:    0.2
 
  budget_per_tenant_per_day_usd: 5.0
  over_budget_behavior:          downgrade("SupportAgentFallback")
 
  tool search_help_docs(term: String) -> String[] = search_help_docs
}
 
llm client SupportAgentFallback {
  provider:    "openai"
  wire_format: "openai"
  model:       "gpt-4.1-mini"
  api_key:     env("OPENAI_API_KEY", "")
  max_tokens:  600
  max_turns:   4
  temperature: 0.2
}
 
llm client routed RoutedSupportAgent {
  primary:             SupportAgent
  fallback:            SupportAgentFallback
  route_by:            cost_vs_latency(target_per_request_usd: 0.05)
  on_primary_outage:   fallback
  on_rate_limit:       fallback
  on_budget_pressure:  fallback
}

Persisted transcripts

Turns land in carrier_llm_conversations and carrier_llm_messages. Supply a conversation_id to continue a prior conversation.

Budgets & routing

budget_per_tenant_per_day_usd clamps spend at call time. Routed wrappers fall back on on_primary_outage, on_rate_limit, and on_budget_pressure.

Realtime streaming

Each client gets /realtime/llm/<client>/ws · sse · poll. Events: chunk, tool_call, completed, failed.

RAG declarations

One typed declaration binds a vector retriever, an embedder, and an llm client. The runtime trims serialized model context to the configured token budget and optionally reranks by score threshold.

src/support/rag.carrier

rag · retriever · llm

rag SupportAnswerer {
  retriever:             Doc.similar_with_scores
  embed_with:            embed_text
  llm:                   SupportAgent
  context_window_tokens: 4000
  rerank:                score_threshold(0.7)
  top_k:                 8
}
 
route POST "/support/answer" public -> String {
  input: SimilarDocRequest
  handler {
    let response = SupportAnswerer.respond(input.query)
    return response.text
  }
}

Generated flow

Carrier retrieves scored matches via Model.similar_with_scores, applies the optional score_threshold, fits serialized context to context_window_tokens, then calls the bound llm client.

Call surface

RagName.respond(question) returns the same LlmResponse shape as llm client, including token counters and an optional realtime stream path.

Agents & chat threads

agent declarations are typed LLM agents with an input type and a bound llm client. ui blocks expose them as chat-thread participants whose identity, thread id, and reply channel are wired from the host chat session — durability lands in carrier_workflow_state.

src/ui/admin.carrier

ui · expose agent · thread_participant

ui AdminConsole {
  framework: yew
  theme:     carrier.default
 
  expose route "/docs"        as list_detail_edit
  expose agent SupportAgent   as thread_participant {
    identity:         from_chat_session
    thread:           injected
    response_channel: thread_reply
  }
 
  safety {
    redact_pii:   auto
    require_auth: auto
    tenant_scope: auto
  }
}

Vector search

Fixed-dimension Vector(N) fields backed by pgvector. Carrier emits the CREATE EXTENSION, the vector(N) DDL, and HNSW/IVFFlat operator-class indexes. Similarity and hybrid helpers are generated when a model declares exactly one vector field.

src/docs/vectors.carrier

Vector(1536) · HNSW

model Doc {
  id: UUID
  title: String
  body: String
  embedding: Vector(1536) @index(hnsw, metric: cosine)
  created_at: Time
  updated_at: Time
}
 
crud Doc at "/docs" {
  list: public
  get:  public
  searchable { title  body }
}
 
native fn embed_text(value: String) -> Vector(1536) = "embed_text"
 
route POST "/docs/search" public -> DocHybridMatch[] {
  input: SimilarDocRequest
  handler {
    return Doc.hybrid_search(
      query: input.query,
      embedding: embed_text(input.query),
      limit: 20
    )
  }
}

Indexes

@index(hnsw, metric: cosine | euclidean | inner_product) or ivfflat. Dimension, index kind, and metric are captured in the manifest and OpenAPI.

Generated helpers

Doc.similar(...), Doc.similar_with_scores(...), Doc.hybrid_search(...). Scores are metric-derived; larger values rank earlier.

Embedding boundary

Carrier does not ship a built-in embedding provider. Wrap your source in a typed native fn embed_text(...) -> Vector(N) or accept vectors from an upstream pipeline.

Typed queues

Redis-backed queues with typed publish schemas, consumers bound to Carrier jobs, configurable concurrency, acknowledgement mode, retry backoff, and dead-letter queues — all in source.

src/queues/orders.carrier

queue · consume · dead_letter

type OrderPlacedPayload {
  order_id: String
  source: String
}
 
job enrich_order_event(payload: OrderPlacedPayload) -> Void {
  logs.info("order received", { order_id: payload.order_id })
}
 
queue OrderEventDeadLetters {
  backend: redis
  subject: "orders.v1.dead"
  publish OrderPlacedDead: OrderPlacedPayload
}
 
queue OrderEvents {
  backend: redis
  subject: "orders.v1"
  publish OrderPlaced: OrderPlacedPayload
 
  consume OrderPlaced by enrich_order_event {
    concurrency: 2
    ack: on_success
    retry: backoff(delay_seconds: 15, max_attempts: 4)
    dead_letter: OrderEventDeadLetters
  }
}

Outbox pattern

Publish inside a transaction { } so message enqueue commits atomically with the DB write that produced it.

Retry & DLQ

retry: backoff(delay_seconds, max_attempts) with dead_letter: routing failed messages into a typed DLQ queue for inspection.

Tenant lifecycle

Declare the tenant model and Carrier generates lifecycle hooks for on_create, on_suspend, and on_delete — plus retention-based purge policies that hold deleted tenants for N days before actually removing their data.

src/tenants/organization.carrier

tenant · on_create · retain 30 days

model Organization {
  org_id: String
  name: String
  suspended_at: Time?
  deleted_at: Time?
}
 
tenant Organization {
  id_field: org_id
  export_format: ndjson
 
  on_create {
    audit.record("tenant.bootstrap", "Organization", tenant.org_id)
  }
 
  on_suspend {
    logs.info("tenant suspended", { org_id: tenant.org_id })
  }
 
  on_delete {
    purge_policy: retain 30 days
    logs.warn("tenant scheduled for purge", { org_id: tenant.org_id })
  }
}

Export

export_format: ndjson produces per-tenant exports for compliance or migration. Tenant and workspace attributes propagate automatically through logs, OTLP, and audit rows.

Purge policy

retain N days holds tenant data for a grace window before the durable purge runs — gives operators a rollback path without hand-written cleanup SQL.

Feature flags

Deterministic feature flags with percentage-based rollouts and tenant-based targeting. Evaluations are stateless, deterministic for a given user/tenant, and emitted into the manifest for tooling.

src/main.carrier

flag · rules · grouped_by

flag support_agent_v2 {
  default: false
  rules {
    tenant in ["org_alpha", "org_beta"]: true
    percentage(20) grouped_by tenant_id: true
  }
}
 
route GET "/flags/support-agent-v2" protect Auth -> Json {
  handler {
    return { enabled: flags.enabled("support_agent_v2") }
  }
}

Evaluation

Call flags.enabled("flag_name") from routes, actions, workflows, or jobs. Rules are evaluated in declaration order; the first match wins.

Realtime streams

Typed events, transport-agnostic delivery. Generated endpoints negotiate WebSocket → SSE → long-poll automatically. watch blocks compile CRUD model changes into typed emitted events that fan out only after the write transaction commits.

src/streams/doctors.carrier

stream · watch · tenant_field

event DoctorRealtimeEvent {
  action: String
  doctor_id: UUID
}
 
stream DoctorEvents
  at "/streams/doctors"
  protect AdminAuth roles [admin]
  -> DoctorRealtimeEvent
{
  tenant_field:    tenant_id
  workspace_field: workspace_id
  group_field:     doctor_id
}
 
watch Doctor {
  on create  emit DoctorRealtimeEvent { action: "created",  doctor_id: value.id }
  on update  emit DoctorRealtimeEvent { action: "updated",  doctor_id: value.id }
  on delete  emit DoctorRealtimeEvent { action: "archived", doctor_id: old.id   }
  on restore emit DoctorRealtimeEvent { action: "restored", doctor_id: value.id }
}

Transport endpoints

GET /.../negotiate · /ws · /sse · /poll. Use group=<id> for channel-scoped fanout and cursor for backlog replay.

Watch semantics

Watched model changes are persisted inside the write transaction and become visible to realtime transports only after commit. No partial visibility of in-flight writes.

Native OTLP telemetry

Declare service.telemetry once and the generated runtime emits spans, metrics, and structured logs for every route, action, job, event handler, workflow, trigger, HTTP client, and LLM call.

src/main.carrier

opentelemetry · otlp_http

service App {
  openapi  { title: "App API"  version: "0.9.0" }
  server   { host: env("HOST", "0.0.0.0")  port: env_int("PORT", 3000) }
 
  telemetry {
    provider:     opentelemetry
    endpoint:     env("OTEL_EXPORTER_OTLP_ENDPOINT", "")
    service_name: "app-api"
    protocol:     otlp_http
    sampling:     parentbased_ratio(0.1)
  }
}
 
route GET "/checkout" protect Auth -> CheckoutResult {
  handler {
    let actor = auth.current_user()
 
    trace.annotate({ actor_id: actor.id, tenant_id: actor.tenant_id })
    metrics.counter("checkout.requests").increment({ tenant_id: actor.tenant_id })
 
    return Checkout.place(input)
  }
}

Stable attributes

carrier.route_name, carrier.workflow_name, carrier.step_name, carrier.tenant_id, and carrier.request_id — pinned in the stability policy.

User-authored

trace.annotate({ ... }) writes scalar attributes onto the current span. metrics.counter(...) and metrics.gauge(...) take scalar attribute objects.

Correlation

Trace context propagates route → action → job enqueue → workflow → downstream HTTP/LLM calls. Durable baggage persists through carrier_jobs and carrier_workflow_state.

Native PDF rendering

pdf.render_html for deterministic text-first PDFs and pdf.render_svg_template for rich visual output with custom fonts and image assets. Both compose with blob.put + blob.signed_url for distribution.

src/certificates.carrier

pdf.render_svg_template · blob

type CertificateContext {
  recipient_name:  String
  completion_date: String
  instructor_name: String
}
 
route POST "/certificates/{recipient_name}" public -> String {
  handler {
    let ctx = CertificateContext {
      recipient_name:  params.recipient_name
      completion_date: "April 23, 2026"
      instructor_name: "Niko Ma"
    }
 
    let pdf = pdf.render_svg_template(
      load_template("ryt200.svg"),
      ctx,
      "/tmp/certs/" + ctx.recipient_name + ".pdf",
      font_paths:   ["/fonts/PlayfairDisplay.ttf"],
      resource_dir: "/tmp/cert-assets",
      title:        "RYT-200 Certificate"
    )
 
    let stored = blob.put(pdf.path, "certs/" + ctx.recipient_name + ".pdf", pdf.content_type)
    return blob.signed_url(stored.key, expires_seconds: 300, download_name: pdf.filename)
  }
}

Two helpers

pdf.render_html(...) for text-first reports; pdf.render_svg_template(...) for SVG-driven visual output with font_paths and resource_dir.

Validation

Carrier validates the typed context against literal SVG templates at compile time. Empty content fails fast with invalid_pdf_content; empty destinations with invalid_pdf_destination.

api_explorer

A compiler-generated Try-it surface that stays typed against Carrier routes, actions, workflows, and federated Carrier service imports. Replayable collection / step / capture / assert checks live next to the explorer declaration.

src/explorers/hospital.carrier

api_explorer · collection

api_explorer HospitalApi {
  upstream: service
  mount_target: full_panel
 
  collection AdmissionFlow {
    step admit {
      call:   action admit_patient
      params: { input: { mrn: "TEST001", age: 65 } }
      capture {
        admission_id: response.id
        admitted_at:  response.admitted_at
      }
    }
 
    step record_vitals {
      call:   route POST "/vitals/{admission_id}"
      params: { admission_id: admission_id }
      input:  { admission_id: admission_id, bp: "130/85", hr: 72 }
      assert: status == 200 && response.recorded_at > admitted_at
    }
  }
 
  safety {
    redact_pii:          auto
    audit_every_request: true
    max_environment:     dev_or_staging
  }
}

Mount paths

Mount path: /ui/api_explorer/{snake_case(name)} with /spec.json, /execute, and /app.wasm companions.

Safety policy

redact_pii: auto, audit_every_request: true, and max_environment compile to runtime guardrails. Collection runs are audited as api_explorer.collection.run.

Federated upstreams

upstream: federated <Service> browses imported manifest actions/workflows. Use environments { ... } for named base URLs with deterministic / mock flags.

In-language tests

test blocks declare HTTP-level scenarios with an optional auth scenario, a given/when/then rhythm, and a typed response helper. Run the whole suite with carrier test.

src/main.carrier

test · given · when · then

test "me endpoint uses authenticated scenario user" {
  auth {
    id: 7
    email: "viewer@example.com"
    name: "Viewer"
    roles: ["user"]
  }
 
  given {
    let health = http.request(method: "GET", path: "/health")
  }
 
  when {
    let me = http.request_as("MeResponse", method: "GET", path: "/me")
  }
 
  then health.status == 200 && me.email == "viewer@example.com"
}

Auth scenarios

auth { id · email · roles } creates an authenticated scenario user for the test. Omit the block for unauthenticated cases.

Typed responses

http.request_as("TypeName", ...) returns a value matching the declared Carrier type. Plain http.request(...) gives you status + body as JSON.

Where each surface lives

Five constructs, one runtime, one binary.

Every surface on this page compiles through the same pipeline. They share auth, policy context, telemetry, audit, and the carrier_* system tables.

Workflow

carrier_workflow_state, carrier_workflow_tasks. Linear/saga/graph execution, typed compensation, per-step retry overrides, durable leases.

LLM client

carrier_llm_conversations, carrier_llm_messages. Typed tools, generated schemas, persistent transcripts, realtime chunk stream per client.

Vector

Vector(N) columns with hnsw / ivfflat indexes. Generated similar_with_scores and hybrid_search.

Queue

Redis queues with typed publish schemas, consumers bound to Carrier jobs, retry backoff, dead-letter routing, outbox pattern via transaction { ... }.

Tenant

Lifecycle hooks — on_create, on_suspend, on_delete — with retain N days purge windows andndjson export.

Realtime

stream, subscription, and watch with WS → SSE → long-poll fallback, tenant/workspace scoping, post-commit fanout.

The platform surface, one compiler away.

Every block on this page is part of the 0.9 release candidate. Language surface, manifest keys, system tables, OTLP attributes, and OpenAPI extensions are covered by the stability policy.

Language reference →Stability policy Speed benchmarks ↗