Loom documentation

Overview

Loom is a Rust-native orchestration platform for Python and TypeScript workloads. One 67 MB binary contains the scheduler, a parallel DAG orchestrator, a warm worker pool, streaming NDJSON ingestion, an embedded SQL engine (Polars), serverless functions, notebooks, an SSE event bus, and an MCP server for AI agents. State lives in a single SQLite (WAL) file.

• Polyglot DAGs — Python, TypeScript, and JavaScript tasks in one graph; results flow across language boundaries.
• ~1 ms task overhead — interpreters stay warm between tasks (29–78× faster than cold spawns).
• Zero infrastructure — no broker, no database server; 16 MB RAM at idle.
• Agent-native — every capability exposed as JSON HTTP + MCP tools.

Quickstart

# run everything (console + API on :7420), password-protected
docker run -p 7420:7420 -v loom-data:/data -e LOOM_PASSWORD=change-me ghcr.io/danielmcsheehy/rust-orchestrator

# or from source
git clone https://github.com/DanielMcSheehy/rust-orchestrator && cd rust-orchestrator
docker compose up --build

# or natively: Rust 1.80+, Python 3.10+, Node 20+ (22+ for TS tasks)
cargo run --release -p loom-server
cd console && npm install && npm run build   # then restart the server

Create and run your first workflow:

curl -X POST localhost:7420/api/workflows -H 'content-type: application/json' -d '{
  "name": "hello",
  "params": {"n": 100},
  "tasks": [
    {"id": "extract", "runtime": "python", "depends_on": [],
     "code": "def handler(params, inputs):\n    return {\"values\": list(range(params[\"n\"]))}\n"},
    {"id": "total", "runtime": "typescript", "depends_on": ["extract"],
     "code": "export function handler(p: any, i: any) {\n  return i.extract.values.reduce((a: number, b: number) => a + b, 0);\n}\n"}
  ]
}'
curl -X POST localhost:7420/api/workflows/<id>/trigger -H 'content-type: application/json' -d '{}'
curl localhost:7420/api/runs/<run_id>     # states, results, logs
curl -N localhost:7420/api/events          # live SSE stream

Concepts

Workflow → Run → TaskRun

A workflow is a named DAG spec (tasks + triggers + params). Triggering it creates a run; each task in the DAG gets a task run. The orchestrator validates the graph at deploy time (duplicate ids, unknown deps, cycles → HTTP 400), then executes in topological layers — every task in a layer runs concurrently, bounded by max_parallel_tasks.

State machine

pending → running → completed | failed | cancelled   (terminal states never transition)

A failed task (after its retries) fails the run; the current layer finishes, and tasks that were never reached are marked cancelled. Workload stack traces land in the task's logs as [trace] lines.

Data flow

Each task receives params (run params merged with task params — task overrides, null keeps base) and inputs — an object mapping each depends_on task id to that task's returned value, including across languages.

Events

Every state change, log line, ingest batch, and function invocation broadcasts a typed event (run_updated, task_updated, log, ingested, function_invoked) on an in-process bus. SSE endpoints, the console, and the SDKs all consume the same stream.

Writing tasks

A task is a self-contained module with one entrypoint. It runs in an isolated worker process, so imports go inside the module and there are no closure captures.

# runtime: python
def handler(params, inputs):
    print("logs stream live to the console and SDKs")
    return {"any": "json"}

// runtime: typescript (Node 22 type-stripping) or javascript
export async function handler(params: any, inputs: any) {
  return { any: "json" };
}

Per-task knobs: depends_on, retries (attempts = retries + 1), timeout_secs (default 300 — the process/sandbox is killed on expiry), params. Return values must be JSON-serializable; keep them small and pass heavy data through datasets (below).

Triggers

HTTP API

JSON in/out; errors are {"error": "message"} with a matching status code.

SDKs

Both SDKs are zero-dependency (Python stdlib / global fetch) and expose the same surface: deploy (upsert by name), trigger & wait, runs, functions, ingest, query, execute, and live event streaming.

Python — pip install -e sdks/python

from loom_sdk import LoomClient, Flow, task

@task
def extract(params, inputs):
    return {"values": list(range(params["n"]))}

@task(depends_on=[extract], retries=2)
def total(params, inputs):
    return sum(inputs["extract"]["values"])

client = LoomClient("http://localhost:7420")
wf  = client.deploy(Flow("sum", params={"n": 100}, tasks=[extract, total]))
run = client.trigger(wf["id"], wait=True)

for event in client.stream_run(run["id"]):   # live SSE
    print(event["type"], event.get("line", ""))

TypeScript — @loom/sdk

import { LoomClient, flow, task } from "@loom/sdk";

const extract = task("extract", async (p) => ({ values: [...] }));
const rank = task("rank", { runtime: "python", code: "..." },
                 { dependsOn: [extract] });  // mixed runtimes

const client = new LoomClient();
const wf = await client.deploy(flow("ranker", [extract, rank]));
for await (const ev of client.streamRun(run.id)) { /* live */ }

Handler-function tasks are serialized as source (Function.prototype.toString() / inspect.getsource) — they must be self-contained; closures don't survive.

In-task bindings

Every task and function gets platform bindings injected into its runtime — import loom in Python, the loom global in JS/TS — no packages needed in the worker environment:

import loom

def handler(params, inputs):
    rows = loom.query("SELECT zone, AVG(value) v FROM telemetry GROUP BY zone")
    loom.ingest("zone-aggregates", rows)      # derived dataset
    return loom.invoke("notify", {"n": len(rows)})

loom.query(sql, limit=) · loom.ingest(dataset, records) · loom.invoke(name, params). Calls go to LOOM_API_URL (set automatically) and bypass HTTP proxies.

MCP — the agent surface

claude mcp add --transport http loom http://localhost:7420/mcp

The built-in Model Context Protocol server (streamable HTTP, stateless JSON-RPC) exposes 15 tools: loom_stats, execute_code, query, ingest, list_datasets, describe_dataset, list_workflows, create_workflow, trigger_workflow, get_run, cancel_run, list_runs, invoke_function, create_function, create_notebook.

Because workflows are plain JSON and the server returns structured validation errors, an agent can loop: write code → create_workflow → trigger → read failure trace → patch → re-trigger — a self-correcting pipeline builder with Loom as the execution substrate.

Datasets & SQL

Ingestion: POST /api/ingest/{dataset} streams NDJSON to disk chunk-by-chunk (~780 MB/s measured; gigabyte bodies never sit in memory). Datasets are append-only NDJSON files plus running totals.

Query: every dataset registers as a SQL table (dashed names also get an underscore alias) on the embedded Polars engine — lazy scans with projection and predicate pushdown, joins across datasets, executed off the async runtime. A GROUP BY over 1.8M rows answers in ~250 ms.

SELECT zone, COUNT(*) AS n, AVG(value) AS avg_v
FROM telemetry WHERE value > 10 GROUP BY zone ORDER BY avg_v DESC

Results are row-capped (default 10k, max 200k) with an explicit truncated flag — the API returns summaries, not datasets.

Connectors

Register once (POST /api/connectors {"name","kind","url"}), then point any query at it: API {"connector": "warehouse"}, SDK client.query(sql, connector="warehouse"), or the dropdown in notebook SQL cells and the Data page.

Notebooks

Observable-style executable documents in the console: markdown cells, code cells (Python/TS/JS, run on the warm pool via /api/execute), and SQL cells (Polars or any connector) — with a chart view (bar/line, x/y pickers) on any tabular result. Cells chain: a cell's output reaches every cell below as inputs.prev (nearest output) and — for named cells — inputs[name], so a Python cell can feed a JS cell can feed a chart. Editors are CodeMirror 6: autocomplete, bracket matching, multi-cursor, ⌘⏎ to run. Cells and their outputs persist with the document, so notebooks reopen with results. Storage is schema-free JSON — the console owns the cell format.

Isolation & the worker pool

Pool rules: a clean finish (including a caught in-task error) returns the worker; timeouts and protocol violations kill it; workers retire after 128 jobs; up to 8 stay warm per runtime. One Node pool serves JS and TS. Container/microVM modes never pool — a fresh sandbox per task is their purpose. Jobs share an interpreter but never a module (fresh import per job, like warm lambdas).

Performance

Sizing: a 512 MB VPS runs light workloads (trim LOOM_WORKER_MAX_IDLE); 1 GB is comfortable for everything documented here; add headroom roughly equal to your largest query's working set.

Deployment

Docker Compose

docker compose up --build     # console + API on :7420, data in a named volume

Coolify

1. + New Resource → Docker Compose, point at the repository.
2. Set Docker Compose Location to /docker-compose.coolify.yml.
3. Assign a domain — the SERVICE_FQDN_LOOM_7420 magic variable routes Coolify's proxy to the server (SSE included). The healthcheck drives status and rolling restarts.

Bare binary

cargo build --release -p loom-server
LOOM_DATA_DIR=/var/lib/loom ./target/release/loom-server

Configuration reference