// SUBSTRATE · v0.2 · alpha
swarm-lib
The substrate your agents have been missing.
A small Python + Bash library that turns long, multi-step agentic work into a queue of atomic tasks that survive compaction, crashes, rate limits, and process restarts. Three primitives — atomic-rename queueing, status.json checkpointing, a generic worker loop — give you durable handoff between fresh LLM contexts. No broker. No daemon. No database. Just POSIX and JSON.
If you've watched an agent hit compaction mid-task, burned an hour of expensive tokens waiting for a subprocess, or shipped a pipeline that quietly dies when the chat thread closes — this is the fix.
The three problems that kill agentic workflows
Every agent system that grows past a toy hits these. They're not bugs in your code. They're structural problems in how LLM-driven work is wired today.
1. Context starvation
You build a workflow as one long conversation: "first do X, then Y, then Z, then summarize." Halfway through Y, the context window approaches its limit, compaction fires, and the model now has a lossy summary instead of the actual artifacts. Z gets a confused result and Y silently drifts.
Root cause: chat history is being used as program state. State that's volatile, lossy under compression, tied to a single process's lifetime.
2. Synchronous tool-call blocking
Your planner runs on the most expensive tier. It decomposes a task into sub-tasks and then... waits. It holds the expensive context window open while subprocesses or external APIs churn for minutes. Tokens burn at idle because the planner can't release its window until the children return.
Root cause: synchronous orchestration. The high-context agent is treated as a coordinator that blocks on its workers.
3. Chat-history-as-state
Your agent runs as a long-lived conversation. It crashes — rate limit, network blip, user closes the tab, the laptop sleeps. When it comes back, there's no durable record of "where am I in the work." The agent either restarts from zero, replays redundantly, or invents a plausible-looking continuation that drifts from reality.
Root cause: no source of truth outside the conversation. If the conversation dies, the work dies.
Three primitives, 30 years of UNIX discipline
swarm-lib gives you the same substrate UNIX shops have used since the 90s — Maildir, cron + lock files, /var/spool/ — applied to LLM-driven agent work.
1 / atomic queueing
Atomic-rename task queueing
Producers stage tasks under pending/<task_id>.json. Consumers race for them via os.replace, which POSIX guarantees is atomic on the same filesystem. Two consumers racing for the same task: exactly one wins. No locks, no broker, no leader election. The filesystem is the coordinator.
2 / durable handoff
status.json checkpointing
Every workflow keeps its state in a single JSON file. Any fresh agent — a new Claude session, an ollama worker, a shell script, a cron job — resumes by reading this file. Compactions, crashes, rate limits, multi-day pauses, machine reboots all become indistinguishable from a clean restart. Chat history is volatile; the file is the contract.
3 / generic worker
worker_loop.sh
A 200-line bash loop that polls a run directory, atomically claims tasks, invokes any handler executable with the task JSON on stdin, and moves results to done/ or failed/. Workers are interchangeable. Any process that reads JSON from stdin and writes to disk is a participant. A background heartbeat keeper enables orphan recovery without a coordinator.
Each primitive gets a proper deep-dive in /docs/concepts.
the core discipline
The Yield Rule
A high-context agent (planner) never blocks waiting on subprocess output. It decomposes the goal into atomic tasks, writes each task's payload to pending/<task_id>.json atomically, and exits immediately, freeing its context window.
Fresh consumer loops pick up the tasks. The planner can be re-invoked later from status.json if needed. No expensive planner sitting idle. No tokens burning on I/O.
What you'd use it for
Five day-to-day patterns. Each one is a specific shape of agentic work where swarm-lib saves real time, tokens, or sanity.
PATTERN 1
Long-running async (the killer app)
Decompose a giant task into N sub-tasks. Walk away. Come back to artifacts. Compaction-immune.
PATTERN 2
Compaction insurance for any multi-step task
Write status.json as you go. Next session reads it, picks up where you left off.
PATTERN 3
Scheduled work that survives restarts
Cron enqueues, worker handles, artifacts land. Retry + state + storage for free.
PATTERN 4
"I am the worker"
Claim a task into your live session, do the work, complete. Useful for review + refinement.
PATTERN 5
Cross-tool federation
n8n, cron, Claude, shell scripts — all reading and writing the same ~/.swarm/ substrate. Each tool stays in its lane; the filesystem is the contract.
Go deeper
Quickstart →
Install, enqueue, run a worker, check the result. Under a minute.
Concepts →
The Yield Rule, file-system-as-orchestrator, handler hygiene, anti-fleet.
Production deployment →
systemd, cron-driven reaper, multi-worker scaling, observability.
Why swarm-lib and not X →
Honest comparison vs Celery, Airflow, Temporal, LangChain, GitHub Actions.
API reference →
Python module surface, swarm-cli subcommands, worker env vars.
Examples →
Seven-step chain (Tribunal-shaped) and HMD triage (cost-routing).
Stop treating chat history as state.
Three primitives, two commits' worth of substrate code, one filesystem. The discipline UNIX had in 1995, applied to the agentic work people are shipping now.