v0.2 · substrate hardened · alpha

swarm-lib

The substrate your agents have been missing.

A small Python + Bash library that turns long, multi-step agentic work into a queue of atomic tasks that survive compaction, crashes, rate limits, and process restarts. Three primitives — atomic-rename queueing, status.json checkpointing, a generic worker loop — give you durable handoff between fresh LLM contexts. No broker. No daemon. No database. Just POSIX and JSON.

If you've watched Claude Code hit compaction mid-task, burned an hour of opus tokens waiting for a subprocess, or shipped an agent pipeline that quietly dies when the chat thread closes — this is the fix.

The three problems that kill agentic workflows

Every agent system that grows past a toy hits these. They're not bugs in your code. They're structural problems in how LLM-driven work is wired today.

Three primitives, 30 years of UNIX discipline

swarm-lib gives you the same substrate UNIX shops have used since the 90s — Maildir, cron + lock files, /var/spool/ — applied to LLM-driven agent work.

os.replace(
    "pending/<task_id>.json",
    "claimed/<worker_id>/<task_id>.json",
)

{
  "schema_version": "0.1",
  "run_id": "audit-r3x2",
  "checkpoint": {
    "summary": "Completed plan; ready for implement stage",
    "next_step": "Invoke implement skill with plan output as input",
    "next_task_id": "t.audit-r3x2.implement",
    "completed_tasks": ["t.audit-r3x2.intent", "t.audit-r3x2.plan"],
    "current_worker": null,
    "timestamp": "2026-05-21T18:45:11Z"
  }
}

60-second tour

Enqueue a task, run a worker against it, check what happened. That's the whole loop.

Install

git clone https://github.com/dpdanpittman/swarm-lib
cd swarm-lib
pip install -e .
# 'swarm-cli' is now on PATH

Enqueue → run → check

# 1. Initialize a run + enqueue a task
mkdir -p ~/.swarm/hello
swarm-cli status-init --run-dir ~/.swarm/hello --run-id hello \
  --summary "Hello-world test run" --next-task-id t.1

swarm-cli enqueue --run-dir ~/.swarm/hello \
  --task-id t.1 --task-type greet \
  --payload '{"who": "world"}'

# 2. Write a handler (gets task JSON on stdin, writes to artifact path)
cat > /tmp/handler.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
TASK_JSON=$(cat)
WHO=$(echo "$TASK_JSON" | jq -r '.payload.who')
echo "# Hello, $WHO" > "$SWARM_ARTIFACT_PATH"
EOF
chmod +x /tmp/handler.sh

# 3. Run a worker — one iteration, then exit
swarm_lib/worker_loop.sh \
  --run-dir ~/.swarm/hello \
  --worker-id w.demo \
  --handler /tmp/handler.sh \
  --max-iterations 1

# 4. Check what happened
swarm-cli ls --run-dir ~/.swarm/hello
cat ~/.swarm/hello/artifacts/t.1.md

From Python

from swarm_lib import claims, status

run_dir = "~/.swarm/hello-py"

status.initialize(run_dir, run_id="hello-py")
claims.enqueue(run_dir, task_id="t.1", task_type="greet",
               payload={"who": "world"})

task = claims.try_claim(run_dir, worker_id="w.demo")
if task is not None:
    # ... do work, write artifacts under run_dir/artifacts/ ...
    claims.complete(task, success=True)

When you actually want this

swarm-lib is purpose-built for these shapes. If your problem looks like one of these, it'll save you real time and tokens.

Why swarm-lib and not X

There are great tools for adjacent problems. Here's the honest comparison.

The honest summary: swarm-lib is less powerful than Celery, Temporal, or Airflow for traditional async work. It's more powerful than those tools for LLM-driven agentic work specifically, because it's purpose-built around the constraints that matter: chat-history-immune, context-window-aware, model-tier routing, interchangeable workers across LLM tools.

What's in v0.2

Substrate is hardened. 42 tests passing, including multi-worker correctness under contention.

Production shape

What a real swarm-lib deployment looks like. systemd for workers, cron for the reaper, the filesystem for everything else.

Worker as a systemd user service

# ~/.config/systemd/user/swarm-worker@.service
[Unit]
Description=swarm-lib worker for ~/.swarm/%i

[Service]
Type=simple
ExecStart=%h/src/swarm-lib/swarm_lib/worker_loop.sh \
  --run-dir %h/.swarm/%i \
  --worker-id w.%H.%i \
  --handler %h/.swarm/handlers/dispatcher.sh \
  --heartbeat-interval 30 --poll-interval 5
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

Cron-driven reaper

# /etc/cron.d/swarm-reap — every 5 minutes, sweep all runs
*/5 * * * * dan for d in ${HOME}/.swarm/*/; do \
  swarm-cli reap --run-dir "$d" --stale-after 300; \
done

Multiple workers on one queue

# Three workers, same run-dir, different worker_ids
swarm_lib/worker_loop.sh --run-dir ~/.swarm/big --worker-id w.1 --handler ./h.sh &
swarm_lib/worker_loop.sh --run-dir ~/.swarm/big --worker-id w.2 --handler ./h.sh &
swarm_lib/worker_loop.sh --run-dir ~/.swarm/big --worker-id w.3 --handler ./h.sh &

Atomic-rename guarantees no double-claims. Multi-worker correctness is proven by the test suite under contention (threaded + subprocess claimants + reaper-during-drain).

Observability without a UI

# Counts + next step + last completed task
swarm-cli ls --run-dir ~/.swarm/audit-r3x2

# Machine-readable for piping
swarm-cli ls --run-dir ~/.swarm/audit-r3x2 --json | jq '.[0].pending'

# Tail a long-running task's progress
tail -f ~/.swarm/audit-r3x2/artifacts/t.implement.log

# Failed tasks in the last hour
find ~/.swarm/audit-r3x2/failed/ -mmin -60

For whom

• → Builders of multi-step agent workflows that don't fit in one context window
• → Anyone running Claude Code, Codex, ollama, or similar long-running LLM agents at scale
• → Teams shipping agent-driven pipelines that need to survive compaction, rate limits, restarts
• → Cross-tool federation: Claude + n8n + cron + shell, all writing to the same substrate
• → Anyone tired of treating chat history as state

Roadmap