tanmay/aetheel-2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add Pi Coding Agent backend runtime, update dashboard UI

Browse Source
This commit is contained in:
Tanmay Karande
2026-03-03 00:24:19 -05:00
parent 453389f55c
commit 4c3a58b680
21 changed files with 4955 additions and 192 deletions
Download Patch File Download Diff File Expand all files Collapse all files

170
docs/FUTURE-FEATURES.md
View File

@@ -2,47 +2,60 @@
Features inspired by nanoclaw that can be implemented in the CLI-based gateway. Excludes Docker containers, WhatsApp channel, and Agent SDK (already ruled out).
## Quick Wins (hours)
---
### 1. Message History Storage
## ✅ Completed
Store inbound/outbound messages in a JSON file per channel. Nanoclaw uses SQLite, but a simple `config/messages/{channelId}.json` would work. This lets the agent see conversation history even after session expiry.
### Message History Storage ✅
Store inbound/outbound messages in a JSON file per channel. Implemented at `config/messages/{channelId}.json` with append-only storage, max 100 messages per channel, auto-trimmed.
How nanoclaw does it: SQLite table with id, chat_jid, sender, sender_name, content, timestamp, is_from_me, is_bot_message columns. Indexed by timestamp.
### Structured Logging ✅
Pino-based structured JSON logging with configurable log levels via `LOG_LEVEL` env var. Pretty-printed in dev, JSON in production.
Our approach: JSON file per channel at `config/messages/{channelId}.json`. Append-only. Include sender name, content, timestamp, and direction (inbound/outbound). Load recent messages into the system prompt or pass as context.
### Conversation Archiving ✅
Every exchange saved as readable markdown in `config/conversations/{channelId}/{YYYY-MM-DD}.md`.
### 2. Structured Logging
### Retry with Backoff ✅
Backend CLI calls retry 3 times with exponential backoff (5s, 10s, 20s) on transient errors. Session corruption errors fail immediately.
Replace `console.log` with a proper logger (pino). Nanoclaw logs everything with structured context (group, duration, event type). Makes debugging way easier on a headless server.
### Idle Timeout for Sessions ✅
Auto-clear sessions after configurable period of inactivity per channel (default 30 min). Tracked via `lastActivityTimestamp` in the session manager.
How nanoclaw does it: Pino logger with pretty-printing, configurable log levels via LOG_LEVEL env var, uncaught exception routing, container logs saved to disk.
### IPC-Based Proactive Messaging ✅
Agent writes JSON files to `config/ipc/outbound/` using the Write tool. Gateway polls every 2 seconds, delivers messages, and deletes processed files.
Our approach: Install pino, replace all console.log/error/warn calls, add log levels, save logs to `config/logs/`. Structured JSON logs make it easy to grep and filter.
### Skills Engine ✅
`config/skills/` directory where each subdirectory contains a `SKILL.md` file. Skills are loaded into the system prompt as additional sections.
### 3. Conversation Archiving
### Config Simplification ✅
Merged `soul.md`, `identity.md`, `user.md`, and `tools.md` into a single `CLAUDE.md` file.
Before sessions get too long, archive transcripts to `config/conversations/`. Nanoclaw does this via a PreCompact hook that saves the full transcript as markdown before Claude compacts its context.
### Mission Control Dashboard ✅
Embedded web-based dashboard served on port 3100 with:
- **Command Center** — Real-time stats, live activity feed, agent config
- **Activity** — Full event history with SSE real-time updates
- **Sessions** — Active Discord channel sessions and message viewer
- **Second Brain** — Knowledge store with facts (notes/URLs/files), categories, tags, search. Plus memory and persona editors, and skills viewer
- **Productivity** — Kanban-style task board (To Do → In Progress → Done) with priorities, projects, and due dates
- **Content Intel** — Content curation (articles, videos, papers, repos) with status tracking (queued → read → archived)
- **Scheduler** — View cron jobs, heartbeat checks, and lifecycle hooks
- **Connections** — Integration statuses
- **Settings** — Config overview and quick todos
Our approach: After each CLI response, optionally append the exchange (prompt + response) to `config/conversations/{channelId}/{date}.md`. This creates a human-readable conversation log separate from Claude's internal session storage.
All backed by local JSON file persistence in `config/` via `local-store.ts` — zero external dependencies.
### 4. Retry with Backoff
### Multi-Backend Support ✅
Abstracted backend interface (`BackendAdapter`) with pluggable CLI adapters. Supports five backends: **Claude Code**, **Codex**, **Gemini CLI**, **OpenCode**, and **Pi**. Switchable via `AGENT_BACKEND` env var. Each adapter normalizes CLI-specific JSON output into a common `BackendEventResult` format. Full property and unit test coverage for all backends.
When the Claude CLI fails, retry with exponential backoff (5s, 10s, 20s, 40s, 80s) up to 5 times. Nanoclaw does this for container failures.
---
How nanoclaw does it: GroupQueue tracks retryCount per group, uses `BASE_RETRY_MS * 2^retryCount` delay, max 5 retries, resets on success.
## Remaining Features
Our approach: Wrap the `executeClaude` call in a retry loop. On transient errors (timeout, CLI crash), wait and retry. On permanent errors (auth failure, invalid session), fail immediately.
### Agent-Managed Tasks via File-Based IPC (Medium Effort)
## Medium Effort (day or two)
The agent can create, pause, resume, and cancel scheduled tasks dynamically during a conversation.
### 5. Agent-Managed Tasks via File-Based IPC
The big one from nanoclaw. The agent can create, pause, resume, and cancel scheduled tasks dynamically during a conversation.
How nanoclaw does it: Custom MCP server with tools like `schedule_task`, `list_tasks`, `pause_task`, `resume_task`, `cancel_task`. Tasks stored in SQLite with schedule_type (cron/interval/once), context_mode (group/isolated), and full lifecycle tracking.
Our approach (simpler): The agent writes task definitions to `config/tasks.json` using the Write tool. The gateway polls this file periodically and schedules/unschedules tasks accordingly. Task format:
Our approach: The agent writes task definitions to `config/tasks.json` using the Write tool. The gateway polls this file periodically and schedules/unschedules tasks accordingly. Task format:
```json
[
@@ -57,115 +70,42 @@ Our approach (simpler): The agent writes task definitions to `config/tasks.json`
]
```
The agent can add/remove/modify entries. The gateway watches the file and updates schedulers.
### 6. Multi-Turn Message Batching
### Multi-Turn Message Batching (Medium Effort)
Instead of sending one message at a time to Claude, batch multiple messages that arrive while the agent is processing.
How nanoclaw does it: Collects all messages since the last agent run timestamp, formats them as XML (`<messages><message sender="..." time="...">content</message></messages>`), sends the batch as a single prompt.
Our approach: When a message arrives while the agent is already processing for that channel, queue it. When the current processing finishes, batch all queued messages into a single prompt with sender/timestamp metadata.
Our approach: When a message arrives while the agent is already processing for that channel, queue it. When the current processing finishes, batch all queued messages into a single prompt with sender/timestamp metadata. This prevents the agent from missing rapid-fire messages.
### 7. Idle Timeout for Sessions
Auto-clear sessions after a configurable period of inactivity per channel.
How nanoclaw does it: 30-minute idle timeout per container. After last output, a timer starts. If no new messages arrive, the container is closed and the session ends.
Our approach: Track `lastActivityTimestamp` per channel in the session manager. Run a periodic check (every 5 minutes) that removes sessions older than the timeout (default 30 min). This prevents stale sessions from accumulating and keeps memory.md relevant.
### 8. Per-Channel Isolation
### Per-Channel Isolation (Medium Effort)
Different channels could have different persona configs. A "work" channel gets a professional persona, a "fun" channel gets a casual one.
How nanoclaw does it: Per-group folders with separate CLAUDE.md, sessions, memory, and container mounts. Each group is fully isolated.
Our approach: Support a `config/channels/{channelId}/` override directory. If it exists, load persona files from there instead of the root config. Falls back to root config for missing files.
Our approach: Support a `config/channels/{channelId}/` override directory. If it exists, load persona files from there instead of the root config. Falls back to root config for missing files. This lets you customize identity.md or soul.md per channel.
## Bigger Features (multi-day)
### 9. IPC-Based Proactive Messaging
Let the agent send messages to Discord channels proactively — not just as a response to a user message.
How nanoclaw does it: Filesystem-based IPC. The agent writes a JSON file to `/workspace/ipc/messages/` with `{type: "message", chatJid: "...", text: "..."}`. The host polls the directory every second and sends the message.
Our approach: Create a `config/ipc/outbound/` directory. The agent writes JSON files there using the Write tool. The gateway polls every 2 seconds, reads new files, sends messages to the specified Discord channel, and deletes the processed files. This enables:
- Agent sending follow-up messages after thinking
- Agent notifying you about something it found during a heartbeat
- Agent sending messages to channels it's not currently chatting in
File format:
```json
{
"type": "message",
"channelId": "1475008084022788312",
"text": "Hey, I found something interesting in your email!"
}
```
### 10. Skills Engine (Simplified)
A system for adding capabilities via markdown skill files that get loaded into the system prompt.
How nanoclaw does it: Full manifest-based system with YAML manifests, dependency resolution, conflict detection, file operations (rename, delete, move), backup/restore, rebase capability, and resolution caching.
Our approach (much simpler): A `config/skills/` directory where each subdirectory contains a `SKILL.md` file. Skills are loaded into the system prompt as additional sections. The agent can reference skills by name. No dependency management — just markdown files that add context.
```
config/skills/
├── web-research/
│ └── SKILL.md → "When asked to research, use WebSearch and WebFetch..."
├── code-review/
│ └── SKILL.md → "When reviewing code, focus on security, performance..."
└── email-helper/
└── SKILL.md → "When checking email, use the Gmail API via..."
```
### 11. SQLite Storage
### SQLite Storage (Bigger Effort)
Replace JSON files with SQLite for messages, sessions, tasks, and state. Better for concurrent access, querying, and doesn't corrupt on partial writes.
How nanoclaw does it: Single SQLite database at `store/messages.db` with tables for chats, messages, scheduled_tasks, task_run_logs, router_state, sessions, registered_groups. Uses better-sqlite3 (synchronous API).
Our approach: Install better-sqlite3, create a single `config/aetheel.db` with tables for sessions, messages, tasks, and state. Migrate existing JSON files on first run.
### 12. Secrets Management
### Secrets Management (Bigger Effort)
Prevent API keys and sensitive data from leaking through agent tool use.
How nanoclaw does it: Secrets passed via stdin (never written to disk), stripped from Bash subprocess environments via a PreToolUse hook, never mounted as files in containers.
Our approach: Add a sanitization step: before passing the prompt to Claude, strip any env var values that look like secrets from the system prompt. Also consider using `--disallowedTools Bash` if the agent doesn't need shell access.
### MCP Server Integration (Bigger Effort)
Expose Aetheel's capabilities as MCP tools so the agent can interact with the dashboard data, manage tasks, and store knowledge programmatically during conversations.
Our approach: Since we use `--dangerously-skip-permissions`, the agent can run Bash commands that might echo environment variables. Add a sanitization step: before passing the prompt to Claude, strip any env var values that look like secrets from the system prompt. Also consider using `--disallowedTools Bash` if the agent doesn't need shell access.
## Priority Recommendation
1. Structured logging (immediate debugging value)
2. Message history storage (conversation context)
3. Retry with backoff (reliability)
4. IPC-based proactive messaging (agent autonomy)
5. Agent-managed tasks (dynamic scheduling)
6. Conversation archiving (audit trail)
7. Everything else as needed
1. Agent-managed dynamic tasks (agent autonomy)
2. Multi-turn message batching (UX improvement)
3. Per-channel isolation (flexibility)
4. MCP Server integration (agent ↔ dashboard)
5. SQLite storage (reliability at scale)
6. Everything else as needed
---
## Completed: Config Simplification
Merged `soul.md`, `identity.md`, `user.md`, and `tools.md` into a single `CLAUDE.md` file. The config directory is now:
```
config/
├── CLAUDE.md ← Persona: identity, personality, user context, tools (all in one)
├── agents.md ← Cron jobs + Hooks (parsed by gateway at startup)
├── heartbeat.md ← Heartbeat checks (parsed by gateway at startup)
├── memory.md ← Long-term memory (agent-writable, auto-created)
└── sessions.json ← Channel → session ID map (auto-generated)
```
Why this split:
- `CLAUDE.md` is pure prompt context — the agent reads it but the gateway doesn't parse it
- `agents.md` and `heartbeat.md` are parsed programmatically by the gateway to set up cron timers and heartbeat intervals
- `memory.md` is the only file the agent writes to — keeping it separate prevents the agent from accidentally overwriting persona config
- Fewer files to manage, one place to edit your persona

162
docs/PROCESS-FLOW.md
View File

@@ -5,7 +5,7 @@ How a Discord message becomes an AI response, step by step.
## The Big Picture
```
Discord User Aetheel Gateway Claude Code CLI
Discord User Aetheel Gateway Backend CLI
│ │ │
│ @Aetheel what's 2+2? │ │
├──────────────────────────────► │ │
@@ -195,21 +195,23 @@ Sections with null or empty content are omitted entirely.
The assembled system prompt is written to a temporary file because it can be thousands of characters — too large for a CLI argument.
**File:** `src/agent-runtime.ts` → `executeClaude()`
**File:** `src/agent-runtime.ts` → `processMessage()`
```
/tmp/aetheel-prompt-1d6c77f1-4a4e-49f8-ae9b-cff6fb47b971.txt
```
This file is deleted after the CLI process completes.
This file is deleted after the CLI process completes (used by the Claude backend; other backends may prepend the system prompt directly to the user prompt).
### Step 8: Spawn Claude CLI
### Step 8: Spawn Backend CLI
The gateway spawns the Claude Code CLI as a child process.
The gateway delegates to the configured **Backend Adapter**, which spawns the corresponding CLI as a child process.
**File:** `src/agent-runtime.ts` → `runClaude()`
**File:** `src/backends/{claude,codex,gemini,opencode,pi}-backend.ts` → `spawnCli()`
The actual command:
The backend is selected via the `AGENT_BACKEND` environment variable (default: `claude`). Each backend adapter translates the common interface into the CLI-specific flags.
#### Example: Claude Code CLI (default)
```bash
claude \
@@ -227,15 +229,55 @@ claude \
--max-turns 25
```
Key flags:
#### Example: Pi CLI
```bash
pi \
-p "what's the weather like?" \
--mode json \
--append-system-prompt "<system prompt text>" \
--no-session \
--no-extensions --no-skills --no-themes
```
#### Example: Codex CLI
```bash
codex exec \
"<system prompt + user prompt>" \
--json \
--dangerously-bypass-approvals-and-sandbox \
--cd /path/to/config
```
#### Example: OpenCode CLI
```bash
opencode run \
"<system prompt + user prompt>" \
--format json
```
#### Example: Gemini CLI
```bash
gemini \
"<system prompt + user prompt>" \
--output-format json \
--approval-mode yolo
```
Key flags (Claude):
- `-p` — Print mode (non-interactive, exits after response)
- `--output-format json` — Returns JSON array of message objects
- `--dangerously-skip-permissions` — No interactive permission prompts
- `--append-system-prompt-file` — Appends our persona/memory to Claude's default prompt
- `--allowedTools` — Which tools Claude can use (one flag per tool)
- `--allowedTools` — Which tools Claude can use (one flag per tool; Claude-only feature)
- `--max-turns` — Prevents runaway agent loops
- `--resume SESSION_ID` — Added when resuming an existing conversation
Other backends use equivalent flags for their CLIs (e.g., Pi uses `--mode json` and `--append-system-prompt`, Codex uses `--json` and `--dangerously-bypass-approvals-and-sandbox`).
The process runs with `cwd` set to the `config/` directory, so Claude can read/write files there (like `memory.md`).
`stdin` is set to `"ignore"` to prevent the CLI from waiting for interactive input.
@@ -259,9 +301,9 @@ When a session ID exists, `--resume 37336c32-73cb-4cf5-9771-1c8f694398ff` is add
The CLI returns a JSON array on stdout. The gateway parses it as chunks arrive.
**File:** `src/agent-runtime.ts` → `runClaude()` stdout handler
**File:** `src/backends/{backend}-backend.ts` → `spawnCli()` stdout handler
Example CLI output:
Example CLI output (Claude):
```json
[
@@ -380,11 +422,11 @@ Instruction: Save important context to memory.md before shutting down.
---
## What Gets Sent to Claude
## What Gets Sent to the Backend
For every event, Claude receives:
For every event, the backend CLI receives:
1. **Default Claude Code system prompt** (built-in, from the CLI)
1. **Default system prompt** (built-in from the CLI — varies by backend)
2. **Appended system prompt** (from our assembled markdown files):
- Identity (who the agent is)
- Personality (how it behaves)
@@ -394,10 +436,12 @@ For every event, Claude receives:
- Tool configuration (API notes)
- Preamble about writing to memory.md
3. **The prompt text** (user message, heartbeat instruction, or cron instruction)
4. **Session history** (if resuming via `--resume`)
5. **Allowed tools** (Read, Write, Edit, Glob, Grep, WebSearch, WebFetch)
4. **Session history** (if resuming via backend-specific session flags)
5. **Allowed tools** (Claude only: Read, Write, Edit, Glob, Grep, WebSearch, WebFetch)
Claude runs in the `config/` directory, so it can read and write files there — including updating `memory.md` with new facts.
> **Note:** How the system prompt is delivered varies by backend. Claude Code uses `--append-system-prompt-file`, Pi uses `--append-system-prompt`, and other backends prepend it to the user prompt.
The backend CLI runs in the `config/` directory, so the agent can read and write files there — including updating `memory.md` with new facts.
---
@@ -408,12 +452,26 @@ src/
├── index.ts ← Entry point: creates GatewayCore, registers shutdown handler
├── gateway-core.ts ← Orchestrator: wires everything, manages lifecycle
├── config.ts ← Reads env vars (DISCORD_BOT_TOKEN, etc.)
├── logger.ts ← Pino structured logger with pretty-printing
├── discord-bot.ts ← Discord.js wrapper: messages, slash commands, typing
├── event-queue.ts ← FIFO queue: all events (message, heartbeat, cron, hook)
├── agent-runtime.ts ← Core engine: reads configs, spawns CLI, parses output
├── agent-runtime.ts ← Core engine: reads configs, delegates to backend adapter
├── backends/ ← Pluggable CLI backend adapters
│ ├── types.ts ← BackendAdapter interface & BackendName union type
│ ├── registry.ts ← resolveBackendName() & createBackend() factory
│ ├── index.ts ← Barrel exports
│ ├── claude-backend.ts ← Claude Code CLI adapter
│ ├── codex-backend.ts ← OpenAI Codex CLI adapter
│ ├── gemini-backend.ts ← Google Gemini CLI adapter
│ ├── opencode-backend.ts ← OpenCode CLI adapter
│ └── pi-backend.ts ← Pi Coding Agent CLI adapter
├── markdown-config-loader.ts ← Reads config/*.md files fresh each event
├── system-prompt-assembler.ts ← Concatenates markdown into system prompt with headers
├── skills-loader.ts ← Loads skills from config/skills/*/SKILL.md
├── session-manager.ts ← Channel → session ID mapping (persisted to JSON)
├── message-history.ts ← Per-channel message storage
├── conversation-archiver.ts ← Markdown conversation logs
├── ipc-watcher.ts ← Polls ipc/outbound/ for proactive messages
├── response-formatter.ts ← Splits long text for Discord's 2000 char limit
├── error-formatter.ts ← Sanitizes errors (strips keys, paths, stacks)
├── heartbeat-scheduler.ts ← setInterval timers from heartbeat.md
@@ -421,16 +479,64 @@ src/
├── hook-manager.ts ← Lifecycle hooks from agents.md
├── bootstrap-manager.ts ← First-run: validates/creates config files
├── channel-queue.ts ← Per-channel sequential processing
└── shutdown-handler.ts ← SIGTERM/SIGINT → graceful shutdown
├── shutdown-handler.ts ← SIGTERM/SIGINT → graceful shutdown
├── dashboard-server.ts ← Embedded HTTP server for Mission Control dashboard
├── dashboard-dev.ts ← Standalone dashboard dev server (mock data)
├── activity-log.ts ← In-memory + file-persisted activity event log
└── local-store.ts ← JSON file persistence layer (brain, tasks, content)
dashboard/ ← Mission Control frontend (served by dashboard-server)
├── index.html ← SPA shell with sidebar navigation
├── styles.css ← Dark-theme design system
└── app.js ← Client-side routing, API calls, rendering
config/
├── identity.md ← Agent name, role, specialization
├── soul.md ← Personality, tone, values
├── agents.md ← Rules, cron jobs, hooks
├── user.md ← Human's info and preferences
├── memory.md ← Long-term memory (agent-writable)
├── tools.md ← Tool configs and notes
├── heartbeat.md ← Proactive check definitions
├── boot.md ← Bootstrap parameters (optional)
└── sessions.json ← Channel → session ID map (auto-generated)
├── CLAUDE.md ← Persona: identity, personality, user context, tools
├── agents.md ← Rules, cron jobs, hooks (parsed at startup)
├── heartbeat.md ← Heartbeat checks (parsed at startup)
├── memory.md ← Long-term memory (agent-writable, auto-created)
├── sessions.json ← Channel → session ID map (auto-generated)
├── brain.json ← Second Brain facts (auto, managed by dashboard)
├── tasks.json ← Productivity tasks (auto, managed by dashboard)
├── content-items.json ← Content Intel items (auto, managed by dashboard)
├── activity-log.json ← Persisted activity log (auto)
├── bot-config.json ← Dashboard bot config (auto)
├── messages/ ← Message history (auto)
├── conversations/ ← Conversation archives (auto)
├── ipc/outbound/ ← Proactive message queue (auto)
└── skills/ ← Skill definitions
```
---
## Dashboard Flow
The Mission Control dashboard runs as an embedded HTTP server inside the gateway process.
### Real-Time Activity (SSE)
```
Agent Runtime processes event
→ ActivityLog.record() called
→ Entry added to in-memory buffer (capped at 200)
→ Entry persisted to config/activity-log.json (capped at 2000)
→ SSE broadcast to all connected dashboard clients
→ Dashboard UI updates live activity feed
```
### Data Persistence Flow
```
Dashboard UI (browser)
→ POST /api/brain { content, type, category, tags }
→ DashboardServer routes to LocalStores
→ JsonStore.append() adds entry with generated ID
→ Debounced write to config/brain.json (300ms)
→ Response with updated data
```
All persistent stores use the same pattern:
- **Read:** Load from JSON file on startup, serve from memory
- **Write:** Append/update in memory, debounced flush to disk
- **Shutdown:** `flushAll()` called to ensure all pending writes complete