feat: add Pi Coding Agent backend runtime, update dashboard UI
This commit is contained in:
Tanmay Karande
@@ -2,47 +2,60 @@
|
||||
|
||||
Features inspired by nanoclaw that can be implemented in the CLI-based gateway. Excludes Docker containers, WhatsApp channel, and Agent SDK (already ruled out).
|
||||
|
||||
## Quick Wins (hours)
|
||||
---
|
||||
|
||||
### 1. Message History Storage
|
||||
## ✅ Completed
|
||||
|
||||
Store inbound/outbound messages in a JSON file per channel. Nanoclaw uses SQLite, but a simple `config/messages/{channelId}.json` would work. This lets the agent see conversation history even after session expiry.
|
||||
### Message History Storage ✅
|
||||
Store inbound/outbound messages in a JSON file per channel. Implemented at `config/messages/{channelId}.json` with append-only storage, max 100 messages per channel, auto-trimmed.
|
||||
|
||||
How nanoclaw does it: SQLite table with id, chat_jid, sender, sender_name, content, timestamp, is_from_me, is_bot_message columns. Indexed by timestamp.
|
||||
### Structured Logging ✅
|
||||
Pino-based structured JSON logging with configurable log levels via `LOG_LEVEL` env var. Pretty-printed in dev, JSON in production.
|
||||
|
||||
Our approach: JSON file per channel at `config/messages/{channelId}.json`. Append-only. Include sender name, content, timestamp, and direction (inbound/outbound). Load recent messages into the system prompt or pass as context.
|
||||
### Conversation Archiving ✅
|
||||
Every exchange saved as readable markdown in `config/conversations/{channelId}/{YYYY-MM-DD}.md`.
|
||||
|
||||
### 2. Structured Logging
|
||||
### Retry with Backoff ✅
|
||||
Backend CLI calls retry 3 times with exponential backoff (5s, 10s, 20s) on transient errors. Session corruption errors fail immediately.
|
||||
|
||||
Replace `console.log` with a proper logger (pino). Nanoclaw logs everything with structured context (group, duration, event type). Makes debugging way easier on a headless server.
|
||||
### Idle Timeout for Sessions ✅
|
||||
Auto-clear sessions after configurable period of inactivity per channel (default 30 min). Tracked via `lastActivityTimestamp` in the session manager.
|
||||
|
||||
How nanoclaw does it: Pino logger with pretty-printing, configurable log levels via LOG_LEVEL env var, uncaught exception routing, container logs saved to disk.
|
||||
### IPC-Based Proactive Messaging ✅
|
||||
Agent writes JSON files to `config/ipc/outbound/` using the Write tool. Gateway polls every 2 seconds, delivers messages, and deletes processed files.
|
||||
|
||||
Our approach: Install pino, replace all console.log/error/warn calls, add log levels, save logs to `config/logs/`. Structured JSON logs make it easy to grep and filter.
|
||||
### Skills Engine ✅
|
||||
`config/skills/` directory where each subdirectory contains a `SKILL.md` file. Skills are loaded into the system prompt as additional sections.
|
||||
|
||||
### 3. Conversation Archiving
|
||||
### Config Simplification ✅
|
||||
Merged `soul.md`, `identity.md`, `user.md`, and `tools.md` into a single `CLAUDE.md` file.
|
||||
|
||||
Before sessions get too long, archive transcripts to `config/conversations/`. Nanoclaw does this via a PreCompact hook that saves the full transcript as markdown before Claude compacts its context.
|
||||
### Mission Control Dashboard ✅
|
||||
Embedded web-based dashboard served on port 3100 with:
|
||||
- **Command Center** — Real-time stats, live activity feed, agent config
|
||||
- **Activity** — Full event history with SSE real-time updates
|
||||
- **Sessions** — Active Discord channel sessions and message viewer
|
||||
- **Second Brain** — Knowledge store with facts (notes/URLs/files), categories, tags, search. Plus memory and persona editors, and skills viewer
|
||||
- **Productivity** — Kanban-style task board (To Do → In Progress → Done) with priorities, projects, and due dates
|
||||
- **Content Intel** — Content curation (articles, videos, papers, repos) with status tracking (queued → read → archived)
|
||||
- **Scheduler** — View cron jobs, heartbeat checks, and lifecycle hooks
|
||||
- **Connections** — Integration statuses
|
||||
- **Settings** — Config overview and quick todos
|
||||
|
||||
Our approach: After each CLI response, optionally append the exchange (prompt + response) to `config/conversations/{channelId}/{date}.md`. This creates a human-readable conversation log separate from Claude's internal session storage.
|
||||
All backed by local JSON file persistence in `config/` via `local-store.ts` — zero external dependencies.
|
||||
|
||||
### 4. Retry with Backoff
|
||||
### Multi-Backend Support ✅
|
||||
Abstracted backend interface (`BackendAdapter`) with pluggable CLI adapters. Supports five backends: **Claude Code**, **Codex**, **Gemini CLI**, **OpenCode**, and **Pi**. Switchable via `AGENT_BACKEND` env var. Each adapter normalizes CLI-specific JSON output into a common `BackendEventResult` format. Full property and unit test coverage for all backends.
|
||||
|
||||
When the Claude CLI fails, retry with exponential backoff (5s, 10s, 20s, 40s, 80s) up to 5 times. Nanoclaw does this for container failures.
|
||||
---
|
||||
|
||||
How nanoclaw does it: GroupQueue tracks retryCount per group, uses `BASE_RETRY_MS * 2^retryCount` delay, max 5 retries, resets on success.
|
||||
## Remaining Features
|
||||
|
||||
Our approach: Wrap the `executeClaude` call in a retry loop. On transient errors (timeout, CLI crash), wait and retry. On permanent errors (auth failure, invalid session), fail immediately.
|
||||
### Agent-Managed Tasks via File-Based IPC (Medium Effort)
|
||||
|
||||
## Medium Effort (day or two)
|
||||
The agent can create, pause, resume, and cancel scheduled tasks dynamically during a conversation.
|
||||
|
||||
### 5. Agent-Managed Tasks via File-Based IPC
|
||||
|
||||
The big one from nanoclaw. The agent can create, pause, resume, and cancel scheduled tasks dynamically during a conversation.
|
||||
|
||||
How nanoclaw does it: Custom MCP server with tools like `schedule_task`, `list_tasks`, `pause_task`, `resume_task`, `cancel_task`. Tasks stored in SQLite with schedule_type (cron/interval/once), context_mode (group/isolated), and full lifecycle tracking.
|
||||
|
||||
Our approach (simpler): The agent writes task definitions to `config/tasks.json` using the Write tool. The gateway polls this file periodically and schedules/unschedules tasks accordingly. Task format:
|
||||
Our approach: The agent writes task definitions to `config/tasks.json` using the Write tool. The gateway polls this file periodically and schedules/unschedules tasks accordingly. Task format:
|
||||
|
||||
```json
|
||||
[
|
||||
@@ -57,115 +70,42 @@ Our approach (simpler): The agent writes task definitions to `config/tasks.json`
|
||||
]
|
||||
```
|
||||
|
||||
The agent can add/remove/modify entries. The gateway watches the file and updates schedulers.
|
||||
|
||||
### 6. Multi-Turn Message Batching
|
||||
### Multi-Turn Message Batching (Medium Effort)
|
||||
|
||||
Instead of sending one message at a time to Claude, batch multiple messages that arrive while the agent is processing.
|
||||
|
||||
How nanoclaw does it: Collects all messages since the last agent run timestamp, formats them as XML (`<messages><message sender="..." time="...">content</message></messages>`), sends the batch as a single prompt.
|
||||
Our approach: When a message arrives while the agent is already processing for that channel, queue it. When the current processing finishes, batch all queued messages into a single prompt with sender/timestamp metadata.
|
||||
|
||||
Our approach: When a message arrives while the agent is already processing for that channel, queue it. When the current processing finishes, batch all queued messages into a single prompt with sender/timestamp metadata. This prevents the agent from missing rapid-fire messages.
|
||||
|
||||
### 7. Idle Timeout for Sessions
|
||||
|
||||
Auto-clear sessions after a configurable period of inactivity per channel.
|
||||
|
||||
How nanoclaw does it: 30-minute idle timeout per container. After last output, a timer starts. If no new messages arrive, the container is closed and the session ends.
|
||||
|
||||
Our approach: Track `lastActivityTimestamp` per channel in the session manager. Run a periodic check (every 5 minutes) that removes sessions older than the timeout (default 30 min). This prevents stale sessions from accumulating and keeps memory.md relevant.
|
||||
|
||||
### 8. Per-Channel Isolation
|
||||
### Per-Channel Isolation (Medium Effort)
|
||||
|
||||
Different channels could have different persona configs. A "work" channel gets a professional persona, a "fun" channel gets a casual one.
|
||||
|
||||
How nanoclaw does it: Per-group folders with separate CLAUDE.md, sessions, memory, and container mounts. Each group is fully isolated.
|
||||
Our approach: Support a `config/channels/{channelId}/` override directory. If it exists, load persona files from there instead of the root config. Falls back to root config for missing files.
|
||||
|
||||
Our approach: Support a `config/channels/{channelId}/` override directory. If it exists, load persona files from there instead of the root config. Falls back to root config for missing files. This lets you customize identity.md or soul.md per channel.
|
||||
|
||||
## Bigger Features (multi-day)
|
||||
|
||||
### 9. IPC-Based Proactive Messaging
|
||||
|
||||
Let the agent send messages to Discord channels proactively — not just as a response to a user message.
|
||||
|
||||
How nanoclaw does it: Filesystem-based IPC. The agent writes a JSON file to `/workspace/ipc/messages/` with `{type: "message", chatJid: "...", text: "..."}`. The host polls the directory every second and sends the message.
|
||||
|
||||
Our approach: Create a `config/ipc/outbound/` directory. The agent writes JSON files there using the Write tool. The gateway polls every 2 seconds, reads new files, sends messages to the specified Discord channel, and deletes the processed files. This enables:
|
||||
- Agent sending follow-up messages after thinking
|
||||
- Agent notifying you about something it found during a heartbeat
|
||||
- Agent sending messages to channels it's not currently chatting in
|
||||
|
||||
File format:
|
||||
```json
|
||||
{
|
||||
"type": "message",
|
||||
"channelId": "1475008084022788312",
|
||||
"text": "Hey, I found something interesting in your email!"
|
||||
}
|
||||
```
|
||||
|
||||
### 10. Skills Engine (Simplified)
|
||||
|
||||
A system for adding capabilities via markdown skill files that get loaded into the system prompt.
|
||||
|
||||
How nanoclaw does it: Full manifest-based system with YAML manifests, dependency resolution, conflict detection, file operations (rename, delete, move), backup/restore, rebase capability, and resolution caching.
|
||||
|
||||
Our approach (much simpler): A `config/skills/` directory where each subdirectory contains a `SKILL.md` file. Skills are loaded into the system prompt as additional sections. The agent can reference skills by name. No dependency management — just markdown files that add context.
|
||||
|
||||
```
|
||||
config/skills/
|
||||
├── web-research/
|
||||
│ └── SKILL.md → "When asked to research, use WebSearch and WebFetch..."
|
||||
├── code-review/
|
||||
│ └── SKILL.md → "When reviewing code, focus on security, performance..."
|
||||
└── email-helper/
|
||||
└── SKILL.md → "When checking email, use the Gmail API via..."
|
||||
```
|
||||
|
||||
### 11. SQLite Storage
|
||||
### SQLite Storage (Bigger Effort)
|
||||
|
||||
Replace JSON files with SQLite for messages, sessions, tasks, and state. Better for concurrent access, querying, and doesn't corrupt on partial writes.
|
||||
|
||||
How nanoclaw does it: Single SQLite database at `store/messages.db` with tables for chats, messages, scheduled_tasks, task_run_logs, router_state, sessions, registered_groups. Uses better-sqlite3 (synchronous API).
|
||||
|
||||
Our approach: Install better-sqlite3, create a single `config/aetheel.db` with tables for sessions, messages, tasks, and state. Migrate existing JSON files on first run.
|
||||
|
||||
### 12. Secrets Management
|
||||
### Secrets Management (Bigger Effort)
|
||||
|
||||
Prevent API keys and sensitive data from leaking through agent tool use.
|
||||
|
||||
How nanoclaw does it: Secrets passed via stdin (never written to disk), stripped from Bash subprocess environments via a PreToolUse hook, never mounted as files in containers.
|
||||
Our approach: Add a sanitization step: before passing the prompt to Claude, strip any env var values that look like secrets from the system prompt. Also consider using `--disallowedTools Bash` if the agent doesn't need shell access.
|
||||
|
||||
### MCP Server Integration (Bigger Effort)
|
||||
|
||||
Expose Aetheel's capabilities as MCP tools so the agent can interact with the dashboard data, manage tasks, and store knowledge programmatically during conversations.
|
||||
|
||||
|
||||
Our approach: Since we use `--dangerously-skip-permissions`, the agent can run Bash commands that might echo environment variables. Add a sanitization step: before passing the prompt to Claude, strip any env var values that look like secrets from the system prompt. Also consider using `--disallowedTools Bash` if the agent doesn't need shell access.
|
||||
|
||||
## Priority Recommendation
|
||||
|
||||
1. Structured logging (immediate debugging value)
|
||||
2. Message history storage (conversation context)
|
||||
3. Retry with backoff (reliability)
|
||||
4. IPC-based proactive messaging (agent autonomy)
|
||||
5. Agent-managed tasks (dynamic scheduling)
|
||||
6. Conversation archiving (audit trail)
|
||||
7. Everything else as needed
|
||||
1. Agent-managed dynamic tasks (agent autonomy)
|
||||
2. Multi-turn message batching (UX improvement)
|
||||
3. Per-channel isolation (flexibility)
|
||||
4. MCP Server integration (agent ↔ dashboard)
|
||||
5. SQLite storage (reliability at scale)
|
||||
6. Everything else as needed
|
||||
|
||||
---
|
||||
|
||||
## Completed: Config Simplification
|
||||
|
||||
Merged `soul.md`, `identity.md`, `user.md`, and `tools.md` into a single `CLAUDE.md` file. The config directory is now:
|
||||
|
||||
```
|
||||
config/
|
||||
├── CLAUDE.md ← Persona: identity, personality, user context, tools (all in one)
|
||||
├── agents.md ← Cron jobs + Hooks (parsed by gateway at startup)
|
||||
├── heartbeat.md ← Heartbeat checks (parsed by gateway at startup)
|
||||
├── memory.md ← Long-term memory (agent-writable, auto-created)
|
||||
└── sessions.json ← Channel → session ID map (auto-generated)
|
||||
```
|
||||
|
||||
Why this split:
|
||||
- `CLAUDE.md` is pure prompt context — the agent reads it but the gateway doesn't parse it
|
||||
- `agents.md` and `heartbeat.md` are parsed programmatically by the gateway to set up cron timers and heartbeat intervals
|
||||
- `memory.md` is the only file the agent writes to — keeping it separate prevents the agent from accidentally overwriting persona config
|
||||
- Fewer files to manage, one place to edit your persona
|
||||
|
||||
Reference in New Issue
Block a user