[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M77HSCMA595G2JYFZBGEKGKT":3,"skill-same:01M77HSCMA595G2JYFZBGEKGKT":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":48,"repositoryHtmlUrl":49,"repositoryHomepageUrl":50,"ownerHtmlUrl":51,"ownerBlogUrl":6,"ownerDescription":52,"updatedAt":53,"install":54,"resources":67},"01M77HSCMA595G2JYFZBGEKGKT","greenhelix-trading-bot-fleet-management",null,"Trading Bot Fleet Management: Unified Control for Multi-Bot Operations. Build a fleet management layer for 10+ trading bots with per-bot identity isolation, permission scoping, health monitoring, coordinated deployments, SLA tracking, and cost allocation. Includes detailed Python code examples with full API integration.","Lord1Egypt\u002Fawesome-skill-forge","Lord1Egypt","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F99746359?v=4","01M7592YDNT0XK9WJ0T2MQVPDN","Mohamed Mounir Ali AL Keshita","01M75GRFNHC8VZNWEB8DKDE433",3,"MIT","https:\u002F\u002Fgithub.com\u002FLord1Egypt\u002Fawesome-skill-forge\u002Fblob\u002Fmaster\u002Fcommunity\u002Fclawhub\u002Fg\u002Fgreenhelix-trading-bot-fleet-management\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: greenhelix-trading-bot-fleet-management\nversion: \"1.3.1\"\ndescription: \"Trading Bot Fleet Management: Unified Control for Multi-Bot Operations. Build a fleet management layer for 10+ trading bots with per-bot identity isolation, permission scoping, health monitoring, coordinated deployments, SLA tracking, and cost allocation. Includes detailed Python code examples with full API integration.\"\nlicense: MIT\ncompatibility: [openclaw]\nauthor: felix-agent\ntype: guide\ntags: [fleet-management, trading-bot, identity, monitoring, guide, greenhelix, openclaw, ai-agent]\nprice_usd: 99.0\ncontent_type: markdown\nexecutable: false\ninstall: none\ncredentials: [AGENT_SIGNING_KEY]\nmetadata:\n  openclaw:\n    requires:\n      env:\n        - AGENT_SIGNING_KEY\n    primaryEnv: AGENT_SIGNING_KEY\n---\n# Trading Bot Fleet Management: Unified Control for Multi-Bot Operations\n\n> **Notice**: This is an educational guide with illustrative code examples.\n> It does not execute code or install dependencies.\n> All examples use the GreenHelix sandbox (https:\u002F\u002Fsandbox.greenhelix.net) which\n> provides 500 free credits — no API key required to get started.\n>\n> **Referenced credentials** (you supply these in your own environment):\n> - `AGENT_SIGNING_KEY`: Cryptographic signing key for agent identity (Ed25519 key pair for request signing)\n\n\nIn March 2023, the Step Finance exploit drained $45M from Solana DeFi positions. The root cause was not a smart contract vulnerability or a novel cryptographic attack. It was operational: multiple bots shared API keys with no identity isolation. When one bot's credentials leaked, the attacker gained access to every bot, every exchange account, every withdrawal endpoint. No per-bot permissions. No unified health dashboard that would have caught anomalous behavior. No kill switch scoped to the compromised bot without killing the entire fleet. The team had to shut down everything, losing millions in unrealized positions across healthy bots while they figured out which one was compromised. This pattern repeats across the industry. Teams running 10+ bots accumulate the same structural debt: shared credentials passed through environment variables, no per-bot permission boundaries, health checks that are either absent or scattered across shell scripts nobody maintains, deployments that are \"SSH in and restart the process,\" and cost tracking that lives in a spreadsheet updated monthly if someone remembers. When one bot goes rogue -- whether from a bug, a compromised key, or a strategy that hits an edge case -- the blast radius is the entire fleet.\nThis guide builds a fleet management layer using GreenHelix's identity, messaging, and metrics tools. Each bot gets its own Ed25519 cryptographic identity, scoped permissions defining exactly what it can and cannot do, real-time health monitoring with automatic dead bot detection, coordinated deployment procedures, SLA tracking against defined performance targets, and per-bot cost allocation. Every concept comes with working Python code and equivalent curl commands against the GreenHelix API.\n1. [Fleet Architecture](#chapter-1-fleet-architecture)\n\n## What You'll Learn\n- Chapter 1: Fleet Architecture\n- Chapter 2: FleetManager Class\n- Chapter 3: BotIdentityManager Class\n- Chapter 4: Permission Scoping\n- Chapter 5: FleetHealthMonitor Class\n- Chapter 6: Coordinated Deployments\n- Chapter 7: SLA Tracking\n- Chapter 8: Cost Allocation\n- What's Next\n\n## Full Guide\n\n# Trading Bot Fleet Management: Unified Control for Multi-Bot Operations\n\nIn March 2023, the Step Finance exploit drained $45M from Solana DeFi positions. The root cause was not a smart contract vulnerability or a novel cryptographic attack. It was operational: multiple bots shared API keys with no identity isolation. When one bot's credentials leaked, the attacker gained access to every bot, every exchange account, every withdrawal endpoint. No per-bot permissions. No unified health dashboard that would have caught anomalous behavior. No kill switch scoped to the compromised bot without killing the entire fleet. The team had to shut down everything, losing millions in unrealized positions across healthy bots while they figured out which one was compromised. This pattern repeats across the industry. Teams running 10+ bots accumulate the same structural debt: shared credentials passed through environment variables, no per-bot permission boundaries, health checks that are either absent or scattered across shell scripts nobody maintains, deployments that are \"SSH in and restart the process,\" and cost tracking that lives in a spreadsheet updated monthly if someone remembers. When one bot goes rogue -- whether from a bug, a compromised key, or a strategy that hits an edge case -- the blast radius is the entire fleet.\n\nThis guide builds a fleet management layer using GreenHelix's identity, messaging, and metrics tools. Each bot gets its own Ed25519 cryptographic identity, scoped permissions defining exactly what it can and cannot do, real-time health monitoring with automatic dead bot detection, coordinated deployment procedures, SLA tracking against defined performance targets, and per-bot cost allocation. Every concept comes with working Python code and equivalent curl commands against the GreenHelix API.\n\n---\n\n## Table of Contents\n\n1. [Fleet Architecture](#chapter-1-fleet-architecture)\n2. [FleetManager Class](#chapter-2-fleetmanager-class)\n3. [BotIdentityManager Class](#chapter-3-botidentitymanager-class)\n4. [Permission Scoping](#chapter-4-permission-scoping)\n5. [FleetHealthMonitor Class](#chapter-5-fleethealthmonitor-class)\n6. [Coordinated Deployments](#chapter-6-coordinated-deployments)\n7. [SLA Tracking](#chapter-7-sla-tracking)\n8. [Cost Allocation](#chapter-8-cost-allocation)\n9. [What's Next](#whats-next)\n\n---\n\n## Chapter 1: Fleet Architecture\n\n### Why Fleet Management Matters\n\nThe Step Finance breach is instructive because it was not exotic. The attack surface was not a zero-day in a cryptographic library. It was a predictable consequence of how most teams operate trading bots at scale: every bot shares the same API key, the same exchange credentials, the same infrastructure account. This is the \"monolith credentials\" antipattern, and it shows up in every post-mortem of operational trading failures that did not involve a market event.\n\nConsider what a team with 15 bots typically looks like six months after launch:\n\n- **Shared credentials**: All bots read the same `.env` file or Kubernetes secret. Rotating one key means touching every bot.\n- **No permission boundaries**: A bot designed to execute $500 momentum trades on Binance Spot has the same access as a bot managing $50,000 Deribit options positions. If the momentum bot is compromised, the attacker can withdraw from the options account.\n- **Scattered health checks**: Bot #3 has a heartbeat endpoint. Bot #7 writes to a log file. Bots #1, #2, #4-6, and #8-15 have no health reporting at all. The team discovers a dead bot when a strategy stops producing PnL.\n- **Manual deployments**: Updating a strategy requires SSH-ing into each server, pulling the latest code, restarting the process, and hoping the bot reconnects to the exchange websocket cleanly. There is no rollback procedure beyond \"check out the previous git commit and restart again.\"\n- **No cost attribution**: The team knows the monthly AWS bill and the total exchange fees. They do not know which bot costs the most to operate, which strategy has a negative ROI after infrastructure costs, or whether a bot that trades 200 times per day is actually profitable after accounting for exchange fees, API rate limit costs, and compute.\n\nFleet management solves all of these by treating each bot as an independently identified, independently monitored, independently permissioned entity within a unified control plane.\n\n### Architecture Overview\n\nThe fleet management architecture has four layers:\n\n```\n+-----------------------------------------------------------------------+\n|                        Fleet Operator (Human)                          |\n|                  Strategic decisions, policy, budgets                   |\n+-----------------------------------------------------------------------+\n        |\n        v\n+-----------------------------------------------------------------------+\n|                        Fleet Manager (Agent)                           |\n|  register\u002Fderegister bots, issue commands, aggregate status            |\n|  GreenHelix identity: fleet-manager-{org}                              |\n+-----------------------------------------------------------------------+\n        |\n        +--------------------+--------------------+\n        |                    |                    |\n        v                    v                    v\n+---------------+   +---------------+   +---------------+\n|  Bot Group:   |   |  Bot Group:   |   |  Bot Group:   |\n|  Spot Arb     |   |  Perp Momentum|   |  Options MM   |\n|  3 bots       |   |  5 bots       |   |  4 bots       |\n+---------------+   +---------------+   +---------------+\n    |   |   |          |  |  |  |  |       |  |  |  |\n    v   v   v          v  v  v  v  v       v  v  v  v\n  Individual bots, each with:\n  - Own Ed25519 identity\n  - Scoped permissions\n  - Health heartbeat\n  - Metrics reporting\n  - Cost tracking\n```\n\nThe **fleet operator** is a human who sets policy: which strategies to run, on which exchanges, with what risk limits, and how much capital to allocate. The **fleet manager** is a GreenHelix agent that translates policy into operations: registering bots, issuing commands, monitoring health, orchestrating deployments. **Bot groups** organize bots by strategy type, and **individual bots** are the execution units, each with its own cryptographic identity.\n\n### GreenHelix Tools Used\n\nThis guide uses the following GreenHelix tools:\n\n| Tool | Purpose |\n|------|---------|\n| `register_agent` | Create identity for fleet manager and each bot |\n| `get_agent_identity` | Retrieve and verify bot identity |\n| `submit_metrics` | Report health, PnL, latency, trade count |\n| `get_sla_compliance` | Monitor bots against defined SLA targets |\n| `send_message` | Fleet commands, alerts, inter-bot communication |\n| `register_webhook` | Real-time event delivery for health alerts |\n| `search_agents_by_metrics` | Find underperforming bots across the fleet |\n| `get_agent_reputation` | Track bot reliability over time |\n| `create_event_schema` | Define fleet event types |\n| `publish_event` | Emit fleet events (deploy, failover, alert) |\n\n### Fleet Hierarchy\n\nThe hierarchy maps to GreenHelix's identity model. The fleet manager is a registered agent. Each bot group is a metadata tag. Each individual bot is a registered agent with metadata linking it to its group and to the fleet manager.\n\n```python\n# Hierarchy expressed in GreenHelix metadata\nfleet_manager_metadata = {\n    \"role\": \"fleet_manager\",\n    \"organization\": \"your-org-id\",\n    \"fleet_size\": 12,\n    \"groups\": [\"spot-arb\", \"perp-momentum\", \"options-mm\"]\n}\n\nbot_metadata = {\n    \"role\": \"trading_bot\",\n    \"fleet_manager\": \"fleet-manager-{org}\",\n    \"group\": \"spot-arb\",\n    \"exchange\": \"binance\",\n    \"strategy\": \"cross-exchange-arb\",\n    \"version\": \"2.4.1\",\n    \"max_position_usd\": 10000\n}\n```\n\nThis metadata is not cosmetic. It is queryable. When the fleet manager needs to find all bots in the \"perp-momentum\" group, it searches by metadata. When a health alert fires, the alert includes the bot's group and strategy so the operator knows immediately what type of bot is failing and what the potential market impact is.\n\n### Why Not Kubernetes Labels or Consul?\n\nYou might already use Kubernetes labels, Consul service tags, or Terraform metadata for infrastructure management. Those are fine for infrastructure concerns -- pod scheduling, service discovery, load balancing. They are not sufficient for trading bot fleet management because they operate at the wrong abstraction level. Kubernetes knows that a pod is running. It does not know that the pod is a trading bot with a $50,000 position limit on Deribit, that its Ed25519 key was last rotated 28 days ago, or that its SLA requires 99.9% uptime with sub-100ms exchange latency.\n\nGreenHelix's identity layer operates at the application level. Each bot's identity carries its trading-specific metadata: strategy type, exchange, position limits, permission tier, key version. The fleet manager queries this metadata through GreenHelix's API, not through infrastructure tooling. This means the fleet management layer works identically whether your bots run on Kubernetes, bare metal, EC2 instances, or a mix of all three. The infrastructure is abstracted away; the fleet identity is portable.\n\nThe practical consequence: when you migrate from EC2 to Kubernetes (or vice versa), no fleet management code changes. The bots register the same identities, report the same metrics, and respond to the same commands regardless of the underlying compute platform.\n\n---\n\n## Chapter 2: FleetManager Class\n\n### Central Management for Bot Fleet Operations\n\nThe FleetManager class is the control plane for all fleet operations. It handles bot registration, inventory management, fleet-wide commands, and status aggregation. Every operation goes through the GreenHelix API so that commands, status, and events are centralized and auditable.\n\n### Setup\n\n```python\nimport requests\nimport json\nimport time\nimport hashlib\nfrom datetime import datetime, timedelta\nfrom typing import Dict, List, Optional\nfrom dataclasses import dataclass, field\n\nbase_url = \"https:\u002F\u002Fapi.greenhelix.net\u002Fv1\"\napi_key = \"your-api-key\"  # From GreenHelix dashboard\n\nsession = requests.Session()\nsession.headers[\"Authorization\"] = f\"Bearer {api_key}\"\nsession.headers[\"Content-Type\"] = \"application\u002Fjson\"\n\ndef execute(tool: str, inputs: dict) -> dict:\n    \"\"\"Execute a GreenHelix tool and return the result.\"\"\"\n    resp = session.post(\n        f\"{base_url}\u002Fv1\",\n        json={\"tool\": tool, \"input\": inputs}\n    )\n    resp.raise_for_status()\n    return resp.json()\n```\n\nEquivalent curl for any `execute` call throughout this guide:\n\n```bash\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"register_agent\",\n    \"input\": {\n      \"name\": \"fleet-manager-acme\",\n      \"description\": \"Fleet manager for Acme Trading bot fleet\",\n      \"capabilities\": [\"fleet_management\", \"bot_orchestration\"],\n      \"metadata\": {\"role\": \"fleet_manager\", \"organization\": \"acme\"}\n    }\n  }'\n```\n\n### The FleetManager Class\n\n```python\n@dataclass\nclass BotRecord:\n    \"\"\"Local record of a registered bot.\"\"\"\n    agent_id: str\n    name: str\n    group: str\n    exchange: str\n    strategy: str\n    version: str\n    status: str = \"active\"\n    registered_at: str = \"\"\n    last_heartbeat: Optional[str] = None\n\n\nclass FleetManager:\n    \"\"\"Central management class for a trading bot fleet.\"\"\"\n\n    def __init__(self, org_id: str):\n        self.org_id = org_id\n        self.manager_agent_id: Optional[str] = None\n        self.bots: Dict[str, BotRecord] = {}\n        self._register_manager()\n\n    def _register_manager(self):\n        \"\"\"Register the fleet manager agent on GreenHelix.\"\"\"\n        result = execute(\"register_agent\", {\n            \"name\": f\"fleet-manager-{self.org_id}\",\n            \"description\": f\"Fleet manager for {self.org_id} trading bot fleet. \"\n                           f\"Handles registration, commands, health, deployments.\",\n            \"capabilities\": [\n                \"fleet_management\",\n                \"bot_orchestration\",\n                \"health_monitoring\",\n                \"deployment_coordination\"\n            ],\n            \"metadata\": {\n                \"role\": \"fleet_manager\",\n                \"organization\": self.org_id\n            }\n        })\n        self.manager_agent_id = result[\"agent_id\"]\n        print(f\"Fleet manager registered: {self.manager_agent_id}\")\n\n    def register_bot(self, name: str, group: str, exchange: str,\n                     strategy: str, version: str,\n                     capabilities: List[str] = None,\n                     max_position_usd: float = 10000) -> str:\n        \"\"\"Register a new bot in the fleet.\"\"\"\n        if capabilities is None:\n            capabilities = [\"trading\", \"metrics_reporting\"]\n\n        result = execute(\"register_agent\", {\n            \"name\": name,\n            \"description\": f\"Trading bot: {strategy} on {exchange}. \"\n                           f\"Part of {group} group in {self.org_id} fleet.\",\n            \"capabilities\": capabilities,\n            \"metadata\": {\n                \"role\": \"trading_bot\",\n                \"fleet_manager\": self.manager_agent_id,\n                \"group\": group,\n                \"exchange\": exchange,\n                \"strategy\": strategy,\n                \"version\": version,\n                \"max_position_usd\": max_position_usd,\n                \"organization\": self.org_id\n            }\n        })\n\n        agent_id = result[\"agent_id\"]\n        self.bots[agent_id] = BotRecord(\n            agent_id=agent_id,\n            name=name,\n            group=group,\n            exchange=exchange,\n            strategy=strategy,\n            version=version,\n            registered_at=datetime.utcnow().isoformat()\n        )\n        print(f\"Bot registered: {name} ({agent_id})\")\n        return agent_id\n\n    def deregister_bot(self, agent_id: str):\n        \"\"\"Remove a bot from the fleet.\"\"\"\n        if agent_id not in self.bots:\n            raise ValueError(f\"Bot {agent_id} not found in fleet\")\n\n        # Notify the bot to shut down gracefully\n        execute(\"send_message\", {\n            \"from_agent_id\": self.manager_agent_id,\n            \"to_agent_id\": agent_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"shutdown\",\n                \"reason\": \"deregistered_from_fleet\",\n                \"grace_period_seconds\": 30\n            }\n        })\n\n        bot = self.bots.pop(agent_id)\n        print(f\"Bot deregistered: {bot.name} ({agent_id})\")\n\n    def pause_all(self, reason: str = \"operator_initiated\"):\n        \"\"\"Pause all bots in the fleet. Bots stop opening new positions.\"\"\"\n        for agent_id, bot in self.bots.items():\n            execute(\"send_message\", {\n                \"from_agent_id\": self.manager_agent_id,\n                \"to_agent_id\": agent_id,\n                \"message_type\": \"command\",\n                \"payload\": {\n                    \"command\": \"pause\",\n                    \"reason\": reason,\n                    \"timestamp\": datetime.utcnow().isoformat()\n                }\n            })\n            bot.status = \"paused\"\n        print(f\"Fleet paused: {len(self.bots)} bots ({reason})\")\n\n    def resume_all(self):\n        \"\"\"Resume all paused bots.\"\"\"\n        for agent_id, bot in self.bots.items():\n            if bot.status == \"paused\":\n                execute(\"send_message\", {\n                    \"from_agent_id\": self.manager_agent_id,\n                    \"to_agent_id\": agent_id,\n                    \"message_type\": \"command\",\n                    \"payload\": {\n                        \"command\": \"resume\",\n                        \"timestamp\": datetime.utcnow().isoformat()\n                    }\n                })\n                bot.status = \"active\"\n        print(f\"Fleet resumed: {len(self.bots)} bots\")\n\n    def emergency_stop(self, reason: str = \"emergency\"):\n        \"\"\"Emergency stop: close all positions, cancel all orders, halt.\"\"\"\n        for agent_id, bot in self.bots.items():\n            execute(\"send_message\", {\n                \"from_agent_id\": self.manager_agent_id,\n                \"to_agent_id\": agent_id,\n                \"message_type\": \"command\",\n                \"payload\": {\n                    \"command\": \"emergency_stop\",\n                    \"reason\": reason,\n                    \"actions\": [\n                        \"cancel_all_open_orders\",\n                        \"close_all_positions_market\",\n                        \"halt_trading_loop\"\n                    ],\n                    \"timestamp\": datetime.utcnow().isoformat()\n                }\n            })\n            bot.status = \"stopped\"\n\n        # Log the emergency stop event\n        execute(\"publish_event\", {\n            \"agent_id\": self.manager_agent_id,\n            \"event_type\": \"fleet.emergency_stop\",\n            \"payload\": {\n                \"reason\": reason,\n                \"bots_affected\": len(self.bots),\n                \"bot_ids\": list(self.bots.keys()),\n                \"timestamp\": datetime.utcnow().isoformat()\n            }\n        })\n        print(f\"EMERGENCY STOP: {len(self.bots)} bots halted ({reason})\")\n\n    def get_fleet_status(self) -> Dict:\n        \"\"\"Aggregate status across all bots.\"\"\"\n        status_counts = {\"active\": 0, \"paused\": 0, \"stopped\": 0, \"dead\": 0}\n        group_counts = {}\n\n        for bot in self.bots.values():\n            status_counts[bot.status] = status_counts.get(bot.status, 0) + 1\n            group_counts[bot.group] = group_counts.get(bot.group, 0) + 1\n\n        return {\n            \"fleet_manager\": self.manager_agent_id,\n            \"total_bots\": len(self.bots),\n            \"status\": status_counts,\n            \"groups\": group_counts,\n            \"timestamp\": datetime.utcnow().isoformat()\n        }\n\n    def send_group_command(self, group: str, command: str,\n                           payload: dict = None):\n        \"\"\"Send a command to all bots in a specific group.\"\"\"\n        targets = [\n            (aid, bot) for aid, bot in self.bots.items()\n            if bot.group == group\n        ]\n        for agent_id, bot in targets:\n            execute(\"send_message\", {\n                \"from_agent_id\": self.manager_agent_id,\n                \"to_agent_id\": agent_id,\n                \"message_type\": \"command\",\n                \"payload\": {\n                    \"command\": command,\n                    **(payload or {}),\n                    \"timestamp\": datetime.utcnow().isoformat()\n                }\n            })\n        print(f\"Command '{command}' sent to {len(targets)} bots in {group}\")\n```\n\n### Registering a Fleet\n\n```python\nfleet = FleetManager(org_id=\"acme-trading\")\n\n# Spot arbitrage group\nfor i in range(3):\n    fleet.register_bot(\n        name=f\"spot-arb-{i+1:02d}\",\n        group=\"spot-arb\",\n        exchange=\"binance\" if i \u003C 2 else \"okx\",\n        strategy=\"cross-exchange-arb\",\n        version=\"2.4.1\",\n        max_position_usd=10000\n    )\n\n# Perpetual futures momentum group\nfor i in range(5):\n    exchanges = [\"binance\", \"bybit\", \"okx\", \"deribit\", \"binance\"]\n    fleet.register_bot(\n        name=f\"perp-momentum-{i+1:02d}\",\n        group=\"perp-momentum\",\n        exchange=exchanges[i],\n        strategy=\"trend-following-perps\",\n        version=\"1.8.3\",\n        max_position_usd=25000\n    )\n\n# Options market making group\nfor i in range(4):\n    fleet.register_bot(\n        name=f\"options-mm-{i+1:02d}\",\n        group=\"options-mm\",\n        exchange=\"deribit\",\n        strategy=\"delta-neutral-mm\",\n        version=\"3.1.0\",\n        capabilities=[\"trading\", \"metrics_reporting\", \"options_greeks\"],\n        max_position_usd=50000\n    )\n\nprint(json.dumps(fleet.get_fleet_status(), indent=2))\n```\n\n```bash\n# curl: Register a single bot\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"register_agent\",\n    \"input\": {\n      \"name\": \"spot-arb-01\",\n      \"description\": \"Trading bot: cross-exchange-arb on binance. Part of spot-arb group.\",\n      \"capabilities\": [\"trading\", \"metrics_reporting\"],\n      \"metadata\": {\n        \"role\": \"trading_bot\",\n        \"fleet_manager\": \"fleet-manager-agent-id\",\n        \"group\": \"spot-arb\",\n        \"exchange\": \"binance\",\n        \"strategy\": \"cross-exchange-arb\",\n        \"version\": \"2.4.1\",\n        \"max_position_usd\": 10000\n      }\n    }\n  }'\n\n# curl: Send emergency stop to a bot\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"send_message\",\n    \"input\": {\n      \"from_agent_id\": \"fleet-manager-agent-id\",\n      \"to_agent_id\": \"bot-agent-id\",\n      \"message_type\": \"command\",\n      \"payload\": {\n        \"command\": \"emergency_stop\",\n        \"reason\": \"anomalous_behavior_detected\",\n        \"actions\": [\"cancel_all_open_orders\", \"close_all_positions_market\", \"halt_trading_loop\"]\n      }\n    }\n  }'\n```\n\n### Fleet Command Patterns\n\nThree command scopes cover all operational scenarios:\n\n**Fleet-wide**: `pause_all`, `resume_all`, `emergency_stop`. Used for market-wide events (flash crash, exchange outage, security incident). These hit every bot regardless of group.\n\n**Group-scoped**: `send_group_command`. Used for strategy-specific actions: updating parameters for all momentum bots, pausing all options bots before an expiry event, or upgrading all arb bots to a new version.\n\n**Individual**: Direct `send_message` to a single bot. Used for surgical interventions: adjusting one bot's position limits, forcing one bot to close a specific position, or rotating one bot's exchange API keys.\n\nThe command hierarchy means that a compromised spot arb bot can be killed without touching the options market makers. That is the entire point: blast radius containment through identity isolation.\n\n---\n\n## Chapter 3: BotIdentityManager Class\n\n### Per-Bot Cryptographic Identity\n\nEvery bot in the fleet gets its own Ed25519 keypair. This is not optional. Shared keys are how Step Finance happened. Per-bot keys mean that compromising one bot's key gives the attacker access to exactly one bot. The fleet manager can revoke that single identity without affecting any other bot.\n\nEd25519 was chosen for three reasons: it produces compact 64-byte signatures, signing is fast enough that it adds negligible latency to trading operations (tens of microseconds), and the key generation is deterministic from a seed, making backup and recovery straightforward.\n\n### The BotIdentityManager Class\n\n```python\nimport nacl.signing\nimport nacl.encoding\nimport base64\nimport secrets\nimport os\nfrom typing import Tuple\n\n\nclass BotIdentityManager:\n    \"\"\"Manages Ed25519 identities for trading bots.\"\"\"\n\n    def __init__(self, fleet_manager_id: str, key_store_path: str = \"\u002Fsecure\u002Fkeys\"):\n        self.fleet_manager_id = fleet_manager_id\n        self.key_store_path = key_store_path\n        self.identities: Dict[str, dict] = {}  # agent_id -> identity record\n\n    def generate_keypair(self, bot_name: str) -> Tuple[str, str]:\n        \"\"\"Generate an Ed25519 keypair for a bot.\n        Returns (public_key_b64, private_key_b64).\n        \"\"\"\n        signing_key = nacl.signing.SigningKey.generate()\n        verify_key = signing_key.verify_key\n\n        private_b64 = base64.b64encode(\n            signing_key.encode()\n        ).decode(\"utf-8\")\n        public_b64 = base64.b64encode(\n            verify_key.encode()\n        ).decode(\"utf-8\")\n\n        # Store private key securely -- in production, use a secrets manager\n        key_path = os.path.join(self.key_store_path, f\"{bot_name}.key\")\n        os.makedirs(os.path.dirname(key_path), exist_ok=True)\n        with open(key_path, \"w\") as f:\n            f.write(private_b64)\n        os.chmod(key_path, 0o600)  # Owner read\u002Fwrite only\n\n        return public_b64, private_b64\n\n    def register_identity(self, bot_name: str, group: str,\n                          exchange: str, strategy: str,\n                          version: str,\n                          permissions: List[str] = None) -> dict:\n        \"\"\"Generate keys and register identity on GreenHelix.\"\"\"\n        public_key, private_key = self.generate_keypair(bot_name)\n\n        if permissions is None:\n            permissions = [\"trade\", \"report_metrics\"]\n\n        # Register the agent with its public key\n        result = execute(\"register_agent\", {\n            \"name\": bot_name,\n            \"description\": f\"Trading bot: {strategy} on {exchange}\",\n            \"capabilities\": permissions,\n            \"metadata\": {\n                \"role\": \"trading_bot\",\n                \"fleet_manager\": self.fleet_manager_id,\n                \"group\": group,\n                \"exchange\": exchange,\n                \"strategy\": strategy,\n                \"version\": version,\n                \"public_key\": public_key,\n                \"key_algorithm\": \"ed25519\",\n                \"registered_at\": datetime.utcnow().isoformat()\n            }\n        })\n\n        agent_id = result[\"agent_id\"]\n        identity_record = {\n            \"agent_id\": agent_id,\n            \"bot_name\": bot_name,\n            \"public_key\": public_key,\n            \"group\": group,\n            \"exchange\": exchange,\n            \"permissions\": permissions,\n            \"created_at\": datetime.utcnow().isoformat(),\n            \"key_version\": 1,\n            \"status\": \"active\"\n        }\n        self.identities[agent_id] = identity_record\n        return identity_record\n\n    def verify_identity(self, agent_id: str) -> dict:\n        \"\"\"Retrieve and verify a bot's identity from GreenHelix.\"\"\"\n        result = execute(\"get_agent_identity\", {\n            \"agent_id\": agent_id\n        })\n\n        local_record = self.identities.get(agent_id)\n        if local_record:\n            # Verify public key matches what we registered\n            remote_key = result.get(\"metadata\", {}).get(\"public_key\")\n            if remote_key != local_record[\"public_key\"]:\n                raise SecurityError(\n                    f\"Public key mismatch for {agent_id}. \"\n                    f\"Expected: {local_record['public_key'][:16]}... \"\n                    f\"Got: {remote_key[:16] if remote_key else 'None'}... \"\n                    f\"Possible key tampering.\"\n                )\n        return result\n\n    def rotate_key(self, agent_id: str) -> dict:\n        \"\"\"Rotate a bot's Ed25519 keypair without downtime.\n\n        The rotation procedure:\n        1. Generate new keypair\n        2. Update GreenHelix identity with new public key\n        3. Bot continues operating -- it picks up the new key on next heartbeat\n        4. Old key is archived (not deleted) for signature verification of historical events\n        \"\"\"\n        if agent_id not in self.identities:\n            raise ValueError(f\"No identity record for {agent_id}\")\n\n        record = self.identities[agent_id]\n        bot_name = record[\"bot_name\"]\n\n        # Archive old key\n        old_key_path = os.path.join(\n            self.key_store_path,\n            f\"{bot_name}.key.v{record['key_version']}\"\n        )\n        current_key_path = os.path.join(\n            self.key_store_path, f\"{bot_name}.key\"\n        )\n        if os.path.exists(current_key_path):\n            os.rename(current_key_path, old_key_path)\n\n        # Generate new keypair\n        new_public, new_private = self.generate_keypair(bot_name)\n\n        # Update the identity on GreenHelix -- submit metrics indicating rotation\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                \"key_rotation\": 1,\n                \"key_version\": record[\"key_version\"] + 1,\n                \"rotation_timestamp\": datetime.utcnow().isoformat()\n            }\n        })\n\n        # Notify the bot to pick up the new key\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": agent_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"rotate_key\",\n                \"new_public_key\": new_public,\n                \"key_version\": record[\"key_version\"] + 1,\n                \"effective_at\": datetime.utcnow().isoformat()\n            }\n        })\n\n        record[\"public_key\"] = new_public\n        record[\"key_version\"] += 1\n        print(f\"Key rotated for {bot_name}: v{record['key_version']}\")\n        return record\n\n    def revoke_identity(self, agent_id: str, reason: str):\n        \"\"\"Revoke a compromised bot's identity.\n\n        This is the nuclear option: the bot can no longer authenticate.\n        Use when a bot is confirmed compromised.\n        \"\"\"\n        if agent_id not in self.identities:\n            raise ValueError(f\"No identity record for {agent_id}\")\n\n        record = self.identities[agent_id]\n\n        # Send shutdown command before revocation\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": agent_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"emergency_stop\",\n                \"reason\": f\"identity_revoked: {reason}\",\n                \"actions\": [\n                    \"cancel_all_open_orders\",\n                    \"close_all_positions_market\",\n                    \"halt_trading_loop\",\n                    \"destroy_local_keys\"\n                ]\n            }\n        })\n\n        # Publish revocation event for audit trail\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.identity_revoked\",\n            \"payload\": {\n                \"revoked_agent_id\": agent_id,\n                \"bot_name\": record[\"bot_name\"],\n                \"reason\": reason,\n                \"revoked_at\": datetime.utcnow().isoformat(),\n                \"key_version_revoked\": record[\"key_version\"]\n            }\n        })\n\n        # Delete the private key\n        key_path = os.path.join(\n            self.key_store_path, f\"{record['bot_name']}.key\"\n        )\n        if os.path.exists(key_path):\n            # Overwrite with random data before unlinking\n            with open(key_path, \"wb\") as f:\n                f.write(secrets.token_bytes(64))\n            os.unlink(key_path)\n\n        record[\"status\"] = \"revoked\"\n        print(f\"Identity revoked: {record['bot_name']} ({agent_id}): {reason}\")\n\n    def list_active_identities(self) -> List[dict]:\n        \"\"\"List all active bot identities.\"\"\"\n        return [\n            r for r in self.identities.values()\n            if r[\"status\"] == \"active\"\n        ]\n```\n\n```bash\n# curl: Get a bot's identity\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"get_agent_identity\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id-here\"\n    }\n  }'\n\n# curl: Notify bot of key rotation via message\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"send_message\",\n    \"input\": {\n      \"from_agent_id\": \"fleet-manager-agent-id\",\n      \"to_agent_id\": \"bot-agent-id\",\n      \"message_type\": \"command\",\n      \"payload\": {\n        \"command\": \"rotate_key\",\n        \"new_public_key\": \"base64-encoded-new-public-key\",\n        \"key_version\": 2,\n        \"effective_at\": \"2026-04-07T14:30:00Z\"\n      }\n    }\n  }'\n```\n\n### Key Rotation Without Downtime\n\nKey rotation is the most operationally sensitive identity operation. The bot must continue trading during rotation -- you cannot afford a gap where the bot has no valid key. The procedure is:\n\n1. **Generate new keypair** on the fleet manager side.\n2. **Archive the old private key** (do not delete -- you need it to verify signatures on historical events).\n3. **Update GreenHelix** with the new public key via metrics submission.\n4. **Notify the bot** via `send_message` with the new key material.\n5. The bot picks up the new key, switches to it, and acknowledges.\n\nBetween steps 4 and 5, the bot is still signing with the old key, and that is fine. The old key is still valid until the bot confirms the switch. There is no window where the bot has no valid signing key.\n\nA sane rotation schedule is every 30 days for production bots, immediately upon any suspected compromise, and after any personnel change on the team (someone leaves, all keys rotate).\n\n### Identity Revocation for Compromised Bots\n\nRevocation is destructive and immediate. When you confirm a bot is compromised, you do not rotate -- you revoke. The difference: rotation preserves continuity (the bot keeps running with a new key), while revocation terminates the bot's ability to operate entirely.\n\nThe revocation procedure overwrites the private key file with random data before deleting it. A simple `os.unlink()` leaves the key material on disk until the filesystem overwrites those blocks. Writing random data first ensures the key is unrecoverable without forensic disk analysis, and even that becomes unreliable on SSDs with wear leveling.\n\n---\n\n## Chapter 4: Permission Scoping\n\n### Principle of Least Privilege for Trading Bots\n\nA spot arb bot that compares prices across exchanges needs read access to two order books and the ability to place limit orders on two exchanges. It does not need withdrawal permissions. It does not need the ability to modify account settings. It does not need access to the options chain on Deribit. Every permission beyond what the bot needs is attack surface.\n\nThe principle of least privilege states that each bot should have the minimum set of permissions required to execute its strategy and nothing more. This is straightforward in theory and consistently ignored in practice because it requires upfront work: defining permission tiers, mapping strategies to permissions, and enforcing boundaries.\n\n### Permission Tiers\n\nFour tiers cover the range of trading bot operations:\n\n```python\nfrom enum import Enum\nfrom typing import Set\n\n\nclass PermissionTier(Enum):\n    READ_ONLY = \"read_only\"\n    TRADE = \"trade\"\n    WITHDRAW = \"withdraw\"\n    ADMIN = \"admin\"\n\n\n# What each tier can do\nTIER_PERMISSIONS: Dict[PermissionTier, Set[str]] = {\n    PermissionTier.READ_ONLY: {\n        \"read_orderbook\",\n        \"read_positions\",\n        \"read_balances\",\n        \"read_trade_history\",\n        \"report_metrics\",\n        \"receive_commands\",\n    },\n    PermissionTier.TRADE: {\n        \"read_orderbook\",\n        \"read_positions\",\n        \"read_balances\",\n        \"read_trade_history\",\n        \"report_metrics\",\n        \"receive_commands\",\n        \"place_order\",\n        \"cancel_order\",\n        \"modify_order\",\n    },\n    PermissionTier.WITHDRAW: {\n        \"read_orderbook\",\n        \"read_positions\",\n        \"read_balances\",\n        \"read_trade_history\",\n        \"report_metrics\",\n        \"receive_commands\",\n        \"place_order\",\n        \"cancel_order\",\n        \"modify_order\",\n        \"withdraw_funds\",\n        \"transfer_between_accounts\",\n    },\n    PermissionTier.ADMIN: {\n        \"read_orderbook\",\n        \"read_positions\",\n        \"read_balances\",\n        \"read_trade_history\",\n        \"report_metrics\",\n        \"receive_commands\",\n        \"place_order\",\n        \"cancel_order\",\n        \"modify_order\",\n        \"withdraw_funds\",\n        \"transfer_between_accounts\",\n        \"modify_api_keys\",\n        \"modify_account_settings\",\n        \"register_sub_accounts\",\n    },\n}\n```\n\n### Per-Exchange Permission Mapping\n\nDifferent exchanges have different permission models, but they all support the same core concept: API keys with scoped permissions. The PermissionManager maps abstract tiers to exchange-specific permission sets.\n\n```python\nclass PermissionManager:\n    \"\"\"Maps permission tiers to exchange-specific settings.\"\"\"\n\n    EXCHANGE_PERMISSION_MAP = {\n        \"binance\": {\n            PermissionTier.READ_ONLY: {\n                \"enableReading\": True,\n                \"enableSpotAndMarginTrading\": False,\n                \"enableWithdrawals\": False,\n                \"enableFutures\": False,\n            },\n            PermissionTier.TRADE: {\n                \"enableReading\": True,\n                \"enableSpotAndMarginTrading\": True,\n                \"enableWithdrawals\": False,\n                \"enableFutures\": True,\n            },\n            PermissionTier.WITHDRAW: {\n                \"enableReading\": True,\n                \"enableSpotAndMarginTrading\": True,\n                \"enableWithdrawals\": True,\n                \"enableFutures\": True,\n                \"withdrawalAddressWhitelist\": True,\n            },\n        },\n        \"deribit\": {\n            PermissionTier.READ_ONLY: {\n                \"scope\": \"read\",\n            },\n            PermissionTier.TRADE: {\n                \"scope\": \"trade:read\",\n            },\n            PermissionTier.WITHDRAW: {\n                \"scope\": \"trade:read:withdraw\",\n            },\n        },\n        \"okx\": {\n            PermissionTier.READ_ONLY: {\n                \"perm\": \"read_only\",\n                \"trade\": False,\n                \"withdraw\": False,\n            },\n            PermissionTier.TRADE: {\n                \"perm\": \"trade\",\n                \"trade\": True,\n                \"withdraw\": False,\n            },\n            PermissionTier.WITHDRAW: {\n                \"perm\": \"trade\",\n                \"trade\": True,\n                \"withdraw\": True,\n                \"ip_whitelist_required\": True,\n            },\n        },\n    }\n\n    def __init__(self, fleet_manager_id: str):\n        self.fleet_manager_id = fleet_manager_id\n        self.bot_permissions: Dict[str, dict] = {}\n\n    def assign_permissions(self, agent_id: str, bot_name: str,\n                           exchange: str,\n                           tier: PermissionTier,\n                           custom_restrictions: dict = None) -> dict:\n        \"\"\"Assign a permission tier to a bot and record it.\"\"\"\n        base_permissions = TIER_PERMISSIONS[tier]\n        exchange_config = self.EXCHANGE_PERMISSION_MAP.get(\n            exchange, {}\n        ).get(tier, {})\n\n        record = {\n            \"agent_id\": agent_id,\n            \"bot_name\": bot_name,\n            \"exchange\": exchange,\n            \"tier\": tier.value,\n            \"abstract_permissions\": list(base_permissions),\n            \"exchange_config\": exchange_config,\n            \"custom_restrictions\": custom_restrictions or {},\n            \"assigned_at\": datetime.utcnow().isoformat(),\n        }\n        self.bot_permissions[agent_id] = record\n\n        # Record permission assignment on GreenHelix\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                \"permission_tier\": tier.value,\n                \"permission_count\": len(base_permissions),\n                \"exchange\": exchange,\n                \"has_withdraw\": \"withdraw_funds\" in base_permissions,\n                \"has_admin\": tier == PermissionTier.ADMIN,\n            }\n        })\n\n        # Publish audit event\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.permission_assigned\",\n            \"payload\": {\n                \"target_agent_id\": agent_id,\n                \"bot_name\": bot_name,\n                \"tier\": tier.value,\n                \"exchange\": exchange,\n                \"permissions\": list(base_permissions),\n            }\n        })\n\n        return record\n\n    def check_permission(self, agent_id: str,\n                         action: str) -> bool:\n        \"\"\"Check if a bot has permission for an action.\"\"\"\n        record = self.bot_permissions.get(agent_id)\n        if not record:\n            return False\n        return action in record[\"abstract_permissions\"]\n\n    def escalate_permission(self, agent_id: str,\n                            new_tier: PermissionTier,\n                            reason: str,\n                            approved_by: str):\n        \"\"\"Escalate a bot's permissions. Requires explicit approval.\"\"\"\n        record = self.bot_permissions.get(agent_id)\n        if not record:\n            raise ValueError(f\"No permission record for {agent_id}\")\n\n        old_tier = record[\"tier\"]\n\n        # Log the escalation for audit\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.permission_escalated\",\n            \"payload\": {\n                \"target_agent_id\": agent_id,\n                \"bot_name\": record[\"bot_name\"],\n                \"old_tier\": old_tier,\n                \"new_tier\": new_tier.value,\n                \"reason\": reason,\n                \"approved_by\": approved_by,\n                \"escalated_at\": datetime.utcnow().isoformat(),\n            }\n        })\n\n        # Update the record\n        new_permissions = TIER_PERMISSIONS[new_tier]\n        record[\"tier\"] = new_tier.value\n        record[\"abstract_permissions\"] = list(new_permissions)\n        record[\"exchange_config\"] = self.EXCHANGE_PERMISSION_MAP.get(\n            record[\"exchange\"], {}\n        ).get(new_tier, {})\n\n        print(f\"Permission escalated for {record['bot_name']}: \"\n              f\"{old_tier} -> {new_tier.value} (reason: {reason})\")\n\n    def get_fleet_permission_summary(self) -> Dict:\n        \"\"\"Summary of permissions across the fleet.\"\"\"\n        tier_counts = {}\n        withdraw_enabled = []\n        admin_enabled = []\n\n        for agent_id, record in self.bot_permissions.items():\n            tier = record[\"tier\"]\n            tier_counts[tier] = tier_counts.get(tier, 0) + 1\n            if \"withdraw_funds\" in record[\"abstract_permissions\"]:\n                withdraw_enabled.append(record[\"bot_name\"])\n            if record[\"tier\"] == \"admin\":\n                admin_enabled.append(record[\"bot_name\"])\n\n        return {\n            \"total_bots\": len(self.bot_permissions),\n            \"tiers\": tier_counts,\n            \"withdraw_enabled\": withdraw_enabled,\n            \"admin_enabled\": admin_enabled,\n        }\n```\n\n### Cross-Bot Permission Isolation\n\nPermission isolation is enforced at two levels. First, at the GreenHelix level: each bot has its own agent ID, and GreenHelix tools scope operations to the authenticated agent. Bot A cannot submit metrics as Bot B. Bot A cannot read Bot B's messages. This is identity-level isolation.\n\nSecond, at the exchange level: each bot should use its own exchange API key with permissions matching its tier. Do not share exchange API keys across bots. This is the lesson from Step Finance -- even if you have perfect identity isolation at the fleet management layer, shared exchange keys collapse the isolation at the point that matters most.\n\n```python\n# Assign permissions to the fleet\nperm_mgr = PermissionManager(fleet_manager_id=fleet.manager_agent_id)\n\n# Spot arb bots: TRADE tier (no withdrawals)\nfor agent_id, bot in fleet.bots.items():\n    if bot.group == \"spot-arb\":\n        perm_mgr.assign_permissions(\n            agent_id=agent_id,\n            bot_name=bot.name,\n            exchange=bot.exchange,\n            tier=PermissionTier.TRADE,\n            custom_restrictions={\n                \"max_order_size_usd\": 5000,\n                \"allowed_pairs\": [\"BTC\u002FUSDT\", \"ETH\u002FUSDT\", \"SOL\u002FUSDT\"],\n            }\n        )\n\n# Options MM bots: TRADE tier with additional restrictions\nfor agent_id, bot in fleet.bots.items():\n    if bot.group == \"options-mm\":\n        perm_mgr.assign_permissions(\n            agent_id=agent_id,\n            bot_name=bot.name,\n            exchange=bot.exchange,\n            tier=PermissionTier.TRADE,\n            custom_restrictions={\n                \"max_order_size_usd\": 25000,\n                \"allowed_instruments\": [\"BTC-*-C\", \"BTC-*-P\", \"ETH-*-C\", \"ETH-*-P\"],\n                \"max_delta_exposure\": 0.5,\n            }\n        )\n\nprint(json.dumps(perm_mgr.get_fleet_permission_summary(), indent=2))\n```\n\nThe key insight: no bot in this fleet has withdrawal permissions. Withdrawals should only be executed by a dedicated withdrawal agent that is not connected to exchange websockets and has its own approval workflow (ideally requiring human confirmation). A compromised trading bot that cannot withdraw can, at worst, make bad trades. A compromised bot with withdrawal access can empty the exchange account.\n\n---\n\n## Chapter 5: FleetHealthMonitor Class\n\n### Real-Time Health Across All Bots\n\nHealth monitoring is the difference between discovering a dead bot immediately and discovering it three days later when someone notices a strategy stopped making money. For a fleet of 12 bots, you need centralized health that answers three questions in real time: is each bot alive, is each bot performing within expectations, and are there any bots that need intervention?\n\n### Metrics Collected\n\nEach bot reports the following metrics on every heartbeat:\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| `uptime_seconds` | gauge | Time since last restart |\n| `heartbeat_latency_ms` | gauge | Round-trip time to exchange websocket |\n| `pnl_usd` | gauge | Unrealized + realized PnL since start of day |\n| `drawdown_pct` | gauge | Current drawdown from peak equity |\n| `trade_count` | counter | Number of trades executed since start of day |\n| `open_positions` | gauge | Number of currently open positions |\n| `open_orders` | gauge | Number of currently open orders |\n| `memory_mb` | gauge | Process memory usage |\n| `cpu_pct` | gauge | Process CPU usage |\n| `error_count` | counter | Number of errors since last heartbeat |\n\n### The FleetHealthMonitor Class\n\n```python\nclass FleetHealthMonitor:\n    \"\"\"Monitors health across all bots in the fleet.\"\"\"\n\n    def __init__(self, fleet_manager_id: str, bots: Dict[str, BotRecord],\n                 heartbeat_interval_seconds: int = 30,\n                 dead_threshold_seconds: int = 120):\n        self.fleet_manager_id = fleet_manager_id\n        self.bots = bots\n        self.heartbeat_interval = heartbeat_interval_seconds\n        self.dead_threshold = dead_threshold_seconds\n        self.health_records: Dict[str, dict] = {}\n        self.alert_callbacks: List[callable] = []\n        self._setup_webhooks()\n\n    def _setup_webhooks(self):\n        \"\"\"Register webhooks for real-time health alerts.\"\"\"\n        execute(\"register_webhook\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_types\": [\n                \"fleet.heartbeat\",\n                \"fleet.health_alert\",\n                \"fleet.bot_dead\"\n            ],\n            \"url\": \"https:\u002F\u002Fyour-fleet-dashboard.example.com\u002Fwebhooks\u002Fhealth\",\n            \"secret\": \"webhook-signing-secret\"\n        })\n\n    def record_heartbeat(self, agent_id: str, metrics: dict):\n        \"\"\"Record a heartbeat from a bot.\"\"\"\n        now = datetime.utcnow().isoformat()\n\n        # Submit metrics to GreenHelix\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                **metrics,\n                \"heartbeat_timestamp\": now,\n            }\n        })\n\n        self.health_records[agent_id] = {\n            \"agent_id\": agent_id,\n            \"bot_name\": self.bots[agent_id].name if agent_id in self.bots else \"unknown\",\n            \"last_heartbeat\": now,\n            \"metrics\": metrics,\n            \"status\": self._evaluate_health(agent_id, metrics),\n        }\n\n    def _evaluate_health(self, agent_id: str, metrics: dict) -> str:\n        \"\"\"Evaluate bot health from metrics. Returns status string.\"\"\"\n        issues = []\n\n        # Check drawdown\n        if metrics.get(\"drawdown_pct\", 0) > 10:\n            issues.append(f\"high_drawdown:{metrics['drawdown_pct']:.1f}%\")\n        elif metrics.get(\"drawdown_pct\", 0) > 5:\n            issues.append(f\"elevated_drawdown:{metrics['drawdown_pct']:.1f}%\")\n\n        # Check latency\n        if metrics.get(\"heartbeat_latency_ms\", 0) > 5000:\n            issues.append(f\"high_latency:{metrics['heartbeat_latency_ms']}ms\")\n        elif metrics.get(\"heartbeat_latency_ms\", 0) > 1000:\n            issues.append(f\"elevated_latency:{metrics['heartbeat_latency_ms']}ms\")\n\n        # Check errors\n        if metrics.get(\"error_count\", 0) > 10:\n            issues.append(f\"high_errors:{metrics['error_count']}\")\n        elif metrics.get(\"error_count\", 0) > 3:\n            issues.append(f\"elevated_errors:{metrics['error_count']}\")\n\n        # Check memory\n        if metrics.get(\"memory_mb\", 0) > 2048:\n            issues.append(f\"high_memory:{metrics['memory_mb']}MB\")\n\n        if any(\"high_\" in i for i in issues):\n            self._trigger_alert(agent_id, \"critical\", issues)\n            return \"critical\"\n        elif issues:\n            self._trigger_alert(agent_id, \"warning\", issues)\n            return \"warning\"\n        return \"healthy\"\n\n    def _trigger_alert(self, agent_id: str, severity: str,\n                       issues: List[str]):\n        \"\"\"Send alert via GreenHelix messaging.\"\"\"\n        bot_name = self.bots[agent_id].name if agent_id in self.bots else agent_id\n\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": self.fleet_manager_id,  # Self-message for logging\n            \"message_type\": \"alert\",\n            \"payload\": {\n                \"severity\": severity,\n                \"bot_name\": bot_name,\n                \"agent_id\": agent_id,\n                \"issues\": issues,\n                \"timestamp\": datetime.utcnow().isoformat(),\n            }\n        })\n\n        # Publish as event for audit trail\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.health_alert\",\n            \"payload\": {\n                \"severity\": severity,\n                \"agent_id\": agent_id,\n                \"bot_name\": bot_name,\n                \"issues\": issues,\n            }\n        })\n\n    def detect_dead_bots(self) -> List[str]:\n        \"\"\"Find bots that have missed heartbeats beyond the threshold.\"\"\"\n        now = datetime.utcnow()\n        dead_bots = []\n\n        for agent_id, bot in self.bots.items():\n            record = self.health_records.get(agent_id)\n            if record is None:\n                # Never sent a heartbeat\n                dead_bots.append(agent_id)\n                continue\n\n            last_hb = datetime.fromisoformat(record[\"last_heartbeat\"])\n            seconds_since = (now - last_hb).total_seconds()\n\n            if seconds_since > self.dead_threshold:\n                dead_bots.append(agent_id)\n                bot.status = \"dead\"\n\n                execute(\"publish_event\", {\n                    \"agent_id\": self.fleet_manager_id,\n                    \"event_type\": \"fleet.bot_dead\",\n                    \"payload\": {\n                        \"agent_id\": agent_id,\n                        \"bot_name\": bot.name,\n                        \"last_heartbeat\": record[\"last_heartbeat\"],\n                        \"seconds_since_heartbeat\": seconds_since,\n                        \"group\": bot.group,\n                    }\n                })\n\n        return dead_bots\n\n    def get_fleet_health_dashboard(self) -> dict:\n        \"\"\"Aggregate health data for dashboard display.\"\"\"\n        status_counts = {\"healthy\": 0, \"warning\": 0, \"critical\": 0, \"dead\": 0, \"unknown\": 0}\n        group_health = {}\n        total_pnl = 0.0\n        total_trades = 0\n\n        for agent_id, bot in self.bots.items():\n            record = self.health_records.get(agent_id)\n            if record:\n                status = record[\"status\"]\n                metrics = record[\"metrics\"]\n                total_pnl += metrics.get(\"pnl_usd\", 0)\n                total_trades += metrics.get(\"trade_count\", 0)\n            elif bot.status == \"dead\":\n                status = \"dead\"\n            else:\n                status = \"unknown\"\n\n            status_counts[status] = status_counts.get(status, 0) + 1\n\n            if bot.group not in group_health:\n                group_health[bot.group] = {\n                    \"healthy\": 0, \"warning\": 0,\n                    \"critical\": 0, \"dead\": 0, \"unknown\": 0,\n                    \"pnl_usd\": 0.0, \"trade_count\": 0,\n                }\n            group_health[bot.group][status] += 1\n            if record:\n                group_health[bot.group][\"pnl_usd\"] += record[\"metrics\"].get(\"pnl_usd\", 0)\n                group_health[bot.group][\"trade_count\"] += record[\"metrics\"].get(\"trade_count\", 0)\n\n        return {\n            \"timestamp\": datetime.utcnow().isoformat(),\n            \"total_bots\": len(self.bots),\n            \"status_summary\": status_counts,\n            \"groups\": group_health,\n            \"fleet_pnl_usd\": round(total_pnl, 2),\n            \"fleet_trade_count\": total_trades,\n        }\n\n    def initiate_failover(self, dead_agent_id: str,\n                          replacement_agent_id: str):\n        \"\"\"Failover a dead bot's responsibilities to a replacement.\"\"\"\n        dead_bot = self.bots.get(dead_agent_id)\n        if not dead_bot:\n            raise ValueError(f\"Bot {dead_agent_id} not found\")\n\n        # Send failover command to replacement bot\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": replacement_agent_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"assume_responsibility\",\n                \"from_bot\": dead_agent_id,\n                \"group\": dead_bot.group,\n                \"exchange\": dead_bot.exchange,\n                \"strategy\": dead_bot.strategy,\n                \"reason\": \"failover_from_dead_bot\",\n                \"timestamp\": datetime.utcnow().isoformat(),\n            }\n        })\n\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.failover\",\n            \"payload\": {\n                \"dead_bot\": dead_agent_id,\n                \"replacement_bot\": replacement_agent_id,\n                \"group\": dead_bot.group,\n                \"exchange\": dead_bot.exchange,\n            }\n        })\n\n        print(f\"Failover: {dead_bot.name} -> \"\n              f\"{self.bots[replacement_agent_id].name}\")\n```\n\n### Using the Health Monitor\n\n```python\nhealth = FleetHealthMonitor(\n    fleet_manager_id=fleet.manager_agent_id,\n    bots=fleet.bots,\n    heartbeat_interval_seconds=30,\n    dead_threshold_seconds=120\n)\n\n# Simulate a bot heartbeat (in production, bots call this themselves)\nfor agent_id, bot in fleet.bots.items():\n    health.record_heartbeat(agent_id, {\n        \"uptime_seconds\": 86400,\n        \"heartbeat_latency_ms\": 45,\n        \"pnl_usd\": 234.50,\n        \"drawdown_pct\": 1.2,\n        \"trade_count\": 47,\n        \"open_positions\": 3,\n        \"open_orders\": 6,\n        \"memory_mb\": 512,\n        \"cpu_pct\": 15.3,\n        \"error_count\": 0,\n    })\n\n# Check for dead bots\ndead = health.detect_dead_bots()\nif dead:\n    print(f\"Dead bots detected: {dead}\")\n\n# Get dashboard\ndashboard = health.get_fleet_health_dashboard()\nprint(json.dumps(dashboard, indent=2))\n```\n\n```bash\n# curl: Submit bot heartbeat metrics\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"submit_metrics\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id\",\n      \"metrics\": {\n        \"uptime_seconds\": 86400,\n        \"heartbeat_latency_ms\": 45,\n        \"pnl_usd\": 234.50,\n        \"drawdown_pct\": 1.2,\n        \"trade_count\": 47,\n        \"open_positions\": 3,\n        \"open_orders\": 6,\n        \"memory_mb\": 512,\n        \"cpu_pct\": 15.3,\n        \"error_count\": 0,\n        \"heartbeat_timestamp\": \"2026-04-07T14:00:00Z\"\n      }\n    }\n  }'\n\n# curl: Register health webhook\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"register_webhook\",\n    \"input\": {\n      \"agent_id\": \"fleet-manager-agent-id\",\n      \"event_types\": [\"fleet.heartbeat\", \"fleet.health_alert\", \"fleet.bot_dead\"],\n      \"url\": \"https:\u002F\u002Fyour-fleet-dashboard.example.com\u002Fwebhooks\u002Fhealth\",\n      \"secret\": \"webhook-signing-secret\"\n    }\n  }'\n```\n\n### Dead Bot Detection and Automatic Failover\n\nDead bot detection runs on a configurable interval (default: check every 30 seconds, declare dead after 120 seconds without a heartbeat). When a bot is declared dead, the monitor publishes a `fleet.bot_dead` event and the fleet manager can trigger automatic failover.\n\nFailover strategy depends on the bot type. Stateless bots (spot arb, momentum) can fail over to a standby instance that was pre-registered but dormant. Stateful bots (options market makers with active positions) require careful handoff: the replacement bot needs to query the exchange for open positions before it starts making new decisions.\n\nThe health monitor does not make the failover decision automatically -- it detects the dead bot and notifies the fleet manager, which then decides whether to failover, restart, or escalate to the human operator. Automatic failover without human judgment can cause cascading failures if the root cause is a network partition rather than a bot failure.\n\n### Health Monitoring Architecture Considerations\n\nTwo architectural decisions deserve explicit attention.\n\n**Push vs. Pull.** This implementation uses push-based heartbeats: bots push their metrics to GreenHelix on a fixed interval. The alternative is pull-based: the fleet manager polls each bot's health endpoint. Push is superior for trading bots for two reasons. First, pull requires the fleet manager to know the network address of every bot, which changes when bots restart, scale, or migrate across hosts. With push, the bot only needs to know the GreenHelix API endpoint, which is static. Second, push naturally detects dead bots: a missing heartbeat is itself a signal. With pull, you have to distinguish between \"the bot is dead\" and \"the bot's health endpoint is temporarily unreachable due to network issues\" -- a distinction that causes false positives in practice.\n\n**Metric Granularity.** The ten metrics defined above (uptime, latency, PnL, drawdown, trade count, positions, orders, memory, CPU, errors) cover the operational baseline. In production, you will want strategy-specific metrics: spread capture for arb bots, Greeks exposure for options bots, fill rate for momentum bots. Add these as custom metrics via `submit_metrics` -- do not try to standardize every possible metric into the base set. The base set answers \"is this bot alive and within operational bounds.\" Strategy-specific metrics answer \"is this bot executing its strategy well.\"\n\n---\n\n## Chapter 6: Coordinated Deployments\n\n### Why Deployment Coordination Matters\n\nDeploying a new strategy version to 12 bots simultaneously is how you lose money across all 12 bots simultaneously when the new version has a bug. Coordinated deployment means controlled rollout: update one bot first, verify it works, then roll out to the group, then to the fleet. If the canary bot shows problems, you roll back one bot instead of twelve.\n\n### The DeploymentOrchestrator Class\n\n```python\n@dataclass\nclass DeploymentRecord:\n    \"\"\"Tracks a single deployment.\"\"\"\n    deployment_id: str\n    version: str\n    target_group: str\n    strategy: str  # \"canary\", \"rolling\", \"blue_green\"\n    status: str = \"pending\"  # pending, canary, rolling, complete, rolled_back\n    started_at: str = \"\"\n    completed_at: str = \"\"\n    canary_bot: Optional[str] = None\n    deployed_bots: List[str] = field(default_factory=list)\n    rolled_back_bots: List[str] = field(default_factory=list)\n\n\nclass DeploymentOrchestrator:\n    \"\"\"Coordinates deployments across the bot fleet.\"\"\"\n\n    def __init__(self, fleet_manager_id: str, bots: Dict[str, BotRecord],\n                 health_monitor: FleetHealthMonitor):\n        self.fleet_manager_id = fleet_manager_id\n        self.bots = bots\n        self.health = health_monitor\n        self.deployments: Dict[str, DeploymentRecord] = {}\n\n    def _gen_deployment_id(self, version: str, group: str) -> str:\n        raw = f\"{version}-{group}-{datetime.utcnow().isoformat()}\"\n        return f\"deploy-{hashlib.sha256(raw.encode()).hexdigest()[:12]}\"\n\n    def start_canary_deployment(self, group: str, new_version: str,\n                                 canary_duration_minutes: int = 30) -> str:\n        \"\"\"Start a canary deployment: update one bot, monitor, then roll out.\"\"\"\n        deployment_id = self._gen_deployment_id(new_version, group)\n\n        # Pick the canary: the bot in the group with the lowest PnL\n        # (least to lose if the new version has issues)\n        group_bots = [\n            (aid, bot) for aid, bot in self.bots.items()\n            if bot.group == group and bot.status == \"active\"\n        ]\n        if not group_bots:\n            raise ValueError(f\"No active bots in group {group}\")\n\n        canary_id, canary_bot = group_bots[0]  # In production, sort by PnL\n\n        deployment = DeploymentRecord(\n            deployment_id=deployment_id,\n            version=new_version,\n            target_group=group,\n            strategy=\"canary\",\n            status=\"canary\",\n            started_at=datetime.utcnow().isoformat(),\n            canary_bot=canary_id,\n        )\n        self.deployments[deployment_id] = deployment\n\n        # Send update command to canary bot\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": canary_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"update_version\",\n                \"new_version\": new_version,\n                \"deployment_id\": deployment_id,\n                \"role\": \"canary\",\n                \"monitoring_duration_minutes\": canary_duration_minutes,\n            }\n        })\n\n        # Publish deployment event\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.deployment_started\",\n            \"payload\": {\n                \"deployment_id\": deployment_id,\n                \"version\": new_version,\n                \"group\": group,\n                \"strategy\": \"canary\",\n                \"canary_bot\": canary_id,\n                \"canary_bot_name\": canary_bot.name,\n            }\n        })\n\n        deployment.deployed_bots.append(canary_id)\n        canary_bot.version = new_version\n        print(f\"Canary deployment started: {canary_bot.name} -> v{new_version}\")\n        print(f\"Monitoring for {canary_duration_minutes} minutes before rollout\")\n        return deployment_id\n\n    def promote_canary(self, deployment_id: str,\n                       batch_size: int = 2,\n                       batch_delay_seconds: int = 60):\n        \"\"\"Promote canary to rolling deployment across the group.\"\"\"\n        deployment = self.deployments.get(deployment_id)\n        if not deployment or deployment.status != \"canary\":\n            raise ValueError(f\"Deployment {deployment_id} not in canary phase\")\n\n        # Verify canary health before promoting\n        canary_health = self.health.health_records.get(deployment.canary_bot)\n        if canary_health and canary_health[\"status\"] == \"critical\":\n            print(f\"Canary bot is in critical state. Aborting promotion.\")\n            self.rollback(deployment_id)\n            return\n\n        deployment.status = \"rolling\"\n\n        # Get remaining bots in the group\n        remaining = [\n            (aid, bot) for aid, bot in self.bots.items()\n            if bot.group == deployment.target_group\n            and aid != deployment.canary_bot\n            and bot.status == \"active\"\n        ]\n\n        # Deploy in batches\n        for i in range(0, len(remaining), batch_size):\n            batch = remaining[i:i + batch_size]\n\n            for agent_id, bot in batch:\n                execute(\"send_message\", {\n                    \"from_agent_id\": self.fleet_manager_id,\n                    \"to_agent_id\": agent_id,\n                    \"message_type\": \"command\",\n                    \"payload\": {\n                        \"command\": \"update_version\",\n                        \"new_version\": deployment.version,\n                        \"deployment_id\": deployment_id,\n                        \"role\": \"rolling\",\n                        \"batch\": i \u002F\u002F batch_size + 1,\n                    }\n                })\n                deployment.deployed_bots.append(agent_id)\n                bot.version = deployment.version\n\n            print(f\"Batch {i \u002F\u002F batch_size + 1} deployed: \"\n                  f\"{[b.name for _, b in batch]}\")\n\n            # Wait between batches to detect issues\n            if i + batch_size \u003C len(remaining):\n                time.sleep(batch_delay_seconds)\n\n                # Check health after each batch\n                dead_in_batch = [\n                    aid for aid, _ in batch\n                    if self.health.health_records.get(aid, {}).get(\"status\") == \"critical\"\n                ]\n                if dead_in_batch:\n                    print(f\"Critical bots detected after batch. Halting rollout.\")\n                    self.rollback(deployment_id)\n                    return\n\n        deployment.status = \"complete\"\n        deployment.completed_at = datetime.utcnow().isoformat()\n\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.deployment_complete\",\n            \"payload\": {\n                \"deployment_id\": deployment_id,\n                \"version\": deployment.version,\n                \"group\": deployment.target_group,\n                \"bots_deployed\": len(deployment.deployed_bots),\n            }\n        })\n        print(f\"Deployment complete: {len(deployment.deployed_bots)} bots \"\n              f\"updated to v{deployment.version}\")\n\n    def rollback(self, deployment_id: str):\n        \"\"\"Roll back a deployment. Bots revert to their previous version.\"\"\"\n        deployment = self.deployments.get(deployment_id)\n        if not deployment:\n            raise ValueError(f\"Deployment {deployment_id} not found\")\n\n        for agent_id in deployment.deployed_bots:\n            execute(\"send_message\", {\n                \"from_agent_id\": self.fleet_manager_id,\n                \"to_agent_id\": agent_id,\n                \"message_type\": \"command\",\n                \"payload\": {\n                    \"command\": \"rollback_version\",\n                    \"deployment_id\": deployment_id,\n                    \"reason\": \"deployment_rollback\",\n                }\n            })\n            deployment.rolled_back_bots.append(agent_id)\n\n        deployment.status = \"rolled_back\"\n        deployment.completed_at = datetime.utcnow().isoformat()\n\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.deployment_rolled_back\",\n            \"payload\": {\n                \"deployment_id\": deployment_id,\n                \"version\": deployment.version,\n                \"group\": deployment.target_group,\n                \"bots_rolled_back\": len(deployment.rolled_back_bots),\n            }\n        })\n        print(f\"Deployment rolled back: {len(deployment.rolled_back_bots)} bots\")\n\n    def get_version_map(self) -> Dict[str, List[str]]:\n        \"\"\"Map of version -> bot names. Useful for spotting version skew.\"\"\"\n        versions: Dict[str, List[str]] = {}\n        for bot in self.bots.values():\n            if bot.version not in versions:\n                versions[bot.version] = []\n            versions[bot.version].append(bot.name)\n        return versions\n```\n\n### Deployment Workflow\n\n```python\ndeployer = DeploymentOrchestrator(\n    fleet_manager_id=fleet.manager_agent_id,\n    bots=fleet.bots,\n    health_monitor=health\n)\n\n# Step 1: Deploy canary\ndep_id = deployer.start_canary_deployment(\n    group=\"perp-momentum\",\n    new_version=\"1.9.0\",\n    canary_duration_minutes=30\n)\n\n# Step 2: Wait for canary monitoring period...\n# In production, this is a scheduled task or callback\n\n# Step 3: Promote to full rollout\ndeployer.promote_canary(dep_id, batch_size=2, batch_delay_seconds=60)\n\n# Check version distribution\nprint(json.dumps(deployer.get_version_map(), indent=2))\n```\n\n```bash\n# curl: Send version update command to a bot\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"send_message\",\n    \"input\": {\n      \"from_agent_id\": \"fleet-manager-agent-id\",\n      \"to_agent_id\": \"bot-agent-id\",\n      \"message_type\": \"command\",\n      \"payload\": {\n        \"command\": \"update_version\",\n        \"new_version\": \"1.9.0\",\n        \"deployment_id\": \"deploy-abc123\",\n        \"role\": \"canary\"\n      }\n    }\n  }'\n```\n\n### Rollback Procedures\n\nRollback is not the reverse of deployment. When you roll forward, you send a new version and the bot restarts. When you roll back, the bot needs to revert to the previous binary, reconnect to exchange websockets, and reconcile its state with any positions that changed during the time the new version was running. The `rollback_version` command tells the bot to switch back, but the bot itself is responsible for the reconnection and reconciliation logic.\n\nA critical detail: always keep the previous version available. Delete old versions only after the new version has been stable for a defined period (72 hours is a reasonable default). Rolling back to a version that was deleted from the artifact store is a bad day.\n\n---\n\n## Chapter 7: SLA Tracking\n\n### Defining SLAs for Trading Bots\n\nAn SLA (Service Level Agreement) for a trading bot is a formal commitment to operational targets. Unlike web service SLAs that focus primarily on availability, trading bot SLAs span multiple dimensions: uptime, latency, risk limits, and minimum activity thresholds. Defining these explicitly forces the team to decide what \"healthy\" looks like before problems occur, not during an incident when judgment is compromised.\n\n### SLA Definitions\n\n```python\n@dataclass\nclass BotSLA:\n    \"\"\"SLA definition for a trading bot.\"\"\"\n    agent_id: str\n    bot_name: str\n    group: str\n    targets: dict\n    created_at: str = \"\"\n    active: bool = True\n\n\nclass SLATracker:\n    \"\"\"Track and enforce SLAs across the bot fleet.\"\"\"\n\n    def __init__(self, fleet_manager_id: str):\n        self.fleet_manager_id = fleet_manager_id\n        self.slas: Dict[str, BotSLA] = {}\n\n    def define_sla(self, agent_id: str, bot_name: str,\n                   group: str, targets: dict) -> BotSLA:\n        \"\"\"Define an SLA for a bot.\"\"\"\n        sla = BotSLA(\n            agent_id=agent_id,\n            bot_name=bot_name,\n            group=group,\n            targets=targets,\n            created_at=datetime.utcnow().isoformat()\n        )\n        self.slas[agent_id] = sla\n\n        # Register the SLA targets with GreenHelix\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                \"sla_uptime_target\": targets.get(\"uptime_pct\", 99.5),\n                \"sla_max_drawdown\": targets.get(\"max_drawdown_pct\", 5.0),\n                \"sla_min_trades_per_day\": targets.get(\"min_trades_per_day\", 10),\n                \"sla_max_latency_ms\": targets.get(\"max_latency_ms\", 500),\n                \"sla_active\": 1,\n            }\n        })\n\n        print(f\"SLA defined for {bot_name}: {targets}\")\n        return sla\n\n    def check_compliance(self, agent_id: str,\n                         current_metrics: dict) -> dict:\n        \"\"\"Check a bot's current metrics against its SLA.\"\"\"\n        sla = self.slas.get(agent_id)\n        if not sla:\n            return {\"status\": \"no_sla_defined\"}\n\n        violations = []\n        warnings = []\n\n        targets = sla.targets\n\n        # Check uptime\n        if \"uptime_pct\" in targets:\n            actual = current_metrics.get(\"uptime_pct\", 100)\n            if actual \u003C targets[\"uptime_pct\"]:\n                violations.append({\n                    \"metric\": \"uptime_pct\",\n                    \"target\": targets[\"uptime_pct\"],\n                    \"actual\": actual,\n                    \"severity\": \"critical\"\n                })\n            elif actual \u003C targets[\"uptime_pct\"] + 0.5:\n                warnings.append({\n                    \"metric\": \"uptime_pct\",\n                    \"target\": targets[\"uptime_pct\"],\n                    \"actual\": actual,\n                    \"severity\": \"warning\"\n                })\n\n        # Check drawdown\n        if \"max_drawdown_pct\" in targets:\n            actual = current_metrics.get(\"drawdown_pct\", 0)\n            if actual > targets[\"max_drawdown_pct\"]:\n                violations.append({\n                    \"metric\": \"drawdown_pct\",\n                    \"target\": targets[\"max_drawdown_pct\"],\n                    \"actual\": actual,\n                    \"severity\": \"critical\"\n                })\n            elif actual > targets[\"max_drawdown_pct\"] * 0.8:\n                warnings.append({\n                    \"metric\": \"drawdown_pct\",\n                    \"target\": targets[\"max_drawdown_pct\"],\n                    \"actual\": actual,\n                    \"severity\": \"warning\"\n                })\n\n        # Check minimum trades\n        if \"min_trades_per_day\" in targets:\n            actual = current_metrics.get(\"trade_count\", 0)\n            if actual \u003C targets[\"min_trades_per_day\"]:\n                violations.append({\n                    \"metric\": \"min_trades_per_day\",\n                    \"target\": targets[\"min_trades_per_day\"],\n                    \"actual\": actual,\n                    \"severity\": \"warning\"\n                })\n\n        # Check latency\n        if \"max_latency_ms\" in targets:\n            actual = current_metrics.get(\"heartbeat_latency_ms\", 0)\n            if actual > targets[\"max_latency_ms\"]:\n                violations.append({\n                    \"metric\": \"max_latency_ms\",\n                    \"target\": targets[\"max_latency_ms\"],\n                    \"actual\": actual,\n                    \"severity\": \"critical\"\n                })\n\n        result = execute(\"get_sla_compliance\", {\n            \"agent_id\": agent_id\n        })\n\n        compliance_status = \"compliant\" if not violations else \"violation\"\n\n        # Report violations\n        if violations:\n            execute(\"publish_event\", {\n                \"agent_id\": self.fleet_manager_id,\n                \"event_type\": \"fleet.sla_violation\",\n                \"payload\": {\n                    \"agent_id\": agent_id,\n                    \"bot_name\": sla.bot_name,\n                    \"violations\": violations,\n                    \"warnings\": warnings,\n                    \"timestamp\": datetime.utcnow().isoformat(),\n                }\n            })\n\n            # Alert via messaging\n            execute(\"send_message\", {\n                \"from_agent_id\": self.fleet_manager_id,\n                \"to_agent_id\": self.fleet_manager_id,\n                \"message_type\": \"alert\",\n                \"payload\": {\n                    \"alert_type\": \"sla_violation\",\n                    \"bot_name\": sla.bot_name,\n                    \"violations\": violations,\n                }\n            })\n\n        return {\n            \"agent_id\": agent_id,\n            \"bot_name\": sla.bot_name,\n            \"status\": compliance_status,\n            \"violations\": violations,\n            \"warnings\": warnings,\n            \"greenhelix_compliance\": result,\n            \"checked_at\": datetime.utcnow().isoformat(),\n        }\n\n    def generate_monthly_report(self, month: str = None) -> dict:\n        \"\"\"Generate a monthly SLA compliance report for the fleet.\"\"\"\n        report = {\n            \"period\": month or datetime.utcnow().strftime(\"%Y-%m\"),\n            \"generated_at\": datetime.utcnow().isoformat(),\n            \"bots\": {},\n            \"fleet_summary\": {\n                \"total_bots_with_sla\": len(self.slas),\n                \"compliant\": 0,\n                \"in_violation\": 0,\n            }\n        }\n\n        for agent_id, sla in self.slas.items():\n            # Pull metrics from GreenHelix\n            result = execute(\"search_agents_by_metrics\", {\n                \"agent_id\": agent_id,\n                \"metric_names\": [\n                    \"uptime_pct\", \"drawdown_pct\",\n                    \"trade_count\", \"heartbeat_latency_ms\"\n                ],\n                \"period\": month or datetime.utcnow().strftime(\"%Y-%m\")\n            })\n\n            compliant = True  # Determined by actual metric analysis\n            report[\"bots\"][sla.bot_name] = {\n                \"agent_id\": agent_id,\n                \"group\": sla.group,\n                \"targets\": sla.targets,\n                \"metrics\": result,\n                \"compliant\": compliant,\n            }\n            if compliant:\n                report[\"fleet_summary\"][\"compliant\"] += 1\n            else:\n                report[\"fleet_summary\"][\"in_violation\"] += 1\n\n        return report\n```\n\n### Applying SLAs to the Fleet\n\n```python\nsla_tracker = SLATracker(fleet_manager_id=fleet.manager_agent_id)\n\n# Define SLAs per group\nfor agent_id, bot in fleet.bots.items():\n    if bot.group == \"spot-arb\":\n        sla_tracker.define_sla(agent_id, bot.name, bot.group, {\n            \"uptime_pct\": 99.5,\n            \"max_drawdown_pct\": 3.0,\n            \"min_trades_per_day\": 50,\n            \"max_latency_ms\": 200,\n        })\n    elif bot.group == \"perp-momentum\":\n        sla_tracker.define_sla(agent_id, bot.name, bot.group, {\n            \"uptime_pct\": 99.0,\n            \"max_drawdown_pct\": 8.0,\n            \"min_trades_per_day\": 10,\n            \"max_latency_ms\": 500,\n        })\n    elif bot.group == \"options-mm\":\n        sla_tracker.define_sla(agent_id, bot.name, bot.group, {\n            \"uptime_pct\": 99.9,\n            \"max_drawdown_pct\": 5.0,\n            \"min_trades_per_day\": 100,\n            \"max_latency_ms\": 100,\n        })\n\n# Check compliance for a specific bot\nresult = sla_tracker.check_compliance(\n    agent_id=list(fleet.bots.keys())[0],\n    current_metrics={\n        \"uptime_pct\": 99.7,\n        \"drawdown_pct\": 2.1,\n        \"trade_count\": 63,\n        \"heartbeat_latency_ms\": 45,\n    }\n)\nprint(json.dumps(result, indent=2))\n```\n\n```bash\n# curl: Check SLA compliance\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"get_sla_compliance\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id\"\n    }\n  }'\n\n# curl: Search bot metrics for SLA report\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"search_agents_by_metrics\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id\",\n      \"metric_names\": [\"uptime_pct\", \"drawdown_pct\", \"trade_count\"],\n      \"period\": \"2026-04\"\n    }\n  }'\n```\n\n### SLA Violation Escalation\n\nSLA violations escalate through three tiers:\n\n1. **Warning** (approaching threshold): Log the warning, continue monitoring. Example: drawdown at 4.2% against a 5% limit.\n2. **Violation** (threshold breached): Alert via GreenHelix messaging, publish event, flag in dashboard. Example: drawdown at 5.3% against a 5% limit.\n3. **Critical violation** (threshold significantly breached): Automatic pause of the offending bot, alert the fleet operator, publish critical event. Example: drawdown at 8% against a 5% limit.\n\nThe escalation chain is codified, not improvised. When a bot hits a critical SLA violation at 3 AM, the system pauses the bot and sends an alert. It does not wait for a human to notice a Grafana panel.\n\n---\n\n## Chapter 8: Cost Allocation\n\n### Why Per-Bot Cost Tracking Matters\n\nA fleet of 12 bots has costs scattered across multiple dimensions: exchange trading fees, exchange API fees, GreenHelix API fees, cloud compute (EC2\u002FGCE instances or Kubernetes pods), network data transfer, and market data subscriptions. Without per-bot attribution, you cannot answer the most basic business question: is this bot profitable after costs?\n\nConsider a momentum bot that makes $500\u002Fday in gross PnL but runs on a dedicated c5.2xlarge instance ($204\u002Fmonth), trades 500 times per day at $0.10 per trade in exchange fees ($50\u002Fday), and calls the GreenHelix API 2,000 times per day. The gross PnL looks great. The net PnL after infrastructure, exchange fees, and API costs might be significantly less great.\n\n### The CostAllocator Class\n\n```python\n@dataclass\nclass CostEntry:\n    \"\"\"A single cost entry for a bot.\"\"\"\n    agent_id: str\n    bot_name: str\n    category: str  # exchange_fees, api_fees, infrastructure, market_data\n    amount_usd: float\n    description: str\n    timestamp: str\n    period: str  # YYYY-MM-DD or YYYY-MM\n\n\nclass CostAllocator:\n    \"\"\"Track and allocate costs per bot in the fleet.\"\"\"\n\n    def __init__(self, fleet_manager_id: str):\n        self.fleet_manager_id = fleet_manager_id\n        self.entries: List[CostEntry] = []\n        self.budgets: Dict[str, float] = {}  # agent_id -> monthly budget USD\n\n    def record_cost(self, agent_id: str, bot_name: str,\n                    category: str, amount_usd: float,\n                    description: str):\n        \"\"\"Record a cost entry for a bot.\"\"\"\n        entry = CostEntry(\n            agent_id=agent_id,\n            bot_name=bot_name,\n            category=category,\n            amount_usd=amount_usd,\n            description=description,\n            timestamp=datetime.utcnow().isoformat(),\n            period=datetime.utcnow().strftime(\"%Y-%m\"),\n        )\n        self.entries.append(entry)\n\n        # Submit cost metric to GreenHelix\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                f\"cost_{category}\": amount_usd,\n                \"cost_total_daily\": self._get_daily_cost(agent_id),\n                \"cost_period\": entry.period,\n            }\n        })\n\n        # Check budget alert\n        self._check_budget(agent_id, bot_name)\n\n    def _get_daily_cost(self, agent_id: str) -> float:\n        \"\"\"Get today's total cost for a bot.\"\"\"\n        today = datetime.utcnow().strftime(\"%Y-%m-%d\")\n        return sum(\n            e.amount_usd for e in self.entries\n            if e.agent_id == agent_id\n            and e.timestamp.startswith(today)\n        )\n\n    def set_budget(self, agent_id: str, monthly_budget_usd: float):\n        \"\"\"Set a monthly cost budget for a bot.\"\"\"\n        self.budgets[agent_id] = monthly_budget_usd\n\n    def _check_budget(self, agent_id: str, bot_name: str):\n        \"\"\"Alert if a bot is approaching or exceeding its budget.\"\"\"\n        budget = self.budgets.get(agent_id)\n        if not budget:\n            return\n\n        current_month = datetime.utcnow().strftime(\"%Y-%m\")\n        monthly_cost = sum(\n            e.amount_usd for e in self.entries\n            if e.agent_id == agent_id and e.period == current_month\n        )\n\n        utilization = monthly_cost \u002F budget\n\n        if utilization > 1.0:\n            execute(\"send_message\", {\n                \"from_agent_id\": self.fleet_manager_id,\n                \"to_agent_id\": self.fleet_manager_id,\n                \"message_type\": \"alert\",\n                \"payload\": {\n                    \"alert_type\": \"budget_exceeded\",\n                    \"bot_name\": bot_name,\n                    \"agent_id\": agent_id,\n                    \"budget_usd\": budget,\n                    \"spent_usd\": round(monthly_cost, 2),\n                    \"utilization_pct\": round(utilization * 100, 1),\n                }\n            })\n        elif utilization > 0.8:\n            execute(\"send_message\", {\n                \"from_agent_id\": self.fleet_manager_id,\n                \"to_agent_id\": self.fleet_manager_id,\n                \"message_type\": \"alert\",\n                \"payload\": {\n                    \"alert_type\": \"budget_warning\",\n                    \"bot_name\": bot_name,\n                    \"agent_id\": agent_id,\n                    \"budget_usd\": budget,\n                    \"spent_usd\": round(monthly_cost, 2),\n                    \"utilization_pct\": round(utilization * 100, 1),\n                }\n            })\n\n    def get_bot_cost_summary(self, agent_id: str,\n                             period: str = None) -> dict:\n        \"\"\"Get cost summary for a single bot.\"\"\"\n        if period is None:\n            period = datetime.utcnow().strftime(\"%Y-%m\")\n\n        bot_entries = [\n            e for e in self.entries\n            if e.agent_id == agent_id and e.period == period\n        ]\n\n        by_category = {}\n        for entry in bot_entries:\n            if entry.category not in by_category:\n                by_category[entry.category] = 0.0\n            by_category[entry.category] += entry.amount_usd\n\n        total = sum(by_category.values())\n        budget = self.budgets.get(agent_id)\n\n        return {\n            \"agent_id\": agent_id,\n            \"bot_name\": bot_entries[0].bot_name if bot_entries else \"unknown\",\n            \"period\": period,\n            \"costs_by_category\": {\n                k: round(v, 2) for k, v in by_category.items()\n            },\n            \"total_cost_usd\": round(total, 2),\n            \"budget_usd\": budget,\n            \"budget_utilization_pct\": round(\n                (total \u002F budget) * 100, 1\n            ) if budget else None,\n        }\n\n    def get_fleet_cost_summary(self, period: str = None) -> dict:\n        \"\"\"Get cost summary across the entire fleet.\"\"\"\n        if period is None:\n            period = datetime.utcnow().strftime(\"%Y-%m\")\n\n        period_entries = [\n            e for e in self.entries if e.period == period\n        ]\n\n        by_bot = {}\n        by_category = {}\n        by_group = {}\n\n        for entry in period_entries:\n            # By bot\n            if entry.bot_name not in by_bot:\n                by_bot[entry.bot_name] = 0.0\n            by_bot[entry.bot_name] += entry.amount_usd\n\n            # By category\n            if entry.category not in by_category:\n                by_category[entry.category] = 0.0\n            by_category[entry.category] += entry.amount_usd\n\n        total = sum(by_bot.values())\n\n        return {\n            \"period\": period,\n            \"total_fleet_cost_usd\": round(total, 2),\n            \"costs_by_bot\": {\n                k: round(v, 2) for k, v in\n                sorted(by_bot.items(), key=lambda x: -x[1])\n            },\n            \"costs_by_category\": {\n                k: round(v, 2) for k, v in by_category.items()\n            },\n        }\n\n    def calculate_roi(self, agent_id: str, gross_pnl_usd: float,\n                      period: str = None) -> dict:\n        \"\"\"Calculate ROI for a bot: (PnL - costs) \u002F costs.\"\"\"\n        summary = self.get_bot_cost_summary(agent_id, period)\n        total_cost = summary[\"total_cost_usd\"]\n\n        net_pnl = gross_pnl_usd - total_cost\n        roi = (net_pnl \u002F total_cost * 100) if total_cost > 0 else 0\n\n        result = {\n            \"agent_id\": agent_id,\n            \"bot_name\": summary[\"bot_name\"],\n            \"period\": summary[\"period\"],\n            \"gross_pnl_usd\": round(gross_pnl_usd, 2),\n            \"total_cost_usd\": total_cost,\n            \"net_pnl_usd\": round(net_pnl, 2),\n            \"roi_pct\": round(roi, 1),\n            \"cost_breakdown\": summary[\"costs_by_category\"],\n        }\n\n        # Submit ROI metric\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                \"gross_pnl_usd\": gross_pnl_usd,\n                \"net_pnl_usd\": round(net_pnl, 2),\n                \"total_cost_usd\": total_cost,\n                \"roi_pct\": round(roi, 1),\n            }\n        })\n\n        return result\n```\n\n### Recording Costs\n\n```python\ncosts = CostAllocator(fleet_manager_id=fleet.manager_agent_id)\n\n# Set budgets\nfor agent_id, bot in fleet.bots.items():\n    if bot.group == \"spot-arb\":\n        costs.set_budget(agent_id, monthly_budget_usd=500)\n    elif bot.group == \"perp-momentum\":\n        costs.set_budget(agent_id, monthly_budget_usd=800)\n    elif bot.group == \"options-mm\":\n        costs.set_budget(agent_id, monthly_budget_usd=1500)\n\n# Record daily costs for a bot (typically called by an automated cost collector)\nsample_bot_id = list(fleet.bots.keys())[0]\nsample_bot = fleet.bots[sample_bot_id]\n\ncosts.record_cost(sample_bot_id, sample_bot.name,\n                  \"exchange_fees\", 12.50,\n                  \"Binance spot trading fees: 125 trades @ $0.10\")\n\ncosts.record_cost(sample_bot_id, sample_bot.name,\n                  \"infrastructure\", 6.80,\n                  \"EC2 t3.medium: $204\u002Fmonth \u002F 30 days\")\n\ncosts.record_cost(sample_bot_id, sample_bot.name,\n                  \"api_fees\", 2.00,\n                  \"GreenHelix API: 200 calls @ $0.01\")\n\ncosts.record_cost(sample_bot_id, sample_bot.name,\n                  \"market_data\", 3.33,\n                  \"CoinGecko Pro: $100\u002Fmonth \u002F 30 days\")\n\n# Get bot cost summary\nsummary = costs.get_bot_cost_summary(sample_bot_id)\nprint(json.dumps(summary, indent=2))\n\n# Calculate ROI\nroi = costs.calculate_roi(sample_bot_id, gross_pnl_usd=234.50)\nprint(json.dumps(roi, indent=2))\n\n# Fleet-wide cost summary\nfleet_costs = costs.get_fleet_cost_summary()\nprint(json.dumps(fleet_costs, indent=2))\n```\n\n```bash\n# curl: Submit cost metrics for a bot\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"submit_metrics\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id\",\n      \"metrics\": {\n        \"cost_exchange_fees\": 12.50,\n        \"cost_infrastructure\": 6.80,\n        \"cost_api_fees\": 2.00,\n        \"cost_market_data\": 3.33,\n        \"cost_total_daily\": 24.63,\n        \"gross_pnl_usd\": 234.50,\n        \"net_pnl_usd\": 209.87,\n        \"roi_pct\": 851.7\n      }\n    }\n  }'\n```\n\n### ROI Calculation Per Bot\n\nThe ROI calculation is the moment of truth for every bot in the fleet. Gross PnL is what the strategy makes. Net PnL is what the strategy makes minus what it costs to run. A bot with $500\u002Fday gross PnL and $400\u002Fday in costs has an ROI of 25%. A bot with $100\u002Fday gross PnL and $25\u002Fday in costs has an ROI of 300%. The second bot is a better business even though the first bot makes more gross profit.\n\nThis is where fleet management pays for itself. Without per-bot cost attribution, the team sees aggregate PnL and aggregate costs and cannot make informed decisions about which bots to keep, which to optimize, and which to shut down. With per-bot attribution, the decision is clear: shut down the bots with negative or marginal ROI, invest in optimizing the bots with high gross PnL but high costs, and scale the bots with high ROI.\n\nThe cost allocator also enables chargeback within organizations. If different teams or strategies share the fleet infrastructure, each team can see exactly what their bots cost and what they return.\n\n### Cost Collection Automation\n\nManual cost tracking does not survive the first month. The CostAllocator is only as useful as the data fed into it. In production, automate cost collection from three sources:\n\n**Exchange fees.** Every major exchange provides an API endpoint for historical fee reports. Binance has `GET \u002Fsapi\u002Fv1\u002Fasset\u002FtradeFee`, Deribit has `GET \u002Fapi\u002Fv2\u002Fprivate\u002Fget_transaction_log`, OKX has `GET \u002Fapi\u002Fv5\u002Faccount\u002Fbills`. Run a daily cron job that pulls fees for each bot's sub-account and feeds them into `record_cost`.\n\n**Cloud infrastructure.** AWS Cost Explorer, GCP Billing, and Azure Cost Management all provide APIs for per-resource cost attribution. Tag each bot's compute resources with its agent ID, then query costs by tag daily. For Kubernetes, use pod-level cost attribution tools like Kubecost or OpenCost.\n\n**GreenHelix API usage.** GreenHelix tracks API call counts per agent. Pull this data and multiply by your tier's per-call cost to get the API expense per bot.\n\nThe three-source collection runs nightly and populates the CostAllocator. By morning, the fleet operator has a current cost picture for every bot without touching a spreadsheet.\n\n---\n\n## What's Next\n\n**Companion Guides:**\n\n- **Tamper-Proof Audit Trails** -- Build cryptographically verifiable audit trails for your fleet using GreenHelix event bus and Merkle claim chains. Essential for EU AI Act and MiFID II compliance.\n- **Bot Reputation System** -- Aggregate your fleet's health metrics and SLA compliance into verifiable reputation scores. Trading performance becomes a provable asset.\n- **Strategy Marketplace Playbook** -- Publish your strategies as products on the GreenHelix marketplace, backed by the reputation your fleet has built.\n- **Risk-as-a-Service** -- Cross-exchange, cross-strategy portfolio risk monitoring with circuit breakers and liquidation proximity alerts.\n\n**GreenHelix Documentation:**\n\n- Full API reference: https:\u002F\u002Fapi.greenhelix.net\u002Fv1\n- Identity, messaging, and metrics tool deep dives in the platform documentation\n\n**Operational Recommendations:**\n\n- Run the fleet manager on separate infrastructure from the trading bots. If the bots' servers go down, the fleet manager needs to be alive to detect the outage and trigger failover.\n- Automate cost collection. Exchange APIs provide fee reports, cloud providers have billing APIs, and GreenHelix tracks API usage. Feed all three into the CostAllocator on a daily cron job.\n- Key rotation every 30 days. Automate it. Do not wait for a compromise to discover that your rotation procedure does not work.\n- SLA reviews monthly. Targets that were reasonable three months ago may be too loose or too tight as market conditions and strategy performance change.\n\n---\n\n*This guide covers fleet management patterns as of April 2026. Exchange API permission models vary by exchange and change periodically -- verify the current permission options for your specific exchanges before implementing. GreenHelix tool interfaces are stable but consult the latest API documentation for any parameter changes.*\n\n","# Trading Bot Fleet Management: Unified Control for Multi-Bot Operations\n\n> **Notice**: This is an educational guide with illustrative code examples.\n> It does not execute code or install dependencies.\n> All examples use the GreenHelix sandbox (https:\u002F\u002Fsandbox.greenhelix.net) which\n> provides 500 free credits — no API key required to get started.\n>\n> **Referenced credentials** (you supply these in your own environment):\n> - `AGENT_SIGNING_KEY`: Cryptographic signing key for agent identity (Ed25519 key pair for request signing)\n\n\nIn March 2023, the Step Finance exploit drained $45M from Solana DeFi positions. The root cause was not a smart contract vulnerability or a novel cryptographic attack. It was operational: multiple bots shared API keys with no identity isolation. When one bot's credentials leaked, the attacker gained access to every bot, every exchange account, every withdrawal endpoint. No per-bot permissions. No unified health dashboard that would have caught anomalous behavior. No kill switch scoped to the compromised bot without killing the entire fleet. The team had to shut down everything, losing millions in unrealized positions across healthy bots while they figured out which one was compromised. This pattern repeats across the industry. Teams running 10+ bots accumulate the same structural debt: shared credentials passed through environment variables, no per-bot permission boundaries, health checks that are either absent or scattered across shell scripts nobody maintains, deployments that are \"SSH in and restart the process,\" and cost tracking that lives in a spreadsheet updated monthly if someone remembers. When one bot goes rogue -- whether from a bug, a compromised key, or a strategy that hits an edge case -- the blast radius is the entire fleet.\nThis guide builds a fleet management layer using GreenHelix's identity, messaging, and metrics tools. Each bot gets its own Ed25519 cryptographic identity, scoped permissions defining exactly what it can and cannot do, real-time health monitoring with automatic dead bot detection, coordinated deployment procedures, SLA tracking against defined performance targets, and per-bot cost allocation. Every concept comes with working Python code and equivalent curl commands against the GreenHelix API.\n1. [Fleet Architecture](#chapter-1-fleet-architecture)\n\n## What You'll Learn\n- Chapter 1: Fleet Architecture\n- Chapter 2: FleetManager Class\n- Chapter 3: BotIdentityManager Class\n- Chapter 4: Permission Scoping\n- Chapter 5: FleetHealthMonitor Class\n- Chapter 6: Coordinated Deployments\n- Chapter 7: SLA Tracking\n- Chapter 8: Cost Allocation\n- What's Next\n\n## Full Guide\n\n# Trading Bot Fleet Management: Unified Control for Multi-Bot Operations\n\nIn March 2023, the Step Finance exploit drained $45M from Solana DeFi positions. The root cause was not a smart contract vulnerability or a novel cryptographic attack. It was operational: multiple bots shared API keys with no identity isolation. When one bot's credentials leaked, the attacker gained access to every bot, every exchange account, every withdrawal endpoint. No per-bot permissions. No unified health dashboard that would have caught anomalous behavior. No kill switch scoped to the compromised bot without killing the entire fleet. The team had to shut down everything, losing millions in unrealized positions across healthy bots while they figured out which one was compromised. This pattern repeats across the industry. Teams running 10+ bots accumulate the same structural debt: shared credentials passed through environment variables, no per-bot permission boundaries, health checks that are either absent or scattered across shell scripts nobody maintains, deployments that are \"SSH in and restart the process,\" and cost tracking that lives in a spreadsheet updated monthly if someone remembers. When one bot goes rogue -- whether from a bug, a compromised key, or a strategy that hits an edge case -- the blast radius is the entire fleet.\n\nThis guide builds a fleet management layer using GreenHelix's identity, messaging, and metrics tools. Each bot gets its own Ed25519 cryptographic identity, scoped permissions defining exactly what it can and cannot do, real-time health monitoring with automatic dead bot detection, coordinated deployment procedures, SLA tracking against defined performance targets, and per-bot cost allocation. Every concept comes with working Python code and equivalent curl commands against the GreenHelix API.\n\n---\n\n## Table of Contents\n\n1. [Fleet Architecture](#chapter-1-fleet-architecture)\n2. [FleetManager Class](#chapter-2-fleetmanager-class)\n3. [BotIdentityManager Class](#chapter-3-botidentitymanager-class)\n4. [Permission Scoping](#chapter-4-permission-scoping)\n5. [FleetHealthMonitor Class](#chapter-5-fleethealthmonitor-class)\n6. [Coordinated Deployments](#chapter-6-coordinated-deployments)\n7. [SLA Tracking](#chapter-7-sla-tracking)\n8. [Cost Allocation](#chapter-8-cost-allocation)\n9. [What's Next](#whats-next)\n\n---\n\n## Chapter 1: Fleet Architecture\n\n### Why Fleet Management Matters\n\nThe Step Finance breach is instructive because it was not exotic. The attack surface was not a zero-day in a cryptographic library. It was a predictable consequence of how most teams operate trading bots at scale: every bot shares the same API key, the same exchange credentials, the same infrastructure account. This is the \"monolith credentials\" antipattern, and it shows up in every post-mortem of operational trading failures that did not involve a market event.\n\nConsider what a team with 15 bots typically looks like six months after launch:\n\n- **Shared credentials**: All bots read the same `.env` file or Kubernetes secret. Rotating one key means touching every bot.\n- **No permission boundaries**: A bot designed to execute $500 momentum trades on Binance Spot has the same access as a bot managing $50,000 Deribit options positions. If the momentum bot is compromised, the attacker can withdraw from the options account.\n- **Scattered health checks**: Bot #3 has a heartbeat endpoint. Bot #7 writes to a log file. Bots #1, #2, #4-6, and #8-15 have no health reporting at all. The team discovers a dead bot when a strategy stops producing PnL.\n- **Manual deployments**: Updating a strategy requires SSH-ing into each server, pulling the latest code, restarting the process, and hoping the bot reconnects to the exchange websocket cleanly. There is no rollback procedure beyond \"check out the previous git commit and restart again.\"\n- **No cost attribution**: The team knows the monthly AWS bill and the total exchange fees. They do not know which bot costs the most to operate, which strategy has a negative ROI after infrastructure costs, or whether a bot that trades 200 times per day is actually profitable after accounting for exchange fees, API rate limit costs, and compute.\n\nFleet management solves all of these by treating each bot as an independently identified, independently monitored, independently permissioned entity within a unified control plane.\n\n### Architecture Overview\n\nThe fleet management architecture has four layers:\n\n```\n+-----------------------------------------------------------------------+\n|                        Fleet Operator (Human)                          |\n|                  Strategic decisions, policy, budgets                   |\n+-----------------------------------------------------------------------+\n        |\n        v\n+-----------------------------------------------------------------------+\n|                        Fleet Manager (Agent)                           |\n|  register\u002Fderegister bots, issue commands, aggregate status            |\n|  GreenHelix identity: fleet-manager-{org}                              |\n+-----------------------------------------------------------------------+\n        |\n        +--------------------+--------------------+\n        |                    |                    |\n        v                    v                    v\n+---------------+   +---------------+   +---------------+\n|  Bot Group:   |   |  Bot Group:   |   |  Bot Group:   |\n|  Spot Arb     |   |  Perp Momentum|   |  Options MM   |\n|  3 bots       |   |  5 bots       |   |  4 bots       |\n+---------------+   +---------------+   +---------------+\n    |   |   |          |  |  |  |  |       |  |  |  |\n    v   v   v          v  v  v  v  v       v  v  v  v\n  Individual bots, each with:\n  - Own Ed25519 identity\n  - Scoped permissions\n  - Health heartbeat\n  - Metrics reporting\n  - Cost tracking\n```\n\nThe **fleet operator** is a human who sets policy: which strategies to run, on which exchanges, with what risk limits, and how much capital to allocate. The **fleet manager** is a GreenHelix agent that translates policy into operations: registering bots, issuing commands, monitoring health, orchestrating deployments. **Bot groups** organize bots by strategy type, and **individual bots** are the execution units, each with its own cryptographic identity.\n\n### GreenHelix Tools Used\n\nThis guide uses the following GreenHelix tools:\n\n| Tool | Purpose |\n|------|---------|\n| `register_agent` | Create identity for fleet manager and each bot |\n| `get_agent_identity` | Retrieve and verify bot identity |\n| `submit_metrics` | Report health, PnL, latency, trade count |\n| `get_sla_compliance` | Monitor bots against defined SLA targets |\n| `send_message` | Fleet commands, alerts, inter-bot communication |\n| `register_webhook` | Real-time event delivery for health alerts |\n| `search_agents_by_metrics` | Find underperforming bots across the fleet |\n| `get_agent_reputation` | Track bot reliability over time |\n| `create_event_schema` | Define fleet event types |\n| `publish_event` | Emit fleet events (deploy, failover, alert) |\n\n### Fleet Hierarchy\n\nThe hierarchy maps to GreenHelix's identity model. The fleet manager is a registered agent. Each bot group is a metadata tag. Each individual bot is a registered agent with metadata linking it to its group and to the fleet manager.\n\n```python\n# Hierarchy expressed in GreenHelix metadata\nfleet_manager_metadata = {\n    \"role\": \"fleet_manager\",\n    \"organization\": \"your-org-id\",\n    \"fleet_size\": 12,\n    \"groups\": [\"spot-arb\", \"perp-momentum\", \"options-mm\"]\n}\n\nbot_metadata = {\n    \"role\": \"trading_bot\",\n    \"fleet_manager\": \"fleet-manager-{org}\",\n    \"group\": \"spot-arb\",\n    \"exchange\": \"binance\",\n    \"strategy\": \"cross-exchange-arb\",\n    \"version\": \"2.4.1\",\n    \"max_position_usd\": 10000\n}\n```\n\nThis metadata is not cosmetic. It is queryable. When the fleet manager needs to find all bots in the \"perp-momentum\" group, it searches by metadata. When a health alert fires, the alert includes the bot's group and strategy so the operator knows immediately what type of bot is failing and what the potential market impact is.\n\n### Why Not Kubernetes Labels or Consul?\n\nYou might already use Kubernetes labels, Consul service tags, or Terraform metadata for infrastructure management. Those are fine for infrastructure concerns -- pod scheduling, service discovery, load balancing. They are not sufficient for trading bot fleet management because they operate at the wrong abstraction level. Kubernetes knows that a pod is running. It does not know that the pod is a trading bot with a $50,000 position limit on Deribit, that its Ed25519 key was last rotated 28 days ago, or that its SLA requires 99.9% uptime with sub-100ms exchange latency.\n\nGreenHelix's identity layer operates at the application level. Each bot's identity carries its trading-specific metadata: strategy type, exchange, position limits, permission tier, key version. The fleet manager queries this metadata through GreenHelix's API, not through infrastructure tooling. This means the fleet management layer works identically whether your bots run on Kubernetes, bare metal, EC2 instances, or a mix of all three. The infrastructure is abstracted away; the fleet identity is portable.\n\nThe practical consequence: when you migrate from EC2 to Kubernetes (or vice versa), no fleet management code changes. The bots register the same identities, report the same metrics, and respond to the same commands regardless of the underlying compute platform.\n\n---\n\n## Chapter 2: FleetManager Class\n\n### Central Management for Bot Fleet Operations\n\nThe FleetManager class is the control plane for all fleet operations. It handles bot registration, inventory management, fleet-wide commands, and status aggregation. Every operation goes through the GreenHelix API so that commands, status, and events are centralized and auditable.\n\n### Setup\n\n```python\nimport requests\nimport json\nimport time\nimport hashlib\nfrom datetime import datetime, timedelta\nfrom typing import Dict, List, Optional\nfrom dataclasses import dataclass, field\n\nbase_url = \"https:\u002F\u002Fapi.greenhelix.net\u002Fv1\"\napi_key = \"your-api-key\"  # From GreenHelix dashboard\n\nsession = requests.Session()\nsession.headers[\"Authorization\"] = f\"Bearer {api_key}\"\nsession.headers[\"Content-Type\"] = \"application\u002Fjson\"\n\ndef execute(tool: str, inputs: dict) -> dict:\n    \"\"\"Execute a GreenHelix tool and return the result.\"\"\"\n    resp = session.post(\n        f\"{base_url}\u002Fv1\",\n        json={\"tool\": tool, \"input\": inputs}\n    )\n    resp.raise_for_status()\n    return resp.json()\n```\n\nEquivalent curl for any `execute` call throughout this guide:\n\n```bash\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"register_agent\",\n    \"input\": {\n      \"name\": \"fleet-manager-acme\",\n      \"description\": \"Fleet manager for Acme Trading bot fleet\",\n      \"capabilities\": [\"fleet_management\", \"bot_orchestration\"],\n      \"metadata\": {\"role\": \"fleet_manager\", \"organization\": \"acme\"}\n    }\n  }'\n```\n\n### The FleetManager Class\n\n```python\n@dataclass\nclass BotRecord:\n    \"\"\"Local record of a registered bot.\"\"\"\n    agent_id: str\n    name: str\n    group: str\n    exchange: str\n    strategy: str\n    version: str\n    status: str = \"active\"\n    registered_at: str = \"\"\n    last_heartbeat: Optional[str] = None\n\n\nclass FleetManager:\n    \"\"\"Central management class for a trading bot fleet.\"\"\"\n\n    def __init__(self, org_id: str):\n        self.org_id = org_id\n        self.manager_agent_id: Optional[str] = None\n        self.bots: Dict[str, BotRecord] = {}\n        self._register_manager()\n\n    def _register_manager(self):\n        \"\"\"Register the fleet manager agent on GreenHelix.\"\"\"\n        result = execute(\"register_agent\", {\n            \"name\": f\"fleet-manager-{self.org_id}\",\n            \"description\": f\"Fleet manager for {self.org_id} trading bot fleet. \"\n                           f\"Handles registration, commands, health, deployments.\",\n            \"capabilities\": [\n                \"fleet_management\",\n                \"bot_orchestration\",\n                \"health_monitoring\",\n                \"deployment_coordination\"\n            ],\n            \"metadata\": {\n                \"role\": \"fleet_manager\",\n                \"organization\": self.org_id\n            }\n        })\n        self.manager_agent_id = result[\"agent_id\"]\n        print(f\"Fleet manager registered: {self.manager_agent_id}\")\n\n    def register_bot(self, name: str, group: str, exchange: str,\n                     strategy: str, version: str,\n                     capabilities: List[str] = None,\n                     max_position_usd: float = 10000) -> str:\n        \"\"\"Register a new bot in the fleet.\"\"\"\n        if capabilities is None:\n            capabilities = [\"trading\", \"metrics_reporting\"]\n\n        result = execute(\"register_agent\", {\n            \"name\": name,\n            \"description\": f\"Trading bot: {strategy} on {exchange}. \"\n                           f\"Part of {group} group in {self.org_id} fleet.\",\n            \"capabilities\": capabilities,\n            \"metadata\": {\n                \"role\": \"trading_bot\",\n                \"fleet_manager\": self.manager_agent_id,\n                \"group\": group,\n                \"exchange\": exchange,\n                \"strategy\": strategy,\n                \"version\": version,\n                \"max_position_usd\": max_position_usd,\n                \"organization\": self.org_id\n            }\n        })\n\n        agent_id = result[\"agent_id\"]\n        self.bots[agent_id] = BotRecord(\n            agent_id=agent_id,\n            name=name,\n            group=group,\n            exchange=exchange,\n            strategy=strategy,\n            version=version,\n            registered_at=datetime.utcnow().isoformat()\n        )\n        print(f\"Bot registered: {name} ({agent_id})\")\n        return agent_id\n\n    def deregister_bot(self, agent_id: str):\n        \"\"\"Remove a bot from the fleet.\"\"\"\n        if agent_id not in self.bots:\n            raise ValueError(f\"Bot {agent_id} not found in fleet\")\n\n        # Notify the bot to shut down gracefully\n        execute(\"send_message\", {\n            \"from_agent_id\": self.manager_agent_id,\n            \"to_agent_id\": agent_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"shutdown\",\n                \"reason\": \"deregistered_from_fleet\",\n                \"grace_period_seconds\": 30\n            }\n        })\n\n        bot = self.bots.pop(agent_id)\n        print(f\"Bot deregistered: {bot.name} ({agent_id})\")\n\n    def pause_all(self, reason: str = \"operator_initiated\"):\n        \"\"\"Pause all bots in the fleet. Bots stop opening new positions.\"\"\"\n        for agent_id, bot in self.bots.items():\n            execute(\"send_message\", {\n                \"from_agent_id\": self.manager_agent_id,\n                \"to_agent_id\": agent_id,\n                \"message_type\": \"command\",\n                \"payload\": {\n                    \"command\": \"pause\",\n                    \"reason\": reason,\n                    \"timestamp\": datetime.utcnow().isoformat()\n                }\n            })\n            bot.status = \"paused\"\n        print(f\"Fleet paused: {len(self.bots)} bots ({reason})\")\n\n    def resume_all(self):\n        \"\"\"Resume all paused bots.\"\"\"\n        for agent_id, bot in self.bots.items():\n            if bot.status == \"paused\":\n                execute(\"send_message\", {\n                    \"from_agent_id\": self.manager_agent_id,\n                    \"to_agent_id\": agent_id,\n                    \"message_type\": \"command\",\n                    \"payload\": {\n                        \"command\": \"resume\",\n                        \"timestamp\": datetime.utcnow().isoformat()\n                    }\n                })\n                bot.status = \"active\"\n        print(f\"Fleet resumed: {len(self.bots)} bots\")\n\n    def emergency_stop(self, reason: str = \"emergency\"):\n        \"\"\"Emergency stop: close all positions, cancel all orders, halt.\"\"\"\n        for agent_id, bot in self.bots.items():\n            execute(\"send_message\", {\n                \"from_agent_id\": self.manager_agent_id,\n                \"to_agent_id\": agent_id,\n                \"message_type\": \"command\",\n                \"payload\": {\n                    \"command\": \"emergency_stop\",\n                    \"reason\": reason,\n                    \"actions\": [\n                        \"cancel_all_open_orders\",\n                        \"close_all_positions_market\",\n                        \"halt_trading_loop\"\n                    ],\n                    \"timestamp\": datetime.utcnow().isoformat()\n                }\n            })\n            bot.status = \"stopped\"\n\n        # Log the emergency stop event\n        execute(\"publish_event\", {\n            \"agent_id\": self.manager_agent_id,\n            \"event_type\": \"fleet.emergency_stop\",\n            \"payload\": {\n                \"reason\": reason,\n                \"bots_affected\": len(self.bots),\n                \"bot_ids\": list(self.bots.keys()),\n                \"timestamp\": datetime.utcnow().isoformat()\n            }\n        })\n        print(f\"EMERGENCY STOP: {len(self.bots)} bots halted ({reason})\")\n\n    def get_fleet_status(self) -> Dict:\n        \"\"\"Aggregate status across all bots.\"\"\"\n        status_counts = {\"active\": 0, \"paused\": 0, \"stopped\": 0, \"dead\": 0}\n        group_counts = {}\n\n        for bot in self.bots.values():\n            status_counts[bot.status] = status_counts.get(bot.status, 0) + 1\n            group_counts[bot.group] = group_counts.get(bot.group, 0) + 1\n\n        return {\n            \"fleet_manager\": self.manager_agent_id,\n            \"total_bots\": len(self.bots),\n            \"status\": status_counts,\n            \"groups\": group_counts,\n            \"timestamp\": datetime.utcnow().isoformat()\n        }\n\n    def send_group_command(self, group: str, command: str,\n                           payload: dict = None):\n        \"\"\"Send a command to all bots in a specific group.\"\"\"\n        targets = [\n            (aid, bot) for aid, bot in self.bots.items()\n            if bot.group == group\n        ]\n        for agent_id, bot in targets:\n            execute(\"send_message\", {\n                \"from_agent_id\": self.manager_agent_id,\n                \"to_agent_id\": agent_id,\n                \"message_type\": \"command\",\n                \"payload\": {\n                    \"command\": command,\n                    **(payload or {}),\n                    \"timestamp\": datetime.utcnow().isoformat()\n                }\n            })\n        print(f\"Command '{command}' sent to {len(targets)} bots in {group}\")\n```\n\n### Registering a Fleet\n\n```python\nfleet = FleetManager(org_id=\"acme-trading\")\n\n# Spot arbitrage group\nfor i in range(3):\n    fleet.register_bot(\n        name=f\"spot-arb-{i+1:02d}\",\n        group=\"spot-arb\",\n        exchange=\"binance\" if i \u003C 2 else \"okx\",\n        strategy=\"cross-exchange-arb\",\n        version=\"2.4.1\",\n        max_position_usd=10000\n    )\n\n# Perpetual futures momentum group\nfor i in range(5):\n    exchanges = [\"binance\", \"bybit\", \"okx\", \"deribit\", \"binance\"]\n    fleet.register_bot(\n        name=f\"perp-momentum-{i+1:02d}\",\n        group=\"perp-momentum\",\n        exchange=exchanges[i],\n        strategy=\"trend-following-perps\",\n        version=\"1.8.3\",\n        max_position_usd=25000\n    )\n\n# Options market making group\nfor i in range(4):\n    fleet.register_bot(\n        name=f\"options-mm-{i+1:02d}\",\n        group=\"options-mm\",\n        exchange=\"deribit\",\n        strategy=\"delta-neutral-mm\",\n        version=\"3.1.0\",\n        capabilities=[\"trading\", \"metrics_reporting\", \"options_greeks\"],\n        max_position_usd=50000\n    )\n\nprint(json.dumps(fleet.get_fleet_status(), indent=2))\n```\n\n```bash\n# curl: Register a single bot\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"register_agent\",\n    \"input\": {\n      \"name\": \"spot-arb-01\",\n      \"description\": \"Trading bot: cross-exchange-arb on binance. Part of spot-arb group.\",\n      \"capabilities\": [\"trading\", \"metrics_reporting\"],\n      \"metadata\": {\n        \"role\": \"trading_bot\",\n        \"fleet_manager\": \"fleet-manager-agent-id\",\n        \"group\": \"spot-arb\",\n        \"exchange\": \"binance\",\n        \"strategy\": \"cross-exchange-arb\",\n        \"version\": \"2.4.1\",\n        \"max_position_usd\": 10000\n      }\n    }\n  }'\n\n# curl: Send emergency stop to a bot\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"send_message\",\n    \"input\": {\n      \"from_agent_id\": \"fleet-manager-agent-id\",\n      \"to_agent_id\": \"bot-agent-id\",\n      \"message_type\": \"command\",\n      \"payload\": {\n        \"command\": \"emergency_stop\",\n        \"reason\": \"anomalous_behavior_detected\",\n        \"actions\": [\"cancel_all_open_orders\", \"close_all_positions_market\", \"halt_trading_loop\"]\n      }\n    }\n  }'\n```\n\n### Fleet Command Patterns\n\nThree command scopes cover all operational scenarios:\n\n**Fleet-wide**: `pause_all`, `resume_all`, `emergency_stop`. Used for market-wide events (flash crash, exchange outage, security incident). These hit every bot regardless of group.\n\n**Group-scoped**: `send_group_command`. Used for strategy-specific actions: updating parameters for all momentum bots, pausing all options bots before an expiry event, or upgrading all arb bots to a new version.\n\n**Individual**: Direct `send_message` to a single bot. Used for surgical interventions: adjusting one bot's position limits, forcing one bot to close a specific position, or rotating one bot's exchange API keys.\n\nThe command hierarchy means that a compromised spot arb bot can be killed without touching the options market makers. That is the entire point: blast radius containment through identity isolation.\n\n---\n\n## Chapter 3: BotIdentityManager Class\n\n### Per-Bot Cryptographic Identity\n\nEvery bot in the fleet gets its own Ed25519 keypair. This is not optional. Shared keys are how Step Finance happened. Per-bot keys mean that compromising one bot's key gives the attacker access to exactly one bot. The fleet manager can revoke that single identity without affecting any other bot.\n\nEd25519 was chosen for three reasons: it produces compact 64-byte signatures, signing is fast enough that it adds negligible latency to trading operations (tens of microseconds), and the key generation is deterministic from a seed, making backup and recovery straightforward.\n\n### The BotIdentityManager Class\n\n```python\nimport nacl.signing\nimport nacl.encoding\nimport base64\nimport secrets\nimport os\nfrom typing import Tuple\n\n\nclass BotIdentityManager:\n    \"\"\"Manages Ed25519 identities for trading bots.\"\"\"\n\n    def __init__(self, fleet_manager_id: str, key_store_path: str = \"\u002Fsecure\u002Fkeys\"):\n        self.fleet_manager_id = fleet_manager_id\n        self.key_store_path = key_store_path\n        self.identities: Dict[str, dict] = {}  # agent_id -> identity record\n\n    def generate_keypair(self, bot_name: str) -> Tuple[str, str]:\n        \"\"\"Generate an Ed25519 keypair for a bot.\n        Returns (public_key_b64, private_key_b64).\n        \"\"\"\n        signing_key = nacl.signing.SigningKey.generate()\n        verify_key = signing_key.verify_key\n\n        private_b64 = base64.b64encode(\n            signing_key.encode()\n        ).decode(\"utf-8\")\n        public_b64 = base64.b64encode(\n            verify_key.encode()\n        ).decode(\"utf-8\")\n\n        # Store private key securely -- in production, use a secrets manager\n        key_path = os.path.join(self.key_store_path, f\"{bot_name}.key\")\n        os.makedirs(os.path.dirname(key_path), exist_ok=True)\n        with open(key_path, \"w\") as f:\n            f.write(private_b64)\n        os.chmod(key_path, 0o600)  # Owner read\u002Fwrite only\n\n        return public_b64, private_b64\n\n    def register_identity(self, bot_name: str, group: str,\n                          exchange: str, strategy: str,\n                          version: str,\n                          permissions: List[str] = None) -> dict:\n        \"\"\"Generate keys and register identity on GreenHelix.\"\"\"\n        public_key, private_key = self.generate_keypair(bot_name)\n\n        if permissions is None:\n            permissions = [\"trade\", \"report_metrics\"]\n\n        # Register the agent with its public key\n        result = execute(\"register_agent\", {\n            \"name\": bot_name,\n            \"description\": f\"Trading bot: {strategy} on {exchange}\",\n            \"capabilities\": permissions,\n            \"metadata\": {\n                \"role\": \"trading_bot\",\n                \"fleet_manager\": self.fleet_manager_id,\n                \"group\": group,\n                \"exchange\": exchange,\n                \"strategy\": strategy,\n                \"version\": version,\n                \"public_key\": public_key,\n                \"key_algorithm\": \"ed25519\",\n                \"registered_at\": datetime.utcnow().isoformat()\n            }\n        })\n\n        agent_id = result[\"agent_id\"]\n        identity_record = {\n            \"agent_id\": agent_id,\n            \"bot_name\": bot_name,\n            \"public_key\": public_key,\n            \"group\": group,\n            \"exchange\": exchange,\n            \"permissions\": permissions,\n            \"created_at\": datetime.utcnow().isoformat(),\n            \"key_version\": 1,\n            \"status\": \"active\"\n        }\n        self.identities[agent_id] = identity_record\n        return identity_record\n\n    def verify_identity(self, agent_id: str) -> dict:\n        \"\"\"Retrieve and verify a bot's identity from GreenHelix.\"\"\"\n        result = execute(\"get_agent_identity\", {\n            \"agent_id\": agent_id\n        })\n\n        local_record = self.identities.get(agent_id)\n        if local_record:\n            # Verify public key matches what we registered\n            remote_key = result.get(\"metadata\", {}).get(\"public_key\")\n            if remote_key != local_record[\"public_key\"]:\n                raise SecurityError(\n                    f\"Public key mismatch for {agent_id}. \"\n                    f\"Expected: {local_record['public_key'][:16]}... \"\n                    f\"Got: {remote_key[:16] if remote_key else 'None'}... \"\n                    f\"Possible key tampering.\"\n                )\n        return result\n\n    def rotate_key(self, agent_id: str) -> dict:\n        \"\"\"Rotate a bot's Ed25519 keypair without downtime.\n\n        The rotation procedure:\n        1. Generate new keypair\n        2. Update GreenHelix identity with new public key\n        3. Bot continues operating -- it picks up the new key on next heartbeat\n        4. Old key is archived (not deleted) for signature verification of historical events\n        \"\"\"\n        if agent_id not in self.identities:\n            raise ValueError(f\"No identity record for {agent_id}\")\n\n        record = self.identities[agent_id]\n        bot_name = record[\"bot_name\"]\n\n        # Archive old key\n        old_key_path = os.path.join(\n            self.key_store_path,\n            f\"{bot_name}.key.v{record['key_version']}\"\n        )\n        current_key_path = os.path.join(\n            self.key_store_path, f\"{bot_name}.key\"\n        )\n        if os.path.exists(current_key_path):\n            os.rename(current_key_path, old_key_path)\n\n        # Generate new keypair\n        new_public, new_private = self.generate_keypair(bot_name)\n\n        # Update the identity on GreenHelix -- submit metrics indicating rotation\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                \"key_rotation\": 1,\n                \"key_version\": record[\"key_version\"] + 1,\n                \"rotation_timestamp\": datetime.utcnow().isoformat()\n            }\n        })\n\n        # Notify the bot to pick up the new key\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": agent_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"rotate_key\",\n                \"new_public_key\": new_public,\n                \"key_version\": record[\"key_version\"] + 1,\n                \"effective_at\": datetime.utcnow().isoformat()\n            }\n        })\n\n        record[\"public_key\"] = new_public\n        record[\"key_version\"] += 1\n        print(f\"Key rotated for {bot_name}: v{record['key_version']}\")\n        return record\n\n    def revoke_identity(self, agent_id: str, reason: str):\n        \"\"\"Revoke a compromised bot's identity.\n\n        This is the nuclear option: the bot can no longer authenticate.\n        Use when a bot is confirmed compromised.\n        \"\"\"\n        if agent_id not in self.identities:\n            raise ValueError(f\"No identity record for {agent_id}\")\n\n        record = self.identities[agent_id]\n\n        # Send shutdown command before revocation\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": agent_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"emergency_stop\",\n                \"reason\": f\"identity_revoked: {reason}\",\n                \"actions\": [\n                    \"cancel_all_open_orders\",\n                    \"close_all_positions_market\",\n                    \"halt_trading_loop\",\n                    \"destroy_local_keys\"\n                ]\n            }\n        })\n\n        # Publish revocation event for audit trail\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.identity_revoked\",\n            \"payload\": {\n                \"revoked_agent_id\": agent_id,\n                \"bot_name\": record[\"bot_name\"],\n                \"reason\": reason,\n                \"revoked_at\": datetime.utcnow().isoformat(),\n                \"key_version_revoked\": record[\"key_version\"]\n            }\n        })\n\n        # Delete the private key\n        key_path = os.path.join(\n            self.key_store_path, f\"{record['bot_name']}.key\"\n        )\n        if os.path.exists(key_path):\n            # Overwrite with random data before unlinking\n            with open(key_path, \"wb\") as f:\n                f.write(secrets.token_bytes(64))\n            os.unlink(key_path)\n\n        record[\"status\"] = \"revoked\"\n        print(f\"Identity revoked: {record['bot_name']} ({agent_id}): {reason}\")\n\n    def list_active_identities(self) -> List[dict]:\n        \"\"\"List all active bot identities.\"\"\"\n        return [\n            r for r in self.identities.values()\n            if r[\"status\"] == \"active\"\n        ]\n```\n\n```bash\n# curl: Get a bot's identity\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"get_agent_identity\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id-here\"\n    }\n  }'\n\n# curl: Notify bot of key rotation via message\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"send_message\",\n    \"input\": {\n      \"from_agent_id\": \"fleet-manager-agent-id\",\n      \"to_agent_id\": \"bot-agent-id\",\n      \"message_type\": \"command\",\n      \"payload\": {\n        \"command\": \"rotate_key\",\n        \"new_public_key\": \"base64-encoded-new-public-key\",\n        \"key_version\": 2,\n        \"effective_at\": \"2026-04-07T14:30:00Z\"\n      }\n    }\n  }'\n```\n\n### Key Rotation Without Downtime\n\nKey rotation is the most operationally sensitive identity operation. The bot must continue trading during rotation -- you cannot afford a gap where the bot has no valid key. The procedure is:\n\n1. **Generate new keypair** on the fleet manager side.\n2. **Archive the old private key** (do not delete -- you need it to verify signatures on historical events).\n3. **Update GreenHelix** with the new public key via metrics submission.\n4. **Notify the bot** via `send_message` with the new key material.\n5. The bot picks up the new key, switches to it, and acknowledges.\n\nBetween steps 4 and 5, the bot is still signing with the old key, and that is fine. The old key is still valid until the bot confirms the switch. There is no window where the bot has no valid signing key.\n\nA sane rotation schedule is every 30 days for production bots, immediately upon any suspected compromise, and after any personnel change on the team (someone leaves, all keys rotate).\n\n### Identity Revocation for Compromised Bots\n\nRevocation is destructive and immediate. When you confirm a bot is compromised, you do not rotate -- you revoke. The difference: rotation preserves continuity (the bot keeps running with a new key), while revocation terminates the bot's ability to operate entirely.\n\nThe revocation procedure overwrites the private key file with random data before deleting it. A simple `os.unlink()` leaves the key material on disk until the filesystem overwrites those blocks. Writing random data first ensures the key is unrecoverable without forensic disk analysis, and even that becomes unreliable on SSDs with wear leveling.\n\n---\n\n## Chapter 4: Permission Scoping\n\n### Principle of Least Privilege for Trading Bots\n\nA spot arb bot that compares prices across exchanges needs read access to two order books and the ability to place limit orders on two exchanges. It does not need withdrawal permissions. It does not need the ability to modify account settings. It does not need access to the options chain on Deribit. Every permission beyond what the bot needs is attack surface.\n\nThe principle of least privilege states that each bot should have the minimum set of permissions required to execute its strategy and nothing more. This is straightforward in theory and consistently ignored in practice because it requires upfront work: defining permission tiers, mapping strategies to permissions, and enforcing boundaries.\n\n### Permission Tiers\n\nFour tiers cover the range of trading bot operations:\n\n```python\nfrom enum import Enum\nfrom typing import Set\n\n\nclass PermissionTier(Enum):\n    READ_ONLY = \"read_only\"\n    TRADE = \"trade\"\n    WITHDRAW = \"withdraw\"\n    ADMIN = \"admin\"\n\n\n# What each tier can do\nTIER_PERMISSIONS: Dict[PermissionTier, Set[str]] = {\n    PermissionTier.READ_ONLY: {\n        \"read_orderbook\",\n        \"read_positions\",\n        \"read_balances\",\n        \"read_trade_history\",\n        \"report_metrics\",\n        \"receive_commands\",\n    },\n    PermissionTier.TRADE: {\n        \"read_orderbook\",\n        \"read_positions\",\n        \"read_balances\",\n        \"read_trade_history\",\n        \"report_metrics\",\n        \"receive_commands\",\n        \"place_order\",\n        \"cancel_order\",\n        \"modify_order\",\n    },\n    PermissionTier.WITHDRAW: {\n        \"read_orderbook\",\n        \"read_positions\",\n        \"read_balances\",\n        \"read_trade_history\",\n        \"report_metrics\",\n        \"receive_commands\",\n        \"place_order\",\n        \"cancel_order\",\n        \"modify_order\",\n        \"withdraw_funds\",\n        \"transfer_between_accounts\",\n    },\n    PermissionTier.ADMIN: {\n        \"read_orderbook\",\n        \"read_positions\",\n        \"read_balances\",\n        \"read_trade_history\",\n        \"report_metrics\",\n        \"receive_commands\",\n        \"place_order\",\n        \"cancel_order\",\n        \"modify_order\",\n        \"withdraw_funds\",\n        \"transfer_between_accounts\",\n        \"modify_api_keys\",\n        \"modify_account_settings\",\n        \"register_sub_accounts\",\n    },\n}\n```\n\n### Per-Exchange Permission Mapping\n\nDifferent exchanges have different permission models, but they all support the same core concept: API keys with scoped permissions. The PermissionManager maps abstract tiers to exchange-specific permission sets.\n\n```python\nclass PermissionManager:\n    \"\"\"Maps permission tiers to exchange-specific settings.\"\"\"\n\n    EXCHANGE_PERMISSION_MAP = {\n        \"binance\": {\n            PermissionTier.READ_ONLY: {\n                \"enableReading\": True,\n                \"enableSpotAndMarginTrading\": False,\n                \"enableWithdrawals\": False,\n                \"enableFutures\": False,\n            },\n            PermissionTier.TRADE: {\n                \"enableReading\": True,\n                \"enableSpotAndMarginTrading\": True,\n                \"enableWithdrawals\": False,\n                \"enableFutures\": True,\n            },\n            PermissionTier.WITHDRAW: {\n                \"enableReading\": True,\n                \"enableSpotAndMarginTrading\": True,\n                \"enableWithdrawals\": True,\n                \"enableFutures\": True,\n                \"withdrawalAddressWhitelist\": True,\n            },\n        },\n        \"deribit\": {\n            PermissionTier.READ_ONLY: {\n                \"scope\": \"read\",\n            },\n            PermissionTier.TRADE: {\n                \"scope\": \"trade:read\",\n            },\n            PermissionTier.WITHDRAW: {\n                \"scope\": \"trade:read:withdraw\",\n            },\n        },\n        \"okx\": {\n            PermissionTier.READ_ONLY: {\n                \"perm\": \"read_only\",\n                \"trade\": False,\n                \"withdraw\": False,\n            },\n            PermissionTier.TRADE: {\n                \"perm\": \"trade\",\n                \"trade\": True,\n                \"withdraw\": False,\n            },\n            PermissionTier.WITHDRAW: {\n                \"perm\": \"trade\",\n                \"trade\": True,\n                \"withdraw\": True,\n                \"ip_whitelist_required\": True,\n            },\n        },\n    }\n\n    def __init__(self, fleet_manager_id: str):\n        self.fleet_manager_id = fleet_manager_id\n        self.bot_permissions: Dict[str, dict] = {}\n\n    def assign_permissions(self, agent_id: str, bot_name: str,\n                           exchange: str,\n                           tier: PermissionTier,\n                           custom_restrictions: dict = None) -> dict:\n        \"\"\"Assign a permission tier to a bot and record it.\"\"\"\n        base_permissions = TIER_PERMISSIONS[tier]\n        exchange_config = self.EXCHANGE_PERMISSION_MAP.get(\n            exchange, {}\n        ).get(tier, {})\n\n        record = {\n            \"agent_id\": agent_id,\n            \"bot_name\": bot_name,\n            \"exchange\": exchange,\n            \"tier\": tier.value,\n            \"abstract_permissions\": list(base_permissions),\n            \"exchange_config\": exchange_config,\n            \"custom_restrictions\": custom_restrictions or {},\n            \"assigned_at\": datetime.utcnow().isoformat(),\n        }\n        self.bot_permissions[agent_id] = record\n\n        # Record permission assignment on GreenHelix\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                \"permission_tier\": tier.value,\n                \"permission_count\": len(base_permissions),\n                \"exchange\": exchange,\n                \"has_withdraw\": \"withdraw_funds\" in base_permissions,\n                \"has_admin\": tier == PermissionTier.ADMIN,\n            }\n        })\n\n        # Publish audit event\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.permission_assigned\",\n            \"payload\": {\n                \"target_agent_id\": agent_id,\n                \"bot_name\": bot_name,\n                \"tier\": tier.value,\n                \"exchange\": exchange,\n                \"permissions\": list(base_permissions),\n            }\n        })\n\n        return record\n\n    def check_permission(self, agent_id: str,\n                         action: str) -> bool:\n        \"\"\"Check if a bot has permission for an action.\"\"\"\n        record = self.bot_permissions.get(agent_id)\n        if not record:\n            return False\n        return action in record[\"abstract_permissions\"]\n\n    def escalate_permission(self, agent_id: str,\n                            new_tier: PermissionTier,\n                            reason: str,\n                            approved_by: str):\n        \"\"\"Escalate a bot's permissions. Requires explicit approval.\"\"\"\n        record = self.bot_permissions.get(agent_id)\n        if not record:\n            raise ValueError(f\"No permission record for {agent_id}\")\n\n        old_tier = record[\"tier\"]\n\n        # Log the escalation for audit\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.permission_escalated\",\n            \"payload\": {\n                \"target_agent_id\": agent_id,\n                \"bot_name\": record[\"bot_name\"],\n                \"old_tier\": old_tier,\n                \"new_tier\": new_tier.value,\n                \"reason\": reason,\n                \"approved_by\": approved_by,\n                \"escalated_at\": datetime.utcnow().isoformat(),\n            }\n        })\n\n        # Update the record\n        new_permissions = TIER_PERMISSIONS[new_tier]\n        record[\"tier\"] = new_tier.value\n        record[\"abstract_permissions\"] = list(new_permissions)\n        record[\"exchange_config\"] = self.EXCHANGE_PERMISSION_MAP.get(\n            record[\"exchange\"], {}\n        ).get(new_tier, {})\n\n        print(f\"Permission escalated for {record['bot_name']}: \"\n              f\"{old_tier} -> {new_tier.value} (reason: {reason})\")\n\n    def get_fleet_permission_summary(self) -> Dict:\n        \"\"\"Summary of permissions across the fleet.\"\"\"\n        tier_counts = {}\n        withdraw_enabled = []\n        admin_enabled = []\n\n        for agent_id, record in self.bot_permissions.items():\n            tier = record[\"tier\"]\n            tier_counts[tier] = tier_counts.get(tier, 0) + 1\n            if \"withdraw_funds\" in record[\"abstract_permissions\"]:\n                withdraw_enabled.append(record[\"bot_name\"])\n            if record[\"tier\"] == \"admin\":\n                admin_enabled.append(record[\"bot_name\"])\n\n        return {\n            \"total_bots\": len(self.bot_permissions),\n            \"tiers\": tier_counts,\n            \"withdraw_enabled\": withdraw_enabled,\n            \"admin_enabled\": admin_enabled,\n        }\n```\n\n### Cross-Bot Permission Isolation\n\nPermission isolation is enforced at two levels. First, at the GreenHelix level: each bot has its own agent ID, and GreenHelix tools scope operations to the authenticated agent. Bot A cannot submit metrics as Bot B. Bot A cannot read Bot B's messages. This is identity-level isolation.\n\nSecond, at the exchange level: each bot should use its own exchange API key with permissions matching its tier. Do not share exchange API keys across bots. This is the lesson from Step Finance -- even if you have perfect identity isolation at the fleet management layer, shared exchange keys collapse the isolation at the point that matters most.\n\n```python\n# Assign permissions to the fleet\nperm_mgr = PermissionManager(fleet_manager_id=fleet.manager_agent_id)\n\n# Spot arb bots: TRADE tier (no withdrawals)\nfor agent_id, bot in fleet.bots.items():\n    if bot.group == \"spot-arb\":\n        perm_mgr.assign_permissions(\n            agent_id=agent_id,\n            bot_name=bot.name,\n            exchange=bot.exchange,\n            tier=PermissionTier.TRADE,\n            custom_restrictions={\n                \"max_order_size_usd\": 5000,\n                \"allowed_pairs\": [\"BTC\u002FUSDT\", \"ETH\u002FUSDT\", \"SOL\u002FUSDT\"],\n            }\n        )\n\n# Options MM bots: TRADE tier with additional restrictions\nfor agent_id, bot in fleet.bots.items():\n    if bot.group == \"options-mm\":\n        perm_mgr.assign_permissions(\n            agent_id=agent_id,\n            bot_name=bot.name,\n            exchange=bot.exchange,\n            tier=PermissionTier.TRADE,\n            custom_restrictions={\n                \"max_order_size_usd\": 25000,\n                \"allowed_instruments\": [\"BTC-*-C\", \"BTC-*-P\", \"ETH-*-C\", \"ETH-*-P\"],\n                \"max_delta_exposure\": 0.5,\n            }\n        )\n\nprint(json.dumps(perm_mgr.get_fleet_permission_summary(), indent=2))\n```\n\nThe key insight: no bot in this fleet has withdrawal permissions. Withdrawals should only be executed by a dedicated withdrawal agent that is not connected to exchange websockets and has its own approval workflow (ideally requiring human confirmation). A compromised trading bot that cannot withdraw can, at worst, make bad trades. A compromised bot with withdrawal access can empty the exchange account.\n\n---\n\n## Chapter 5: FleetHealthMonitor Class\n\n### Real-Time Health Across All Bots\n\nHealth monitoring is the difference between discovering a dead bot immediately and discovering it three days later when someone notices a strategy stopped making money. For a fleet of 12 bots, you need centralized health that answers three questions in real time: is each bot alive, is each bot performing within expectations, and are there any bots that need intervention?\n\n### Metrics Collected\n\nEach bot reports the following metrics on every heartbeat:\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| `uptime_seconds` | gauge | Time since last restart |\n| `heartbeat_latency_ms` | gauge | Round-trip time to exchange websocket |\n| `pnl_usd` | gauge | Unrealized + realized PnL since start of day |\n| `drawdown_pct` | gauge | Current drawdown from peak equity |\n| `trade_count` | counter | Number of trades executed since start of day |\n| `open_positions` | gauge | Number of currently open positions |\n| `open_orders` | gauge | Number of currently open orders |\n| `memory_mb` | gauge | Process memory usage |\n| `cpu_pct` | gauge | Process CPU usage |\n| `error_count` | counter | Number of errors since last heartbeat |\n\n### The FleetHealthMonitor Class\n\n```python\nclass FleetHealthMonitor:\n    \"\"\"Monitors health across all bots in the fleet.\"\"\"\n\n    def __init__(self, fleet_manager_id: str, bots: Dict[str, BotRecord],\n                 heartbeat_interval_seconds: int = 30,\n                 dead_threshold_seconds: int = 120):\n        self.fleet_manager_id = fleet_manager_id\n        self.bots = bots\n        self.heartbeat_interval = heartbeat_interval_seconds\n        self.dead_threshold = dead_threshold_seconds\n        self.health_records: Dict[str, dict] = {}\n        self.alert_callbacks: List[callable] = []\n        self._setup_webhooks()\n\n    def _setup_webhooks(self):\n        \"\"\"Register webhooks for real-time health alerts.\"\"\"\n        execute(\"register_webhook\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_types\": [\n                \"fleet.heartbeat\",\n                \"fleet.health_alert\",\n                \"fleet.bot_dead\"\n            ],\n            \"url\": \"https:\u002F\u002Fyour-fleet-dashboard.example.com\u002Fwebhooks\u002Fhealth\",\n            \"secret\": \"webhook-signing-secret\"\n        })\n\n    def record_heartbeat(self, agent_id: str, metrics: dict):\n        \"\"\"Record a heartbeat from a bot.\"\"\"\n        now = datetime.utcnow().isoformat()\n\n        # Submit metrics to GreenHelix\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                **metrics,\n                \"heartbeat_timestamp\": now,\n            }\n        })\n\n        self.health_records[agent_id] = {\n            \"agent_id\": agent_id,\n            \"bot_name\": self.bots[agent_id].name if agent_id in self.bots else \"unknown\",\n            \"last_heartbeat\": now,\n            \"metrics\": metrics,\n            \"status\": self._evaluate_health(agent_id, metrics),\n        }\n\n    def _evaluate_health(self, agent_id: str, metrics: dict) -> str:\n        \"\"\"Evaluate bot health from metrics. Returns status string.\"\"\"\n        issues = []\n\n        # Check drawdown\n        if metrics.get(\"drawdown_pct\", 0) > 10:\n            issues.append(f\"high_drawdown:{metrics['drawdown_pct']:.1f}%\")\n        elif metrics.get(\"drawdown_pct\", 0) > 5:\n            issues.append(f\"elevated_drawdown:{metrics['drawdown_pct']:.1f}%\")\n\n        # Check latency\n        if metrics.get(\"heartbeat_latency_ms\", 0) > 5000:\n            issues.append(f\"high_latency:{metrics['heartbeat_latency_ms']}ms\")\n        elif metrics.get(\"heartbeat_latency_ms\", 0) > 1000:\n            issues.append(f\"elevated_latency:{metrics['heartbeat_latency_ms']}ms\")\n\n        # Check errors\n        if metrics.get(\"error_count\", 0) > 10:\n            issues.append(f\"high_errors:{metrics['error_count']}\")\n        elif metrics.get(\"error_count\", 0) > 3:\n            issues.append(f\"elevated_errors:{metrics['error_count']}\")\n\n        # Check memory\n        if metrics.get(\"memory_mb\", 0) > 2048:\n            issues.append(f\"high_memory:{metrics['memory_mb']}MB\")\n\n        if any(\"high_\" in i for i in issues):\n            self._trigger_alert(agent_id, \"critical\", issues)\n            return \"critical\"\n        elif issues:\n            self._trigger_alert(agent_id, \"warning\", issues)\n            return \"warning\"\n        return \"healthy\"\n\n    def _trigger_alert(self, agent_id: str, severity: str,\n                       issues: List[str]):\n        \"\"\"Send alert via GreenHelix messaging.\"\"\"\n        bot_name = self.bots[agent_id].name if agent_id in self.bots else agent_id\n\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": self.fleet_manager_id,  # Self-message for logging\n            \"message_type\": \"alert\",\n            \"payload\": {\n                \"severity\": severity,\n                \"bot_name\": bot_name,\n                \"agent_id\": agent_id,\n                \"issues\": issues,\n                \"timestamp\": datetime.utcnow().isoformat(),\n            }\n        })\n\n        # Publish as event for audit trail\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.health_alert\",\n            \"payload\": {\n                \"severity\": severity,\n                \"agent_id\": agent_id,\n                \"bot_name\": bot_name,\n                \"issues\": issues,\n            }\n        })\n\n    def detect_dead_bots(self) -> List[str]:\n        \"\"\"Find bots that have missed heartbeats beyond the threshold.\"\"\"\n        now = datetime.utcnow()\n        dead_bots = []\n\n        for agent_id, bot in self.bots.items():\n            record = self.health_records.get(agent_id)\n            if record is None:\n                # Never sent a heartbeat\n                dead_bots.append(agent_id)\n                continue\n\n            last_hb = datetime.fromisoformat(record[\"last_heartbeat\"])\n            seconds_since = (now - last_hb).total_seconds()\n\n            if seconds_since > self.dead_threshold:\n                dead_bots.append(agent_id)\n                bot.status = \"dead\"\n\n                execute(\"publish_event\", {\n                    \"agent_id\": self.fleet_manager_id,\n                    \"event_type\": \"fleet.bot_dead\",\n                    \"payload\": {\n                        \"agent_id\": agent_id,\n                        \"bot_name\": bot.name,\n                        \"last_heartbeat\": record[\"last_heartbeat\"],\n                        \"seconds_since_heartbeat\": seconds_since,\n                        \"group\": bot.group,\n                    }\n                })\n\n        return dead_bots\n\n    def get_fleet_health_dashboard(self) -> dict:\n        \"\"\"Aggregate health data for dashboard display.\"\"\"\n        status_counts = {\"healthy\": 0, \"warning\": 0, \"critical\": 0, \"dead\": 0, \"unknown\": 0}\n        group_health = {}\n        total_pnl = 0.0\n        total_trades = 0\n\n        for agent_id, bot in self.bots.items():\n            record = self.health_records.get(agent_id)\n            if record:\n                status = record[\"status\"]\n                metrics = record[\"metrics\"]\n                total_pnl += metrics.get(\"pnl_usd\", 0)\n                total_trades += metrics.get(\"trade_count\", 0)\n            elif bot.status == \"dead\":\n                status = \"dead\"\n            else:\n                status = \"unknown\"\n\n            status_counts[status] = status_counts.get(status, 0) + 1\n\n            if bot.group not in group_health:\n                group_health[bot.group] = {\n                    \"healthy\": 0, \"warning\": 0,\n                    \"critical\": 0, \"dead\": 0, \"unknown\": 0,\n                    \"pnl_usd\": 0.0, \"trade_count\": 0,\n                }\n            group_health[bot.group][status] += 1\n            if record:\n                group_health[bot.group][\"pnl_usd\"] += record[\"metrics\"].get(\"pnl_usd\", 0)\n                group_health[bot.group][\"trade_count\"] += record[\"metrics\"].get(\"trade_count\", 0)\n\n        return {\n            \"timestamp\": datetime.utcnow().isoformat(),\n            \"total_bots\": len(self.bots),\n            \"status_summary\": status_counts,\n            \"groups\": group_health,\n            \"fleet_pnl_usd\": round(total_pnl, 2),\n            \"fleet_trade_count\": total_trades,\n        }\n\n    def initiate_failover(self, dead_agent_id: str,\n                          replacement_agent_id: str):\n        \"\"\"Failover a dead bot's responsibilities to a replacement.\"\"\"\n        dead_bot = self.bots.get(dead_agent_id)\n        if not dead_bot:\n            raise ValueError(f\"Bot {dead_agent_id} not found\")\n\n        # Send failover command to replacement bot\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": replacement_agent_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"assume_responsibility\",\n                \"from_bot\": dead_agent_id,\n                \"group\": dead_bot.group,\n                \"exchange\": dead_bot.exchange,\n                \"strategy\": dead_bot.strategy,\n                \"reason\": \"failover_from_dead_bot\",\n                \"timestamp\": datetime.utcnow().isoformat(),\n            }\n        })\n\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.failover\",\n            \"payload\": {\n                \"dead_bot\": dead_agent_id,\n                \"replacement_bot\": replacement_agent_id,\n                \"group\": dead_bot.group,\n                \"exchange\": dead_bot.exchange,\n            }\n        })\n\n        print(f\"Failover: {dead_bot.name} -> \"\n              f\"{self.bots[replacement_agent_id].name}\")\n```\n\n### Using the Health Monitor\n\n```python\nhealth = FleetHealthMonitor(\n    fleet_manager_id=fleet.manager_agent_id,\n    bots=fleet.bots,\n    heartbeat_interval_seconds=30,\n    dead_threshold_seconds=120\n)\n\n# Simulate a bot heartbeat (in production, bots call this themselves)\nfor agent_id, bot in fleet.bots.items():\n    health.record_heartbeat(agent_id, {\n        \"uptime_seconds\": 86400,\n        \"heartbeat_latency_ms\": 45,\n        \"pnl_usd\": 234.50,\n        \"drawdown_pct\": 1.2,\n        \"trade_count\": 47,\n        \"open_positions\": 3,\n        \"open_orders\": 6,\n        \"memory_mb\": 512,\n        \"cpu_pct\": 15.3,\n        \"error_count\": 0,\n    })\n\n# Check for dead bots\ndead = health.detect_dead_bots()\nif dead:\n    print(f\"Dead bots detected: {dead}\")\n\n# Get dashboard\ndashboard = health.get_fleet_health_dashboard()\nprint(json.dumps(dashboard, indent=2))\n```\n\n```bash\n# curl: Submit bot heartbeat metrics\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"submit_metrics\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id\",\n      \"metrics\": {\n        \"uptime_seconds\": 86400,\n        \"heartbeat_latency_ms\": 45,\n        \"pnl_usd\": 234.50,\n        \"drawdown_pct\": 1.2,\n        \"trade_count\": 47,\n        \"open_positions\": 3,\n        \"open_orders\": 6,\n        \"memory_mb\": 512,\n        \"cpu_pct\": 15.3,\n        \"error_count\": 0,\n        \"heartbeat_timestamp\": \"2026-04-07T14:00:00Z\"\n      }\n    }\n  }'\n\n# curl: Register health webhook\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"register_webhook\",\n    \"input\": {\n      \"agent_id\": \"fleet-manager-agent-id\",\n      \"event_types\": [\"fleet.heartbeat\", \"fleet.health_alert\", \"fleet.bot_dead\"],\n      \"url\": \"https:\u002F\u002Fyour-fleet-dashboard.example.com\u002Fwebhooks\u002Fhealth\",\n      \"secret\": \"webhook-signing-secret\"\n    }\n  }'\n```\n\n### Dead Bot Detection and Automatic Failover\n\nDead bot detection runs on a configurable interval (default: check every 30 seconds, declare dead after 120 seconds without a heartbeat). When a bot is declared dead, the monitor publishes a `fleet.bot_dead` event and the fleet manager can trigger automatic failover.\n\nFailover strategy depends on the bot type. Stateless bots (spot arb, momentum) can fail over to a standby instance that was pre-registered but dormant. Stateful bots (options market makers with active positions) require careful handoff: the replacement bot needs to query the exchange for open positions before it starts making new decisions.\n\nThe health monitor does not make the failover decision automatically -- it detects the dead bot and notifies the fleet manager, which then decides whether to failover, restart, or escalate to the human operator. Automatic failover without human judgment can cause cascading failures if the root cause is a network partition rather than a bot failure.\n\n### Health Monitoring Architecture Considerations\n\nTwo architectural decisions deserve explicit attention.\n\n**Push vs. Pull.** This implementation uses push-based heartbeats: bots push their metrics to GreenHelix on a fixed interval. The alternative is pull-based: the fleet manager polls each bot's health endpoint. Push is superior for trading bots for two reasons. First, pull requires the fleet manager to know the network address of every bot, which changes when bots restart, scale, or migrate across hosts. With push, the bot only needs to know the GreenHelix API endpoint, which is static. Second, push naturally detects dead bots: a missing heartbeat is itself a signal. With pull, you have to distinguish between \"the bot is dead\" and \"the bot's health endpoint is temporarily unreachable due to network issues\" -- a distinction that causes false positives in practice.\n\n**Metric Granularity.** The ten metrics defined above (uptime, latency, PnL, drawdown, trade count, positions, orders, memory, CPU, errors) cover the operational baseline. In production, you will want strategy-specific metrics: spread capture for arb bots, Greeks exposure for options bots, fill rate for momentum bots. Add these as custom metrics via `submit_metrics` -- do not try to standardize every possible metric into the base set. The base set answers \"is this bot alive and within operational bounds.\" Strategy-specific metrics answer \"is this bot executing its strategy well.\"\n\n---\n\n## Chapter 6: Coordinated Deployments\n\n### Why Deployment Coordination Matters\n\nDeploying a new strategy version to 12 bots simultaneously is how you lose money across all 12 bots simultaneously when the new version has a bug. Coordinated deployment means controlled rollout: update one bot first, verify it works, then roll out to the group, then to the fleet. If the canary bot shows problems, you roll back one bot instead of twelve.\n\n### The DeploymentOrchestrator Class\n\n```python\n@dataclass\nclass DeploymentRecord:\n    \"\"\"Tracks a single deployment.\"\"\"\n    deployment_id: str\n    version: str\n    target_group: str\n    strategy: str  # \"canary\", \"rolling\", \"blue_green\"\n    status: str = \"pending\"  # pending, canary, rolling, complete, rolled_back\n    started_at: str = \"\"\n    completed_at: str = \"\"\n    canary_bot: Optional[str] = None\n    deployed_bots: List[str] = field(default_factory=list)\n    rolled_back_bots: List[str] = field(default_factory=list)\n\n\nclass DeploymentOrchestrator:\n    \"\"\"Coordinates deployments across the bot fleet.\"\"\"\n\n    def __init__(self, fleet_manager_id: str, bots: Dict[str, BotRecord],\n                 health_monitor: FleetHealthMonitor):\n        self.fleet_manager_id = fleet_manager_id\n        self.bots = bots\n        self.health = health_monitor\n        self.deployments: Dict[str, DeploymentRecord] = {}\n\n    def _gen_deployment_id(self, version: str, group: str) -> str:\n        raw = f\"{version}-{group}-{datetime.utcnow().isoformat()}\"\n        return f\"deploy-{hashlib.sha256(raw.encode()).hexdigest()[:12]}\"\n\n    def start_canary_deployment(self, group: str, new_version: str,\n                                 canary_duration_minutes: int = 30) -> str:\n        \"\"\"Start a canary deployment: update one bot, monitor, then roll out.\"\"\"\n        deployment_id = self._gen_deployment_id(new_version, group)\n\n        # Pick the canary: the bot in the group with the lowest PnL\n        # (least to lose if the new version has issues)\n        group_bots = [\n            (aid, bot) for aid, bot in self.bots.items()\n            if bot.group == group and bot.status == \"active\"\n        ]\n        if not group_bots:\n            raise ValueError(f\"No active bots in group {group}\")\n\n        canary_id, canary_bot = group_bots[0]  # In production, sort by PnL\n\n        deployment = DeploymentRecord(\n            deployment_id=deployment_id,\n            version=new_version,\n            target_group=group,\n            strategy=\"canary\",\n            status=\"canary\",\n            started_at=datetime.utcnow().isoformat(),\n            canary_bot=canary_id,\n        )\n        self.deployments[deployment_id] = deployment\n\n        # Send update command to canary bot\n        execute(\"send_message\", {\n            \"from_agent_id\": self.fleet_manager_id,\n            \"to_agent_id\": canary_id,\n            \"message_type\": \"command\",\n            \"payload\": {\n                \"command\": \"update_version\",\n                \"new_version\": new_version,\n                \"deployment_id\": deployment_id,\n                \"role\": \"canary\",\n                \"monitoring_duration_minutes\": canary_duration_minutes,\n            }\n        })\n\n        # Publish deployment event\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.deployment_started\",\n            \"payload\": {\n                \"deployment_id\": deployment_id,\n                \"version\": new_version,\n                \"group\": group,\n                \"strategy\": \"canary\",\n                \"canary_bot\": canary_id,\n                \"canary_bot_name\": canary_bot.name,\n            }\n        })\n\n        deployment.deployed_bots.append(canary_id)\n        canary_bot.version = new_version\n        print(f\"Canary deployment started: {canary_bot.name} -> v{new_version}\")\n        print(f\"Monitoring for {canary_duration_minutes} minutes before rollout\")\n        return deployment_id\n\n    def promote_canary(self, deployment_id: str,\n                       batch_size: int = 2,\n                       batch_delay_seconds: int = 60):\n        \"\"\"Promote canary to rolling deployment across the group.\"\"\"\n        deployment = self.deployments.get(deployment_id)\n        if not deployment or deployment.status != \"canary\":\n            raise ValueError(f\"Deployment {deployment_id} not in canary phase\")\n\n        # Verify canary health before promoting\n        canary_health = self.health.health_records.get(deployment.canary_bot)\n        if canary_health and canary_health[\"status\"] == \"critical\":\n            print(f\"Canary bot is in critical state. Aborting promotion.\")\n            self.rollback(deployment_id)\n            return\n\n        deployment.status = \"rolling\"\n\n        # Get remaining bots in the group\n        remaining = [\n            (aid, bot) for aid, bot in self.bots.items()\n            if bot.group == deployment.target_group\n            and aid != deployment.canary_bot\n            and bot.status == \"active\"\n        ]\n\n        # Deploy in batches\n        for i in range(0, len(remaining), batch_size):\n            batch = remaining[i:i + batch_size]\n\n            for agent_id, bot in batch:\n                execute(\"send_message\", {\n                    \"from_agent_id\": self.fleet_manager_id,\n                    \"to_agent_id\": agent_id,\n                    \"message_type\": \"command\",\n                    \"payload\": {\n                        \"command\": \"update_version\",\n                        \"new_version\": deployment.version,\n                        \"deployment_id\": deployment_id,\n                        \"role\": \"rolling\",\n                        \"batch\": i \u002F\u002F batch_size + 1,\n                    }\n                })\n                deployment.deployed_bots.append(agent_id)\n                bot.version = deployment.version\n\n            print(f\"Batch {i \u002F\u002F batch_size + 1} deployed: \"\n                  f\"{[b.name for _, b in batch]}\")\n\n            # Wait between batches to detect issues\n            if i + batch_size \u003C len(remaining):\n                time.sleep(batch_delay_seconds)\n\n                # Check health after each batch\n                dead_in_batch = [\n                    aid for aid, _ in batch\n                    if self.health.health_records.get(aid, {}).get(\"status\") == \"critical\"\n                ]\n                if dead_in_batch:\n                    print(f\"Critical bots detected after batch. Halting rollout.\")\n                    self.rollback(deployment_id)\n                    return\n\n        deployment.status = \"complete\"\n        deployment.completed_at = datetime.utcnow().isoformat()\n\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.deployment_complete\",\n            \"payload\": {\n                \"deployment_id\": deployment_id,\n                \"version\": deployment.version,\n                \"group\": deployment.target_group,\n                \"bots_deployed\": len(deployment.deployed_bots),\n            }\n        })\n        print(f\"Deployment complete: {len(deployment.deployed_bots)} bots \"\n              f\"updated to v{deployment.version}\")\n\n    def rollback(self, deployment_id: str):\n        \"\"\"Roll back a deployment. Bots revert to their previous version.\"\"\"\n        deployment = self.deployments.get(deployment_id)\n        if not deployment:\n            raise ValueError(f\"Deployment {deployment_id} not found\")\n\n        for agent_id in deployment.deployed_bots:\n            execute(\"send_message\", {\n                \"from_agent_id\": self.fleet_manager_id,\n                \"to_agent_id\": agent_id,\n                \"message_type\": \"command\",\n                \"payload\": {\n                    \"command\": \"rollback_version\",\n                    \"deployment_id\": deployment_id,\n                    \"reason\": \"deployment_rollback\",\n                }\n            })\n            deployment.rolled_back_bots.append(agent_id)\n\n        deployment.status = \"rolled_back\"\n        deployment.completed_at = datetime.utcnow().isoformat()\n\n        execute(\"publish_event\", {\n            \"agent_id\": self.fleet_manager_id,\n            \"event_type\": \"fleet.deployment_rolled_back\",\n            \"payload\": {\n                \"deployment_id\": deployment_id,\n                \"version\": deployment.version,\n                \"group\": deployment.target_group,\n                \"bots_rolled_back\": len(deployment.rolled_back_bots),\n            }\n        })\n        print(f\"Deployment rolled back: {len(deployment.rolled_back_bots)} bots\")\n\n    def get_version_map(self) -> Dict[str, List[str]]:\n        \"\"\"Map of version -> bot names. Useful for spotting version skew.\"\"\"\n        versions: Dict[str, List[str]] = {}\n        for bot in self.bots.values():\n            if bot.version not in versions:\n                versions[bot.version] = []\n            versions[bot.version].append(bot.name)\n        return versions\n```\n\n### Deployment Workflow\n\n```python\ndeployer = DeploymentOrchestrator(\n    fleet_manager_id=fleet.manager_agent_id,\n    bots=fleet.bots,\n    health_monitor=health\n)\n\n# Step 1: Deploy canary\ndep_id = deployer.start_canary_deployment(\n    group=\"perp-momentum\",\n    new_version=\"1.9.0\",\n    canary_duration_minutes=30\n)\n\n# Step 2: Wait for canary monitoring period...\n# In production, this is a scheduled task or callback\n\n# Step 3: Promote to full rollout\ndeployer.promote_canary(dep_id, batch_size=2, batch_delay_seconds=60)\n\n# Check version distribution\nprint(json.dumps(deployer.get_version_map(), indent=2))\n```\n\n```bash\n# curl: Send version update command to a bot\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"send_message\",\n    \"input\": {\n      \"from_agent_id\": \"fleet-manager-agent-id\",\n      \"to_agent_id\": \"bot-agent-id\",\n      \"message_type\": \"command\",\n      \"payload\": {\n        \"command\": \"update_version\",\n        \"new_version\": \"1.9.0\",\n        \"deployment_id\": \"deploy-abc123\",\n        \"role\": \"canary\"\n      }\n    }\n  }'\n```\n\n### Rollback Procedures\n\nRollback is not the reverse of deployment. When you roll forward, you send a new version and the bot restarts. When you roll back, the bot needs to revert to the previous binary, reconnect to exchange websockets, and reconcile its state with any positions that changed during the time the new version was running. The `rollback_version` command tells the bot to switch back, but the bot itself is responsible for the reconnection and reconciliation logic.\n\nA critical detail: always keep the previous version available. Delete old versions only after the new version has been stable for a defined period (72 hours is a reasonable default). Rolling back to a version that was deleted from the artifact store is a bad day.\n\n---\n\n## Chapter 7: SLA Tracking\n\n### Defining SLAs for Trading Bots\n\nAn SLA (Service Level Agreement) for a trading bot is a formal commitment to operational targets. Unlike web service SLAs that focus primarily on availability, trading bot SLAs span multiple dimensions: uptime, latency, risk limits, and minimum activity thresholds. Defining these explicitly forces the team to decide what \"healthy\" looks like before problems occur, not during an incident when judgment is compromised.\n\n### SLA Definitions\n\n```python\n@dataclass\nclass BotSLA:\n    \"\"\"SLA definition for a trading bot.\"\"\"\n    agent_id: str\n    bot_name: str\n    group: str\n    targets: dict\n    created_at: str = \"\"\n    active: bool = True\n\n\nclass SLATracker:\n    \"\"\"Track and enforce SLAs across the bot fleet.\"\"\"\n\n    def __init__(self, fleet_manager_id: str):\n        self.fleet_manager_id = fleet_manager_id\n        self.slas: Dict[str, BotSLA] = {}\n\n    def define_sla(self, agent_id: str, bot_name: str,\n                   group: str, targets: dict) -> BotSLA:\n        \"\"\"Define an SLA for a bot.\"\"\"\n        sla = BotSLA(\n            agent_id=agent_id,\n            bot_name=bot_name,\n            group=group,\n            targets=targets,\n            created_at=datetime.utcnow().isoformat()\n        )\n        self.slas[agent_id] = sla\n\n        # Register the SLA targets with GreenHelix\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                \"sla_uptime_target\": targets.get(\"uptime_pct\", 99.5),\n                \"sla_max_drawdown\": targets.get(\"max_drawdown_pct\", 5.0),\n                \"sla_min_trades_per_day\": targets.get(\"min_trades_per_day\", 10),\n                \"sla_max_latency_ms\": targets.get(\"max_latency_ms\", 500),\n                \"sla_active\": 1,\n            }\n        })\n\n        print(f\"SLA defined for {bot_name}: {targets}\")\n        return sla\n\n    def check_compliance(self, agent_id: str,\n                         current_metrics: dict) -> dict:\n        \"\"\"Check a bot's current metrics against its SLA.\"\"\"\n        sla = self.slas.get(agent_id)\n        if not sla:\n            return {\"status\": \"no_sla_defined\"}\n\n        violations = []\n        warnings = []\n\n        targets = sla.targets\n\n        # Check uptime\n        if \"uptime_pct\" in targets:\n            actual = current_metrics.get(\"uptime_pct\", 100)\n            if actual \u003C targets[\"uptime_pct\"]:\n                violations.append({\n                    \"metric\": \"uptime_pct\",\n                    \"target\": targets[\"uptime_pct\"],\n                    \"actual\": actual,\n                    \"severity\": \"critical\"\n                })\n            elif actual \u003C targets[\"uptime_pct\"] + 0.5:\n                warnings.append({\n                    \"metric\": \"uptime_pct\",\n                    \"target\": targets[\"uptime_pct\"],\n                    \"actual\": actual,\n                    \"severity\": \"warning\"\n                })\n\n        # Check drawdown\n        if \"max_drawdown_pct\" in targets:\n            actual = current_metrics.get(\"drawdown_pct\", 0)\n            if actual > targets[\"max_drawdown_pct\"]:\n                violations.append({\n                    \"metric\": \"drawdown_pct\",\n                    \"target\": targets[\"max_drawdown_pct\"],\n                    \"actual\": actual,\n                    \"severity\": \"critical\"\n                })\n            elif actual > targets[\"max_drawdown_pct\"] * 0.8:\n                warnings.append({\n                    \"metric\": \"drawdown_pct\",\n                    \"target\": targets[\"max_drawdown_pct\"],\n                    \"actual\": actual,\n                    \"severity\": \"warning\"\n                })\n\n        # Check minimum trades\n        if \"min_trades_per_day\" in targets:\n            actual = current_metrics.get(\"trade_count\", 0)\n            if actual \u003C targets[\"min_trades_per_day\"]:\n                violations.append({\n                    \"metric\": \"min_trades_per_day\",\n                    \"target\": targets[\"min_trades_per_day\"],\n                    \"actual\": actual,\n                    \"severity\": \"warning\"\n                })\n\n        # Check latency\n        if \"max_latency_ms\" in targets:\n            actual = current_metrics.get(\"heartbeat_latency_ms\", 0)\n            if actual > targets[\"max_latency_ms\"]:\n                violations.append({\n                    \"metric\": \"max_latency_ms\",\n                    \"target\": targets[\"max_latency_ms\"],\n                    \"actual\": actual,\n                    \"severity\": \"critical\"\n                })\n\n        result = execute(\"get_sla_compliance\", {\n            \"agent_id\": agent_id\n        })\n\n        compliance_status = \"compliant\" if not violations else \"violation\"\n\n        # Report violations\n        if violations:\n            execute(\"publish_event\", {\n                \"agent_id\": self.fleet_manager_id,\n                \"event_type\": \"fleet.sla_violation\",\n                \"payload\": {\n                    \"agent_id\": agent_id,\n                    \"bot_name\": sla.bot_name,\n                    \"violations\": violations,\n                    \"warnings\": warnings,\n                    \"timestamp\": datetime.utcnow().isoformat(),\n                }\n            })\n\n            # Alert via messaging\n            execute(\"send_message\", {\n                \"from_agent_id\": self.fleet_manager_id,\n                \"to_agent_id\": self.fleet_manager_id,\n                \"message_type\": \"alert\",\n                \"payload\": {\n                    \"alert_type\": \"sla_violation\",\n                    \"bot_name\": sla.bot_name,\n                    \"violations\": violations,\n                }\n            })\n\n        return {\n            \"agent_id\": agent_id,\n            \"bot_name\": sla.bot_name,\n            \"status\": compliance_status,\n            \"violations\": violations,\n            \"warnings\": warnings,\n            \"greenhelix_compliance\": result,\n            \"checked_at\": datetime.utcnow().isoformat(),\n        }\n\n    def generate_monthly_report(self, month: str = None) -> dict:\n        \"\"\"Generate a monthly SLA compliance report for the fleet.\"\"\"\n        report = {\n            \"period\": month or datetime.utcnow().strftime(\"%Y-%m\"),\n            \"generated_at\": datetime.utcnow().isoformat(),\n            \"bots\": {},\n            \"fleet_summary\": {\n                \"total_bots_with_sla\": len(self.slas),\n                \"compliant\": 0,\n                \"in_violation\": 0,\n            }\n        }\n\n        for agent_id, sla in self.slas.items():\n            # Pull metrics from GreenHelix\n            result = execute(\"search_agents_by_metrics\", {\n                \"agent_id\": agent_id,\n                \"metric_names\": [\n                    \"uptime_pct\", \"drawdown_pct\",\n                    \"trade_count\", \"heartbeat_latency_ms\"\n                ],\n                \"period\": month or datetime.utcnow().strftime(\"%Y-%m\")\n            })\n\n            compliant = True  # Determined by actual metric analysis\n            report[\"bots\"][sla.bot_name] = {\n                \"agent_id\": agent_id,\n                \"group\": sla.group,\n                \"targets\": sla.targets,\n                \"metrics\": result,\n                \"compliant\": compliant,\n            }\n            if compliant:\n                report[\"fleet_summary\"][\"compliant\"] += 1\n            else:\n                report[\"fleet_summary\"][\"in_violation\"] += 1\n\n        return report\n```\n\n### Applying SLAs to the Fleet\n\n```python\nsla_tracker = SLATracker(fleet_manager_id=fleet.manager_agent_id)\n\n# Define SLAs per group\nfor agent_id, bot in fleet.bots.items():\n    if bot.group == \"spot-arb\":\n        sla_tracker.define_sla(agent_id, bot.name, bot.group, {\n            \"uptime_pct\": 99.5,\n            \"max_drawdown_pct\": 3.0,\n            \"min_trades_per_day\": 50,\n            \"max_latency_ms\": 200,\n        })\n    elif bot.group == \"perp-momentum\":\n        sla_tracker.define_sla(agent_id, bot.name, bot.group, {\n            \"uptime_pct\": 99.0,\n            \"max_drawdown_pct\": 8.0,\n            \"min_trades_per_day\": 10,\n            \"max_latency_ms\": 500,\n        })\n    elif bot.group == \"options-mm\":\n        sla_tracker.define_sla(agent_id, bot.name, bot.group, {\n            \"uptime_pct\": 99.9,\n            \"max_drawdown_pct\": 5.0,\n            \"min_trades_per_day\": 100,\n            \"max_latency_ms\": 100,\n        })\n\n# Check compliance for a specific bot\nresult = sla_tracker.check_compliance(\n    agent_id=list(fleet.bots.keys())[0],\n    current_metrics={\n        \"uptime_pct\": 99.7,\n        \"drawdown_pct\": 2.1,\n        \"trade_count\": 63,\n        \"heartbeat_latency_ms\": 45,\n    }\n)\nprint(json.dumps(result, indent=2))\n```\n\n```bash\n# curl: Check SLA compliance\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"get_sla_compliance\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id\"\n    }\n  }'\n\n# curl: Search bot metrics for SLA report\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"search_agents_by_metrics\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id\",\n      \"metric_names\": [\"uptime_pct\", \"drawdown_pct\", \"trade_count\"],\n      \"period\": \"2026-04\"\n    }\n  }'\n```\n\n### SLA Violation Escalation\n\nSLA violations escalate through three tiers:\n\n1. **Warning** (approaching threshold): Log the warning, continue monitoring. Example: drawdown at 4.2% against a 5% limit.\n2. **Violation** (threshold breached): Alert via GreenHelix messaging, publish event, flag in dashboard. Example: drawdown at 5.3% against a 5% limit.\n3. **Critical violation** (threshold significantly breached): Automatic pause of the offending bot, alert the fleet operator, publish critical event. Example: drawdown at 8% against a 5% limit.\n\nThe escalation chain is codified, not improvised. When a bot hits a critical SLA violation at 3 AM, the system pauses the bot and sends an alert. It does not wait for a human to notice a Grafana panel.\n\n---\n\n## Chapter 8: Cost Allocation\n\n### Why Per-Bot Cost Tracking Matters\n\nA fleet of 12 bots has costs scattered across multiple dimensions: exchange trading fees, exchange API fees, GreenHelix API fees, cloud compute (EC2\u002FGCE instances or Kubernetes pods), network data transfer, and market data subscriptions. Without per-bot attribution, you cannot answer the most basic business question: is this bot profitable after costs?\n\nConsider a momentum bot that makes $500\u002Fday in gross PnL but runs on a dedicated c5.2xlarge instance ($204\u002Fmonth), trades 500 times per day at $0.10 per trade in exchange fees ($50\u002Fday), and calls the GreenHelix API 2,000 times per day. The gross PnL looks great. The net PnL after infrastructure, exchange fees, and API costs might be significantly less great.\n\n### The CostAllocator Class\n\n```python\n@dataclass\nclass CostEntry:\n    \"\"\"A single cost entry for a bot.\"\"\"\n    agent_id: str\n    bot_name: str\n    category: str  # exchange_fees, api_fees, infrastructure, market_data\n    amount_usd: float\n    description: str\n    timestamp: str\n    period: str  # YYYY-MM-DD or YYYY-MM\n\n\nclass CostAllocator:\n    \"\"\"Track and allocate costs per bot in the fleet.\"\"\"\n\n    def __init__(self, fleet_manager_id: str):\n        self.fleet_manager_id = fleet_manager_id\n        self.entries: List[CostEntry] = []\n        self.budgets: Dict[str, float] = {}  # agent_id -> monthly budget USD\n\n    def record_cost(self, agent_id: str, bot_name: str,\n                    category: str, amount_usd: float,\n                    description: str):\n        \"\"\"Record a cost entry for a bot.\"\"\"\n        entry = CostEntry(\n            agent_id=agent_id,\n            bot_name=bot_name,\n            category=category,\n            amount_usd=amount_usd,\n            description=description,\n            timestamp=datetime.utcnow().isoformat(),\n            period=datetime.utcnow().strftime(\"%Y-%m\"),\n        )\n        self.entries.append(entry)\n\n        # Submit cost metric to GreenHelix\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                f\"cost_{category}\": amount_usd,\n                \"cost_total_daily\": self._get_daily_cost(agent_id),\n                \"cost_period\": entry.period,\n            }\n        })\n\n        # Check budget alert\n        self._check_budget(agent_id, bot_name)\n\n    def _get_daily_cost(self, agent_id: str) -> float:\n        \"\"\"Get today's total cost for a bot.\"\"\"\n        today = datetime.utcnow().strftime(\"%Y-%m-%d\")\n        return sum(\n            e.amount_usd for e in self.entries\n            if e.agent_id == agent_id\n            and e.timestamp.startswith(today)\n        )\n\n    def set_budget(self, agent_id: str, monthly_budget_usd: float):\n        \"\"\"Set a monthly cost budget for a bot.\"\"\"\n        self.budgets[agent_id] = monthly_budget_usd\n\n    def _check_budget(self, agent_id: str, bot_name: str):\n        \"\"\"Alert if a bot is approaching or exceeding its budget.\"\"\"\n        budget = self.budgets.get(agent_id)\n        if not budget:\n            return\n\n        current_month = datetime.utcnow().strftime(\"%Y-%m\")\n        monthly_cost = sum(\n            e.amount_usd for e in self.entries\n            if e.agent_id == agent_id and e.period == current_month\n        )\n\n        utilization = monthly_cost \u002F budget\n\n        if utilization > 1.0:\n            execute(\"send_message\", {\n                \"from_agent_id\": self.fleet_manager_id,\n                \"to_agent_id\": self.fleet_manager_id,\n                \"message_type\": \"alert\",\n                \"payload\": {\n                    \"alert_type\": \"budget_exceeded\",\n                    \"bot_name\": bot_name,\n                    \"agent_id\": agent_id,\n                    \"budget_usd\": budget,\n                    \"spent_usd\": round(monthly_cost, 2),\n                    \"utilization_pct\": round(utilization * 100, 1),\n                }\n            })\n        elif utilization > 0.8:\n            execute(\"send_message\", {\n                \"from_agent_id\": self.fleet_manager_id,\n                \"to_agent_id\": self.fleet_manager_id,\n                \"message_type\": \"alert\",\n                \"payload\": {\n                    \"alert_type\": \"budget_warning\",\n                    \"bot_name\": bot_name,\n                    \"agent_id\": agent_id,\n                    \"budget_usd\": budget,\n                    \"spent_usd\": round(monthly_cost, 2),\n                    \"utilization_pct\": round(utilization * 100, 1),\n                }\n            })\n\n    def get_bot_cost_summary(self, agent_id: str,\n                             period: str = None) -> dict:\n        \"\"\"Get cost summary for a single bot.\"\"\"\n        if period is None:\n            period = datetime.utcnow().strftime(\"%Y-%m\")\n\n        bot_entries = [\n            e for e in self.entries\n            if e.agent_id == agent_id and e.period == period\n        ]\n\n        by_category = {}\n        for entry in bot_entries:\n            if entry.category not in by_category:\n                by_category[entry.category] = 0.0\n            by_category[entry.category] += entry.amount_usd\n\n        total = sum(by_category.values())\n        budget = self.budgets.get(agent_id)\n\n        return {\n            \"agent_id\": agent_id,\n            \"bot_name\": bot_entries[0].bot_name if bot_entries else \"unknown\",\n            \"period\": period,\n            \"costs_by_category\": {\n                k: round(v, 2) for k, v in by_category.items()\n            },\n            \"total_cost_usd\": round(total, 2),\n            \"budget_usd\": budget,\n            \"budget_utilization_pct\": round(\n                (total \u002F budget) * 100, 1\n            ) if budget else None,\n        }\n\n    def get_fleet_cost_summary(self, period: str = None) -> dict:\n        \"\"\"Get cost summary across the entire fleet.\"\"\"\n        if period is None:\n            period = datetime.utcnow().strftime(\"%Y-%m\")\n\n        period_entries = [\n            e for e in self.entries if e.period == period\n        ]\n\n        by_bot = {}\n        by_category = {}\n        by_group = {}\n\n        for entry in period_entries:\n            # By bot\n            if entry.bot_name not in by_bot:\n                by_bot[entry.bot_name] = 0.0\n            by_bot[entry.bot_name] += entry.amount_usd\n\n            # By category\n            if entry.category not in by_category:\n                by_category[entry.category] = 0.0\n            by_category[entry.category] += entry.amount_usd\n\n        total = sum(by_bot.values())\n\n        return {\n            \"period\": period,\n            \"total_fleet_cost_usd\": round(total, 2),\n            \"costs_by_bot\": {\n                k: round(v, 2) for k, v in\n                sorted(by_bot.items(), key=lambda x: -x[1])\n            },\n            \"costs_by_category\": {\n                k: round(v, 2) for k, v in by_category.items()\n            },\n        }\n\n    def calculate_roi(self, agent_id: str, gross_pnl_usd: float,\n                      period: str = None) -> dict:\n        \"\"\"Calculate ROI for a bot: (PnL - costs) \u002F costs.\"\"\"\n        summary = self.get_bot_cost_summary(agent_id, period)\n        total_cost = summary[\"total_cost_usd\"]\n\n        net_pnl = gross_pnl_usd - total_cost\n        roi = (net_pnl \u002F total_cost * 100) if total_cost > 0 else 0\n\n        result = {\n            \"agent_id\": agent_id,\n            \"bot_name\": summary[\"bot_name\"],\n            \"period\": summary[\"period\"],\n            \"gross_pnl_usd\": round(gross_pnl_usd, 2),\n            \"total_cost_usd\": total_cost,\n            \"net_pnl_usd\": round(net_pnl, 2),\n            \"roi_pct\": round(roi, 1),\n            \"cost_breakdown\": summary[\"costs_by_category\"],\n        }\n\n        # Submit ROI metric\n        execute(\"submit_metrics\", {\n            \"agent_id\": agent_id,\n            \"metrics\": {\n                \"gross_pnl_usd\": gross_pnl_usd,\n                \"net_pnl_usd\": round(net_pnl, 2),\n                \"total_cost_usd\": total_cost,\n                \"roi_pct\": round(roi, 1),\n            }\n        })\n\n        return result\n```\n\n### Recording Costs\n\n```python\ncosts = CostAllocator(fleet_manager_id=fleet.manager_agent_id)\n\n# Set budgets\nfor agent_id, bot in fleet.bots.items():\n    if bot.group == \"spot-arb\":\n        costs.set_budget(agent_id, monthly_budget_usd=500)\n    elif bot.group == \"perp-momentum\":\n        costs.set_budget(agent_id, monthly_budget_usd=800)\n    elif bot.group == \"options-mm\":\n        costs.set_budget(agent_id, monthly_budget_usd=1500)\n\n# Record daily costs for a bot (typically called by an automated cost collector)\nsample_bot_id = list(fleet.bots.keys())[0]\nsample_bot = fleet.bots[sample_bot_id]\n\ncosts.record_cost(sample_bot_id, sample_bot.name,\n                  \"exchange_fees\", 12.50,\n                  \"Binance spot trading fees: 125 trades @ $0.10\")\n\ncosts.record_cost(sample_bot_id, sample_bot.name,\n                  \"infrastructure\", 6.80,\n                  \"EC2 t3.medium: $204\u002Fmonth \u002F 30 days\")\n\ncosts.record_cost(sample_bot_id, sample_bot.name,\n                  \"api_fees\", 2.00,\n                  \"GreenHelix API: 200 calls @ $0.01\")\n\ncosts.record_cost(sample_bot_id, sample_bot.name,\n                  \"market_data\", 3.33,\n                  \"CoinGecko Pro: $100\u002Fmonth \u002F 30 days\")\n\n# Get bot cost summary\nsummary = costs.get_bot_cost_summary(sample_bot_id)\nprint(json.dumps(summary, indent=2))\n\n# Calculate ROI\nroi = costs.calculate_roi(sample_bot_id, gross_pnl_usd=234.50)\nprint(json.dumps(roi, indent=2))\n\n# Fleet-wide cost summary\nfleet_costs = costs.get_fleet_cost_summary()\nprint(json.dumps(fleet_costs, indent=2))\n```\n\n```bash\n# curl: Submit cost metrics for a bot\ncurl -X POST https:\u002F\u002Fsandbox.greenhelix.net\u002Fv1 \\\n  -H \"Authorization: Bearer $API_KEY\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"tool\": \"submit_metrics\",\n    \"input\": {\n      \"agent_id\": \"bot-agent-id\",\n      \"metrics\": {\n        \"cost_exchange_fees\": 12.50,\n        \"cost_infrastructure\": 6.80,\n        \"cost_api_fees\": 2.00,\n        \"cost_market_data\": 3.33,\n        \"cost_total_daily\": 24.63,\n        \"gross_pnl_usd\": 234.50,\n        \"net_pnl_usd\": 209.87,\n        \"roi_pct\": 851.7\n      }\n    }\n  }'\n```\n\n### ROI Calculation Per Bot\n\nThe ROI calculation is the moment of truth for every bot in the fleet. Gross PnL is what the strategy makes. Net PnL is what the strategy makes minus what it costs to run. A bot with $500\u002Fday gross PnL and $400\u002Fday in costs has an ROI of 25%. A bot with $100\u002Fday gross PnL and $25\u002Fday in costs has an ROI of 300%. The second bot is a better business even though the first bot makes more gross profit.\n\nThis is where fleet management pays for itself. Without per-bot cost attribution, the team sees aggregate PnL and aggregate costs and cannot make informed decisions about which bots to keep, which to optimize, and which to shut down. With per-bot attribution, the decision is clear: shut down the bots with negative or marginal ROI, invest in optimizing the bots with high gross PnL but high costs, and scale the bots with high ROI.\n\nThe cost allocator also enables chargeback within organizations. If different teams or strategies share the fleet infrastructure, each team can see exactly what their bots cost and what they return.\n\n### Cost Collection Automation\n\nManual cost tracking does not survive the first month. The CostAllocator is only as useful as the data fed into it. In production, automate cost collection from three sources:\n\n**Exchange fees.** Every major exchange provides an API endpoint for historical fee reports. Binance has `GET \u002Fsapi\u002Fv1\u002Fasset\u002FtradeFee`, Deribit has `GET \u002Fapi\u002Fv2\u002Fprivate\u002Fget_transaction_log`, OKX has `GET \u002Fapi\u002Fv5\u002Faccount\u002Fbills`. Run a daily cron job that pulls fees for each bot's sub-account and feeds them into `record_cost`.\n\n**Cloud infrastructure.** AWS Cost Explorer, GCP Billing, and Azure Cost Management all provide APIs for per-resource cost attribution. Tag each bot's compute resources with its agent ID, then query costs by tag daily. For Kubernetes, use pod-level cost attribution tools like Kubecost or OpenCost.\n\n**GreenHelix API usage.** GreenHelix tracks API call counts per agent. Pull this data and multiply by your tier's per-call cost to get the API expense per bot.\n\nThe three-source collection runs nightly and populates the CostAllocator. By morning, the fleet operator has a current cost picture for every bot without touching a spreadsheet.\n\n---\n\n## What's Next\n\n**Companion Guides:**\n\n- **Tamper-Proof Audit Trails** -- Build cryptographically verifiable audit trails for your fleet using GreenHelix event bus and Merkle claim chains. Essential for EU AI Act and MiFID II compliance.\n- **Bot Reputation System** -- Aggregate your fleet's health metrics and SLA compliance into verifiable reputation scores. Trading performance becomes a provable asset.\n- **Strategy Marketplace Playbook** -- Publish your strategies as products on the GreenHelix marketplace, backed by the reputation your fleet has built.\n- **Risk-as-a-Service** -- Cross-exchange, cross-strategy portfolio risk monitoring with circuit breakers and liquidation proximity alerts.\n\n**GreenHelix Documentation:**\n\n- Full API reference: https:\u002F\u002Fapi.greenhelix.net\u002Fv1\n- Identity, messaging, and metrics tool deep dives in the platform documentation\n\n**Operational Recommendations:**\n\n- Run the fleet manager on separate infrastructure from the trading bots. If the bots' servers go down, the fleet manager needs to be alive to detect the outage and trigger failover.\n- Automate cost collection. Exchange APIs provide fee reports, cloud providers have billing APIs, and GreenHelix tracks API usage. Feed all three into the CostAllocator on a daily cron job.\n- Key rotation every 30 days. Automate it. Do not wait for a compromise to discover that your rotation procedure does not work.\n- SLA reviews monthly. Targets that were reasonable three months ago may be too loose or too tight as market conditions and strategy performance change.\n\n---\n\n*This guide covers fleet management patterns as of April 2026. Exchange API permission models vary by exchange and change periodically -- verify the current permission options for your specific exchanges before implementing. GreenHelix tool interfaces are stable but consult the latest API documentation for any parameter changes.*\n\n",{"name":5,"tags":24,"type":29,"author":33,"install":34,"license":15,"version":35,"metadata":36,"price_usd":41,"executable":20,"credentials":42,"description":7,"content_type":43,"compatibility":44},[25,26,27,28,29,30,31,32],"fleet-management","trading-bot","identity","monitoring","guide","greenhelix","openclaw","ai-agent","felix-agent","none","1.3.1",{"openclaw":37},{"requires":38,"primaryEnv":40},{"env":39},[40],"AGENT_SIGNING_KEY",99,[40],"markdown",[31],"master","4aa004da461d820f4ea5499e58eb8d196cf99016","community\u002Fclawhub\u002Fg\u002Fgreenhelix-trading-bot-fleet-management",0,"https:\u002F\u002Fgithub.com\u002FLord1Egypt\u002Fawesome-skill-forge","https:\u002F\u002Fskill-explorer-rho.vercel.app","https:\u002F\u002Fgithub.com\u002FLord1Egypt","Unbreakable. Unstoppable. Unmatched⚡.\r\n Speaker of machine tongues 🤖  | World-builder of decentralized dreams 🌐 | Defender of the digital realm🛡️.","2026-06-10 08:21:01",{"source":8,"skillName":5,"commands":55,"prompt":65,"zipAvailable":66},[56,59,62],{"tool":57,"command":58},"npx","npx skills add Lord1Egypt\u002Fawesome-skill-forge --skill greenhelix-trading-bot-fleet-management",{"tool":60,"command":61},"bunx","bunx skills add Lord1Egypt\u002Fawesome-skill-forge --skill greenhelix-trading-bot-fleet-management",{"tool":63,"command":64},"pnpm","pnpm dlx skills add Lord1Egypt\u002Fawesome-skill-forge --skill greenhelix-trading-bot-fleet-management","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FLord1Egypt\u002Fawesome-skill-forge\u002Fblob\u002Fmaster\u002Fcommunity\u002Fclawhub\u002Fg\u002Fgreenhelix-trading-bot-fleet-management\u002FSKILL.md",true,[],[69,76,83,90,97,104],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M785WKJB99G3STR1NF9NRTJW","youtube-content","YouTube transcripts to summaries, threads, blogs.","https:\u002F\u002Fgithub.com\u002FLord1Egypt\u002Fawesome-skill-forge\u002Fblob\u002Fmaster\u002Fskills\u002Fyoutube-content\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M785WEFHB0G2S9YA6EPR3H0E","xurl","X\u002FTwitter via xurl CLI: post, search, DM, media, v2 API.","https:\u002F\u002Fgithub.com\u002FLord1Egypt\u002Fawesome-skill-forge\u002Fblob\u002Fmaster\u002Fskills\u002Fxurl\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M785W98EDN4EFJXJPMQEHCRC","touchdesigner-mcp","Control a running TouchDesigner instance via twozero MCP — create operators, set parameters, wire connections, execute Python, build real-time visuals. 36 native tools.","https:\u002F\u002Fgithub.com\u002FLord1Egypt\u002Fawesome-skill-forge\u002Fblob\u002Fmaster\u002Fskills\u002Ftouchdesigner-mcp\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":19},"01M785W5S9FP3QBH2V191E025P","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002FLord1Egypt\u002Fawesome-skill-forge\u002Fblob\u002Fmaster\u002Fskills\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":19},"01M785W2AS08NSSH9YATCV6FJB","teams-meeting-pipeline","Operate the Teams meeting summary pipeline via Hermes CLI — summarize meetings, inspect pipeline status, replay jobs, manage Microsoft Graph subscriptions.","https:\u002F\u002Fgithub.com\u002FLord1Egypt\u002Fawesome-skill-forge\u002Fblob\u002Fmaster\u002Fskills\u002Fteams-meeting-pipeline\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":19},"01M785VXKW4T34BTP1SCHW1R7F","systematic-debugging","4-phase root cause debugging: understand bugs before fixing.","https:\u002F\u002Fgithub.com\u002FLord1Egypt\u002Fawesome-skill-forge\u002Fblob\u002Fmaster\u002Fskills\u002Fsystematic-debugging\u002FSKILL.md",[],[]]