Wire Protocol

This document specifies the complete wire format between agent, exchange, and cli. Definitions live in crates/proto, and all three share the same code: fields are defined in one place, so inconsistencies surface at compile time rather than during a production session.

It serves operators inspecting network traffic to diagnose problems, and developers changing the protocol or writing third-party clients.

Contents

• Overview
• Transport and Endpoints
• Protocol Version
• Control Messages
  • DeviceUp (agent → exchange)
  • DeviceDown (exchange → agent)
  • OperatorUp (cli → exchange)
  • OperatorDown (exchange → cli)
  • Shared Data Structures
• Binary Data Frames
• Session ID Translation Between Two Namespaces
• Handshake Sequence
• Error Handling and Failure Categories
• Keepalive and Timeouts
• Backpressure
• Android Screen Control
• End-to-End Sealed Sessions

Overview

The protocol has two layers sharing one WebSocket connection:

Embedding streaming data in JSON as base64 increases its size by approximately 33%. PTY traffic consists of frequent small packets echoing each keystroke, so this overhead directly affects interaction latency. Control messages are infrequent, making JSON readability more valuable: packet captures are immediately understandable.

By convention, each WebSocket binary frame carries exactly one data frame. Extra or missing bytes are errors.

Transport and Endpoints

The two paths use separate credential types. Device tokens cannot access the operator endpoint, and vice versa.

Operator credentials use a request header rather than a URL query string because URLs appear in reverse proxy logs, browser history, and Referer headers. Device tokens use the first frame rather than a request header because some embedded clients cannot easily customize WebSocket handshake headers. The first-frame approach also allows protocol version mismatches and invalid tokens to use the same response message.

Production deployments must use wss://: both credential types appear in plaintext during the handshake.

Protocol Version

pub const PROTOCOL_VERSION: u16 = 1;

Versions are compared in both directions during the handshake. A mismatch causes immediate rejection with an explicit explanation. This prevents version mismatches from appearing as an inexplicable parsing failure in a later message, which would be difficult to diagnose.

• Device side: agent includes its version in register. On a mismatch, the server returns register_ack with ok: false, describes both versions in error, and closes the connection.
• Operator side: the server proactively sends welcome after connection establishment. If the CLI detects a mismatch, it reports an error and instructs the user to upgrade to a version matching server_version.

Unknown message type values are always rejected, never silently interpreted as a known type.

Control Messages

All control messages are JSON objects tagged by the type field (serde internally tagged enums). Tag values and field names use snake_case. session_id is transparently serialized as a plain number.

DeviceUp (agent → exchange)

A second register on an already registered connection is treated as protocol misuse and logged as a warning.

Example:

{"type":"register","protocol_version":1,"device_id":"lab-router","token":"a1b2….c2Vj…","info":{"hostname":"OpenWrt","model":"ath79/generic","openwrt_version":"OpenWrt 23.05.5","kernel":"5.15.137","arch":"mips","agent_version":"0.1.0"}}
{"type":"exec_exit","session_id":31,"code":0,"timed_out":false}

DeviceDown (exchange → agent)

OperatorUp (cli → exchange)

Authentication happens during the WebSocket handshake, so there are no login messages.

The session_id in all four open_* messages is allocated by the CLI itself (see Session ID Translation Between Two Namespaces), allowing subsequent frames immediately after the request without waiting for an ID acknowledgment.

OperatorDown (exchange → cli)

Shared Data Structures

ExecHoldInfo: job_id, kind (exec / download / upload / file), age_secs, session_id?, command_preview, status, timeout_secs, started_at_unix_ms, exit_code?, timed_out. For compatibility, missing kind defaults to exec, and missing age_secs defaults to 0. List responses calculate age from the current clock. Command/file path summaries are limited to approximately 120 characters. Non-exec jobs support kill only, not tail/cat/attach.

DeviceSessionInfo: session_id: u32 (device-side ID), operator_session_id: u32, kind: string, hold: bool, age_secs: u64, deadline_unix_ms: u64. Exchange answers session-list and forced-cancellation requests directly without opening another device session; read-only accounts receive forbidden.

DeviceInfo

Static device information, reported once in register and stored in devices.info. Everything except agent_version is optional: most sources are unavailable when agent runs outside OpenWrt, such as on a development machine or in a container. Registration should still succeed rather than startup failing. Missing fields are omitted during serialization.

Metrics

Collected again for each heartbeat, using only values available directly from /proc without forking subprocesses. Failure to read one item does not fail the whole collection: losing one metric is preferable to failing the whole heartbeat and incorrectly reporting the device offline.

InterfaceStats: { name: string, rx_bytes: u64, tx_bytes: u64 }

Binary Data Frames

 0        1                    5                    9
 +--------+--------------------+--------------------+
 |  kind  |     session_id     |    payload_len     |  payload (payload_len 字节)
 |   u8   |    u32 (大端)      |    u32 (大端)      |
 +--------+--------------------+--------------------+
 └──────────── 9 字节定长头 ────────────┘

Stream Types

If agent receives ExecStdout / ExecStderr downstream, it logs a warning and discards the frame: this indicates an implementation error at the peer.

Design Notes

payload_len is indeed redundant with the WebSocket length field. It remains for two reasons: decoding can detect truncation (a declaration of 11 bytes with only 5 supplied must be rejected rather than silently accepted as a shorter packet), and the frame format can remain unchanged if raw TCP is needed later.

The 1 MiB payload limit is not intended for normal traffic: PTY and command output are incremental streams far smaller than that. It prevents a peer from exhausting server memory with a header claiming a 2 GiB payload.

Empty payloads are valid: zero-byte reads occasionally occur in PTY sessions. Such a frame consists of the 9-byte header alone.

Payloads contain arbitrary bytes, including non-UTF-8 sequences, and must be preserved exactly.

Decoding Errors

A data-frame decoding failure affects only that frame: exchange logs a warning and discards it without disconnecting.

Session ID Translation Between Two Namespaces

This is the part of the protocol most easily misunderstood.

In One Sentence

The initiator allocates session_id, which is unique only within its own connection. Operator and device connections each maintain independent ID namespaces, and exchange translates between them. One logical session has two different IDs.

  remote-cli                    exchange                      remote-agent
                                                        （设备 lab-router）

  ┌────────────┐        ┌────────────────────┐            ┌────────────┐
  │ 运维编号   │        │      路由表        │            │ 设备编号   │
  │  空间      │        │                    │            │  空间      │
  │            │        │  lab-router:       │            │            │
  │  session 7 │───────▶│    31 → (op-A, 7)  │───────────▶│ session 31 │
  │            │◀───────│                    │◀───────────│            │
  └────────────┘        │  op-A:             │            └────────────┘
                        │    7 → (lab-router,│
   由 CLI 自己的        │         31)        │             由 exchange
   SessionIdGenerator   └────────────────────┘             的 DeviceHandle
   分配（1,2,3…）                                          分配（1,2,3…）

Why This Design

Avoid a round trip. If the server allocated all IDs, the CLI would have to wait for an acknowledgment saying its session ID is 31 before sending keyboard input after opening a PTY. CLI allocation allows data frames immediately after open_pty.

Avoid ID collisions between operators. Two operators may simultaneously use local ID 1 on the same device. The device must distinguish them to avoid sending one person's output to another. exchange allocates device-side IDs separately for each device connection, mapping the two session 1 instances to device-side 31 and 32.

Keep operator behavior from influencing device-side IDs. exchange controls device-side IDs entirely; no operator can specify or guess them.

Two Tables

exchange maintains one table per connection, with both tables always updated together:

The reverse index enables cleanup: when an operator disconnects, the server must find and terminate its sessions on every device, otherwise processes and PTYs with no output recipient would remain.

Where Translation Happens

Data-frame translation is the only rewrite: exchange does not parse, buffer, or change any payload byte. DataFrame::with_session_id replaces the ID in place without reallocating the payload.

Allocation Rules

Both allocators increment from 1 and skip 0, the no-session sentinel used, for example, in connection-level error messages.

• CLI (SessionIdGenerator): wraps on overflow and skips 0. Using four billion sessions in one connection is unrealistic, but wrapping is better suited to a long-running process than a panic.
• Device (DeviceHandle::open_session): after wrapping, if an ID belongs to a live session, continues to the next free slot.

exchange rejects operator ID 0 and reuse of an active session ID. Device connection validity, concurrency limits, and registration in both indexes are checked within the same critical section, preventing simultaneous session opens from multiple operators from exceeding the limit.

When No Route Exists

Silently discard without an error. A peer still sending trailing data just after session closure is a normal race. This applies in both directions:

• Device replies for a session that no longer exists → debug log; the operator may already have disconnected.
• Operator sends data for an unowned session → trace log and discard.

Session Cleanup Paths

The reverse index binds to a specific device connection; it does not redirect by device ID to a new connection after reconnection. Multiple operator connections for one account each own an index, so disconnecting one does not cancel another connection's sessions.

Acknowledgments for explicit cancellation: exec returns exec_exit (code: null, timed_out: false), PTY returns pty_closed (code: null), and ubus returns session_failed (reason: device_error). This confirms exchange has closed the session, not that the remote process has exited; subsequent trailing data and exit messages from the device are discarded.

At the concurrency limit, Exchange first reclaims routes whose operators are gone or whose deadlines have passed under the same session lock, then checks the limit again. The exec deadline is the actual dispatched time limit plus a 15-second grace period; files use 3600 seconds plus 15 seconds; other types use 24 hours. Hold exec within its deadline is not killed when the operator leaves. Reclamation/explicit kill_device_session sends the corresponding cancellation to the device and immediately removes both indexes without waiting for a device response. Available operators receive exec_exit (timed_out:true) for exec or session_failed (device_error, session reclaimed) for other types. Audit updates are asynchronous and best effort.

Device Reconnection and epoch

Each device connection receives a random epoch (UUID) during registration. When the same device reconnects, the new connection replaces the old one, rather than being rejected. It is impossible to reliably distinguish a duplicate connection from a dead connection the server has not detected yet (a half-open connection); rejecting the new one would prevent reconnection until the old one timed out.

Cleanup for the old connection may execute after registration of the new one. Deregistration must therefore check epoch: remove the connection table entry only if its current epoch matches the connection being cleaned up. Without this check, old-connection cleanup would immediately mark the newly connected device offline.

Handshake Sequence

Device Side

  remote-agent                                        exchange
      │                                                 │
      │  ①  WebSocket 握手（wss://host/ws/device）      │
      │────────────────────────────────────────────────▶│
      │◀────────────────────────────────────────────────│
      │            101 Switching Protocols              │
      │                                                 │
      │                                          ┌──────┴──────┐
      │                                          │ 启动 15 秒  │
      │                                          │ 注册计时器  │
      │                                          └──────┬──────┘
      │  ②  {"type":"register","protocol_version":1,    │
      │      "device_id":"…","token":"…","info":{…}}    │
      │────────────────────────────────────────────────▶│
      │                                                 │
      │                              ┌──────────────────┴──────────────────┐
      │                              │ a. 比对 protocol_version            │
      │                              │ b. split_token → 前缀查 devices 表  │
      │                              │ c. 校验 enabled                     │
      │                              │ d. argon2 verify（前缀不符也照做，  │
      │                              │    避免用耗时区分前缀是否存在）     │
      │                              └──────────────────┬──────────────────┘
      │                                                 │
      │  ③  {"type":"register_ack","ok":true,           │
      │      "error":null,"heartbeat_interval_secs":30} │
      │◀────────────────────────────────────────────────│
      │                                                 │
      │                                          ┌──────┴──────┐
      │                                          │ 登记连接表  │
      │                                          │ 顶替旧连接  │
      │                                          │ 回写 info 与│
      │                                          │ last_seen_at│
      │                                          └──────┬──────┘
      │                                                 │
      │  ④  退避曲线复位，写状态文件 state=connected     │
      │                                                 │
      │══════════════ 转发循环 ═══════════════════════════│
      │                                                 │
      │  heartbeat（每 30 秒，携带 Metrics）             │
      │────────────────────────────────────────────────▶│
      │  WebSocket Ping（随每次 heartbeat 发出）         │
      │────────────────────────────────────────────────▶│
      │                                     WebSocket Pong
      │◀────────────────────────────────────────────────│
      │                                     WebSocket Ping（每 30 秒）
      │◀────────────────────────────────────────────────│
      │  Pong（tungstenite 排队，agent 显式 flush）       │
      │────────────────────────────────────────────────▶│

Failure paths:

• First frame is not text, is invalid JSON, or is not register → log a warning and close immediately without an ack.
• No register within 15 seconds → close.
• Version mismatch → return register_ack{ok:false, error:"协议版本不匹配：设备为 X，服务端为 Y"} and close.
• Invalid token / nonexistent device / disabled device → always return the same "设备 ID 或令牌无效" and close. Do not distinguish causes: doing so would make the device inventory enumerable.

agent handles these two failure categories separately:

Backoff resets only after successful registration, not merely TCP establishment. Otherwise, immediate rejection after connection would degrade into dense retries at millisecond intervals.

Jitter is essential. When a server restarts, hundreds or thousands of devices detect the disconnect at once. With an identical deterministic retry curve, they remain synchronized and hit the newly started server in waves, turning a restart into a sustained cascading failure.

Operator Side

  remote-cli                                          exchange
      │                                                 │
      │  ①  WebSocket 握手（wss://host/ws/operator）    │
      │      Authorization: Bearer <prefix>.<secret>    │
      │────────────────────────────────────────────────▶│
      │                                                 │
      │                              ┌──────────────────┴──────────────────┐
      │                              │ 在 upgrade 之前完成鉴权：           │
      │                              │ split_token → 前缀查 operators 表   │
      │                              │ → argon2 verify → 校验 enabled      │
      │                              └──────────────────┬──────────────────┘
      │                                                 │
      │      鉴权失败 → HTTP 401「凭证无效」（不升级）   │
      │◀────────────────────────────────────────────────│
      │                                                 │
      │◀────────────────────────────────────────────────│
      │            101 Switching Protocols              │
      │                                                 │
      │  ②  {"type":"welcome","protocol_version":1,     │
      │      "server_version":"0.1.0"}                  │
      │◀────────────────────────────────────────────────│
      │                                                 │
      │  ③  比对 protocol_version（15 秒超时）          │
      │      不一致 → 报错退出，提示升级 CLI            │
      │                                                 │
      │══════════════ 可以发起会话 ═══════════════════════│
      │                                                 │
      │  {"type":"open_pty","session_id":1,             │
      │   "device_id":"lab-router","rows":40,"cols":120}│
      │────────────────────────────────────────────────▶│
      │                                                 │──▶ 查设备在线
      │                                                 │──▶ 查并发上限
      │                                                 │──▶ 查 readonly 角色
      │                                                 │──▶ 写审计（先于下发）
      │                                                 │──▶ 分配设备侧编号
      │  二进制帧（PtyData, session 1, 键盘输入）        │
      │────────────────────────────────────────────────▶│──▶ 改写为设备侧编号
      │                                                 │
      │  二进制帧（PtyData, session 1, 屏幕输出）        │
      │◀────────────────────────────────────────────────│◀── 改写为运维侧编号

The CLI can send the next data frame without waiting for any acknowledgment. This is precisely why the initiator allocates session IDs.

Error Handling and Failure Categories

SessionFailure

Used in the reason field of session_failed; JSON values use snake_case.

is_retryable() deliberately excludes forbidden and device_error: retrying a permission issue or internal device error is pointless and only causes unnecessary CLI activity.

The CLI appends an actionable hint after the original message. For example, device_offline suggests checking device status with remote-cli device list. device_error is the exception: the device's own error is the most accurate explanation and should not be diluted by boilerplate.

Failure Semantics by Layer

The principle: keep failures local whenever possible, since a device connection may carry simultaneous sessions for multiple operators.

Keepalive and Timeouts

An idle shell client still replies with Pong. agent explicitly flushes the response automatically queued by the library without waiting for the next business heartbeat, so lack of keyboard input does not cause a false disconnect (#42). --heartbeat-secs must be greater than 0; invalid values are rejected during startup argument parsing.

Backpressure

exchange maintains a bounded send queue of 512 messages for each connection.

A full queue means the peer cannot keep up, usually because a PTY connection has stalled or the client is suspended. The response is to disconnect that connection proactively rather than keep buffering: unbounded buffering would turn one stalled client into a memory leak affecting the whole service. 512 messages provide ample headroom for interactive PTY traffic (normally queue length is in single digits); hitting the limit generally means the peer has stopped consuming.

Delivery always uses nonblocking try_send. The routing path holds a connection-table lock; any await would tie lock duration to peer consumption speed, allowing a slow client to stall the entire table.

The two delivery failures are handled differently:

Connection close signals bypass the send queue, allowing cleanup even when the queue is full or writes are blocked. Device replacement after reconnect and token rotation use the same mechanism.

agent also has an outbound queue of capacity 256. A full queue means the network cannot send data at all, so further accumulation is pointless; disconnect and reconnect. When the PTY input queue (64) is full, keystrokes are simply discarded: accumulating small interactive inputs serves no purpose.

Android Screen Control

Defined in crates/proto/src/android.rs, using the existing two-namespace session routing, authentication, concurrency limits, and auditing. The request in open_android / android_req is a JSON object discriminated by action:

AndroidResult is discriminated by kind; all fields are required:

Nodes contain strings class_name, package, view_id, text, description, bounds: [left, top, right, bottom] (signed screen pixel coordinates), and booleans clickable, editable, focused, password. At most 200 nodes, with each string limited to 256 UTF-16 units. truncated indicates only the node limit. Password values and descriptions are empty, and the agent's own configuration subtree is omitted.

Screenshot longest edge is limited to 1600 pixels. width/height are image dimensions; screen_width/screen_height are dimensions before scaling. Gesture coordinates use the original dimensions. JSON returned by the Java bridge is limited to 2 MiB, with an additional 1024 bytes allowed for the WebSocket envelope. No new binary frame type is added.

{"type":"open_android","session_id":1,"device_id":"phone","request":{"action":"input_text","text":"Hello\nWorld"}}
{"type":"android_req","session_id":31,"request":{"action":"input_text","text":"Hello\nWorld"}}
{"type":"android_resp","session_id":31,"result":{"kind":"ok"}}
{"type":"android_resp","session_id":1,"result":{"kind":"ok"}}

screen_capture_enabled indicates currently available screenshot authorization (MediaProjection or Android 11+ accessibility); all actual screen operations still require controls_enabled. Insufficient permission, inaccessible windows, and failed operations return agent session_error, translated by Exchange to session_failed. Non-Android agents return an explicit unsupported error.

Agent handles at most 4 Android requests concurrently and encodes only one screenshot at a time. Java operations wait at most 10 seconds (up to 5 seconds for a screen frame); agent request timeout is 15 seconds, CLI timeout 20 seconds. On operator cancellation or disconnection, Exchange sends android_cancel; agent closes the local socket and the APK cancels queued work that has not run yet. Completed gestures or text changes are not rolled back.

The audit action is android; detail stores the action and coordinates. input_text stores only characters (Unicode character count), and successful results record only outcome: "completed". Text content, UI trees, and images are not written to audit records. readonly accounts cannot initiate even status, tree, or screenshots.

File Session Extension

Agents supporting file operations advertise the files capability. Exchange translates open_file {session_id, device_id, request} to file_req {session_id, request}. request.action is list, download, or upload; field definitions are in crates/proto/src/files.rs.

Bidirectional control messages use JSON: the controller sends file_input {session_id, input}, and the device returns file_event {session_id, event}. input.kind is chunk {offset, data}, ack {offset}, or finish; event.kind is ready {size}, chunk {offset, data}, progress {received}, complete {size}, or list {path, entries, truncated}. Chunk contents use standard Base64, each chunk is at most 64 KiB, and the sender waits for acknowledgment before continuing.

A download's initial ready requires ack {offset: 0}; subsequent chunks acknowledge the cumulative byte count. Upload starts with ready, and each progress drives the next chunk; after sending everything, finish commits it. complete and list are terminal messages; errors use existing session_error / session_failed. close_session sends file_cancel; cancellation and disconnection both remove session routes. File messages are forwarded only on file sessions, preserving isolation between operator and device session IDs.

Protocol version remains 1; the new messages require matching Exchange and Agent upgrades. For operation limits and LAN interfaces, see File Transfer and Direct LAN Connections.

Multi-Platform Extensions

Protocol version remains 1. Registration info adds optional platform (openwrt / linux / freebsd / netbsd / windows / android / macos), os_version, and capabilities (exec / pty / ubus / android_control / files / sensors / net). OpenWrt reports openwrt_version.

Heartbeat metrics.unavailable is an optional string array identifying currently unavailable uptime, load, memory, cpu, and interfaces metrics. If absent, parse using the old format. Values corresponding to listed metrics are only compatibility placeholders; clients should show them as unavailable. Windows does not provide Unix load averages, and Android may restrict some /proc sources through system permissions.

Commands are interpreted by the target device's shell: /bin/sh on Linux/OpenWrt/macOS, PowerShell on Windows, and /system/bin/sh on Android. PTY frame format and control messages are identical across the three systems; Windows uses ConPTY. Non-OpenWrt devices return session_error for ubus requests.

End-to-End Sealed Sessions

Protocol version remains 1. New messages: OperatorUp open_sealed {session_id, device_id, hello}, sealed_input {session_id, packet}; DeviceDown sealed_open {session_id, hello}, sealed_input, sealed_close {session_id, detach}; DeviceUp / OperatorDown sealed_accept {session_id, accept}, sealed_event {session_id, packet, last}. New binary frame type 4 = Sealed (payload counter ‖ AEAD(inner_kind ‖ bytes)), DeviceInfo.seal_key, and capability sealed.

For sealed sessions, Exchange performs only ID translation, concurrency limiting, and routing. It neither parses nor audits payloads (audit records contain only public keys, authorization level and expiry, and final outcome), and does not decide permissions beyond checking whether the account can see the device. Routes are released on last:true. Sealed routes accept only Sealed frames; ordinary routes reject Sealed frames. agent discards any plaintext sent to sealed session IDs. For authorization, handshake, key derivation, threat model, see End-to-End Sealed Sessions.