10. 心跳监控系统 _Heartbeat
10.1 设计原理
#mermaid-svg-cTO2GHoGNsGriCif{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-cTO2GHoGNsGriCif .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-cTO2GHoGNsGriCif .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-cTO2GHoGNsGriCif .error-icon{fill:#552222;}#mermaid-svg-cTO2GHoGNsGriCif .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-cTO2GHoGNsGriCif .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-cTO2GHoGNsGriCif .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-cTO2GHoGNsGriCif .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-cTO2GHoGNsGriCif .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-cTO2GHoGNsGriCif .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-cTO2GHoGNsGriCif .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-cTO2GHoGNsGriCif .marker{fill:#333333;stroke:#333333;}#mermaid-svg-cTO2GHoGNsGriCif .marker.cross{stroke:#333333;}#mermaid-svg-cTO2GHoGNsGriCif svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-cTO2GHoGNsGriCif p{margin:0;}#mermaid-svg-cTO2GHoGNsGriCif .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-cTO2GHoGNsGriCif .cluster-label text{fill:#333;}#mermaid-svg-cTO2GHoGNsGriCif .cluster-label span{color:#333;}#mermaid-svg-cTO2GHoGNsGriCif .cluster-label span p{background-color:transparent;}#mermaid-svg-cTO2GHoGNsGriCif .label text,#mermaid-svg-cTO2GHoGNsGriCif span{fill:#333;color:#333;}#mermaid-svg-cTO2GHoGNsGriCif .node rect,#mermaid-svg-cTO2GHoGNsGriCif .node circle,#mermaid-svg-cTO2GHoGNsGriCif .node ellipse,#mermaid-svg-cTO2GHoGNsGriCif .node polygon,#mermaid-svg-cTO2GHoGNsGriCif .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-cTO2GHoGNsGriCif .rough-node .label text,#mermaid-svg-cTO2GHoGNsGriCif .node .label text,#mermaid-svg-cTO2GHoGNsGriCif .image-shape .label,#mermaid-svg-cTO2GHoGNsGriCif .icon-shape .label{text-anchor:middle;}#mermaid-svg-cTO2GHoGNsGriCif .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-cTO2GHoGNsGriCif .rough-node .label,#mermaid-svg-cTO2GHoGNsGriCif .node .label,#mermaid-svg-cTO2GHoGNsGriCif .image-shape .label,#mermaid-svg-cTO2GHoGNsGriCif .icon-shape .label{text-align:center;}#mermaid-svg-cTO2GHoGNsGriCif .node.clickable{cursor:pointer;}#mermaid-svg-cTO2GHoGNsGriCif .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-cTO2GHoGNsGriCif .arrowheadPath{fill:#333333;}#mermaid-svg-cTO2GHoGNsGriCif .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-cTO2GHoGNsGriCif .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-cTO2GHoGNsGriCif .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-cTO2GHoGNsGriCif .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-cTO2GHoGNsGriCif .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-cTO2GHoGNsGriCif .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-cTO2GHoGNsGriCif .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-cTO2GHoGNsGriCif .cluster text{fill:#333;}#mermaid-svg-cTO2GHoGNsGriCif .cluster span{color:#333;}#mermaid-svg-cTO2GHoGNsGriCif div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-cTO2GHoGNsGriCif .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-cTO2GHoGNsGriCif rect.text{fill:none;stroke-width:0;}#mermaid-svg-cTO2GHoGNsGriCif .icon-shape,#mermaid-svg-cTO2GHoGNsGriCif .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-cTO2GHoGNsGriCif .icon-shape p,#mermaid-svg-cTO2GHoGNsGriCif .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-cTO2GHoGNsGriCif .icon-shape rect,#mermaid-svg-cTO2GHoGNsGriCif .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-cTO2GHoGNsGriCif .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-cTO2GHoGNsGriCif .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-cTO2GHoGNsGriCif :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
是(有进展)
否(无进展)
否
是 (idle>=15cycles=450sin_tool>=40cycles=1200s)
_Heartbeat(child, parent, idx)
last_seen = {iter:0, tool:None, ts:None, stale:0}
start()
schedule(self.tick, _HEARTBEAT_INTERVAL=30s)通过periodic_schedulerNOT one daemon thread per child
tick() 每30秒
child.get_activity_summary()获取(iter, tool, activity_ts)
iter变了 ORtool变了 ORactivity_ts推进?
last_seen.update(iter, tool, stale=0)更新ts
last_seen.stale += 1
touch(desc)刷新parent._last_activity_ts防止gateway不活动超时
stale >= 阈值?
return False停止心跳让gateway超时触发
return None继续下一周期
10.2 停滞检测阈值
| idle (无tool) | 15 cycles | 450秒 (7.5分钟) | tight:真正wedged的child不应掩盖gateway超时 |
| in-tool (有tool) | 40 cycles | 1200秒 (20分钟) | 高:合法的长工具需要时间完成 |
进度定义(三选一即视为有进展):
last_activity_ts在慢模型等待时也会刷新(direct_api_call心跳),所以慢模型不会在idle阈值看起来停滞。
10.3 关键设计:共享periodic_scheduler
def start(self) –> None:
from agent.periodic_scheduler import schedule
from tools.delegate_tool import _HEARTBEAT_INTERVAL
self.handle = schedule(self.tick, _HEARTBEAT_INTERVAL)
- 不是每个child一个daemon thread:所有心跳通过共享的agent.periodic_scheduler调度
- 首次schedule失败(OS线程耗尽)→handle为None→stop()是no-op
- stop()调用handle.cancel(wait=5),镜像旧thread join:一个in-flight tick完成
11. 结果构建与摘要预算
11.1 _finalize_child_results() 完整终结化
def _finalize_child_results(results, task_list, children, parent_agent):
with _parent_finalization_lock(parent_agent): # per-parent锁序列化生命周期副作用
_apply_summary_budget(results, parent_agent) # 1. 摘要预算
child_by_index = {index: child for index, _task, child in children}
_notify_memory_manager(results, task_list, child_by_index, parent_agent) # 2. 记忆通知
children_cost_total = _fire_subagent_stop_hooks(results, child_by_index, parent_agent) # 3. 钩子+成本弹出
_rollup_children_cost(parent_agent, children_cost_total) # 4. 成本汇总
11.2 摘要预算算法
#mermaid-svg-NVDk3vx8AatbfWOU{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-NVDk3vx8AatbfWOU .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-NVDk3vx8AatbfWOU .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-NVDk3vx8AatbfWOU .error-icon{fill:#552222;}#mermaid-svg-NVDk3vx8AatbfWOU .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-NVDk3vx8AatbfWOU .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-NVDk3vx8AatbfWOU .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-NVDk3vx8AatbfWOU .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-NVDk3vx8AatbfWOU .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-NVDk3vx8AatbfWOU .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-NVDk3vx8AatbfWOU .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-NVDk3vx8AatbfWOU .marker{fill:#333333;stroke:#333333;}#mermaid-svg-NVDk3vx8AatbfWOU .marker.cross{stroke:#333333;}#mermaid-svg-NVDk3vx8AatbfWOU svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-NVDk3vx8AatbfWOU p{margin:0;}#mermaid-svg-NVDk3vx8AatbfWOU .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-NVDk3vx8AatbfWOU .cluster-label text{fill:#333;}#mermaid-svg-NVDk3vx8AatbfWOU .cluster-label span{color:#333;}#mermaid-svg-NVDk3vx8AatbfWOU .cluster-label span p{background-color:transparent;}#mermaid-svg-NVDk3vx8AatbfWOU .label text,#mermaid-svg-NVDk3vx8AatbfWOU span{fill:#333;color:#333;}#mermaid-svg-NVDk3vx8AatbfWOU .node rect,#mermaid-svg-NVDk3vx8AatbfWOU .node circle,#mermaid-svg-NVDk3vx8AatbfWOU .node ellipse,#mermaid-svg-NVDk3vx8AatbfWOU .node polygon,#mermaid-svg-NVDk3vx8AatbfWOU .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-NVDk3vx8AatbfWOU .rough-node .label text,#mermaid-svg-NVDk3vx8AatbfWOU .node .label text,#mermaid-svg-NVDk3vx8AatbfWOU .image-shape .label,#mermaid-svg-NVDk3vx8AatbfWOU .icon-shape .label{text-anchor:middle;}#mermaid-svg-NVDk3vx8AatbfWOU .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-NVDk3vx8AatbfWOU .rough-node .label,#mermaid-svg-NVDk3vx8AatbfWOU .node .label,#mermaid-svg-NVDk3vx8AatbfWOU .image-shape .label,#mermaid-svg-NVDk3vx8AatbfWOU .icon-shape .label{text-align:center;}#mermaid-svg-NVDk3vx8AatbfWOU .node.clickable{cursor:pointer;}#mermaid-svg-NVDk3vx8AatbfWOU .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-NVDk3vx8AatbfWOU .arrowheadPath{fill:#333333;}#mermaid-svg-NVDk3vx8AatbfWOU .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-NVDk3vx8AatbfWOU .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-NVDk3vx8AatbfWOU .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-NVDk3vx8AatbfWOU .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-NVDk3vx8AatbfWOU .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-NVDk3vx8AatbfWOU .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-NVDk3vx8AatbfWOU .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-NVDk3vx8AatbfWOU .cluster text{fill:#333;}#mermaid-svg-NVDk3vx8AatbfWOU .cluster span{color:#333;}#mermaid-svg-NVDk3vx8AatbfWOU div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-NVDk3vx8AatbfWOU .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-NVDk3vx8AatbfWOU rect.text{fill:none;stroke-width:0;}#mermaid-svg-NVDk3vx8AatbfWOU .icon-shape,#mermaid-svg-NVDk3vx8AatbfWOU .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-NVDk3vx8AatbfWOU .icon-shape p,#mermaid-svg-NVDk3vx8AatbfWOU .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-NVDk3vx8AatbfWOU .icon-shape rect,#mermaid-svg-NVDk3vx8AatbfWOU .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-NVDk3vx8AatbfWOU .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-NVDk3vx8AatbfWOU .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-NVDk3vx8AatbfWOU :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
否
是
_apply_summary_budget(results, parent_agent)
筛选有summary的条目
static_ceiling = cfg.max_summary_chars默认24000
_parent_summary_char_budget()动态headroom预算
context_length = compressor.context_length
_parent_prompt_size_tokens()parent的当前prompt大小
headroom = context_length – used – max_tokens
per_summary = headroom * 0.5 / n_summaries * 4 chars/token
max(2000, per_summary)
cap = min(static_ceiling, dynamic_budget)
for each summary:
len(summary) > cap?
保持原样
_trim_summary_with_footer()
75% head + 25% tail按行边界对齐
_spill_summary_to_file()全文写入cache/delegation/
附加footer:read_file path=… offset=…
entry.summary_truncated = Trueentry.summary_full_path = spill_path
完成
关键参数:
- _SUMMARY_HEADROOM_FRACTION = 0.5:父代理剩余上下文headroom的50%分配给摘要
- _MIN_SUMMARY_CHARS = 2000:即使父代理接近满,单摘要至少2000字符
- DEFAULT_MAX_SUMMARY_CHARS = 24000:硬性上限
- cap = min(static_ceiling, dynamic_budget):取较小值
_parent_prompt_size_tokens() 的关键修正(L225-236):
size = getattr(parent_agent, "_last_prompt_size_tokens", None) # aggregator的最后一次prompt_tokens
if isinstance(size, (int, float)) and size > 0:
return int(size)
last_usage = getattr(parent_agent, "_last_turn_usage", None) or {}
used = last_usage.get("prompt_tokens") if isinstance(last_usage, dict) else None
# 注意:session_prompt_tokens是整个会话的累计SUM,几百次调用后超过任何上下文窗口
# 用它会让每个摘要都坍缩到2000字符floor
11.3 _fire_subagent_stop_hooks() 钩子+成本弹出
def _fire_subagent_stop_hooks(results, child_by_index, parent_agent) –> float:
children_cost_total = 0.0
for entry in results:
child_role = entry.pop("_child_role", None) # 弹出模型不可见字段
child_cost = entry.pop("_child_cost_usd", 0.0)
if child_cost:
children_cost_total += float(child_cost)
if invoke_hook is not None:
child = child_by_index.get(entry.get("task_index", –1))
invoke_hook(
"subagent_stop",
parent_session_id=getattr(parent_agent, "session_id", None),
parent_turn_id=getattr(parent_agent, "_current_turn_id", "") or "",
child_session_id=getattr(child, "session_id", None),
child_role=child_role,
child_summary=entry.get("summary"),
child_status=entry.get("status"),
tool_call_history=_subagent_stop_tool_call_history(entry.get("tool_trace")),
duration_ms=int((entry.get("duration_seconds") or 0) * 1000),
)
return children_cost_total
- _child_role和_child_cost_usd在序列化前弹出(模型不可见)
- tool_call_history经过元数据only脱敏(输入摘要重新清洗,URL secrets剥离)
- subagent_stop钩子让插件可以监控子代理生命周期
11.4 成本汇总
def _rollup_children_cost(parent_agent, children_cost_total: float):
if children_cost_total <= 0.0:
return
current = float(getattr(parent_agent, "session_estimated_cost_usd", 0.0) or 0.0)
parent_agent.session_estimated_cost_usd = current + children_cost_total
if getattr(parent_agent, "session_cost_source", "none") in {None, "", "none"}:
parent_agent.session_cost_source = "subagent"
if getattr(parent_agent, "session_cost_status", "unknown") in {None, "", "unknown"}:
parent_agent.session_cost_status = "estimated"
- 子代理成本折入父代理的会话总成本
- source/status仅在父代理没有自己的值时设置(不覆盖)
12. 控制面:list / steer / stop
12.1 控制面架构
#mermaid-svg-hGZxRyafEQwD0Wms{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-hGZxRyafEQwD0Wms .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-hGZxRyafEQwD0Wms .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-hGZxRyafEQwD0Wms .error-icon{fill:#552222;}#mermaid-svg-hGZxRyafEQwD0Wms .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-hGZxRyafEQwD0Wms .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-hGZxRyafEQwD0Wms .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-hGZxRyafEQwD0Wms .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-hGZxRyafEQwD0Wms .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-hGZxRyafEQwD0Wms .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-hGZxRyafEQwD0Wms .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-hGZxRyafEQwD0Wms .marker{fill:#333333;stroke:#333333;}#mermaid-svg-hGZxRyafEQwD0Wms .marker.cross{stroke:#333333;}#mermaid-svg-hGZxRyafEQwD0Wms svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-hGZxRyafEQwD0Wms p{margin:0;}#mermaid-svg-hGZxRyafEQwD0Wms .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-hGZxRyafEQwD0Wms .cluster-label text{fill:#333;}#mermaid-svg-hGZxRyafEQwD0Wms .cluster-label span{color:#333;}#mermaid-svg-hGZxRyafEQwD0Wms .cluster-label span p{background-color:transparent;}#mermaid-svg-hGZxRyafEQwD0Wms .label text,#mermaid-svg-hGZxRyafEQwD0Wms span{fill:#333;color:#333;}#mermaid-svg-hGZxRyafEQwD0Wms .node rect,#mermaid-svg-hGZxRyafEQwD0Wms .node circle,#mermaid-svg-hGZxRyafEQwD0Wms .node ellipse,#mermaid-svg-hGZxRyafEQwD0Wms .node polygon,#mermaid-svg-hGZxRyafEQwD0Wms .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-hGZxRyafEQwD0Wms .rough-node .label text,#mermaid-svg-hGZxRyafEQwD0Wms .node .label text,#mermaid-svg-hGZxRyafEQwD0Wms .image-shape .label,#mermaid-svg-hGZxRyafEQwD0Wms .icon-shape .label{text-anchor:middle;}#mermaid-svg-hGZxRyafEQwD0Wms .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-hGZxRyafEQwD0Wms .rough-node .label,#mermaid-svg-hGZxRyafEQwD0Wms .node .label,#mermaid-svg-hGZxRyafEQwD0Wms .image-shape .label,#mermaid-svg-hGZxRyafEQwD0Wms .icon-shape .label{text-align:center;}#mermaid-svg-hGZxRyafEQwD0Wms .node.clickable{cursor:pointer;}#mermaid-svg-hGZxRyafEQwD0Wms .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-hGZxRyafEQwD0Wms .arrowheadPath{fill:#333333;}#mermaid-svg-hGZxRyafEQwD0Wms .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-hGZxRyafEQwD0Wms .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-hGZxRyafEQwD0Wms .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hGZxRyafEQwD0Wms .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-hGZxRyafEQwD0Wms .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hGZxRyafEQwD0Wms .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-hGZxRyafEQwD0Wms .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-hGZxRyafEQwD0Wms .cluster text{fill:#333;}#mermaid-svg-hGZxRyafEQwD0Wms .cluster span{color:#333;}#mermaid-svg-hGZxRyafEQwD0Wms div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-hGZxRyafEQwD0Wms .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-hGZxRyafEQwD0Wms rect.text{fill:none;stroke-width:0;}#mermaid-svg-hGZxRyafEQwD0Wms .icon-shape,#mermaid-svg-hGZxRyafEQwD0Wms .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hGZxRyafEQwD0Wms .icon-shape p,#mermaid-svg-hGZxRyafEQwD0Wms .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-hGZxRyafEQwD0Wms .icon-shape rect,#mermaid-svg-hGZxRyafEQwD0Wms .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hGZxRyafEQwD0Wms .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-hGZxRyafEQwD0Wms .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-hGZxRyafEQwD0Wms :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
list
steer
stop
是
否
是
否
模型调用 delegate_task(action=…)
_handle_control_action()
action
_list_payload(parent_agent)列出当前会话的活跃子代理
检查 subagent_id + message
检查 subagent_id
_owns_subagent_record()Tier1: weakref链Tier2: durable lineage
{subagent_id, parent_id, goal, model,status, running_seconds, accepting_steer,live_transcript}
返回JSON
_owns_subagent_record?+ record.accepting_steer?
steer_subagent(sid, message)→ agent.steer(text)
错误:不在此会话的spawn树中
返回:steer已排队子代理在下次迭代边界看到
_owns_subagent_record?
interrupt_subagent(sid)→ request_hard_interrupt(agent)
错误:不在此会话的spawn树中
返回:interrupt_requested子代理在下次迭代边界停止部分结果仍返回
12.2 steer_subagent() 详细逻辑
def steer_subagent(subagent_id, text, *, owner_session_id=None, owner_transport=None, owner_session_record=None):
if not text or not text.strip():
return False
with _active_subagents_lock:
record = _active_subagents.get(subagent_id)
if not record or not record.get("accepting_steer", False):
return False # 不存在或已关闭steer接受
# 权限验证:gateway调用者必须匹配owner_session_id + transport + session_record
if owner_session_id is not None and (
record.get("owner_session_id") != owner_session_id
or owner_transport is None
or not _subagent_transport_matches(record, owner_transport)
or owner_session_record is None
or record.get("owner_session_record") is not owner_session_record
):
return False # 权限不匹配
agent = record.get("agent")
if agent is None:
return False
try:
return bool(agent.steer(text)) # AIAgent.steer()追加到子代理最后的工具结果
except Exception:
return False
steer语义:
- 不切断当前工具调用:text追加到子代理下次迭代边界的最后工具结果
- 线性化:注册表锁保证acceptance和delivery的原子性
- missed_steer:如果子代理在delivery边界前完成,text进入entry的missed_steer字段
12.3 _owns_subagent_record() 双层权限
def _owns_subagent_record(record, parent_agent) –> bool:
# Tier 1: 身份——_delegate_parent_ref weakref链到达parent_agent
if _is_descendant_of(record.get("agent"), parent_agent):
return True
# Tier 2: 持久lineage——owner_agent_session_id匹配(跨parent_agent重建)
owner_sid = str(record.get("owner_agent_session_id") or "")
parent_sid = str(getattr(parent_agent, "session_id", "") or "")
if not owner_sid or not parent_sid:
return False
if owner_sid == parent_sid:
return True
# 压缩旋转:比较lineage tips
return _resolve_session_lineage(owner_sid, parent_agent) in {parent_sid, ...}
Tier 2存在的原因:
- CLI在会话中设置self.agent = None(路由变更、凭证刷新、/model、MoA one-shots)并构建新AIAgent
- 子代理保持对旧AIAgent的weakref,identity chain断裂
- Delivery按持久session_id路由,控制必须使用同一spine
12.4 _subagent_transport_matches() 传输权限
def _subagent_transport_matches(record, transport) –> bool:
if record.get("owner_transport") is None:
return False # 非gateway会话无RPC权限
owner = record.get("owner_session_record")
bound = owner.get("transport") if isinstance(owner, dict) else record.get("owner_transport")
return bound is transport or (isinstance(bound, FanoutTransport) and bound.contains(transport))
- 权限跟随拥有会话的LIVE传输槽(读取时检查,非记录时快照)
- 每个reattach路径(prompt.submit、queued drain、resume、activate、viewer failover)都修改传输槽
- per-record副本需要在每个reattach站点同步,有两个被遗漏(#106663)
13. 工具集隔离机制
13.1 被阻止的工具
DELEGATE_BLOCKED_TOOLS = frozenset([
"delegate_task", # 无递归委派(leaf角色)
"clarify", # 无用户交互
"memory", # 无MEMORY.md写入
"send_message", # 无跨平台副作用
"cronjob_manage", # 无以父代理名义调度更多工作
])
13.2 _resolve_child_toolsets() 解析流程
#mermaid-svg-qhm4SwA7HRWYCW2c{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-qhm4SwA7HRWYCW2c .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-qhm4SwA7HRWYCW2c .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-qhm4SwA7HRWYCW2c .error-icon{fill:#552222;}#mermaid-svg-qhm4SwA7HRWYCW2c .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-qhm4SwA7HRWYCW2c .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-qhm4SwA7HRWYCW2c .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-qhm4SwA7HRWYCW2c .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-qhm4SwA7HRWYCW2c .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-qhm4SwA7HRWYCW2c .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-qhm4SwA7HRWYCW2c .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-qhm4SwA7HRWYCW2c .marker{fill:#333333;stroke:#333333;}#mermaid-svg-qhm4SwA7HRWYCW2c .marker.cross{stroke:#333333;}#mermaid-svg-qhm4SwA7HRWYCW2c svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-qhm4SwA7HRWYCW2c p{margin:0;}#mermaid-svg-qhm4SwA7HRWYCW2c .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-qhm4SwA7HRWYCW2c .cluster-label text{fill:#333;}#mermaid-svg-qhm4SwA7HRWYCW2c .cluster-label span{color:#333;}#mermaid-svg-qhm4SwA7HRWYCW2c .cluster-label span p{background-color:transparent;}#mermaid-svg-qhm4SwA7HRWYCW2c .label text,#mermaid-svg-qhm4SwA7HRWYCW2c span{fill:#333;color:#333;}#mermaid-svg-qhm4SwA7HRWYCW2c .node rect,#mermaid-svg-qhm4SwA7HRWYCW2c .node circle,#mermaid-svg-qhm4SwA7HRWYCW2c .node ellipse,#mermaid-svg-qhm4SwA7HRWYCW2c .node polygon,#mermaid-svg-qhm4SwA7HRWYCW2c .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-qhm4SwA7HRWYCW2c .rough-node .label text,#mermaid-svg-qhm4SwA7HRWYCW2c .node .label text,#mermaid-svg-qhm4SwA7HRWYCW2c .image-shape .label,#mermaid-svg-qhm4SwA7HRWYCW2c .icon-shape .label{text-anchor:middle;}#mermaid-svg-qhm4SwA7HRWYCW2c .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-qhm4SwA7HRWYCW2c .rough-node .label,#mermaid-svg-qhm4SwA7HRWYCW2c .node .label,#mermaid-svg-qhm4SwA7HRWYCW2c .image-shape .label,#mermaid-svg-qhm4SwA7HRWYCW2c .icon-shape .label{text-align:center;}#mermaid-svg-qhm4SwA7HRWYCW2c .node.clickable{cursor:pointer;}#mermaid-svg-qhm4SwA7HRWYCW2c .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-qhm4SwA7HRWYCW2c .arrowheadPath{fill:#333333;}#mermaid-svg-qhm4SwA7HRWYCW2c .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-qhm4SwA7HRWYCW2c .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-qhm4SwA7HRWYCW2c .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-qhm4SwA7HRWYCW2c .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-qhm4SwA7HRWYCW2c .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-qhm4SwA7HRWYCW2c .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-qhm4SwA7HRWYCW2c .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-qhm4SwA7HRWYCW2c .cluster text{fill:#333;}#mermaid-svg-qhm4SwA7HRWYCW2c .cluster span{color:#333;}#mermaid-svg-qhm4SwA7HRWYCW2c div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-qhm4SwA7HRWYCW2c .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-qhm4SwA7HRWYCW2c rect.text{fill:none;stroke-width:0;}#mermaid-svg-qhm4SwA7HRWYCW2c .icon-shape,#mermaid-svg-qhm4SwA7HRWYCW2c .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-qhm4SwA7HRWYCW2c .icon-shape p,#mermaid-svg-qhm4SwA7HRWYCW2c .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-qhm4SwA7HRWYCW2c .icon-shape rect,#mermaid-svg-qhm4SwA7HRWYCW2c .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-qhm4SwA7HRWYCW2c .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-qhm4SwA7HRWYCW2c .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-qhm4SwA7HRWYCW2c :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
有
None
有
无
有
无
是
否
是
否
_resolve_child_toolsets(parent, toolsets, effective_role)
parent.enabled_toolsets?
parent_toolsets = set(parent.enabled_toolsets)
parent有valid_tool_names?
从tool names反推toolsets
parent_toolsets = DEFAULT_TOOLSETS['terminal', 'file', 'web']
显式toolsets参数?
_expand_parent_toolsets()添加tools是parent子集的toolset
继承parent的enabled set
child_toolsets = [t for t in toolsets if t in expanded_parent]
inherit_mcp_toolsets?
追加parent的MCP toolsets
继续
_strip_blocked_tools()移除全部工具被阻止的toolset+ delegation/kanban
effective_role == 'orchestrator'?
从disabled中移除delegation在enabled中添加delegation
保持delegation在disabled中
child_disabled = inherited_disabled +_blocked_toolsets_for_role(role) + ['kanban']
return (child_toolsets, child_disabled_toolsets)
关键设计:
- 子代理永不获得父代理缺少的工具:显式toolsets与parent的(复合展开后的)集合取交集
- 双重剥离:整体被阻止的toolset在这里剥离 + 混合bundle中的单个被阻止工具通过disabled_toolsets在composite展开后减去
- orchestrator角色例外:delegationtoolset被无条件重新添加(角色授予,非继承)
13.3 composite toolset展开
def _expand_parent_toolsets(parent_toolsets: set) –> set:
"""添加tools是parent子集的toolset"""
parent_tool_names = {t for ts_name in parent_toolsets
for t in (TOOLSETS.get(ts_name) or {}).get("tools", [])}
expanded = set(parent_toolsets)
if parent_tool_names:
expanded.update(
ts_name for ts_name, ts_def in TOOLSETS.items()
if ts_name not in expanded and ts_def.get("tools")
and set(ts_def["tools"]).issubset(parent_tool_names)
)
return expanded
问题:parent在hermes-clicomposite上,child请求web/terminal——bare name intersection会拒绝它们,因为web ≠ hermes-cli。展开后,web的tools是hermes-clitools的子集→被添加到expanded集合→child的web在expanded中→通过。
14. 结构化输出验证
14.1 验证流程
#mermaid-svg-tW67sLIJd1HREJNT{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-tW67sLIJd1HREJNT .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-tW67sLIJd1HREJNT .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-tW67sLIJd1HREJNT .error-icon{fill:#552222;}#mermaid-svg-tW67sLIJd1HREJNT .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-tW67sLIJd1HREJNT .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-tW67sLIJd1HREJNT .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-tW67sLIJd1HREJNT .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-tW67sLIJd1HREJNT .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-tW67sLIJd1HREJNT .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-tW67sLIJd1HREJNT .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-tW67sLIJd1HREJNT .marker{fill:#333333;stroke:#333333;}#mermaid-svg-tW67sLIJd1HREJNT .marker.cross{stroke:#333333;}#mermaid-svg-tW67sLIJd1HREJNT svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-tW67sLIJd1HREJNT p{margin:0;}#mermaid-svg-tW67sLIJd1HREJNT .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-tW67sLIJd1HREJNT .cluster-label text{fill:#333;}#mermaid-svg-tW67sLIJd1HREJNT .cluster-label span{color:#333;}#mermaid-svg-tW67sLIJd1HREJNT .cluster-label span p{background-color:transparent;}#mermaid-svg-tW67sLIJd1HREJNT .label text,#mermaid-svg-tW67sLIJd1HREJNT span{fill:#333;color:#333;}#mermaid-svg-tW67sLIJd1HREJNT .node rect,#mermaid-svg-tW67sLIJd1HREJNT .node circle,#mermaid-svg-tW67sLIJd1HREJNT .node ellipse,#mermaid-svg-tW67sLIJd1HREJNT .node polygon,#mermaid-svg-tW67sLIJd1HREJNT .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-tW67sLIJd1HREJNT .rough-node .label text,#mermaid-svg-tW67sLIJd1HREJNT .node .label text,#mermaid-svg-tW67sLIJd1HREJNT .image-shape .label,#mermaid-svg-tW67sLIJd1HREJNT .icon-shape .label{text-anchor:middle;}#mermaid-svg-tW67sLIJd1HREJNT .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-tW67sLIJd1HREJNT .rough-node .label,#mermaid-svg-tW67sLIJd1HREJNT .node .label,#mermaid-svg-tW67sLIJd1HREJNT .image-shape .label,#mermaid-svg-tW67sLIJd1HREJNT .icon-shape .label{text-align:center;}#mermaid-svg-tW67sLIJd1HREJNT .node.clickable{cursor:pointer;}#mermaid-svg-tW67sLIJd1HREJNT .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-tW67sLIJd1HREJNT .arrowheadPath{fill:#333333;}#mermaid-svg-tW67sLIJd1HREJNT .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-tW67sLIJd1HREJNT .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-tW67sLIJd1HREJNT .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-tW67sLIJd1HREJNT .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-tW67sLIJd1HREJNT .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-tW67sLIJd1HREJNT .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-tW67sLIJd1HREJNT .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-tW67sLIJd1HREJNT .cluster text{fill:#333;}#mermaid-svg-tW67sLIJd1HREJNT .cluster span{color:#333;}#mermaid-svg-tW67sLIJd1HREJNT div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-tW67sLIJd1HREJNT .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-tW67sLIJd1HREJNT rect.text{fill:none;stroke-width:0;}#mermaid-svg-tW67sLIJd1HREJNT .icon-shape,#mermaid-svg-tW67sLIJd1HREJNT .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-tW67sLIJd1HREJNT .icon-shape p,#mermaid-svg-tW67sLIJd1HREJNT .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-tW67sLIJd1HREJNT .icon-shape rect,#mermaid-svg-tW67sLIJd1HREJNT .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-tW67sLIJd1HREJNT .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-tW67sLIJd1HREJNT .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-tW67sLIJd1HREJNT :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
否
是
是
否
否
是
_validate_child_output_schema()
child._delegate_output_schemais dict?
返回: 无schema, valid=None, 无分支
validate_output(first_text, schema)jsonschema验证
valid?
返回: schema, valid=True, 0 retries
first_text非空且非interrupted?
返回: schema, valid=False, 0 retries
child.run_conversation(build_retry_message(errors))EXACTLY ONE bounded retry
_retry_result
result.final_response = retry_textresult.api_calls += retry_api_callsresult.messages += retry_messages
validate_output(retry_text, schema)
返回: schema, valid=?, 1 retry
14.2 retry设计要点
def build_retry_message(errors: List[str]) –> str:
error_block = "\\n".join(f"- {e}" for e in errors)
return ("Your previous final response was rejected by the output contract "
"validator. Validation errors:\\n" f"{error_block}\\n\\n"
"Reply with ONLY the corrected JSON object matching the OUTPUT "
"CONTRACT schema from your task context. No prose, no explanations.")
- 恰好一次重试:多次重试让前沿模型丢弃第一次正确的字段
- Schema不再重新粘贴:子代理已在context中持有contract
- 错误逐字传递:验证错误原文传入retry消息
14.3 JSON候选提取
def extract_json_candidate(text: str) –> str:
raw = (text or "").strip()
if raw.startswith("```"): # 剥离markdown fence
raw = raw.split("\\n", 1)[–1]
if raw.rstrip().endswith("```"):
raw = raw.rstrip()[:–3]
raw = raw.strip()
if raw.lower().startswith("json\\n"):
raw = raw.split("\\n", 1)[1]
for opener, closer in (("{", "}"), ("[", "]")): # 提取最外层{…}/[…]
if raw.startswith(opener):
return raw
start = raw.find(opener)
end = raw.rfind(closer)
if start >= 0 and end > start:
return raw[start:end + 1]
return raw
15. 异步持久化账本与崩溃恢复
15.1 SQLite表结构
— async_delegations 表 (state.db)
CREATE TABLE async_delegations (
delegation_id TEXT PRIMARY KEY,
origin_session TEXT, — 会话路由键
origin_ui_session_id TEXT, — TUI标签ID
parent_session_id TEXT, — 父会话持久ID
state TEXT, — running/finalizing/completed/error/stalled/unknown
dispatched_at REAL,
updated_at REAL,
completed_at REAL,
event_json TEXT, — 完成事件JSON
result_json TEXT, — 结果JSON
delivery_state TEXT, — pending/delivered/dropped
delivery_attempts INTEGER,
delivery_claim TEXT,
delivery_claimed_at REAL,
delivered_at REAL,
owner_pid INTEGER, — 派发进程PID
owner_started_at REAL, — 进程启动时间(区分PID复用)
task_json TEXT, — 任务载荷JSON
origin_session_id TEXT
);
15.2 投递状态机
#mermaid-svg-srqUKBoGyHtY5D8z{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-srqUKBoGyHtY5D8z .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-srqUKBoGyHtY5D8z .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-srqUKBoGyHtY5D8z .error-icon{fill:#552222;}#mermaid-svg-srqUKBoGyHtY5D8z .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-srqUKBoGyHtY5D8z .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-srqUKBoGyHtY5D8z .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-srqUKBoGyHtY5D8z .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-srqUKBoGyHtY5D8z .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-srqUKBoGyHtY5D8z .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-srqUKBoGyHtY5D8z .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-srqUKBoGyHtY5D8z .marker{fill:#333333;stroke:#333333;}#mermaid-svg-srqUKBoGyHtY5D8z .marker.cross{stroke:#333333;}#mermaid-svg-srqUKBoGyHtY5D8z svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-srqUKBoGyHtY5D8z p{margin:0;}#mermaid-svg-srqUKBoGyHtY5D8z defs #statediagram-barbEnd{fill:#333333;stroke:#333333;}#mermaid-svg-srqUKBoGyHtY5D8z g.stateGroup text{fill:#9370DB;stroke:none;font-size:10px;}#mermaid-svg-srqUKBoGyHtY5D8z g.stateGroup text{fill:#333;stroke:none;font-size:10px;}#mermaid-svg-srqUKBoGyHtY5D8z g.stateGroup .state-title{font-weight:bolder;fill:#131300;}#mermaid-svg-srqUKBoGyHtY5D8z g.stateGroup rect{fill:#ECECFF;stroke:#9370DB;}#mermaid-svg-srqUKBoGyHtY5D8z g.stateGroup line{stroke:#333333;stroke-width:1;}#mermaid-svg-srqUKBoGyHtY5D8z .transition{stroke:#333333;stroke-width:1;fill:none;}#mermaid-svg-srqUKBoGyHtY5D8z .stateGroup .composit{fill:white;border-bottom:1px;}#mermaid-svg-srqUKBoGyHtY5D8z .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px;}#mermaid-svg-srqUKBoGyHtY5D8z .state-note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-srqUKBoGyHtY5D8z .state-note text{fill:black;stroke:none;font-size:10px;}#mermaid-svg-srqUKBoGyHtY5D8z .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5;}#mermaid-svg-srqUKBoGyHtY5D8z .edgeLabel .label rect{fill:#ECECFF;opacity:0.5;}#mermaid-svg-srqUKBoGyHtY5D8z .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-srqUKBoGyHtY5D8z .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-srqUKBoGyHtY5D8z .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-srqUKBoGyHtY5D8z .edgeLabel .label text{fill:#333;}#mermaid-svg-srqUKBoGyHtY5D8z .label div .edgeLabel{color:#333;}#mermaid-svg-srqUKBoGyHtY5D8z .stateLabel text{fill:#131300;font-size:10px;font-weight:bold;}#mermaid-svg-srqUKBoGyHtY5D8z .node circle.state-start{fill:#333333;stroke:#333333;}#mermaid-svg-srqUKBoGyHtY5D8z .node .fork-join{fill:#333333;stroke:#333333;}#mermaid-svg-srqUKBoGyHtY5D8z .node circle.state-end{fill:#9370DB;stroke:white;stroke-width:1.5;}#mermaid-svg-srqUKBoGyHtY5D8z .end-state-inner{fill:white;stroke-width:1.5;}#mermaid-svg-srqUKBoGyHtY5D8z .node rect{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-srqUKBoGyHtY5D8z .node polygon{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-srqUKBoGyHtY5D8z #statediagram-barbEnd{fill:#333333;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-cluster rect{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-srqUKBoGyHtY5D8z .cluster-label,#mermaid-svg-srqUKBoGyHtY5D8z .nodeLabel{color:#131300;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-cluster rect.outer{rx:5px;ry:5px;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-state .divider{stroke:#9370DB;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-state .title-state{rx:5px;ry:5px;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-cluster.statediagram-cluster .inner{fill:white;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-cluster.statediagram-cluster-alt .inner{fill:#f0f0f0;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-cluster .inner{rx:0;ry:0;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-state rect.basic{rx:5px;ry:5px;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#f0f0f0;}#mermaid-svg-srqUKBoGyHtY5D8z .note-edge{stroke-dasharray:5;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-note rect{fill:#fff5ad;stroke:#aaaa33;stroke-width:1px;rx:0;ry:0;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-note rect{fill:#fff5ad;stroke:#aaaa33;stroke-width:1px;rx:0;ry:0;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-note text{fill:black;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram-note .nodeLabel{color:black;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagram .edgeLabel{color:red;}#mermaid-svg-srqUKBoGyHtY5D8z #dependencyStart,#mermaid-svg-srqUKBoGyHtY5D8z #dependencyEnd{fill:#333333;stroke:#333333;stroke-width:1;}#mermaid-svg-srqUKBoGyHtY5D8z .statediagramTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-srqUKBoGyHtY5D8z :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
_persist_dispatch()
complete_completion_delivery()原子确认成功投递
drop_completion_delivery()目标永久消失(/new/reset)
release_completion_delivery()attempts >= MAX(8)
defer_completion_delivery()不消耗attempt返回pending
pending
delivered
dropped
claim_completion_delivery():原子claim防止多consumer竞争delivery_attempts + 1300秒后旧claim可被抢占
15.3 崩溃恢复流程
#mermaid-svg-IZX2skRZz371Q2dY{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-IZX2skRZz371Q2dY .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-IZX2skRZz371Q2dY .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-IZX2skRZz371Q2dY .error-icon{fill:#552222;}#mermaid-svg-IZX2skRZz371Q2dY .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-IZX2skRZz371Q2dY .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-IZX2skRZz371Q2dY .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-IZX2skRZz371Q2dY .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-IZX2skRZz371Q2dY .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-IZX2skRZz371Q2dY .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-IZX2skRZz371Q2dY .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-IZX2skRZz371Q2dY .marker{fill:#333333;stroke:#333333;}#mermaid-svg-IZX2skRZz371Q2dY .marker.cross{stroke:#333333;}#mermaid-svg-IZX2skRZz371Q2dY svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-IZX2skRZz371Q2dY p{margin:0;}#mermaid-svg-IZX2skRZz371Q2dY .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-IZX2skRZz371Q2dY .cluster-label text{fill:#333;}#mermaid-svg-IZX2skRZz371Q2dY .cluster-label span{color:#333;}#mermaid-svg-IZX2skRZz371Q2dY .cluster-label span p{background-color:transparent;}#mermaid-svg-IZX2skRZz371Q2dY .label text,#mermaid-svg-IZX2skRZz371Q2dY span{fill:#333;color:#333;}#mermaid-svg-IZX2skRZz371Q2dY .node rect,#mermaid-svg-IZX2skRZz371Q2dY .node circle,#mermaid-svg-IZX2skRZz371Q2dY .node ellipse,#mermaid-svg-IZX2skRZz371Q2dY .node polygon,#mermaid-svg-IZX2skRZz371Q2dY .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-IZX2skRZz371Q2dY .rough-node .label text,#mermaid-svg-IZX2skRZz371Q2dY .node .label text,#mermaid-svg-IZX2skRZz371Q2dY .image-shape .label,#mermaid-svg-IZX2skRZz371Q2dY .icon-shape .label{text-anchor:middle;}#mermaid-svg-IZX2skRZz371Q2dY .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-IZX2skRZz371Q2dY .rough-node .label,#mermaid-svg-IZX2skRZz371Q2dY .node .label,#mermaid-svg-IZX2skRZz371Q2dY .image-shape .label,#mermaid-svg-IZX2skRZz371Q2dY .icon-shape .label{text-align:center;}#mermaid-svg-IZX2skRZz371Q2dY .node.clickable{cursor:pointer;}#mermaid-svg-IZX2skRZz371Q2dY .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-IZX2skRZz371Q2dY .arrowheadPath{fill:#333333;}#mermaid-svg-IZX2skRZz371Q2dY .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-IZX2skRZz371Q2dY .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-IZX2skRZz371Q2dY .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IZX2skRZz371Q2dY .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-IZX2skRZz371Q2dY .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IZX2skRZz371Q2dY .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-IZX2skRZz371Q2dY .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-IZX2skRZz371Q2dY .cluster text{fill:#333;}#mermaid-svg-IZX2skRZz371Q2dY .cluster span{color:#333;}#mermaid-svg-IZX2skRZz371Q2dY div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-IZX2skRZz371Q2dY .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-IZX2skRZz371Q2dY rect.text{fill:none;stroke-width:0;}#mermaid-svg-IZX2skRZz371Q2dY .icon-shape,#mermaid-svg-IZX2skRZz371Q2dY .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IZX2skRZz371Q2dY .icon-shape p,#mermaid-svg-IZX2skRZz371Q2dY .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-IZX2skRZz371Q2dY .icon-shape rect,#mermaid-svg-IZX2skRZz371Q2dY .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IZX2skRZz371Q2dY .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-IZX2skRZz371Q2dY .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-IZX2skRZz371Q2dY :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
是
否
是
否
是
否
进程重启
recover_abandoned_delegations()
SELECT * FROM async_delegationsWHERE state IN ('running','finalizing')
for each row:
owner_pid存在且进程存活且启动时间匹配?
跳过:进程仍在运行
分类为unknown
有partial results?
_recovered_results()已记录的children保持真实结果未记录的标记unknown
所有children标记unknown
UPDATE state='unknown',event_json=…, result_json=…
restore_undelivered_completions()
SELECT * FROM async_delegationsWHERE state != 'running'AND delivery_state='pending'AND event_json IS NOT NULL
for each row:
age > 48h?
UPDATE delivery_state='dropped'WARNING: 太旧不重放
evt.restored = Truetarget_queue.put(evt)
恢复完成
关键恢复参数:
- _MAX_RETAINED_COMPLETED = 50:内存中保留的已完成记录上限
- _DURABLE_RETENTION_SECONDS = 7 * 24 * 3600:SQLite中delivered记录保留7天
- _MAX_DURABLE_PENDING = 1000:pending记录上限
- _MAX_DELIVERY_ATTEMPTS = 8:投递尝试上限,超过后terminally dropped
- _MAX_COMPLETION_REPLAY_AGE_S = 48 * 3600:48小时以上的pending不重放
15.4 _transaction() 连接管理
@contextmanager
def _transaction() –> Iterator[sqlite3.Connection]:
"""Open a connection, commit/rollback on exit, and ALWAYS close it"""
conn = _connect()
try:
with conn: # commit/rollback
yield conn
finally:
conn.close() # __enter__/__exit__只提交/回滚,不关闭连接!
关键修复(#69567/#69594):
- with conn:只提交/回滚事务,不关闭连接
- 单用with _connect()会泄漏连接和WAL/SHM文件描述符
- 长期运行的gateway会耗尽RLIMIT_NOFILE
- finally: conn.close()确保每次都关闭
15.5 record_unit_child() 增量持久化
def record_unit_child(delegation_id: str, entry: Dict[str, Any]) –> None:
"""为多子代理单元的每个完成的child持久化结果,崩溃后只丢失未完成的"""
with _DB_LOCK, _transaction() as conn:
row = conn.execute("SELECT result_json FROM async_delegations WHERE state='running'", ...).fetchone()
if row is None:
return
partial = json.loads(row[0] or "{}") or {}
results = [r for r in partial.get("results") or [] if r.get("task_index") != entry.get("task_index")]
results.append(entry) # 替换同task_index的旧记录
conn.execute("UPDATE result_json=?, updated_at=? WHERE state='running'", ...)
场景:1393个agent的批量运行中,wave-1的所有child在08:29的401风暴中死亡,父代理在09:36才收到"unknown outcome"块——66分钟无任何运行。增量持久化让已完成的children结果在崩溃后仍然可用。
16. 停滞监控 _stale_monitor_loop
16.1 监控流程
#mermaid-svg-fUIf5QWu1zLSqJUG{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-fUIf5QWu1zLSqJUG .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-fUIf5QWu1zLSqJUG .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-fUIf5QWu1zLSqJUG .error-icon{fill:#552222;}#mermaid-svg-fUIf5QWu1zLSqJUG .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-fUIf5QWu1zLSqJUG .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-fUIf5QWu1zLSqJUG .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-fUIf5QWu1zLSqJUG .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-fUIf5QWu1zLSqJUG .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-fUIf5QWu1zLSqJUG .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-fUIf5QWu1zLSqJUG .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-fUIf5QWu1zLSqJUG .marker{fill:#333333;stroke:#333333;}#mermaid-svg-fUIf5QWu1zLSqJUG .marker.cross{stroke:#333333;}#mermaid-svg-fUIf5QWu1zLSqJUG svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-fUIf5QWu1zLSqJUG p{margin:0;}#mermaid-svg-fUIf5QWu1zLSqJUG .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-fUIf5QWu1zLSqJUG .cluster-label text{fill:#333;}#mermaid-svg-fUIf5QWu1zLSqJUG .cluster-label span{color:#333;}#mermaid-svg-fUIf5QWu1zLSqJUG .cluster-label span p{background-color:transparent;}#mermaid-svg-fUIf5QWu1zLSqJUG .label text,#mermaid-svg-fUIf5QWu1zLSqJUG span{fill:#333;color:#333;}#mermaid-svg-fUIf5QWu1zLSqJUG .node rect,#mermaid-svg-fUIf5QWu1zLSqJUG .node circle,#mermaid-svg-fUIf5QWu1zLSqJUG .node ellipse,#mermaid-svg-fUIf5QWu1zLSqJUG .node polygon,#mermaid-svg-fUIf5QWu1zLSqJUG .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-fUIf5QWu1zLSqJUG .rough-node .label text,#mermaid-svg-fUIf5QWu1zLSqJUG .node .label text,#mermaid-svg-fUIf5QWu1zLSqJUG .image-shape .label,#mermaid-svg-fUIf5QWu1zLSqJUG .icon-shape .label{text-anchor:middle;}#mermaid-svg-fUIf5QWu1zLSqJUG .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-fUIf5QWu1zLSqJUG .rough-node .label,#mermaid-svg-fUIf5QWu1zLSqJUG .node .label,#mermaid-svg-fUIf5QWu1zLSqJUG .image-shape .label,#mermaid-svg-fUIf5QWu1zLSqJUG .icon-shape .label{text-align:center;}#mermaid-svg-fUIf5QWu1zLSqJUG .node.clickable{cursor:pointer;}#mermaid-svg-fUIf5QWu1zLSqJUG .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-fUIf5QWu1zLSqJUG .arrowheadPath{fill:#333333;}#mermaid-svg-fUIf5QWu1zLSqJUG .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-fUIf5QWu1zLSqJUG .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-fUIf5QWu1zLSqJUG .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-fUIf5QWu1zLSqJUG .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-fUIf5QWu1zLSqJUG .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-fUIf5QWu1zLSqJUG .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-fUIf5QWu1zLSqJUG .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-fUIf5QWu1zLSqJUG .cluster text{fill:#333;}#mermaid-svg-fUIf5QWu1zLSqJUG .cluster span{color:#333;}#mermaid-svg-fUIf5QWu1zLSqJUG div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-fUIf5QWu1zLSqJUG .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-fUIf5QWu1zLSqJUG rect.text{fill:none;stroke-width:0;}#mermaid-svg-fUIf5QWu1zLSqJUG .icon-shape,#mermaid-svg-fUIf5QWu1zLSqJUG .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-fUIf5QWu1zLSqJUG .icon-shape p,#mermaid-svg-fUIf5QWu1zLSqJUG .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-fUIf5QWu1zLSqJUG .icon-shape rect,#mermaid-svg-fUIf5QWu1zLSqJUG .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-fUIf5QWu1zLSqJUG .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-fUIf5QWu1zLSqJUG .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-fUIf5QWu1zLSqJUG :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
是
否
stalling
是
否
running + has progress_fn
否
是
是
否
是
否
其他
否
是
_dispatch() with progress_fn
_ensure_stale_monitor()
monitor_thread存活?
返回
启动daemon threadname='async-delegate-stale-monitor'
while not _monitor_stop.wait(30s):
_sweep_stale_locked(now)
for each record:
status
now – _interrupted_at >= 120s?
加入expired列表强制终结化
保持stalling
_started?
排队中,不计停滞
token, in_tool = progress_fn()
token != _progress_token?
update _progress_token, _progress_ts=now有进展
quiet_for = now – _progress_ts
limit = 1200s if in_tool else 450s
quiet_for >= limit?
status='stalling'_interrupted_at=now记录stall元数据
保持running
跳过
_call_interrupt(interrupt_fn)调用子代理中断
_finalize(id, _stalled_result, 'stalled')强制终结化
any_monitorable?
return监控线程退出
16.2 阈值参数
| _STALE_CHECK_INTERVAL | 30秒 | 监控线程扫描间隔 |
| _STALE_IDLE_SECONDS | 450秒 (7.5分钟) | idle状态停滞阈值 |
| _STALE_IN_TOOL_SECONDS | 1200秒 (20分钟) | in-tool状态停滞阈值 |
| _STALL_GRACE_SECONDS | 120秒 (2分钟) | stalling→强制终结化的宽限窗口 |
16.3 _stalled_result() 合成结果
def _stalled_result(delegation_id, event_record) –> Dict[str, Any]:
error = (
f"Async delegation {delegation_id} stalled: the detached subagent stopped making progress "
"(no new API calls, tool activity, or streamed tokens), did not respond to interruption, and never "
"produced a completion event. The worker may be wedged inside a model API call…"
)
stall_meta = {
"stalled_after_quiet_seconds": event_record.get("_stall_quiet_seconds"),
"stall_threshold_seconds": event_record.get("_stall_threshold_seconds"),
"stall_phase": "in_tool" if stall_in_tool else "idle" if stall_in_tool is not None else None,
"stall_grace_seconds": _STALL_GRACE_SECONDS,
}
if event_record.get("is_batch"):
return {**_batch_crash(error, duration), **stall_meta}
return {**_single_crash(error, duration), "status": "stalled", "exit_reason": "stalled", **stall_meta}
结构化stall元数据让父代理/UI区分stall-monitor kill和其他失败,无需解析error字符串。
17. 实时转录日志
17.1 LiveTranscriptWriter 架构
#mermaid-svg-jU4pY3AGOEsWqH6a{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-jU4pY3AGOEsWqH6a .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-jU4pY3AGOEsWqH6a .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-jU4pY3AGOEsWqH6a .error-icon{fill:#552222;}#mermaid-svg-jU4pY3AGOEsWqH6a .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-jU4pY3AGOEsWqH6a .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-jU4pY3AGOEsWqH6a .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-jU4pY3AGOEsWqH6a .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-jU4pY3AGOEsWqH6a .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-jU4pY3AGOEsWqH6a .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-jU4pY3AGOEsWqH6a .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-jU4pY3AGOEsWqH6a .marker{fill:#333333;stroke:#333333;}#mermaid-svg-jU4pY3AGOEsWqH6a .marker.cross{stroke:#333333;}#mermaid-svg-jU4pY3AGOEsWqH6a svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-jU4pY3AGOEsWqH6a p{margin:0;}#mermaid-svg-jU4pY3AGOEsWqH6a .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-jU4pY3AGOEsWqH6a .cluster-label text{fill:#333;}#mermaid-svg-jU4pY3AGOEsWqH6a .cluster-label span{color:#333;}#mermaid-svg-jU4pY3AGOEsWqH6a .cluster-label span p{background-color:transparent;}#mermaid-svg-jU4pY3AGOEsWqH6a .label text,#mermaid-svg-jU4pY3AGOEsWqH6a span{fill:#333;color:#333;}#mermaid-svg-jU4pY3AGOEsWqH6a .node rect,#mermaid-svg-jU4pY3AGOEsWqH6a .node circle,#mermaid-svg-jU4pY3AGOEsWqH6a .node ellipse,#mermaid-svg-jU4pY3AGOEsWqH6a .node polygon,#mermaid-svg-jU4pY3AGOEsWqH6a .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-jU4pY3AGOEsWqH6a .rough-node .label text,#mermaid-svg-jU4pY3AGOEsWqH6a .node .label text,#mermaid-svg-jU4pY3AGOEsWqH6a .image-shape .label,#mermaid-svg-jU4pY3AGOEsWqH6a .icon-shape .label{text-anchor:middle;}#mermaid-svg-jU4pY3AGOEsWqH6a .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-jU4pY3AGOEsWqH6a .rough-node .label,#mermaid-svg-jU4pY3AGOEsWqH6a .node .label,#mermaid-svg-jU4pY3AGOEsWqH6a .image-shape .label,#mermaid-svg-jU4pY3AGOEsWqH6a .icon-shape .label{text-align:center;}#mermaid-svg-jU4pY3AGOEsWqH6a .node.clickable{cursor:pointer;}#mermaid-svg-jU4pY3AGOEsWqH6a .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-jU4pY3AGOEsWqH6a .arrowheadPath{fill:#333333;}#mermaid-svg-jU4pY3AGOEsWqH6a .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-jU4pY3AGOEsWqH6a .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-jU4pY3AGOEsWqH6a .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-jU4pY3AGOEsWqH6a .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-jU4pY3AGOEsWqH6a .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-jU4pY3AGOEsWqH6a .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-jU4pY3AGOEsWqH6a .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-jU4pY3AGOEsWqH6a .cluster text{fill:#333;}#mermaid-svg-jU4pY3AGOEsWqH6a .cluster span{color:#333;}#mermaid-svg-jU4pY3AGOEsWqH6a div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-jU4pY3AGOEsWqH6a .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-jU4pY3AGOEsWqH6a rect.text{fill:none;stroke-width:0;}#mermaid-svg-jU4pY3AGOEsWqH6a .icon-shape,#mermaid-svg-jU4pY3AGOEsWqH6a .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-jU4pY3AGOEsWqH6a .icon-shape p,#mermaid-svg-jU4pY3AGOEsWqH6a .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-jU4pY3AGOEsWqH6a .icon-shape rect,#mermaid-svg-jU4pY3AGOEsWqH6a .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-jU4pY3AGOEsWqH6a .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-jU4pY3AGOEsWqH6a .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-jU4pY3AGOEsWqH6a :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
是
否
是
否
create_live_transcripts()
LiveTranscriptWriter(delegation_id, task_index, goal, context)
创建目录 cache/delegation/live//
写入 HH:MM:SS role | text
event('user', 'kickoff: ' + goal)
_ok = True, path = task-N.log
observe(event_type, tool_name, preview, args, kwargs)
_OBSERVERS demux:
tool.started → tool_start(name, args_preview)
tool.completed → tool_result(name, result, duration, is_error)
_thinking → thinking(text)
reasoning.available → thinking(text)
subagent.text → add_stream_delta(delta)
subagent.start → event('start', goal)
subagent.complete → _on_complete(status, duration, summary)
event(role, text)
_stream_buf.append(delta)
_stream_len >= 4000?
flush_stream()
等待更多delta
_redact(text)redact_sensitive_text(force=True)
写入失败?
_ok = False后续调用变为debug-logged no-op
继续
17.2 安全设计
每行文本通过_redact()强制脱敏:
def _redact(text: str) –> str:
if not text:
return text
try:
from agent.redact import redact_sensitive_text
return redact_sensitive_text(text, force=True) or ""
except Exception:
return "[line withheld: redaction unavailable]" # 失败时扣留而非泄露
- force=True:即使全局toggle关闭也强制脱敏(安全边界)
- redactor不可用时扣留整行而非泄露
- cache/delegation被只读挂载到远程终端后端,每行都必须凭证脱敏
17.3 截断预算
| assistant_text | 600 chars | 助手回复文本 |
| thinking | 300 chars | 推理文本 |
| tool args | 220 chars | 工具参数预览 |
| tool result | 400 chars | 工具结果预览 |
| kickoff | 500 chars | 启动消息 |
| stream buffer | 4000 chars | 流式delta缓冲flush阈值 |
转录日志是紧凑的操作视图;完整文本在SessionDB transcript和summary spill文件中。
17.4 保留策略
LIVE_RETENTION_DAYS = 7
7天保留,通过update_manifest_statuses()更新manifest中的状态,在后续dispatch时修剪旧目录。
18. 设计亮点与关键决策
18.1 十大设计亮点
#mermaid-svg-l5A4xwWZJQA1Enkx{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-l5A4xwWZJQA1Enkx .error-icon{fill:#552222;}#mermaid-svg-l5A4xwWZJQA1Enkx .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-l5A4xwWZJQA1Enkx .marker{fill:#333333;stroke:#333333;}#mermaid-svg-l5A4xwWZJQA1Enkx .marker.cross{stroke:#333333;}#mermaid-svg-l5A4xwWZJQA1Enkx svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-l5A4xwWZJQA1Enkx p{margin:0;}#mermaid-svg-l5A4xwWZJQA1Enkx .edge{stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .section–1 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section–1 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section–1 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section–1 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section–1 text{fill:#ffffff;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth–1{stroke-width:17;}#mermaid-svg-l5A4xwWZJQA1Enkx .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-0 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-0 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-0 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-0 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-0 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-0{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-0{stroke-width:14;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-1 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-1 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-1 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-1 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-1 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-1{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-1{stroke-width:11;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-2 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-2 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-2 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-2 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-2 text{fill:#ffffff;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-2{stroke-width:8;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-3 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-3 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-3 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-3 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-3 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-3{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-3{stroke-width:5;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-4 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-4 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-4 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-4 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-4 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-4{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-4{stroke-width:2;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-5 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-5 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-5 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-5 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-5 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-5{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-5{stroke-width:-1;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-6 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-6 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-6 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-6 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-6 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-6{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-6{stroke-width:-4;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-7 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-7 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-7 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-7 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-7 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-7{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-7{stroke-width:-7;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-8 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-8 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-8 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-8 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-8 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-8{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-8{stroke-width:-10;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-9 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-9 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-9 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-9 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-9 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-9{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-9{stroke-width:-13;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-10 rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-10 path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-10 circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-10 polygon,#mermaid-svg-l5A4xwWZJQA1Enkx .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-10 text{fill:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .node-icon-10{font-size:40px;color:black;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .edge-depth-10{stroke-width:-16;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled circle,#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:lightgray;}#mermaid-svg-l5A4xwWZJQA1Enkx .disabled text{fill:#efefef;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-root rect,#mermaid-svg-l5A4xwWZJQA1Enkx .section-root path,#mermaid-svg-l5A4xwWZJQA1Enkx .section-root circle,#mermaid-svg-l5A4xwWZJQA1Enkx .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-l5A4xwWZJQA1Enkx .section-root text{fill:#ffffff;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-root span{color:#ffffff;}#mermaid-svg-l5A4xwWZJQA1Enkx .section-2 span{color:#ffffff;}#mermaid-svg-l5A4xwWZJQA1Enkx .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-l5A4xwWZJQA1Enkx .edge{fill:none;}#mermaid-svg-l5A4xwWZJQA1Enkx .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-l5A4xwWZJQA1Enkx :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
子代理委派设计亮点
隔离性
独立session_id/task_id
专用SessionDB句柄
skip_memory/skip_context_files
可选
可控性
工具集双重剥离
depth-derived角色
暂停开关
非交互审批回调
可观测
实时转录日志
进度回调relay
tool_trace追踪
心跳监控
可恢复
SQLite持久化账本
增量child持久化
崩溃恢复
48h重放窗口
可控制
list/steer/stop
双层权限验证
传输槽权限
missed_steer报告
成本可控
headroom
无ceiling
深度上限
压缩阈值覆盖
Cache TTL降级
安全性
凭证脱敏转录
凭证池隔离
URL secret剥离
输出Schema验证
健壮性
daemon线程池
0.5s轮询中断
超时诊断转储
OpenSSL安全
停滞监控+宽限窗口
18.2 关键设计决策总结
| 1 | 子代理用专用SessionDB而非共享父代理的 | 父句柄可能在其生命周期中被关闭,后台child仍在daemon线程刷新→transcript静默丢失 | #81267 |
| 2 | 容量满时拒绝而非排队 | 防止失控模型堆积无界后台工作 | 设计决策 |
| 3 | 0.5秒轮询而非as_completed() | wedged child不能永远阻塞父代理 | 健壮性 |
| 4 | 超时后延迟关闭而非立即关闭 | worker通常parked在OpenSSL read中,跨线程FD释放破坏原生状态 | #29507/#70773 |
| 5 | heartbeat用共享periodic_scheduler | 不是每个child一个daemon thread | 资源效率 |
| 6 | 摘要预算用_last_prompt_size_tokens而非session_prompt_tokens | 后者是累计SUM,几百次调用后超过任何上下文窗口 | 实测发现 |
| 7 | _transaction()确保conn.close() | with conn:只提交/回滚不关闭,泄漏WAL/SHM fd | #69567/#69594 |
| 8 | record_unit_child()增量持久化 | 1393-agent运行中66分钟无任何运行的dead wave | 实测场景 |
| 9 | restored事件标记restored=True | 新会话不应继承死会话的delegation结果 | #64484 |
| 10 | custom端点按身份匹配凭证池 | 不同custom端点误共享池会覆盖子代理的base_url | #7833 |
| 11 | ACP command必须在PATH上 | 否则子代理静默降级到其他transport | #80450 |
| 12 | 传输权限读取时检查而非记录时快照 | 每个reattach路径都修改传输槽,per-record副本需同步 | #106663 |
| 13 | Schema验证恰好一次重试 | 多次重试让前沿模型丢弃第一次正确的字段 | 模型行为 |
| 14 | 子代理不使用1h cache tier | 子代理每几秒调用、几分钟后消失,1h tier 2x写入无保留收益 | #14971 |
| 15 | push_task_failure_notice即时通知 | 不等整个批次完成就通知失败的sibling | 1393-agent场景 |
18.3 完整端到端时序图
Pluginscompletion_queueSQLiteRegistryHeartbeatAIAgent(子)DaemonThreadPoolExecutor_build_childrendelegate_taskAIAgent(父)用户Pluginscompletion_queueSQLiteRegistryHeartbeatAIAgent(子)DaemonThreadPoolExecutor_build_childrendelegate_taskAIAgent(父)用户#mermaid-svg-sal0OyuNRgLzmduS{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-sal0OyuNRgLzmduS .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-sal0OyuNRgLzmduS .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-sal0OyuNRgLzmduS .error-icon{fill:#552222;}#mermaid-svg-sal0OyuNRgLzmduS .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-sal0OyuNRgLzmduS .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-sal0OyuNRgLzmduS .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-sal0OyuNRgLzmduS .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-sal0OyuNRgLzmduS .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-sal0OyuNRgLzmduS .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-sal0OyuNRgLzmduS .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-sal0OyuNRgLzmduS .marker{fill:#333333;stroke:#333333;}#mermaid-svg-sal0OyuNRgLzmduS .marker.cross{stroke:#333333;}#mermaid-svg-sal0OyuNRgLzmduS svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-sal0OyuNRgLzmduS p{margin:0;}#mermaid-svg-sal0OyuNRgLzmduS .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-sal0OyuNRgLzmduS text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-sal0OyuNRgLzmduS .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-sal0OyuNRgLzmduS .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-sal0OyuNRgLzmduS .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-sal0OyuNRgLzmduS .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-sal0OyuNRgLzmduS #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-sal0OyuNRgLzmduS .sequenceNumber{fill:white;}#mermaid-svg-sal0OyuNRgLzmduS #sequencenumber{fill:#333;}#mermaid-svg-sal0OyuNRgLzmduS #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-sal0OyuNRgLzmduS .messageText{fill:#333;stroke:none;}#mermaid-svg-sal0OyuNRgLzmduS .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-sal0OyuNRgLzmduS .labelText,#mermaid-svg-sal0OyuNRgLzmduS .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-sal0OyuNRgLzmduS .loopText,#mermaid-svg-sal0OyuNRgLzmduS .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-sal0OyuNRgLzmduS .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-sal0OyuNRgLzmduS .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-sal0OyuNRgLzmduS .noteText,#mermaid-svg-sal0OyuNRgLzmduS .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-sal0OyuNRgLzmduS .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-sal0OyuNRgLzmduS .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-sal0OyuNRgLzmduS .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-sal0OyuNRgLzmduS .actorPopupMenu{position:absolute;}#mermaid-svg-sal0OyuNRgLzmduS .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-sal0OyuNRgLzmduS .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-sal0OyuNRgLzmduS .actor-man circle,#mermaid-svg-sal0OyuNRgLzmduS line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-sal0OyuNRgLzmduS :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}守卫检查: pause/depth/concurrencyloop[每个任务 (主线程顺序)]_units_of() → 默认1个单元(全部3个task)daemon worker线程par[3个并行worker]父代理Turn结束后drain"分析3个文件"delegate_task(tasks=[t0, t1, t2])_resolve_delegation_credentials()_normalize_task_list() / _coerce_task_schemas()create_live_transcripts() → 3个transcript writers_capture_origin() → 捕获session身份_build_children(task_list, …)_build_child_preserving_parent_tools()_resolve_child_toolsets() → enabled/disabled_build_child_system_prompt(goal, context)_resolve_child_runtime() → provider/base_url/api_keyAIAgent(**rt, quiet_mode=True, skip_memory=True, …)_apply_child_cache_ttl() → 1h降级5m_apply_child_compression_cap()child._delegate_parent_ref = weakref(parent)_attach_child(parent, child)invoke_hook("subagent_start", …)children = [(0,t0,c0), (1,t1,c1), (2,t2,c2)]_Batch(…)_run_batch(batch, background=True)_dispatch_background(batch)_dispatch_unit(unit, unit_id, slot_key, routing)_persist_dispatch(record) → INSERT async_delegationsexecutor.submit(_worker){"status": "dispatched", "delegation_id": "deleg_xxx"}"已派发3个子代理到后台…"_execute_and_aggregate(unit)_run_children_parallel(batch)_register_child(c0, …)heartbeat.start()c0.run_conversation(goal)(独立的对话循环…)touch("subagent 0 working") (每30s)_register_child(c1, …)heartbeat.start()c1.run_conversation(goal)(独立的对话循环…)touch("subagent 1 working") (每30s)_register_child(c2, …)heartbeat.start()c2.run_conversation(goal)(独立的对话循环…)touch("subagent 2 working") (每30s)result = {final_response, api_calls, messages, …}_validate_child_output_schema()_build_result_entry() → {status, summary, tokens, cost, tool_trace}_finalize_child_results()_apply_summary_budget() → 摘要裁剪+spillinvoke_hook("subagent_stop", …)_rollup_children_cost(parent, total)_unregister_subagent(sid)heartbeat.stop()child.close() → 释放SessionDB/终端/browser_persist_completion(event, result)completion_queue.put(event)新Turn: type="async_delegation" event处理完成事件(results)"3个子代理完成: [结果摘要]"
附录A:文件清单与行数统计
| tools/delegate_tool.py | 745 | 入口、_build_child_agent、_run_single_child、_build_children、schema注册 |
| tools/delegate_tool_dispatch.py | 432 | _Batch、_run_batch、_dispatch_background、_execute_and_aggregate、_run_children_parallel |
| tools/delegate_tool_child_run.py | 836 | _Heartbeat、_ChildRun、_register_child、_build_result_entry、_validate_child_output_schema、_defer_close_after_timeout |
| tools/delegate_tool_config.py | 529 | 配置旋钮、_resolve_delegation_credentials、_resolve_child_credential_pool、_resolve_child_runtime |
| tools/delegate_tool_registry.py | 310 | _active_subagents注册表、steer/stop/list控制面、_owns_subagent_record权限 |
| tools/delegate_tool_toolsets.py | 114 | _resolve_child_toolsets、DELEGATE_BLOCKED_TOOLS、composite展开 |
| tools/delegate_tool_results.py | 410 | _apply_summary_budget、_fire_subagent_stop_hooks、_rollup_children_cost、_finalize_child_results |
| tools/async_delegation.py | 1012 | _dispatch、_finalize、_push_completion_event、SQLite账本、崩溃恢复、停滞监控 |
| tools/delegation_live_log.py | 315 | LiveTranscriptWriter、实时转录、凭证脱敏 |
| tools/delegation_output_schema.py | 116 | coerce_output_schema、validate_output、build_retry_message |
| tools/daemon_pool.py | 56 | DaemonThreadPoolExecutor |
| tools/subagent_worktree.py | 193 | git worktree隔离(可选) |
| 总计 | 5,072 |
附录B:关键常量速查
| _DEFAULT_MAX_CONCURRENT_CHILDREN | 10 | config.py | 并发子代理默认上限 |
| MAX_DEPTH | 1 | config.py | 默认spawn深度(扁平) |
| _MIN_SPAWN_DEPTH | 1 | config.py | spawn深度floor |
| DEFAULT_MAX_ITERATIONS | 250 | delegate_tool.py | 子代理最大迭代数 |
| _HEARTBEAT_INTERVAL | 30s | delegate_tool.py | 心跳间隔 |
| _HEARTBEAT_STALE_CYCLES_IDLE | 15 | delegate_tool.py | idle停滞周期(450s) |
| _HEARTBEAT_STALE_CYCLES_IN_TOOL | 40 | delegate_tool.py | in-tool停滞周期(1200s) |
| DEFAULT_CHILD_TIMEOUT | None | config.py | 默认无超时 |
| DEFAULT_MAX_SUMMARY_CHARS | 24000 | results.py | 摘要硬上限 |
| _SUMMARY_HEADROOM_FRACTION | 0.5 | results.py | headroom分配比例 |
| _MIN_SUMMARY_CHARS | 2000 | results.py | 摘要floor |
| _DEFAULT_MAX_ASYNC_CHILDREN | 3 | async_delegation.py | 异步子代理默认上限 |
| _MAX_RETAINED_COMPLETED | 50 | async_delegation.py | 内存已完成记录上限 |
| _DURABLE_RETENTION_SECONDS | 7天 | async_delegation.py | SQLite保留期 |
| _MAX_DELIVERY_ATTEMPTS | 8 | async_delegation.py | 投递尝试上限 |
| _MAX_COMPLETION_REPLAY_AGE_S | 48h | async_delegation.py | 重放年龄上限 |
| _STALE_CHECK_INTERVAL | 30s | async_delegation.py | 停滞监控间隔 |
| _STALE_IDLE_SECONDS | 450s | async_delegation.py | 异步idle停滞阈值 |
| _STALE_IN_TOOL_SECONDS | 1200s | async_delegation.py | 异步in-tool停滞阈值 |
| _STALL_GRACE_SECONDS | 120s | async_delegation.py | stalling→强制终结宽限 |
| LIVE_RETENTION_DAYS | 7 | delegation_live_log.py | 转录日志保留 |
| _CHILD_CAP_MIN | 16000 | delegate_tool.py | 压缩阈值floor |
| MAX_SCHEMA_RETRIES | 1 | delegation_output_schema.py | Schema验证重试次数 |
| _RECENT_SUBAGENTS_CAP | 200 | registry.py | 近期完成子代理记录上限 |
网硕互联帮助中心




评论前必须登录!
注册