byted-util-volcengine-detect-retry

Files (1)

• SKILL.md

SKILL.md (first lines)

# Volcengine Detect Retry

Reduce Volcengine Cloud Detect (云拨测) operations toil. This skill confirms whether a
failing dial point is a real outage or just noise, and — when a dial point itself is broken
— suggests a similar replacement node.

**This skill ships no script of its own.** It is pure orchestration: every action is one of
the `byted-util-volcengine-cloud-detect` skill's `volc_detect.py` CLI commands
(`get-task`, `get-result`, `list-nodes`, `fuzzy-search-nodes`, `create-task`, `delete-task`).
Follow the workflow below by invoking those commands in sequence and applying the decision
logic to their JSON output.

**Requires**: `byted-util-volcengine-cloud-detect` skill. Do not reimplement any Cloud Detect API call here.

## Locate The Base CLI

Resolve `volc_detect.py` dynamically (never hard-code an absolute path):

```bash
VOLC_DETECT="${VOLC_DETECT:-$(find "${SKILLS_DIR:-$HOME/Code/Skill}" -type f \
  -path '*/byted-util-volcengine-cloud-detect/scripts/volc_detect.py' 2>/dev/null | head -n 1)}"
test -n "$VOLC_DETECT" || { echo "volc_detect.py not found"; exit 1; }
alias vd="python3 \"$VOLC_DETECT\""
```

All commands below are written as `python3 "$VOLC_DETECT" <command>`.

## Prerequisites

Volcengine credentials must be present (the base skill enforces this). If either is missing,
follow the base skill's Prerequisites section to collect and persist them:

```bash
test -n "$VOLC_ACCESSKEY" && test -n "$VOLC_SECRETKEY"
```

## Decision Logic

For each dial point whose **latest** sample in the scan window failed:

1. **Retry 5 times** (configurable) against the **same** monitored target via a temporary
   `create-task`.
   - **All retries succeed** → transient glitch → **ignore** (`transient_issue`).
   - **Any retry fails** → go to control validation.
2. **Control validation**: a temporary `create-task` dials a trusted domain from the **same** node.
   - China nodes → `https://cn.bing.com`
   - Overseas nodes → `https://www.google.com`
   - **Control succeeds** → `real_failure` (target is genuinely down for that node) → **ALERT**.
   - **Control fails** → `node_issue` (the dial point itself is broken) → **find similar
     replacement nodes and suggest replacing it**.
3. Temporary retry/control tasks are **always** deleted (`delete-task`).

By default only **LastMile** dial points (`client_info.type == "1"`, node name suffix `（LM）`)
are auto-validated, because they are the noisy ones. Include IDC/Private only on request.

Replacement-node priority (`node_issue` only):
1. **Mandatory**: same node type (LastMile / IDC / Private).
2. **Preferred**: same province, then same country.

## Success Rule

A result sample is **success** when `basic_detail.usability_info.status` is `"success"`
(or boolean `true`); it is a **failure** when status is `"fail"` (the `usability_info.reason`
array, e.g. `["主机未找到(601)"]`, and `basic_detail.error_msg` explain why). If no
`usability_info` is present, fall back to `http_detail.http_code` in `200–499` = reachable.

## Workflow: Scan One Periodic Task

### Step 1 — Fetch the task and its recent samples

```bash
# Task must be Running (status == 2) to be actively monitoring.
python3 "$VOLC_DETECT" get-task --task-id <TASK_ID>

# Pull the last hour of results (API limits the range to <= 1 hour, PageSize <= 500).
NOW=$(date +%s); START=$((NOW-3600))
python3 "$VOLC_DETECT" get-result --task-id <TASK_ID> \
  --start-time $START --end-time $NOW --page-size 500
```

### Step 2 — Identify failing nodes (latest sample per node)

From the `Data` array, group samples by node and keep the **latest** (max `basic_detail.timestamp`)
per node. A node is identified by its `basic_detail.client_info` (`region` / `city` / `isp` / `type`).
Map that back to a node `id` + full name with `list-nodes` (match `region`, `city`, `isp`
substrings; if `client_info.type` is present, also match node type to disambiguate LM vs IDC).

```bash
python3 "$VOLC_DETECT" list-nodes
```

Keep only nodes whose latest sample is a **failure**. By default, drop non-LastMile nodes
(`client_info.type != "1"`) unless the user asked to include IDC/Private. If a `client_info`
maps to multiple nodes (ambiguous) or none, mark it `inconclusive` and do not auto-page.

### Step 3 — Retry each failing node 5× against the same target

`get-task` gives the monitored `address`. Create a temporary task on the failing node only:

```bash
ADDRESS="<the task's address>"; LINE_ID=<failing node id>
NOW=$(date +%s); FINISH=$((NOW + 5*60 + 120))   # 5 rounds @ 60s + margin
python3 "$VOLC_DETECT" create-task \
  --address "$ADDRESS" --name "retry_${LINE_ID}_${NOW}" \
  --type 1 --node-count 1 --interval-seconds 60 \
  --finish-time $FINISH --line-ids $LINE_ID --http-method 1
```

Wait for ~5 samples (poll `get-task` until `status == 6`, or sleep `rounds × interval`),
fetch the results, then **delete** the temporary task:

```bash
END=$(date +%s); START=$((END-3600))
python3 "$VOLC_DETECT" get-result --task-id <RETRY_TASK_ID> \
  --start-time $START --end-time $END --page-size 500
python3 "$VOLC_DETECT" delete-task --task-id <RETRY_TASK_ID>
```

- **All retry samples succeed** → `transient_issue` → ignore. Done for this node.

From bytedance/agentkit-samples, Apache-2.0.

More from bytedance/agentkit-samples