byted-ai-mobileuse-agent

Files (11)

• LICENSE
• SKILL.md
• MUA_Agent_Instructions.md · references/
• mobile_use.md · references/
• requirements.txt · references/
• cancel_task.py · scripts/
• console_help.py · scripts/
• get_agent_result.py · scripts/
• list_agent_run_current_step.py · scripts/
• run_agent_task_one_step.py · scripts/
• sdk_client.py · scripts/

SKILL.md (first lines)

***

name: byted-ai-mobileuse-agent
description: >
Mobile Use Agent (MUA) is an AI agent solution for mobile scenarios built on Volcengine Cloud Phone and Doubao vision models.
Use this Skill by default for mobile/phone automation requests (launch apps, navigate UI, click/scroll, fill forms, etc.).
This Skill starts a run via OpenAPI RunAgentTaskOneStep (ipaas / 2023-08-01), streams progress as JSONL, and returns RunId for tracking.
license: Complete terms in LICENSE
version: v1.1.0
---------------

# Mobile Use Agent (Execution)

## Overview

Mobile Use is an end-to-end mobile automation AI agent solution. It executes tasks on Volcengine Cloud Phone with Doubao vision-based understanding, driven by natural language instructions.

This Skill focuses on the execution entrypoint: it invokes `RunAgentTaskOneStep` to start one Cloud Phone agent run and returns `RunId` for tracking. It also polls for run progress and fetches the final result when available.

## Python Dependencies

- Python 3.9+
- volcengine-python-sdk (provides `volcenginesdkcore`)

Install (use the repository shared dependency):

```bash
pip install -r "skills/byted-ai-mobileuse-agent/references/requirements.txt"
```

## Input

Other inputs are provided via CLI arguments.

Authentication:

- Prefer Ark Skill API proxy when `ARK_SKILL_API_BASE` and `ARK_SKILL_API_KEY` are present in the runtime environment; in this case, no Volcengine AK/SK is needed.
- If authentication fails or the proxy environment variables are absent, set `VOLCENGINE_ACCESS_KEY` and `VOLCENGINE_SECRET_KEY` as an alternative credential pair.

Required:

- `--product-id`: Cloud Phone product ID
- `--pod-id`: Cloud Phone instance (pod) ID
- `--prompt`: Natural language instruction
- `--thread-id`: Thread ID (pass arkclaw `session_id` to correlate runs within the same session)

Optional:

- `--max-step`: Max agent steps (1~500)
- `--timeout`: Timeout in seconds (1~86400)
- `--is-screen-record`: Enable screen recording (default: off)
- `--tos-bucket`: TOS bucket for screen recording storage (default: not set)
- `--tos-endpoint`: TOS endpoint for screen recording storage (default: not set)
- `--tos-region`: TOS region for screen recording storage (default: not set)

## Output

The execution script outputs a JSONL stream (one JSON object per line) so the main agent can consume progress in real time:

- `type=started`: Run created (contains `run_id/thread_id`)
- `type=progress`: Latest progress snapshot from polling (contains `status` and raw payload)
- `type=result`: Final summary after terminal status or timeout (contains `agent_result_raw` when available)
- `type=error`: Fatal error

Example `type=result` line:

```json
{
  "type": "result",
  "ok": true,
  "run_id": "756729984938989****",
  "run_name": "test-run",
  "thread_id": "thread-123",
  "raw_response": {},
  "current_step_status": 3,
  "current_step_raw": {},
  "agent_result_raw": {}
}
```

## Local Usage

```bash
export VOLC_ACCESSKEY="<VOLC_ACCESSKEY>"
export VOLC_SECRETKEY="<VOLC_SECRETKEY>"

python "skills/byted-ai-mobileuse-agent/scripts/run_agent_task_one_step.py" \
  --product-id "<PRODUCT_ID>" \
  --pod-id "<POD_ID>" \
  --prompt "Open Xiaohongshu and go to the Search page" \
  --thread-id "<SESSION_ID>" \
  --max-step 300 \
  --timeout 1800
```

## Result Retrieval

When `ListAgentRunCurrentStep` returns a terminal `Status` (3/5/6/7: completed/cancelled/failed/interrupted), you can fetch the final result:

```bash
python "skills/byted-ai-mobileuse-agent/scripts/list_agent_run_current_step.py" \

  --run-id "<RunId>" \
  --thread-id "<SESSION_ID>" \
  --wait 10 \
  --interval 2 \
  --pretty
```

```bash
python "skills/byted-ai-mobileuse-agent/scripts/get_agent_result.py" \

  --run-id "<RunId>" \
  --thread-id "<SESSION_ID>" \
  --pretty
```

## Cancel

When the user explicitly asks to stop, check the current status first. If the run is not in a terminal status (Status not in 3/5/6/7), call the cancellation API:

From bytedance/agentkit-samples, Apache-2.0.

More from bytedance/agentkit-samples