mfs-ingest

Files (24)

• SKILL.md
• bigquery.md · reference/connectors/
• discord.md · reference/connectors/
• feishu.md · reference/connectors/
• file.md · reference/connectors/
• gdrive.md · reference/connectors/
• github.md · reference/connectors/
• gmail.md · reference/connectors/
• hubspot.md · reference/connectors/
• jira.md · reference/connectors/
• linear.md · reference/connectors/
• mongo.md · reference/connectors/
• mysql.md · reference/connectors/
• notion.md · reference/connectors/
• postgres.md · reference/connectors/
• s3.md · reference/connectors/
• slack.md · reference/connectors/
• snowflake.md · reference/connectors/
• web.md · reference/connectors/
• zendesk.md · reference/connectors/
• credentials.md · reference/
• error-codes.md · reference/
• troubleshooting.md · reference/
• update-flow.md · reference/

SKILL.md (first lines)

# MFS — register / update / re-sync data sources

## 1. What this skill does

Walks the user through getting a data source into MFS so it's searchable.
The work splits into:

1. Picking the right connector scheme.
2. Collecting credentials (preferring `env:VAR` / `file:/path` indirection
   over plaintext).
3. Writing a connector TOML.
4. Calling `mfs add <uri> --config <toml>` and monitoring the returned job.

Each connector has its own field set, credential acquisition story, and
gotchas. Per-connector details live in
`reference/connectors/<scheme>.md` — **read the matching one before
collecting fields** for any scheme.

## Step 0: Pre-flight (always run first)

```bash
mfs --version            # missing? `cargo install mfs-cli` (see install row below)
mfs status               # server reachable? connectors/jobs visible?
mfs config show          # endpoint/profile/client id/server-info debugging
mfs connector list       # what's already configured?
```

Branch on the result:

| Signal | Action |
|---|---|
| `mfs` not found | install the CLI (Rust): `cargo install mfs-cli`, or the shell installer from the project's GitHub releases page. |
| `mfs status` connection refused | the configured server is down. Tell the user how to bring it up — pre-release, the server runs from source: `git clone https://github.com/zilliztech/mfs.git && cd mfs/server/python && uv sync && uv run mfs-server setup && uv run mfs-server run` — and wait. Work only through the configured endpoint rather than pointing the CLI at a different server. |
| `mfs status` returns 401 unauthorized | the user's `MFS_API_TOKEN` is missing/wrong. Use `mfs config show` to confirm the endpoint/profile, then set the intended token source and retry. |
| server up + `connector list` empty | first-ever connector; jump to **§B (greenfield walk-through)** when intent matches |
| server up + N connectors registered | proceed to Step 1 intent classification |

## Step 1: Classify intent (the central decision)

Read the user's most recent message. Pick exactly one row:

| User said... | Intent | Jump to |
|---|---|---|
| "add postgres prod-db to MFS" + credentials available (env / file / about to paste) | **A. Zero-friction add** | §A |
| "I want to add postgres / slack / X" (no specifics, vague) | **B. Greenfield walk-through** | §B |
| "re-sync github", "re-index slack", "pull latest from jira" | **C. Force re-ingest** | §C |
| "update my slack token", "change postgres host", "switch to new DSN" | **D. Edit existing config** | §D |
| "what connectors do I have", "list registered sources" | **E. List** | §E |
| "find X" / "search Y" / "grep Z" / "cat W" | **wrong skill** | redirect to `mfs-find`, stop |
| "is X indexed yet" / "did the sync finish" / "search returns nothing" | **wrong skill or boundary** | suggest `mfs-find` for query-side diagnosis; if user says it's an ingest issue, jump to §C or §F |
| Truly unclear after a re-read | **F. Clarify** | §F |

### Mid-flow redirect

If at any point the user changes intent ("wait, just list what I have" /
"actually let me just re-sync the existing one"), abandon the current §
and jump to the new one. Don't insist on finishing the original branch.

---

## §A. Zero-friction add

User knows what to add and has credentials handy. Aim for: ≤3 questions
to the user, then write toml + run `mfs add`.

1. **Parse the URI** from the user's message. Shape: `<scheme>://<alias>`.
   Scheme is required and is the connector type (`postgres`, `slack`, …).
   Alias is the human-readable instance ID — gets used as the toml
   filename and the connector's row in metadata.
   - If only `<scheme>` was given (no alias), ASK: "What should I call
     this instance? (free-form; appears as the URI host part, e.g.
     `postgres://**prod-db**`)"
   - **`file` takes a bare path, not an alias.** The target is a local
     path: `mfs add /abs/path` (the URI is derived as `file://local/abs/path`).
     The path is client-side: on the same host the server reads it directly; on
     a different host the CLI bundles and uploads the tree (`--upload` /
     `--no-upload` to force).

2. **Read the matching `reference/connectors/<scheme>.md`** for the
   required field set, and `reference/credentials.md` for how credentials
   work. For each credential field, put a **reference** in the toml —
   `env:VAR_NAME` or `file:/abs/path` — which the server resolves against its
   own environment / filesystem at ingest time. Make sure that value is present
   where the server runs (client and server share a machine on a loopback
   endpoint; otherwise it lives on the server — ask the user if unsure).

3. **Write the toml** to a temp path:
   ```toml
   # mfs-server connector config — <scheme>
   # URI: <uri>
   <field1> = "<value or env:VAR>"
   <field2> = "<value>"
   ...
   ```
   Use a path like `/tmp/mfs-<alias>.toml` so it doesn't pollute the
   user's cwd.

4. **Run `mfs add` in estimate-confirm mode** for external sources where
   cost matters (databases >100k rows, GitHub repos with many issues,
   large Slack workspaces, full website crawls):
   ```bash
   mfs add <uri> --config /tmp/mfs-<alias>.toml
   ```
   For non-local targets, the current CLI automatically calls
   `/v1/connectors/estimate` and prompts `Continue? [y/N]` unless `--yes`
   is set. There is no standalone `--estimate` flag. Show the estimate to
   the user and only answer yes when the user has approved.

   For small / unambiguous sources (single repo of docs, one CRM with
   <10k records, a defined Slack channel), the same command is still the
   normal add path. Use `--yes` only when the user has already accepted
   skipping the estimate confirmation.

   **Whole-account enumerators** (gdrive = the entire Drive; feishu user-mode docs =
   the entire My Space): if the estimate is large, don't just confirm a full index —
   first propose narrowing by time. Re-estimate with a recent start date (`POST
   /v1/connectors/estimate` with a `since` field) to show the smaller count, then add
   with that bound:
   ```bash

From zilliztech/mfs, Apache-2.0.

More from zilliztech/mfs