scaffold-connector

Files (10)

• GUIDE.md
• SKILL.md
• connector-profile.schema.json
• dashboard-rest.yaml · examples/
• database-sqlalchemy.yaml · examples/
• pipeline-sdk.yaml · examples/
• architecture-decision-tree.md · references/
• capability-mapping.md · references/
• connection-type-guide.md · references/
• standards

SKILL.md (first lines)

# OpenMetadata Connector Building Skill

## When to Activate

When a user asks to build, create, add, or scaffold a new connector, source, or integration for OpenMetadata.

## Core Insight

**One JSON Schema definition cascades through 6 layers**: Python Pydantic models, Java models, UI forms (RJSF auto-render), API validation, test fixtures, and documentation. Define the schema once — everything else is generated or guided.

## Workflow: 7 Phases

### Phase 0: ENVIRONMENT — Set Up Python Dev Environment

Before any `make` or `python` commands, set up the environment from the repo root:

```bash
python3.11 -m venv env
source env/bin/activate
make install_dev generate
```

Always activate before running commands: `source env/bin/activate`

### Phase 1: SCAFFOLD — Generate Boilerplate

Run the scaffold CLI to collect inputs and generate files:

```bash
source env/bin/activate
metadata scaffold-connector
```

Interactive mode collects: connector name, service type, connection type, auth types, capabilities, docs URL, SDK package, API endpoints, implementation notes, Docker image, container port.

Non-interactive mode:
```bash
metadata scaffold-connector \
  --name my_db \
  --service-type database \
  --connection-type sqlalchemy \
  --scheme "mydb+pymydb" \
  --auth-types basic \
  --capabilities metadata lineage usage profiler \
  --docs-url "https://docs.example.com/api" \
  --sdk-package "mydb-sdk" \
  --docker-image "mydb/mydb:latest" \
  --docker-port 5432
```

**Output**: JSON Schema + test connection JSON + Python files + `CONNECTOR_CONTEXT.md` as an AI working document. SQLAlchemy database connectors get concrete code templates; all others get skeleton files with pointers to reference connectors.

**CONNECTOR_CONTEXT.md handling**: The scaffold generates `CONNECTOR_CONTEXT.md` in the connector directory as a working document for any AI tool (Claude Code, Cursor, Codex, Copilot, Windsurf). It is **gitignored** — it stays local and is never committed to the repo. No cleanup needed.

### Phase 2: CLASSIFY — Understand the Source

The scaffold classifies along 3 dimensions. Verify the choices:

**Dimension 1 — Service Type** (determines directory + base class):

| Service Type | Base Class | Reference |
|---|---|---|
| `database` | `CommonDbSourceService` | `mysql/` |
| `dashboard` | `DashboardServiceSource` | `metabase/` |
| `pipeline` | `PipelineServiceSource` | `airflow/` |
| `messaging` | `MessagingServiceSource` | `kafka/` |
| `mlmodel` | `MlModelServiceSource` | `mlflow/` |
| `storage` | `StorageServiceSource` | `s3/` |
| `search` | `SearchServiceSource` | `elasticsearch/` |
| `api` | `ApiServiceSource` | `rest/` |

**Dimension 2 — Connection Type** (database only):
- `sqlalchemy` → `BaseConnection[Config, Engine]` + SQLAlchemy dialect
- `rest_api` → `get_connection()` + custom REST client (ref: `salesforce/`)
- `sdk_client` → `get_connection()` + vendor SDK wrapper

**Dimension 3 — Capabilities** (determines extra files):
`metadata` (always), `lineage`, `usage`, `profiler`, `stored_procedures`, `data_diff`

Read the source-type-specific standard at `${CLAUDE_SKILL_DIR}/standards/source_types/{service_type}.md` for detailed patterns.

### Phase 3: RESEARCH — API/SDK Discovery

Read the `CONNECTOR_CONTEXT.md` generated by the scaffold. Then research the source's API/SDK.

**If you can dispatch sub-agents** (Claude Code): Launch a `connector-researcher` agent:
```
Agent: openmetadata-skills:connector-researcher
Prompt: "Research {source_name} for an OpenMetadata {service_type} connector.
Find: API docs, auth methods, key endpoints, pagination, rate limits, SDK packages."
```

**If you cannot dispatch sub-agents**: Perform the research yourself using WebSearch and WebFetch.

### Phase 4: IMPLEMENT — Fill in the TODO Items

The scaffold generates files with `# TODO` markers. Read the relevant standards before implementing:
- `${CLAUDE_SKILL_DIR}/standards/connection.md` — Connection patterns
- `${CLAUDE_SKILL_DIR}/standards/patterns.md` — Error handling, pagination, auth
- `${CLAUDE_SKILL_DIR}/standards/performance.md` — Pagination, lookup optimization, anti-patterns
- `${CLAUDE_SKILL_DIR}/standards/memory.md` — Memory management, streaming, OOM prevention
- `${CLAUDE_SKILL_DIR}/standards/source_types/{service_type}.md` — Service-specific patterns

**SQLAlchemy database**: Templates are mostly complete. Customize `_get_client()` if needed.
**Non-SQLAlchemy**: Study the reference connector, then implement each skeleton file.

**Critical for JSON Schema**:
- Make auth fields (`username`, `password`, `token`) **required** when the service needs authentication by default. If omitting a field means an opaque 401 at runtime, make it required so the UI validates upfront.
- Include SSL/TLS config (`verifySSL` + `sslConfig` `$ref`) for any connector that communicates over HTTPS — enterprise deployments use internal CAs.
- **SSL must be wired end-to-end**: schema → `connection.py` (resolve with `get_verify_ssl_fn`) → `client.py` (`session.verify = verify_ssl`). Missing wiring triggers SonarQube Security Review failure.
- See `${CLAUDE_SKILL_DIR}/standards/schema.md` for the `$ref` patterns and required fields guidance.

**Critical for Pydantic API models (models.py)**:
- Always set `model_config = ConfigDict(populate_by_name=True)` when using `Field(alias=...)` — without this, constructing instances with Python attribute names raises `ValidationError`.
- See `${CLAUDE_SKILL_DIR}/standards/code_style.md` for the full pattern.

**Critical for non-database connectors (client.py)**:
- Every list endpoint MUST implement pagination if the API supports it. Check the API docs.
- Missing pagination causes silent data loss — only the first page is ingested.

From open-metadata/OpenMetadata, Apache-2.0.

More from open-metadata/OpenMetadata