confluent-cloud-flink-sql

Files (8)

• SKILL.md
• evals.json · evals/
• cli-reference.md · references/
• dialect-traps.md · references/
• formats-and-serialization.md · references/
• reserved-words.md · references/
• sql-patterns-cc.md · references/
• troubleshooting-cc.md · references/

SKILL.md (first lines)

# Confluent Cloud Flink SQL

Enforce the CC-Flink-vs-OSS-Flink dialect boundary and the CLI-driven verification loop for any Confluent Cloud Flink SQL work. Apache Flink OSS training data is a trap — CC rejects or silently mishandles a long list of otherwise-valid Flink SQL constructs.

Scope note: this skill's reference material is built around the OSS-vs-CC dialect boundary — traps in constructs that exist in both dialects but behave differently. It does not yet catalog CC-only DDL that has no OSS counterpart (e.g. `CREATE MATERIALIZED TABLE`, `CREATE MODEL`/`AI_COMPLETE`, `CREATE AGENT`, `USE CATALOG`). For those, verify directly against the [CC Flink SQL reference](https://docs.confluent.io/cloud/current/flink/reference/overview.md) rather than expecting a trap entry here.

## Non-negotiables

1. **CC Flink ≠ Apache Flink.** Verify every API, SQL construct, and runtime behavior against:
   - [Confluent Cloud Flink docs](https://docs.confluent.io/cloud/current/flink/overview.md)
   - [CC Flink SQL reference](https://docs.confluent.io/cloud/current/flink/reference/overview.md)
   - [Confluent Terraform provider](https://registry.terraform.io/providers/confluentinc/confluent/latest/docs)
   - Live `confluent flink shell` against the user's compute pool
2. **No mocks in verification.** Integration claims require real `confluent` CLI runs. Unit tests may mock; anything calling itself "end-to-end verification" may not.
3. **Record decisions somewhere durable.** Ask the user where they want dialect traps and verification notes tracked (e.g. `docs/flink-dialect-traps.md` in their project) before writing any new file — don't assume a `docs/` layout.
4. **Secrets never in repo.** `CREATE CONNECTION` parameters are Terraform-injected, never hardcoded. Gitignore `.tfvars`, `.tfstate*`, `*.secret*` from day one.
5. **EXPLAIN before CREATE.** Always `EXPLAIN` a query before `statement create` — catches parse/type errors without consuming CFUs.
6. **Don't invent identifiers.** Use `<placeholder>` for any topic, table, statement, or resource name you haven't verified.

## Stream Processing Workflow

When a user asks to build, modify, or troubleshoot a Flink SQL streaming solution, first understand the streaming problem and identify the required data and processing semantics before generating SQL.

Use this workflow:

1. Understand the requirement
2. Discover existing Confluent resources
3. Establish the required streaming semantics (when needed)
4. Propose the processing design (when needed)
5. Confirm the design before modifying resources
6. Generate and validate Flink SQL
7. Execute the statement when explicitly approved
8. Observe and verify the result

Do not ask clarifying questions if a request is complete (i.e., all required information, context, and parameters are provided or retrievable via available MCP tools). Skip directly to implementation and validation. 

Only ask questions when critical information is missing and cannot be resolved through existing context or tools.

### 1. Understand the Requirement

Before generating Flink SQL, determine what the user is trying to accomplish.

Establish, when relevant:

- What streaming problem is being solved?
- What data/events should be processed?
- What are the input sources?
- What should the output represent?
- Where should the output go?
- What business conditions determine which records are processed?
- What are the required time semantics?
- What should happen with duplicates, late events, or invalid data?
- Are there important assumptions that could change the result?

Ask only questions that materially affect the pipeline design.

Do not ask for information that is already available in the conversation.

### 2. Discover Existing Confluent Resources

When MCP tools are available, prefer inspecting existing Confluent resources over asking the user for information that can be discovered.

For referenced input resources:

1. Find the relevant topic/table.
2. Inspect its schema.
3. Inspect representative records when necessary.
4. Identify relevant keys and timestamp fields.
5. Confirm the fields required by the proposed transformation.
6. Identify the target resource if it already exists.

Do not invent topic names, table names, column names, schemas, or timestamp fields.

If multiple resources could match the request, present the candidates and ask the user to select one.

### 3. Establish Streaming Semantics (when needed)

Before generating SQL, determine the semantics that materially affect the result. Skip this step if all relevant semantics are already established from the user's prompt or the discovery step.

Examples include:

- Event time vs processing time
- Window duration and behavior
- Grouping keys
- Join keys and temporal semantics
- Deduplication key and ordering
- Late-event handling
- Output granularity
- Whether the result represents individual events, aggregates, alerts, or another derived stream

### 4. Propose the Processing Design (when needed)

For non-trivial requests, summarize the proposed design before generating executable SQL. For simple, fully specified requests, this can be reduced to a brief summary or skipped entirely.

Use:

**Input**
- Relevant topic/table(s)
- Relevant fields

**Processing**
- Main transformations
- Joins/enrichment
- Aggregation/windowing
- Other required operations

**Output**
- Target topic/table
- Output semantics

**Assumptions**
- Material assumptions that have not been explicitly specified

Ask the user to confirm or correct the design before creating or modifying resources.

### 5. Plan Before Execute

Before creating, modifying, or deleting a Confluent resource, present the execution plan.

From confluentinc/agent-skills, Apache-2.0.

More from confluentinc/agent-skills