edgedelta-pipelines

Files (20)

• .gitignore
• CHANGELOG.md
• README.md
• SECURITY.md
• SKILL.md
• templates-summary.md · assets/examples/
• best-practices.md · assets/references/
• sequence-processors.md · assets/references/
• validation-rules.md · assets/references/
• deploy_pipeline.py · assets/scripts/
• inspect_environment.py · assets/scripts/
• pipeline_builder.py · assets/scripts/
• validate_pipeline.py · assets/scripts/
• template-1-log-pii-masking.yaml · assets/templates/
• template-2-otlp-dual.yaml · assets/templates/
• template-3-mixed-telemetry.yaml · assets/templates/
• template-4-api-json.yaml · assets/templates/
• template-5-multi-api.yaml · assets/templates/
• template-6-prometheus-scraper.yaml · assets/templates/
• template-7-lookup-enrichment.yaml · assets/templates/

SKILL.md (first lines)

# EdgeDelta Pipelines Skill

Creates, validates, and deploys EdgeDelta pipeline v3 configurations. Provides production-tested templates, validation tools, environment inspection, and interactive pipeline building.

## When to Use This Skill

Activate this skill when the user:
- Wants to create an EdgeDelta pipeline
- Asks about EdgeDelta configuration or monitoring
- Needs to collect logs, metrics, or traces
- Wants to validate a pipeline YAML
- Asks "what can I monitor" or wants environment inspection
- Mentions pipelines, telemetry, observability with EdgeDelta context

## Core Capabilities

1. **Quick Deploy**: Choose from 7 production-tested templates
2. **Custom Builder**: Interactive pipeline creation with environment inspection
3. **Validation**: Check pipelines against EdgeDelta rules before deployment
4. **Environment Discovery**: Inspect K8s/Linux/Windows for monitoring opportunities
5. **Direct Deployment**: Deploy pipelines via EdgeDelta API

## Quick Reference Searches

For fast lookups without loading full skill context:

```bash
# Find template by use case
grep -n "Use case:" SKILL.md

# Find template architecture
grep -n "Architecture:" SKILL.md

# Find validation rules
grep -n "validation" assets/scripts/validate_pipeline.py

# Find processor examples
grep -rn "type: generic_mask" assets/templates/

# Find API deployment script
ls assets/scripts/deploy_pipeline.py

# See all templates
ls assets/templates/template-*.yaml
```

## Available Workflows

### Workflow 1: Quick Template Deployment

**When**: User wants to quickly deploy a pipeline

**Steps**:
1. Ask for EdgeDelta credentials:
   - Organization ID
   - API Token
   - Check `~/.edgedelta.env or project .env file` first
   - Store in conversation context

2. Present template options:
   - **Template 1**: Log Ingestion with PII Masking
     - Use case: Application logs with compliance requirements
     - Architecture: file_input → sequence (3x generic_mask + extract_metric) → ed_output

   - **Template 2**: OTLP Dual Receiver
     - Use case: OpenTelemetry data collection
     - Architecture: otlp_input (gRPC+HTTP) → sequence (transform + metrics) → ed_output

   - **Template 3**: Mixed Telemetry Processing
     - Use case: Logs AND metrics with different processing
     - Architecture: file+otlp inputs → parallel sequences → ed_output

   - **Template 4**: API Pull with JSON Processing
     - Use case: REST API polling (ServiceNow, CMDB)
     - Architecture: http_pull → sequence (json_unroll + transform) → ed_output

   - **Template 5**: Multi-API with Aggregation
     - Use case: Multiple API sources (Duo Security pattern)
     - Architecture: 3x http_pull → 3x sequences → aggregator → ed_output

   - **Template 6**: Prometheus Metrics Scraper
     - Use case: Scrape Prometheus exporters (node_exporter, custom metrics)
     - Architecture: prometheus_input → sequence (transform) → ed_output
     - Features: Multi-target scraping, relabel_configs, cluster monitoring

   - **Template 7**: Lookup Enrichment
     - Use case: Enrich logs with reference data (user metadata, IP geo, product catalogs)
     - Architecture: compound (input → sequence (parse + lookup + transform) → route → matched/unmatched outputs)
     - Features: CSV/DB lookups, conditional routing, enrichment validation

3. Ask customization questions based on selected template:
   - Tag name
   - File paths (Template 1, 3)
   - Ports (Template 2, 3)
   - API endpoints (Template 4, 5)
   - Pull intervals (Template 4, 5)
   - Prometheus targets (Template 6)
   - Scrape intervals (Template 6)
   - Lookup CSV path and key fields (Template 7)
   - Enrichment attributes (Template 7)

4. Read the template from `assets/templates/template-{number}-{name}.yaml`

5. Customize the template with user's values

6. Run validation: `python3 assets/scripts/validate_pipeline.py /tmp/customized.yaml`

7. If valid, deploy: `python3 assets/scripts/deploy_pipeline.py /tmp/customized.yaml <org_id> <api_token>`

8. Provide deployment details:
   - Pipeline ID
   - App URL: https://app.edgedelta.com/pipelines/{id}
   - Install command

### Workflow 2: Environment Inspection + Custom Pipeline

**When**: User wants to build based on their environment OR asks "what can I monitor"

**Steps**:

From edgedelta/skills, Apache-2.0.