bigbrain-media-ingest

Files (4)

• SKILL.md
• openai.yaml · agents/
• validate-skill.sh · scripts/
• cases.md · tests/

SKILL.md (first lines)

# BigBrain Media Ingest

Preserve time-based media and create a comprehensive, readable account of what the source communicates from beginning to end.

## Contract Checklist

- Read and follow [the shared ingestion completeness checklist](../bigbrain-ingest/references/ingestion-completeness.md) before any Brain write, including when this skill is invoked directly. Complete that checklist before declaring ingestion complete.
- Read live `filing_rules` before choosing a destination or writing.
- Deduplicate by stable source ID, exact title, proposed path, and direct reads of plausible matches.
- Preserve the complete timestamped transcript when one is available or can reasonably be produced, subject to the YouTube sponsor-read redaction rule below.
- Store the transcript under the owning collection's flat `.raw/` path with exactly one same-basename Markdown sidecar.
- For YouTube, preserve the exact video title and channel as source metadata, then derive the sidecar title and normalized basename from them.
- When YouTube provides chapters, preserve every chapter label and its order as the sidecar's primary section structure.
- Run the deterministic YouTube metadata and sidecar checks before writing; if a check fails, stop and let the chat show the error.
- Make the sidecar a comprehensive chronological prose exposition, not a condensed thematic summary or an analytical rewrite.
- Write as a third-person observer taking detailed, readable meeting minutes: preserve the source's sequence, substance, reported facts, definitions, examples, evidence, counterarguments, qualifications, and key short quotes without forcing them into a rigid schema.
- Attribute claims to the source and distinguish what the source reports from independently verified facts. Do not add an unlabelled agent take.
- Apply [the Fact Sheet standard](../bigbrain-ingest/references/fact-sheet.md) to the verified transcript. When substantial recall-worthy facts qualify, put a short attributed `## Fact Sheet` near the top of the sidecar, before the chronological account; omit it otherwise.
- Reconcile canonical subjects and every identifiable mentioned entity using the shared checklist and `$bigbrain-page`; keep the source sidecar on its specialized shape.
- Rely on the owning Brain MCP for automatic propagation and backup, and independently read back the sidecar, raw transcript, and canonical updates. Run manual sync or maintenance only if verification identifies a failed side effect that requires repair.

## YouTube Identity and Chapter Standard

For every YouTube transcript ingest:

- Store the raw values separately in frontmatter as `youtube_title`, `channel`, and `youtube_id`.
- Set the sidecar title to `<exact YouTube video title> - <exact YouTube channel name>` only after preserving and validating the raw values.
- Do not use a generated topic label in place of the exact YouTube video title.
- Derive the sidecar and raw transcript basename from `<slugified video title>-<slugified channel name>`.
- Keep the channel name as the final suffix in both the display title and normalized basename.
- Preserve source capitalization and punctuation in the display title where BigBrain permits it. Normalize only the basename.
- Store the YouTube ID in frontmatter for deduplication and provenance.
- If distinct videos share a title and channel, place a short ID disambiguator before the channel suffix.
- If live metadata cannot resolve the exact title or channel, stop before writing. Do not invent either value.
- If the YouTube description contains a `Timestamps:` or `Chapters:` block, parse its timestamp and label pairs before drafting the exposition.
- Use one primary heading for every parsed chapter, preserving the exact label and source order. Agent-authored headings may appear only as subordinate headings beneath the source chapter.
- A `## Fact Sheet` may appear before the first chapter as an editorial preface; it is not a chapter heading and must not change the validated chapter heading sequence.
- Apply the YouTube skill's sponsor-read rule before filing the transcript: remove promotional reads only, insert timestamped placeholders, and disclose the redacted ranges in the sidecar coverage note. Parse chapters from the exact retrieved description before redacting promotional copy and links in the saved description. Preserve substantive editorial material and every published chapter label, including sponsor-labelled chapters.
- If no chapters are present, label the sidecar metadata with `chapter_source: none` and make any fallback chronology visibly agent-authored.

## Source Exposition Standard

The sidecar should feel like a written version of the recording prepared by an attentive third-person observer. Follow the source's natural progression and use readable paragraphs and descriptive headings where they help. Do not turn the page into a form, ledger, database, or repeated checklist.

As the source unfolds, naturally preserve the material a future reader would need to understand what happened and what was communicated. This can include what was discussed, reported facts and figures, definitions and terminology, examples and case studies, speaker or narrator claims, supporting evidence, counterarguments, qualifications, transitions in the argument, and concise key quotes. These are prompts for faithful writing, not required headings or a rigid schema.

Aim for comprehensive coverage rather than maximal compression. A reader should be able to follow the full source without watching it, while the raw transcript remains the authority for exact wording. Timestamps are optional navigation aids for especially useful passages, user highlights, or key quotes; timestamp ranges are not required throughout the exposition.

Keep the exposition observational. It may explain how one part leads to the next, but it should not replace the source's structure with a new framework or silently derive its own conclusions. If agent analysis would add durable value, place it on the appropriate canonical concept page or in a clearly labelled, brief analytical note that cannot be confused with the source account.

## Workflow

1. Resolve the source and destination:
   - Read the shared ingestion completeness checklist linked above and use it throughout this workflow, even on direct invocation.
   - identify the source URL or local media, exact title, publisher or channel, stable source ID, duration, publication date, speakers when known, and primary subject
   - call live `filing_rules` and select the owning collection by primary subject rather than media format
   - preserve `youtube_title`, `channel`, `youtube_id`, and the raw description before drafting or constructing write arguments
   - run `validateYouTubeMetadata` and `buildYouTubeIdentity` from `src/bigbrain/youtube-ingest.js`; a thrown error stops the run before any raw or page write
   - Anti-patterns: filing by format, guessing metadata, using a generated topic label as the source identity, constructing the display title before validating raw metadata
2. Deduplicate before retrieval or mutation:
   - search for the stable source ID, exact title, proposed sidecar path, and close source matches
   - directly read plausible matches and update the existing source record when the stable ID already exists
   - use `findYouTubeRecordById` and `assertExistingYouTubeRecordCompatible` before mutation; same-ID records are updated in place or left for direct review, never duplicated
   - if more than one existing record has the ID, stop and resolve the duplicate before writing
   - distinct IDs with the same title and channel receive an ID disambiguator before the channel suffix
   - Anti-patterns: creating from semantic-search absence, duplicate pages for one source, reading only snippets, overwriting a same-ID conflict
3. Retrieve complete source support:
   - prefer published manual captions, then published automatic captions, then a clearly labelled local transcription
   - preserve the complete available substantive transcript and record coverage, method, diarization status, and accuracy caveats; for YouTube, apply the sponsor-read exception above and record each redacted range
   - retain source metadata and any user-highlighted passages or interpretations
   - parse the YouTube description's chapter block with `parseYouTubeChapters` before prose synthesis
   - Anti-patterns: excerpt-only capture, calling a partial transcript complete, presenting machine captions as human-verified, treating chapters as optional context
4. Write the raw transcript and chronological sidecar:
   - upload the transcript to `<collection>/.raw/<basename>.<ext>` and create the sole same-basename sidecar
   - write a faithful chronological exposition using the Source Exposition Standard, preceded by the optional Fact Sheet only when the linked standard's threshold is met
   - when chapters exist, use `chapter.heading` values as the primary section headings and place the detailed prose under the corresponding source section
   - when chapters do not exist, set `chapter_source: none` and label fallback headings as agent-authored chronology
   - preserve source claims as attributed claims and integrate concise key quotes only where their wording materially matters
   - link the sidecar to its raw transcript and any directly relevant existing pages
   - Anti-patterns: condensed thematic summary, rigid fact ledger or glossary schema, merged or renamed source chapters, agent-authored thesis replacing the source, long transcript copy in the Markdown page, fact bullets without source location or qualification
5. Reconcile canonical coverage:
   - Apply the shared checklist to the complete transcript and source sidecar, using `$bigbrain-page` for canonical creation and updates.
   - Keep the chronological source exposition on the sidecar and supported reusable knowledge on its canonical subject page.
   - Link already-covered entities without manufacturing new facts or duplicate updates.
   - Anti-patterns: treating entity reconciliation as optional, creating a separate concept page for every chapter or fact, copying the exposition into a concept page, upgrading source claims into facts
6. Verify through the owning Brain:
   - Complete the shared checklist before declaring success.
   - directly read back the canonical page and source sidecar when canonical changes were written
   - read back the sidecar and run `validateYouTubeSidecar` against the retrieved metadata, description, frontmatter, and body before treating the run as complete
   - confirm exact source metadata, derived title, stable source ID, raw-file link, chapter count, chapter labels, chapter order, chronological coverage, and evidence caveats
   - read back the raw transcript, compare its bytes when practical, and confirm it appears in the raw-file listing
   - read back any changed canonical page and verify its link to the source sidecar
   - a failed read-back assertion is a failed run; do not report a successful write as sufficient evidence of source fidelity
   - Anti-patterns: declaring success from a write response, skipping raw-file verification, skipping the metadata or chapter audit, reporting a canonical update that was not read back

## Anti-Patterns

- Treating the sidecar as a short summary, list of takeaways, or new analytical framework.
- Reorganizing the source primarily by themes when that loses its original progression.
- Replacing a published YouTube chapter map with an agent-authored thematic taxonomy.
- Treating the derived display title as proof that the exact source title was preserved.
- Requiring timestamp ranges, a fact ledger, a glossary, or a coverage matrix as fixed sidecar sections when the source does not provide them.
- Omitting quantitative claims, terminology, examples, evidence, counterarguments, or key quotes merely because they are inconvenient to summarize.
- Creating duplicate entity pages or leaving mentioned entities unreconciled because they appear only briefly.
- Confusing source-reported claims with independently verified facts.
- Reporting completion without same-MCP read-back or adding routine manual sync after successful MCP writes.

## Output

Briefly report the source title, sidecar and raw artifact, preserved coverage and any transcription gaps, canonical pages created, updated, or linked as already covered, and the read-back result. Mention user-highlighted passages when applicable. Do not require a fixed output schema or repeat the internal checklist.

From life-efficient/bigbrain, MIT.

More from life-efficient/bigbrain