moq-kit

Files (9)

• SKILL.md
• data-tracks-android.md · references/
• data-tracks-ios.md · references/
• playback-android.md · references/
• playback-ios.md · references/
• publishing-android.md · references/
• publishing-ios.md · references/
• screen-capture-android.md · references/
• screen-capture-ios.md · references/

SKILL.md (first lines)

# moq-kit

Native Swift and Kotlin SDKs for Media over QUIC live streaming: connect to a relay, discover broadcasts, play catalog-described streams with sub-second latency, publish camera/microphone/screen tracks, and exchange raw data tracks. Wraps the UniFFI bindings generated from `moq-ffi` (Luke Curley's `moq-dev/moq`); targets the `moq-lite` protocol, not the IETF `moq-transport` draft.

**Requirements:** iOS 16+ / macOS 13+ (Xcode 16+ to build) / Android minSdk 29, compileSdk 35. A moq-lite relay to connect to. The references cover iOS and Android; macOS is declared by the Swift package but untested here.

moq-kit is an **active preview**: public APIs, packaging, and codec coverage can still change between releases. Pin an exact version and check signatures against the installed artifact rather than assuming a shape.

**Install — iOS (SPM):**

```swift
.package(url: "https://github.com/software-mansion-labs/moq-kit", exact: "<latest release>")
// product: .product(name: "MoQKit", package: "moq-kit")
```

**Install — Android (Maven Central):**

```kotlin
dependencies { implementation("com.swmansion.moqkit:moqkit:<latest release>") }
```

Fill in `<latest release>` from https://github.com/software-mansion-labs/moq-kit/releases. Both pins are exact on purpose — loosen the SPM one (`.upToNextMinor(from:)`) only if you mean to track preview releases as they land.

## Platform detection

Reference files are split per platform — read only the ones for the project at hand:

- `Package.swift`, `*.xcodeproj`, or a `Podfile` → the `*-ios.md` references (Swift).
- `build.gradle`/`build.gradle.kts` with `com.swmansion.moqkit` → the `*-android.md` references (Kotlin).
- Cross-platform work (feature parity, bindings on top of both SDKs) → read both; the APIs are deliberately kept equivalent.

## Mental model

- A **`Session`** owns one QUIC connection to a relay and serves both subscribing and publishing. `connect()` once; a session is **one-shot** — after `close()` (or a connection error) create a new `Session`. Swift: `actor`, `connect() async throws`. Kotlin: optionally takes a `parentScope` (defaults to an internal IO scope; pass `viewModelScope`/`lifecycleScope` so cancelling it tears the session down); `connect()` is the only suspend call.
- **Subscribing:** `session.subscribe(prefix:)` → stream of `Broadcast` → `broadcast.catalogs()` → pick playable tracks → `Player(catalog:videoTrackName:audioTrackName:targetBuffering:)` → `play()`. Rendering is platform-specific: iOS adds `player.videoLayer` (an `AVSampleBufferDisplayLayer`) to the view hierarchy; Android calls `player.setSurface(surface)` with a `Surface` you own.
- **Publishing:** create capture sources (`CameraCapture`, `MicrophoneCapture`, `MultiCameraCapture`, `ScreenCapture`) and **start them yourself** → `Publisher()` + `addVideoTrack`/`addAudioTrack`/`addDataTrack` → `session.publish(path, publisher)` → `publisher.start()`. A publisher is single-use; add all tracks before `start()`.
- **Data tracks** (`DataTrackEmitter` → `addDataTrack`; consume with `broadcast.subscribeTrack(name:)`) bypass the media catalog — publisher and subscriber agree on the track name out of band.
- Everything is observable: `session.state`, `publisher.state`/`events`, player events/stats. Kotlin uses `StateFlow`/`SharedFlow`/cold `Flow`; Swift uses `AsyncStream` for session/publisher state and diagnostics, but callback subscriptions (`subscribeEvents`/`subscribeStats`) for player events/stats.

## Quick start — watch

**iOS:**

```swift
let session = Session(url: "http://localhost:4443/anon")
try await session.connect()
let subscription = try await session.subscribe(prefix: "live")
for await broadcast in subscription.broadcasts {
    for await catalog in broadcast.catalogs() {
        let video = catalog.playableVideoTracks.first?.name
        let audio = catalog.playableAudioTracks.first?.name
        guard video != nil || audio != nil else { continue }
        let player = try await MainActor.run {
            try Player(catalog: catalog, videoTrackName: video, audioTrackName: audio)
        }
        try await player.play() // render via player.videoLayer — see playback-ios.md
    }
}
```

**Android:**

```kotlin
scope.launch {
    val session = Session(url = "http://localhost:4443/anon", parentScope = scope)
    session.connect()
    session.subscribe(prefix = "live").broadcasts.collect { broadcast ->
        broadcast.catalogs().collect { catalog ->
            val video = catalog.playableVideoTracks.firstOrNull()?.name
            val audio = catalog.playableAudioTracks.firstOrNull()?.name
            if (video == null && audio == null) return@collect
            val player = Player(catalog, video, audio, parentScope = scope)
            player.setSurface(surfaceView.holder.surface)
            player.play()
        }
    }
}
```

Minimal on purpose: the nested loops handle one broadcast at a time — spawn a `Task`/`launch` per broadcast in real code — and each catalog update should tear down the previous player before creating the new one (players hold native handles).

## Quick start — go live

Same four steps on both platforms, and the order is load-bearing (rule 3):

**iOS:**

```swift
try await camera.start()                                   // 1. captures — publisher.start() won't do this
let publisher = try Publisher()                            // 2. every track before start()
publisher.addVideoTrack(name: "camera", source: camera, config: VideoEncoderConfig())
try await session.publish(path: "live/ios", publisher: publisher)  // 3. register
try await publisher.start()                                // 4. go live
```

**Android:**

```kotlin
camera.start(context, lifecycleOwner)
val publisher = Publisher()
publisher.addVideoTrack(name = "camera", source = camera, config = VideoEncoderConfig())
session.publish(path = "live/android", publisher = publisher)
publisher.start()
```

Captures, previews, encoder configs, and permissions are in the publishing references.

## Shared concepts (both platforms)

- **Catalogs:** `broadcast.catalogs()` streams catalog updates — each element **replaces** the previous; the stream ending means the broadcast went away (mark it offline). It often ends by **throwing** rather than completing — a producer that disappears surfaces as `MoqException$Mux: json: remote error: code=…` out of `collect`, so handle the exception and the clean completion identically (both demo apps and react-native-moq do). Always select from `playableVideoTracks` / `playableAudioTracks` (device-filtered on Android and for iOS audio; iOS video playability is codec-family recognition only), not the raw lists; per-track `isPlayable`/`unsupportedReason` explain exclusions (they live on the concrete `VideoTrackInfo`/`AudioTrackInfo`, not the base `TrackInfo`). Video codec strings: `"avc1"`, `"hev1"`, `"av01"`; audio: `"mp4a.40.2"`/`"aac"` (Android-published tracks advertise `"aac"`), `"opus"`. AV1 is **playback-only** — neither platform publishes it, and Apple-side decode is iPhone 15 Pro-class hardware, not a general capability.
- **Broadcast paths are prefix-relative:** subscribing with prefix `"live"` reports broadcast `live/game` as `path: "game"` — subscribe with `""` to see full paths.
- **Three observation channels on `Player`** — don't mix them up: **events** (lifecycle milestones — `player.init`, `playback.start`, `track.ready`, `track.switch`, `track.stall.start`, `rebuffer.end`, `decode.error` — wire values; Kotlin matches the `PlayerEventName` enum; not replayed), **stats** (`PlaybackStats` at ~1 s cadence — latency, fps, bitrates, `StallStats`, `timeToFirst`, buffer depths, drops, switches), and **diagnostics** (typed per-frame `PipelineEvent` — `frameDropped` with `DropStage`/`DropReason`, `stallStarted`, `latencySample`, … — bounded at 256 drop-oldest — Android hands every caller the same shared flow — and never backpressures playback; for logging/telemetry only).
- **Publisher states:** `Idle / Publishing / Stopped / Error(message)` — but Android never actually enters `Error`, and it does not react to the session dying either: an Android publisher sits in `Publishing` indefinitely after its session is gone, so the session's own state is the only signal that the broadcast stopped. Wire failure UI to **events**: `TrackStarted` / `TrackStopped` / a per-track error (Kotlin `TrackError(name, message)`, Swift `.error(name, message)`); per-track `PublishedTrack.state` (`Idle / Starting / Active / Stopped`) with `PublishedTrack.stop()` to end one track. `session.unpublish(path)` calls `publisher.stop()`.
- **`TrackDelivery`** for raw track subscription: `monotonic` (default) skips late groups — right for newest-value-wins live state; `arrival` delivers every group in order — right for chat and anything lossless.

## Critical rules

1. **Sessions, publishers, and players are single-use.** A `Session` connects once (`alreadyConnected`/`IllegalStateException` on reuse); after `close()` make a new one. A stopped `Publisher` cannot be restarted. Never restart a torn-down `Player` on either platform — create a fresh one.
2. **Media track names are local labels, catalog names are muxer-generated.** The `name:` passed to `addVideoTrack`/`addAudioTrack` (defaults `"video"`/`"audio"`) only labels `PublishedTrack` and publisher events; subscribers must discover real track names from `Catalog.videoTracks`/`audioTracks`. **Exception: data tracks** — `addDataTrack(name:)` (default `"data"`) IS the name subscribers pass to `subscribeTrack(name:)`.

From software-mansion-labs/skills, Apache-2.0.

More from software-mansion-labs/skills