Nyxmix

Recent edits with author (human or agent) — use it to see what the human changed.

Render a waveform or spectrogram PNG of the mix, an asset or a clip, so you can look at the audio.

Fundamental-frequency (pitch) track of an asset, measured with YIN every 10 ms on a mono mix. Returns a summary (medianHz, minHz/maxHz after a 5-frame median so single octave slips do not count, p10Hz/p90Hz, voicedFraction, meanConfidence) and the track as [t, hz|null, confidence] decimated to maxPoints (default 120; median of the voiced frames per point). Use it to check a voice (typical speaking f0: ~85–180 Hz lower voices, ~165–255 Hz higher voices), find where someone is speaking (voiced) versus noise, or verify a pitch shift. With clipId and no start/end, measures the clip's source window…

Read the session: tracks, clips (with times), effects, assets, markers. format "summary" (default) is compact text; "json" is the full document.

Read the TIMELINE transcript: words from every clip on unmuted tracks, mapped to timeline seconds (so it reflects all edits; re-read after cutting, because later times shift). format "text" (default): one line per phrase, "[start–end] words…" (with "Track:" when several tracks speak), filler words shown as ‹um›, low-confidence words suffixed with ~. format "words": JSON {words:[{start,end,word,probability,trackId,clipId,filler?}]} for exact word times.

The session's reference video: name, duration, size, fps, offset (timeline seconds where frame 0 sits) and the soundtrack asset id. Timeline time = offset + video time.

Add an audio file to the session bin from a public URL or base64 data. Any common format; converted losslessly to 32-bit float at the project rate. Returns the asset (id, duration). Then place it with clip.add.

Attach a reference video to the session (one per session; a new one replaces the old) so audio can be cut to picture. Give a public URL or base64 data. NyxMix makes a browser proxy and a filmstrip, and extracts the soundtrack into the bin as an asset (audioAssetId). place:true also adds a "Video sound" track with the soundtrack clip at the video offset. Slow for long videos (it transcodes).

Install a real-time effect written in Faust (https://faust.grame.fr), compiled to WebAssembly on the server. Example: import("stdfaust.lib"); cutoff = hslider("cutoff [unit:Hz]", 2000, 50, 18000, 1); process = par(i, 2, fi.lowpass(4, cutoff)); Every hslider/vslider/nentry/checkbox/button becomes a parameter named after its label in camelCase (e.g. "Wet Dry" → wetDry) with the Faust min/max/init. A mono program runs once per channel; a stereo one (2 in/2 out) on L/R. The full standard library is available: re.zita_rev1_stereo / dm.zita_light (reverb), dm.freeverb_demo, co.compressor_stere…

Install a custom real-time effect: manifest {id, label, description, params:{name:{minimum,maximum,default,unit}}} plus code = an AudioWorklet module that calls registerProcessor("plugin-<id>", Class) with one k-rate AudioParam per param. It is test-rendered before it is accepted, then usable everywhere as effect type "plugin:<id>" (preview and export). Only works if the operator enabled plugins.

Search the native effect plugins installed on the server — LV2 and VST3 (LSP, x42, Calf, ZAM, Dragonfly, DISTRHO, MDA, …: EQs, compressors, limiters, gates, reverbs, delays, filters, distortion, modulation, stereo tools). They run server-side as destructive processes via apply_lv2_plugin. Filter by free text (all words must match name/brand/category/uri) and/or category; with no filters returns category counts plus the first results. Output: one line per plugin "#index [Category] Name (brand, in→out channels, N controls) uri".

Installed effect plugins (use them with effect.add type "plugin:<id>") and whether installing is enabled here.

List editing sessions.

Proposals for a session with their status (open, accepted, rejected) and who decided.

Presets people and agents have saved (built-in ones are applied with preset_apply).

Controls of one LV2 or VST3 plugin (by uri or #index from list_lv2_plugins): symbol, label, range, unit, default, toggles and enumerations. Pass values by symbol in apply_lv2_plugin.controls; anything not passed keeps its default.

Noise-print noise reduction (like Audition's Capture Noise Print + Noise Reduction). Best for steady noise: hiss, fan, air-con, hum, room tone. Give start/end of a stretch that contains ONLY the noise (room tone before someone speaks, ≥ 0.1 s, ideally 0.5–2 s): the print is learned from it, stored on the asset, and applied to the whole asset in one call. Without start/end the stored print of this asset (or the asset it was made from, or profileFrom) is reused. learnOnly:true only captures the print and returns its description (level, band balance, tonal peaks such as 50/60 Hz hum). Applying cr…

Manual pitch correction: shift a voice by a curve of points {time (asset/source seconds), cents} — linear between points, 0 outside, so put 0-cent points around the part you fix. E.g. a note 30 cents flat from 12.40–12.90 s: [{time:12.3,cents:0},{time:12.4,cents:30},{time:12.9,cents:30},{time:13.0,cents:0}]. Use get_pitch first to see where the voice is off. Formants are kept (WORLD vocoder); only the covered span is resynthesised.

Description is built at runtime.

Suggest a batch of edits WITHOUT applying them. The human hears a preview and accepts or rejects it in the UI. Use this for anything subjective or destructive-feeling (big cuts, re-ordering, heavy processing choices) or when the human asked to review first. Same op format as `edit`. label = one line the human reads; rationale = why (short). Returns the proposal; check its status later with list_proposals.

Redo.

Make a music clip fit a target length by jumping between musically similar beats (like Audition's Remix) instead of stretching: tempo and pitch stay untouched. Result is ordinary clips with crossfades at the jumps (one undo step), so a human can adjust them. Typical use: fit a bed to an episode or a 30 s ad. Returns the achieved length (within about a beat of the target).

Description is built at runtime.

Export the mix. Returns a download URL plus measured loudness. bounce:true instead adds the mixdown to the bin as a new asset.

Save the effect chain of a track (or the master, without trackId) as a named preset that works in every session — e.g. after dialling in a voice sound the human likes.

Save a named snapshot of the session (e.g. before a risky pass) that the human can return to.

Slide the video against the timeline: offset = timeline seconds where the video starts. Does not move audio clips (move the soundtrack clip with clip.move if they should stay in sync).

Audition-style spectral editing of an asset on a time × frequency region. The region is ONE of: a box (start, end, optional lowHz/highHz); a lasso polygon ([[t, hz], …], for shapes that follow a sweep or a slanted chirp); or brush strokes ({points:[[t, hz], …], radiusMs, radiusSemitones}, for painting along a whistle, a harmonic or onto single clicks). action "attenuate" scales the region by gainDb with smooth edges (e.g. take a whistle at 2.9–3.2 kHz down by 30 dB); action "heal" rebuilds it, per frequency, from the audio right before and after it — use it for coughs, clicks, mouth noises, a …

Look at an asset's spectrogram rendered by NyxMix's own STFT, with an exact pixel → (time, Hz) mapping (returned as text next to the image) so you can read off rectangles for spectral_edit. Log-frequency axis (bottom = minHz, top = maxHz), no axis decorations. With clipId and no start/end, shows the clip's source window. All times are seconds in asset/source time, not timeline time. A clip maps source to timeline as timeline = clip.start + (source − clip.offset), i.e. source = clip.offset + (timeline − clip.start).

Time-stretch one clip to an exact length (duration in seconds) or by a ratio, keeping its pitch (Rubber Band, high quality) — e.g. fit a music bed to 30.0 s. Creates a new file and points the clip at it; one undo step.

Speech-to-text with word-level timestamps (Whisper, runs locally). Transcribes one asset, or (without assetId) every asset used on the timeline that has no transcript yet. Results are cached per asset, so calling again is cheap; force:true re-runs. Same-length processed copies (denoise etc.) reuse their source's transcript. Takes roughly 0.1–0.3× the audio length; the first run also downloads the model. If it is still running after ~50 s this returns {status:"running"} — call get_transcript later. Afterwards use get_transcript to read and cut_transcript / remove_filler_words to edit by text.

Undo the last edit (by anyone).

Look at the picture: returns the video frame at a timeline time as a JPEG image (e.g. to check who is on screen, or find where a shot changes before cutting the audio there).