Closed captions and subtitles

#CEA-608 captions in the video

US broadcast recordings, MPEG-TS captures and many disc rips carry captions inside the video stream — ATSC A/53 cc_data in H.264 or HEVC SEI messages — rather than as a subtitle track, which is why they look like they have none. PrismCore decodes them during the remux read into the same segmented WebVTT renditions every other text track uses, so CC1…CC4 appear as real AVMediaSelectionOptions and survive Picture in Picture, AirPlay and external displays.

• Modes: pop-on, roll-up (2, 3 and 4 rows) and paint-on, with the control codes, preamble addressing and the basic, special and extended character sets.
• Names: the channel, plus the video track's language where the container declares one — "English (CC1)" or "CC1".
• Timing: caption bytes are reordered from decode order to presentation order before they reach the decoder (a window sized to H.264's maximum reorder depth). Without that, on any source with B-frames an erase lands before the flip it was meant to end and the screen shows the caption before last. A caption has no end time on the wire, so cues end at the erase or flip that removes them, split at segment boundaries and capped at 10 s.
• Cost: a bounded scan before the first segment (at most 28 video packets, stopping at the first printed character) decides whether a source has captions at all. A source without them pays nothing in the copy loop. The scan is skipped for a codec with no SEI and for an input that cannot be rewound.
• Styling (colour, italics, underline) and cell-accurate positioning are dropped on purpose, so the system caption renderer applies the viewer's own accessibility style.

#How captions are signalled: CLOSED-CAPTIONS

Since 3.5.2 every variant in the master playlist says CLOSED-CAPTIONS=NONE. Without it AVPlayer finds the in-band 608 captions in the video's SEI by itself and lists them — on iOS sometimes switched on — next to the WebVTT rendition PrismCore already serves of the same service, and next to whatever the host draws.

A host that wants AVPlayer's own 608 rendering can opt back in:

let session = try PrismCoreSession(
    url: mkvURL,
    display: await DisplayCapabilities.current(),
    inBandClosedCaptions: true
)

With inBandClosedCaptions: true each caption service the startup scan found is declared as #EXT-X-MEDIA:TYPE=CLOSED-CAPTIONS,GROUP-ID="cc",…,DEFAULT=NO,AUTOSELECT=NO,INSTREAM-ID="CCn", and the WebVTT renditions stay — so the menu lists both. A source the scan found no captions in (or never scanned) still gets NONE. Masters only: the media-playlist shape has no EXT-X-STREAM-INF line to say either on.

#Preferred subtitle language

let session = try PrismCoreSession(
    url: mkvURL,
    display: await DisplayCapabilities.current(),
    preferredSubtitleLanguage: "cs"
)

The matching rendition becomes the only one marked DEFAULT=YES,AUTOSELECT=YES, which makes AVKit engage it at load instead of starting with subtitles off. A full rendition wins over a forced one of the same language, and FORCED itself is untouched. Matching is tolerant (cze, ces, cs, cs-CZ; a bare tag matches a regioned one; und matches nothing), and no match changes nothing. Without a preference, no subtitle rendition is marked DEFAULT.

#The cue tap: drawing your own overlay

setTimedTextCueHandler(_:) streams every embedded cue — text, OCR'd bitmap and CEA-608 — as the remux produces it. Caption cues use synthetic stream indices (CC1 is -1 … CC4 is -4), which cannot collide with a demuxed track's.

await session.setTimedTextCueHandler { cue in
    // Times are on the AVPlayer timeline (rebased to the presentation origin).
    print(cue.streamIndex, cue.start, cue.end, cue.text)
}

A handler registered late — or registered again to pick up a new subtitle delay — is first replayed everything produced so far, so by default the session keeps every cue for its lifetime. For long captioned streams, bound that history:

let session = try PrismCoreSession(
    url: mkvURL,
    display: await DisplayCapabilities.current(),
    subtitleCueHistory: .bounded(maxCues: 1_024, maxBytes: 1 << 20)
)
let stats = session.subtitleCueHistoryStats   // retained cues and bytes, evictions

Live delivery is never bounded, only the replay. Deduplication covers the retained window only, so a host that opts in must tolerate a repeated cue after a long seek back. evictedCues counts what is no longer replayable and droppedBeforeOrigin what never went out.

Bitmap tracks that were not turned into renditions (the source also has a text track) can still be OCR'd into the tap on request: requestBitmapSubtitleOCR(streamIndex:), for tracks whose SubtitleTrackInfo.isOCRReadable is true.

TimedTextCue.placement carries the placement the source asked for (TextCuePlacement: a numpad alignment, plus an anchor from ASS \pos or a WebVTT line:/position: pair); nil means the host's default, which is what nearly every cue wants.

#Subtitle delay

let change = await session.setSubtitleDelaySeconds(0.5)   // positive shows text later

The delay is clamped to ±10 s. Before start() it returns .inForce; afterwards .appliesToNewSegments — cues delivered to the tap and WebVTT segments written from then on use it, while segments already written keep their times. The software pipeline applies it at read time.

#On the software path

SoftwarePlaybackPipeline offers embedded text tracks only (selectableSubtitleTracks); bitmap, caption and external tracks are not selectable there. Text starts off, forced tracks included. Select with selectSubtitleTrack(streamIndex:) (nil is off) and poll activeSubtitleCues from your UI update — an empty array means clear the overlay. Cue times on this path use the source axis, matching pipeline.currentTime. The look-ahead cache holds at most 1,024 cues / 1 MiB of text.