19 KiB
Media generation in @opencode/ai — public API direction
Status: proposal. Branch media-support.
Goal
@opencode/ai becomes the one package you reach for to generate anything: text, images, video, speech, transcripts, and later music and realtime. The LLM surface already exists and is shaped by three constraints: Effect-first, used by OpenCode Core, usable externally. Media has a different priority order: external DX first, Effect and Promise as peers, Core as one consumer among many.
The design below is derived from a survey of the raw provider APIs (OpenAI, Gemini/Veo/Imagen, xAI, Stability, BFL, fal, Replicate, Runway, Luma, Kling, MiniMax, ElevenLabs, Deepgram, Cartesia, AssemblyAI, Lyria) and of existing multi-provider SDKs.
What the survey forces
- Three execution shapes, everywhere. Inline sync (OpenAI images, all TTS, Gemini), async job with polling or webhook (every video provider, BFL, fal, Replicate, AssemblyAI), and bidirectional streams (ElevenLabs/Cartesia/Deepgram WS, realtime). Video has no sync provider at all.
- Output is never just bytes. base64, signed URLs with TTLs from 10 minutes (BFL) to 2 days (Veo), URLs that need auth plus redirect (Veo), separate download endpoints (Sora
/content?variant=), raw bodies (Stability, TTS). Multi-output is the norm. - Inputs have roles. First/last frame, mask, style/subject reference, source video for edit/extend, reference audio, prior generation id, provider-side file handles (
file_id,gs://,runway://,mm_file://). - Partial streaming is modality-specific. Images: a few whole partial frames. Audio: ordered chunks plus timestamp events. Jobs: status/progress/logs. Video: none.
- Usage is a union: tokens, seconds, characters (often only in headers), credits, compute time.
- Moderation can be partial success (Veo strips audio but returns video). Deprecations are constant (Sora API shuts down 2026-09-24, Imagen on Gemini API 2026-08-17).
Where existing SDKs are weak and we should not be
- No streaming TTS.
- Video handles are experimental start/status pairs; the polling loop lives inside the generate call.
- Unsupported inputs become silent warnings arrays, so a request can succeed while dropping your mask.
nis fanned out into hidden parallel calls, which obscures cost and idempotency.- Each modality has its own bespoke result type; the file abstraction is a lazy base64/bytes pair with no URL, expiry, or provider ref.
- Effect's own
unstable/aihas no media generation. Nothing in the Effect ecosystem owns this.
Design principles
- Same shape as LLM.
X.request(...)→ Schema class;X.generate(request)/X.stream(request);XClient.Service+layer; typedAIError. If you knowLLM, you knowVideo. - Execution shape is route policy, not API shape.
Image.generatereturns an image whether the provider is inline or queued. Job control is available uniformly when you want it. - Errors, not warnings. Unsupported common fields fail at the protocol boundary with a typed
AIError, as the LLM routes do today. Provider-side partial results (filtered audio, moderated sample) surface asnoticeson the response, never as silent drops. - One asset type in, one asset type out, shared with LLM messages and tool results.
- Typed per-model options, no hidden fan-out, no implicit retries that spend money.
- Promise API is one mechanism for the whole package, not a media-only wrapper.
- One construction path per model. Media models come from per-modality selectors on the configured facade (
openai.image("gpt-image-2")), the same shape asopenai.responses("gpt-5").
Public API
Model selection
A model value is built as OpenAI.configure({ apiKey }).responses("gpt-5") or .image("gpt-image-2"): configure fixes credentials, endpoint, and defaults; the selector fixes which of the provider's APIs to hit and binds the typed providerOptions generic. Media follows the same shape with one selector per modality — openai.image(id) today, .video(id) / .speech(id) / .transcription(id) as those modalities land — mirroring openai.responses(id). Image.request accepts ImageModel only, exactly as LLM.request accepts LanguageModel.
import { OpenAI, Google } from "@opencode/ai/providers"
const openai = OpenAI.configure({ apiKey }) // OpenAI(...) alone uses env auth (OPENAI_API_KEY)
LLM.request({ model: openai.responses("gpt-5"), prompt })
Image.request({ model: openai.image("gpt-image-2"), prompt })
Video.request({ model: google.video("veo-3.1-generate-preview"), prompt })
Speech.request({ model: openai.speech("gpt-4o-mini-tts"), text })
Transcription.request({ model: openai.transcription("gpt-4o-transcribe"), audio })
The request namespace and the selector share one word (Image.request + .image(...)). That redundancy is accepted: a callable facade returning a lazily resolved ref would be a second way to construct the same model, and the type machinery to infer providerOptions through it is not worth one word. Where a provider has two APIs for one modality, the selectors stay explicit (openai.chat, a future google.imagen), and one default per modality per provider is part of the facade definition (OpenAI image → Images API, Google image → Gemini-native since Imagen on the Gemini API shuts down 2026-08-17). Provider package entrypoints keep model(modelID, settings) per modality-specific path, e.g. @opencode/ai/providers/openai/responses.
Media — the asset type
Replaces MediaPart.data: string | Uint8Array, ImageInput, GeneratedImage, and aligns Tool.FileContent.
import { Media } from "@opencode/ai"
Media.Source =
| { type: "bytes"; data: Uint8Array; mediaType: string }
| { type: "base64"; data: string; mediaType: string }
| { type: "url"; url: string; mediaType?: string; expiresAt?: number; headers?: Record<string, string> }
| { type: "ref"; provider: ProviderID; id: string; mediaType?: string } // file_id, gs://, runway://, prior generation
class Media.Asset {
readonly source: Media.Source
readonly mediaType: string // always resolved (sniffed when the provider omits it)
readonly kind: "image" | "video" | "audio" | "document" | "other"
readonly info?: { width?; height?; durationSeconds?; sampleRate?; channels?; encoding?; format? }
readonly expiresAt?: number
readonly providerMetadata?: ProviderMetadata
bytes(): Effect<Uint8Array, AIError, RequestExecutor.Service> // downloads/decodes lazily, cached
base64(): Effect<string, AIError, RequestExecutor.Service>
dataUrl(): Effect<string, AIError, RequestExecutor.Service>
materialize(): Effect<Media.Asset, AIError, RequestExecutor.Service> // url/ref → bytes, before the URL dies
}
Media.bytes(data, mediaType?) Media.base64(data, mediaType?)
Media.url(url, options?) Media.ref(provider, id)
Media.file(path) // Bun/Node: reads + sniffs; Effect FileSystem variant for layers
Media.write(asset, path) // convenience, uses FileSystem
Raw-PCM outputs (Gemini TTS, Cartesia raw, Deepgram WS) carry info.encoding/sampleRate/channels because there is no container header.
Modality namespaces
Each namespace mirrors LLM exactly.
import { Image, Video, Speech, Transcription } from "@opencode/ai"
import { OpenAI, Google, ElevenLabs, Fal } from "@opencode/ai/providers"
Image
const request = Image.request({
model: openai.image("gpt-image-2"),
prompt: "A robot tending a rooftop garden",
images: [Media.file("./ref.png")], // references / edit sources
mask: Media.file("./mask.png"),
n: 2,
size: "1536x1024", // or aspectRatio: "3:2"
seed: 7,
format: "webp",
providerOptions: { quality: "high", background: "transparent" }, // typed per model
})
const response = yield* Image.generate(request) // ImageResponse
response.image // Media.Asset (first)
response.images // Media.Asset[]
response.usage // Usage union (see below)
response.notices // moderation / partial-result notices
yield* Image.stream(request) // Stream<ImageEvent>
// ImageEvent: generation-queued | generation-progress | image-partial { index, image } | image { index, image } | finish { usage }
Editing is not a separate function; images/mask on the request select the edit path in the route (OpenAI /images/edits, Gemini multimodal parts, xAI /images/edits). Routes that cannot honor mask fail with Unsupported.
Video
const request = Video.request({
model: google.video("veo-3.1-generate-preview"),
prompt: "Panning wide shot of a calico kitten sleeping in the sunshine",
frames: { first: Media.file("./start.png"), last: Media.file("./end.png") },
references: [Media.url("https://…/style.png")],
video: Media.ref("openai", "video_123"), // edit / extend / remix source
durationSeconds: 8,
aspectRatio: "16:9",
resolution: "1080p",
audio: true,
n: 1,
providerOptions: { personGeneration: "dont_allow", negativePrompt: "text, watermark" },
})
// Simple: wait for it.
const response = yield* Video.generate(request, { poll: { interval: "10 seconds", timeout: "10 minutes" } })
response.video // Media.Asset (url with expiresAt, or bytes when the route downloads)
yield* response.video.materialize() // pull bytes before the URL expires
// Explicit generation control.
const generation = yield* Video.start(request) // Generation<VideoResponse>
generation.id; generation.status; generation.progress; generation.token // token is serializable JSON
yield* generation.await({ poll }) // VideoResponse
yield* generation.cancel()
// Resume from another process.
const resumed = yield* Video.resume(model, token) // Generation<VideoResponse>
// Progress as a stream.
yield* Video.stream(request) // Stream<VideoEvent>: generation-queued { position } | generation-progress { progress, logs } | video { index, video } | finish
Webhooks: Video.complete(model, token, webhook) finishes a generation from a webhook payload without polling. Token shape is route-owned and opaque (Veo operation name, fal response_url, Runway task id).
Speech (TTS)
const request = Speech.request({
model: elevenlabs.speech("eleven_v3"),
text: "Hello from OpenCode.",
voice: "JBFqnCBsd6RMkjVDRZzb", // name, uuid, or { id } — provider-normalized
format: "mp3", // mp3 | wav | pcm | opus | aac | flac | (string & {})
speed: 1.0,
language: "en",
instructions: "Warm, unhurried.",
providerOptions: { stability: 0.5 },
})
const response = yield* Speech.generate(request) // SpeechResponse: audio: Media.Asset, timestamps?, usage
yield* Speech.stream(request) // Stream<SpeechEvent>: audio-delta { chunk } | timestamps { words } | finish
Streaming TTS is first-class on day one: OpenAI stream_format: sse, ElevenLabs /stream, Cartesia SSE, Deepgram chunked. Input-streaming TTS (WS, text arrives incrementally) is a later Speech.session(...) scoped resource, not part of generate.
Transcription (STT)
const request = Transcription.request({
model: openai.transcription("gpt-4o-transcribe"),
audio: Media.file("./call.wav"),
language: "en",
prompt: "Names: Shoubhit, OpenCode.",
timestamps: "word", // none | segment | word
diarize: true,
providerOptions: { chunkingStrategy: "auto" },
})
const response = yield* Transcription.generate(request)
response.text; response.segments; response.words; response.language; response.durationSeconds
yield* Transcription.stream(request) // Stream<TranscriptionEvent>: text-delta | segment | finish
Realtime STT over WebSocket is the same future session shape as input-streaming TTS.
Generation — shared async execution
class Generation<Response> {
readonly id: string
readonly model: MediaModel
readonly token: unknown // route-owned serializable JSON
readonly status: "queued" | "running" | "completed" | "failed" | "cancelled" | "expired"
readonly progress?: number // 0..1, normalized
readonly position?: number
readonly expiresAt?: number
refresh(): Effect<Generation<Response>, AIError>
await(options?: { poll?: Poll }): Effect<Response, AIError>
cancel(): Effect<void, AIError>
events(options?): Stream<GenerationEvent, AIError>
}
Poll = { interval?: Duration; timeout?: Duration; schedule?: Schedule } // route may override from provider hints (`openai-poll-after-ms`)
Generation is not video-specific. Image routes on BFL, fal, and Replicate are queued; Image.start exists for them. A route declares itself inline or queued; generate on a queued route is start then await.
Usage
Usage =
| { type: "tokens"; input; output; total; details? }
| { type: "seconds"; seconds }
| { type: "characters"; characters }
| { type: "credits"; credits }
| { type: "compute"; seconds }
Header-only usage (ElevenLabs character-cost, Deepgram dg-char-count) is lifted into usage by the route.
Promise API — @opencode/ai/promise
Mirrors the packages/plugin/src/effect and packages/plugin/src/promise split that already exists in this repo. One mechanism for LLM and media.
import { AI } from "@opencode/ai/promise"
const ai = AI.make() // ManagedRuntime over RequestExecutor.fetchLayer + all clients
// AI.make({ layer }) to inject a custom executor / recorder / middleware
const image = await ai.image.generate({ model, prompt })
await image.image.bytes()
for await (const event of ai.speech.stream({ model, text, voice })) { … }
const generation = await ai.video.start({ model, prompt })
const video = await generation.await({ poll: { interval: 10_000 }, signal })
const resumed = ai.video.resume(model, JSON.parse(saved))
const text = await ai.llm.generate({ model, prompt }) // closes today's gap: LLM has no promise API either
for await (const event of ai.llm.stream(request)) { … }
await ai.dispose()
Streams become AsyncIterable via Stream.toAsyncIterable. AIError is thrown as-is. AbortSignal maps to interruption. Nothing in src/* except this entrypoint knows about promises.
Providers
Existing facades gain per-modality selectors; the modality routes each facade provides:
| Facade | llm | image | video | speech | transcription | other |
|---|---|---|---|---|---|---|
OpenAI |
responses (default), chat | Images API | Sora (deprecated 2026-09-24) | ✓ | ✓ | |
Google |
Gemini | Gemini-native (default), imagen |
Veo | Gemini TTS | Gemini transcribe | |
XAI |
✓ | ✓ | ✓ | |||
ElevenLabs |
✓ | Scribe | soundEffect, music | |||
Fal |
✓ | ✓ | ||||
Replicate, Runway, Luma, Kling, MiniMax, Deepgram, Cartesia, AssemblyAI, BlackForestLabs, Stability |
per provider |
New facades follow the existing one-file-per-provider rule. Package entrypoints are modality-specific, such as @opencode/ai/providers/openai/images, and return the concrete model.
ImageModel<Options> already gives typed providerOptions per model; VideoModel, SpeechModel, TranscriptionModel follow the same generic. A shared MediaModel union is what Generation and the promise client key on.
Routes and protocols
Media does not fit the LLM four-axis route (SSE frames → event state machine) except for streaming TTS/STT. Reuse Endpoint, Auth, Framing, RequestExecutor, and add media protocol kinds:
MediaProtocol.inline—body.from(request)(JSON, multipart, or query),response.decode(response)(JSON, or binary body →Media.Asset).MediaProtocol.queued—start,status,result,cancel, optionaldownload,pollHint,tokenschema.MediaProtocol.stream— framing +stepstate machine emitting modality events, same discipline as LLM protocols.
Route.make for media composes one protocol kind with endpoint/auth. The existing ImageRoute { generate(request, execute) } is the ad-hoc version of inline and gets folded in.
LLM integration
MediaPartbecomes{ type: "media"; media: Media.Asset; … }so protocols branch onkindand can passurl/refsources through natively (OpenAIimage_url, GeminifileData).- New
LLMEvents:media { media: Media.Asset }so Gemini inline image output is first-class instead of dropped. OpenAI Responsesimage_generation_callkeeps its single carrier — the provider-executedtool-resultwithfilecontent — because Core consumes hosted tool-result content today and has nomediaevent handling yet; it switches to themediacarrier when Core adopts the event, so the image is never emitted twice. Message.assistant([...])accepts media parts; Gemini multi-turn image editing replays them.Tool.FileContentaligns withMedia.Source.
Decisions
All settled:
- Per-modality selectors (
openai.image(id),.video,.speech,.transcription) name media models, mirroringopenai.responses(id). The one-word overlap with the request namespace is accepted over a callable-facadeModelRefas a second construction path. providerOptionseverywhere (rename currentImage.options) for consistency with LLM.- No hidden
nfan-out.nlowers natively; routes that cannot don > 1fail typed. Callers useEffect.all/Promise.allexplicitly. - Errors over warnings for unsupported common fields;
noticesfor provider-side partial results only. Media.Assetis a class (lazy bytes, cached) withMedia.Sourceas the serializable Schema for wire/persistence.Asset.from(source)/asset.sourceround-trip losslessly. Same pattern asLanguageModeltoday.- Promise entrypoint:
@opencode/ai/promiseexportingAI.make(options?: { layer? })plus a module-level defaultaifor scripts, covering LLM too. - Modality set for v1:
Image,Video,Speech,Transcription.Music/SoundEffectandsession(bidirectional WS, realtime) are designed-for but deferred. - Sora is skipped (API shuts down 2026-09-24). Video launches with Veo, xAI, fal, Runway.
Build order
Foundation + Image ship together as the reference implementation, serially. Video, Speech, and Transcription then proceed in parallel on separate branches. Image jobs and partial streaming come last, after Video has hardened Generation.
Phasing
- Foundation — per-modality selectors,
Media,Generation,Poll,Usageunion,MediaProtocolkinds,@opencode/ai/promisewithllm+image. Port the five existing image protocols onto it. UnifyMediaPartand add themediaLLM event (fixes Gemini image output being dropped). - Video — Veo, xAI, fal, Runway first. Then Luma, Kling, MiniMax, Replicate.
- Speech + Transcription — OpenAI, ElevenLabs, Gemini TTS, Deepgram, Cartesia, AssemblyAI. Streaming TTS from the start.
- Image queued routes and partials — BFL, fal, Replicate, Stability; OpenAI
partial_imagesstreaming. - Later — ElevenLabs music/SFX, Lyria,
Speech.session/Transcription.session, realtime.
Core adoption (session attachments beyond png/jpeg/gif/webp/pdf, image-generation tool, TUI rendering) comes after phase 1 and is a Core concern.