refactor(ai): move speech, transcription, video and realtime to experimental (#60484)

This commit is contained in:
Aiden Cline
2026-10-30 00:58:12 -05:00
committed by GitHub
Unverified
parent e505852682
commit ec64c1872e
31 changed files with 263 additions and 231 deletions
+87 -71
View File
@@ -1,6 +1,7 @@
# @opencode/ai
Schema-first APIs for text, images, video, speech, transcription, and decisions, built with Effect.
Schema-first APIs for text, images, and decisions, built with Effect. Video, speech, transcription, extraction, and
realtime voice are [experimental](#experimental).
`@opencode/ai` is the Promise API: one managed runtime behind the LLM and media methods, plus asset helpers. `ai.file`
and `ai.write` load `node:fs/promises` on first use, so no Effect `FileSystem` is needed:
@@ -165,8 +166,8 @@ Raw usage is preserved in `usage.providerMetadata.google`. `inputTokens` follows
`contextTokens` uses its full `raw_prompt_token` count when supplied. These can differ substantially with server-side
continuation. Explicit caches, hosted tools, and generated media are not supported by this initial protocol.
The same configured facade names image, video, speech, and transcription models. `Image.generate` resolves the
provider's image route from the model and returns `Media.Asset`s with lazily decoded bytes:
The same configured facade names image models (and the experimental video, speech, and transcription models).
`Image.generate` resolves the provider's image route from the model and returns `Media.Asset`s with lazily decoded bytes:
```ts
import { NodeFileSystem } from "@effect/platform-node"
@@ -309,38 +310,6 @@ CloudflareAIGateway.configure({ accountId }).decision("typesafe/jev")
OpenAI and Vercel AI Gateway use OpenAI's `/v1/decisions` API. TypeSafe, OpenCode, OpenRouter, Baseten and
DigitalOcean use System One (`/systemone`), and Cloudflare sends the System One body through `/ai/run`.
### Extraction (experimental)
Extraction models read typed values out of text instead of answering questions with probabilities. A field of type
`text`, `number`, `date`, `time`, `datetime`, `duration`, `list`, `estimate` or `forecast` gets an answer typed to
match; other types are sent and answered as-is. Only Cloudflare's Clairic serves it today, so it lives under
`@opencode/ai/experimental`.
```ts
import { ai } from "@opencode/ai"
import { Extraction } from "@opencode/ai/experimental"
import { CloudflareWorkersAI } from "@opencode/ai/providers"
const response = await ai.run(
Extraction.generate({
model: CloudflareWorkersAI.configure({ accountId }).experimental.extraction("@cf/cloudflare/clairic"),
input: "Invoice INV-2291, issued 2026-09-02, net 30. Total $1,080.00. Contact: Dana Ruiz, Priya Shah.",
fields: {
invoiceNumber: { type: "text" },
dueDate: { type: "date", instructions: "Issue date plus the net payment days" },
total: { type: "number", decimals: 2, unit: "USD" },
contacts: { type: "list", items: "text" },
},
}),
)
response.answers.total.value // 1080, or null when the input does not say
```
Value answers carry `guess`, `nullProbability` and `confidence`; text answers carry a `span` into `input`. Estimates and
forecasts return `median`, `p10` and `p90`. Field options other than `type`, `instructions` and `nullThreshold` go to
Clairic as written. Effect users provide `RequestExecutor.fetchLayer` (or `AIClient.layer`). Images are not supported
yet.
## Alibaba Cloud Model Studio
`Alibaba` provides standard Model Studio inference. Configure a region explicitly, then select
@@ -910,7 +879,42 @@ const program = Effect.gen(function* () {
The hosted result is represented as a provider-executed tool call and a tool result whose content carries the generated image as a file. Gemini image-capable models instead emit a first-class `media` `LLMEvent` for inline image output (`response.message` then carries a `media` part). Retaining `response.message` preserves the generated image for continuation on both routes.
## Video generation
## Public API
Everything below is exported from `@opencode/ai/effect` unless noted.
- **`LLM.request({...})`** — build a provider-neutral `LLMRequest`. Accepts ergonomic inputs (`system: string`, `prompt: string`) that normalize into the canonical Schema classes.
- **`LLM.generate` / `LLM.stream`** — run direct input or an `LLMRequest` through `LLMClient` for one-import use.
- **`LLM.loop`** — stream model calls, running `tool(...)` tools between them until the model stops calling tools.
- **`Message.user(...)` / `Message.assistant(...)` / `Message.tool(...)`** — message constructors from the canonical schema model.
- **`LanguageModel.make(...)` / `ToolCallPart.make(...)` / `ToolResultPart.make(...)` / `ToolDefinition.make(...)`** — model and tool-related constructors from the canonical schema model.
- **`LLMEvent.is.*`** — typed guards (`is.textDelta`, `is.toolCall`, `is.finish`, …) for filtering streams.
- **`Image.request` / `generate` / `stream` / `start` / `resume`** — images over inline, streaming (partial previews), and queued routes through a provider-neutral request and response model.
- **`ImageClient`** — Effect service and layer for image execution, parallel to `LLMClient`.
- **`Media`** — the shared asset type (`Media.Asset`, `Media.Source`) and constructors used by messages, tool results, and media requests.
- **`Generation`** — provider-neutral handle for an in-flight media generation (`await`, `refresh`, `cancel`, `events`) used by queued media routes.
- **`Decision.request` / `Decision.generate`** — typed predicate, choice, and score questions over text, JSON, and media input; `DecisionClient` is its Effect service and layer.
- **`AIClient.layer` / `AIClient.layerWith(executor)`** — every modality client plus the request executor in one layer.
- **`@opencode/ai`** (Promise root) — `AI.make({ layer?, clients? })` and a default `ai` client exposing `llm`, `image`, and `decision` (plus `experimental.video`, `experimental.speech`, and `experimental.transcription`) as Promise / `AsyncIterable` APIs, `run(effect)` for any package Effect, plus `file`, `write`, `bytes`, `base64`, and `materialize` for assets.
- **`@opencode/ai/experimental`** — APIs that may change without compatibility guarantees: `Video`, `Speech`, `Transcription`, `Extraction`, and `Realtime`. See [Experimental](#experimental).
## Experimental
These APIs may change without compatibility guarantees. The Effect API is exported from `@opencode/ai/experimental`,
the Promise client exposes it under `ai.experimental`, and test layers come from `@opencode/ai/experimental/testing`.
- **`Video`**, **`Speech`**, and **`Transcription`** — media generation and speech-to-text, with `VideoClient`,
`SpeechClient`, and `TranscriptionClient` as their Effect services. Promise: `ai.experimental.video`, `.speech`, and
`.transcription`. Testing: `TestVideo`, `TestSpeech`, and `TestTranscription`.
- **`Extraction`** — typed values read out of text.
- **`Realtime`** — `Realtime.connect` starts an OpenAI, xAI, or Gemini voice session and returns a WebRTC answer SDP or
a short-lived WebSocket URL and token.
The provider selectors for these modalities (`.video(...)`, `.speech(...)`, `.transcription(...)`,
`.experimental.extraction(...)`) and their protocol entrypoints (such as `@opencode/ai/protocols/openai-speech`) are
part of the same experimental surface.
### Video generation
Video mirrors `Image` with one difference: every provider is asynchronous, so the route is a submit-then-poll
`Generation`. Models come from `.video(...)` selectors on the `Google` (Veo), `XAI`, and `Runway` facades.
@@ -919,7 +923,7 @@ Common fields (`frames`, `references`, `video`, `durationSeconds`, `aspect`, `re
under `providerOptions`, inferred from the selected model.
```ts
import { Video } from "@opencode/ai/effect"
import { Video } from "@opencode/ai/experimental"
import { Google, Runway } from "@opencode/ai/providers"
const google = Google.configure({ apiKey: process.env.GOOGLE_GENERATIVE_AI_API_KEY })
@@ -981,9 +985,9 @@ Provider notes:
- **Runway** expects pixel ratios in `aspect` for most models (`"1280:720"`), pins `X-Runway-Version`, reports
`usage: { type: "credits" }`, and its output URLs expire after 24–48 hours.
The promise client exposes the same surface: `ai.video.start(...)` resolves to a handle with `await`, `events`,
`result`, `refresh`, `cancel`, and `token`; `ai.video.generate`, `ai.video.resume(model, token)`, and
`ai.video.stream` mirror the Effect API. The handle's `status` and `progress` are a snapshot from when it was
The promise client exposes the same surface: `ai.experimental.video.start(...)` resolves to a handle with `await`, `events`,
`result`, `refresh`, `cancel`, and `token`; `ai.experimental.video.generate`, `ai.experimental.video.resume(model, token)`, and
`ai.experimental.video.stream` mirror the Effect API. The handle's `status` and `progress` are a snapshot from when it was
created; `refresh()` resolves to a new handle. Every promise method and stream accepts `{ signal }`: like `fetch`,
aborting rejects the Promise or throws from the `for await` loop with `signal.reason` (an `AbortError` `DOMException`
unless `abort(reason)` passed one), while `break` stops a stream without throwing.
@@ -991,13 +995,13 @@ unless `abort(reason)` passed one), while `break` stops a stream without throwin
```ts
import { ai } from "@opencode/ai"
const generation = await ai.video.start({ model, prompt })
const generation = await ai.experimental.video.start({ model, prompt })
for await (const event of generation.events({ poll: { interval: 10_000 } })) console.log(event.type)
const video = await generation.result({ signal })
await ai.write(video.video, "./kite.mp4")
```
## Speech generation
### Speech generation
Speech (text-to-speech) is one request whose response is parsed incrementally, so every route supports both
`Speech.generate` (the whole file) and `Speech.stream` (audio chunks as they arrive). Models come from `.speech(...)`
@@ -1006,7 +1010,8 @@ selectors on the `OpenAI`, `Google` (Gemini TTS), `ElevenLabs`, `Cartesia`, and
before any network call; provider-native controls live under `providerOptions`, inferred from the selected model.
```ts
import { Media, Speech, SpeechClient, SpeechEvent } from "@opencode/ai/effect"
import { Media } from "@opencode/ai/effect"
import { Speech, SpeechEvent } from "@opencode/ai/experimental"
import { ElevenLabs, OpenAI } from "@opencode/ai/providers"
const openai = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY })
@@ -1068,20 +1073,20 @@ Provider notes:
and `providerOptions` lower to query parameters (`encoding`, `container`, `sample_rate`, `bit_rate`); `pcm` is
`linear16` without a container. Auth is `Authorization: Token <DEEPGRAM_API_KEY>`.
The promise client mirrors the Effect API; `ai.speech.stream` is an `AsyncIterable`.
The promise client mirrors the Effect API; `ai.experimental.speech.stream` is an `AsyncIterable`.
```ts
import { ai } from "@opencode/ai"
const response = await ai.speech.generate({ model, text: "Hello from OpenCode.", voice: "coral" })
const response = await ai.experimental.speech.generate({ model, text: "Hello from OpenCode.", voice: "coral" })
await ai.write(response.audio, "hello.mp3")
for await (const event of ai.speech.stream({ model, text: "Hello from OpenCode.", voice: "coral" })) {
for await (const event of ai.experimental.speech.stream({ model, text: "Hello from OpenCode.", voice: "coral" })) {
if (event.type === "audio-delta") player.write(event.chunk)
}
```
## Transcription
### Transcription
Transcription (speech-to-text) is the one modality whose providers use every route kind: OpenAI and Gemini stream,
Deepgram and ElevenLabs answer inline, and AssemblyAI is queued. `Transcription.generate` and `Transcription.stream`
@@ -1093,7 +1098,8 @@ any network call; a route may return more than asked.
```ts
import { Console, Effect, Stream } from "effect"
import { Media, Transcription, TranscriptionEvent } from "@opencode/ai/effect"
import { Media } from "@opencode/ai/effect"
import { Transcription, TranscriptionEvent } from "@opencode/ai/experimental"
import { AssemblyAI, Deepgram, OpenAI } from "@opencode/ai/providers"
const openai = OpenAI.configure({ apiKey: process.env.OPENAI_API_KEY })
@@ -1146,33 +1152,43 @@ The promise client mirrors the Effect API:
```ts
const audio = await ai.file("./call.mp3")
const text = (await ai.transcription.generate({ model, audio })).text
for await (const event of ai.transcription.stream({ model, audio })) if (event.type === "text-delta") write(event.delta)
const generation = await ai.transcription.start({ model: assemblyai, audio })
const text = (await ai.experimental.transcription.generate({ model, audio })).text
for await (const event of ai.experimental.transcription.stream({ model, audio }))
if (event.type === "text-delta") write(event.delta)
const generation = await ai.experimental.transcription.start({ model: assemblyai, audio })
const transcript = await generation.await({ poll: { interval: 3_000 } })
```
## Public API
### Extraction
Everything below is exported from `@opencode/ai/effect` unless noted.
Extraction models read typed values out of text instead of answering questions with probabilities. A field of type
`text`, `number`, `date`, `time`, `datetime`, `duration`, `list`, `estimate` or `forecast` gets an answer typed to
match; other types are sent and answered as-is. Only Cloudflare's Clairic serves it today.
- **`LLM.request({...})`** — build a provider-neutral `LLMRequest`. Accepts ergonomic inputs (`system: string`, `prompt: string`) that normalize into the canonical Schema classes.
- **`LLM.generate` / `LLM.stream`** — run direct input or an `LLMRequest` through `LLMClient` for one-import use.
- **`LLM.loop`** — stream model calls, running `tool(...)` tools between them until the model stops calling tools.
- **`Message.user(...)` / `Message.assistant(...)` / `Message.tool(...)`** — message constructors from the canonical schema model.
- **`LanguageModel.make(...)` / `ToolCallPart.make(...)` / `ToolResultPart.make(...)` / `ToolDefinition.make(...)`** — model and tool-related constructors from the canonical schema model.
- **`LLMEvent.is.*`** — typed guards (`is.textDelta`, `is.toolCall`, `is.finish`, …) for filtering streams.
- **`Image.request` / `generate` / `stream` / `start` / `resume`** — images over inline, streaming (partial previews), and queued routes through a provider-neutral request and response model.
- **`ImageClient`** — Effect service and layer for image execution, parallel to `LLMClient`.
- **`Media`** — the shared asset type (`Media.Asset`, `Media.Source`) and constructors used by messages, tool results, and media requests.
- **`Generation`** — provider-neutral handle for an in-flight media generation (`await`, `refresh`, `cancel`, `events`) used by queued media routes.
- **`Video.request` / `generate` / `stream` / `start` / `resume`** — queued video generation through a provider-neutral request; `VideoClient` is its Effect service and layer.
- **`Speech.request` / `Speech.generate` / `Speech.stream`** — text-to-speech through a provider-neutral request; `SpeechClient` is its Effect service and layer.
- **`Transcription.request` / `generate` / `stream` / `start` / `resume`** — speech-to-text over inline, streaming, and queued routes; `TranscriptionClient` is its Effect service and layer.
- **`Decision.request` / `Decision.generate`** — typed predicate, choice, and score questions over text, JSON, and media input; `DecisionClient` is its Effect service and layer.
- **`AIClient.layer` / `AIClient.layerWith(executor)`** — every modality client plus the request executor in one layer.
- **`@opencode/ai`** (Promise root) — `AI.make({ layer?, clients? })` and a default `ai` client exposing `llm`, `image`, `video`, `speech`, `transcription`, and `decision` as Promise / `AsyncIterable` APIs, `run(effect)` for any package Effect, plus `file`, `write`, `bytes`, `base64`, and `materialize` for assets.
- **`@opencode/ai/experimental`** — APIs that may still change: `Extraction` (typed values read out of text) and `Realtime` (`Realtime.connect` starts an OpenAI, xAI, or Gemini voice session and returns a WebRTC answer SDP or a short-lived WebSocket URL and token).
```ts
import { ai } from "@opencode/ai"
import { Extraction } from "@opencode/ai/experimental"
import { CloudflareWorkersAI } from "@opencode/ai/providers"
const response = await ai.run(
Extraction.generate({
model: CloudflareWorkersAI.configure({ accountId }).experimental.extraction("@cf/cloudflare/clairic"),
input: "Invoice INV-2291, issued 2026-09-02, net 30. Total $1,080.00. Contact: Dana Ruiz, Priya Shah.",
fields: {
invoiceNumber: { type: "text" },
dueDate: { type: "date", instructions: "Issue date plus the net payment days" },
total: { type: "number", decimals: 2, unit: "USD" },
contacts: { type: "list", items: "text" },
},
}),
)
response.answers.total.value // 1080, or null when the input does not say
```
Value answers carry `guess`, `nullProbability` and `confidence`; text answers carry a `span` into `input`. Estimates and
forecasts return `median`, `p10` and `p90`. Field options other than `type`, `instructions` and `nullThreshold` go to
Clairic as written. Effect users provide `RequestExecutor.fetchLayer` (or `AIClient.layer`). Images are not supported
yet.
## Testing
@@ -1218,7 +1234,7 @@ For explicit compaction, script a `CompactionResponse` through `push`, `always`,
For `compact(request, { mechanism: "trigger" })`, script a `CompactionCheckpointResponse` instead. It carries `checkpoint`, `responseID`, and optional `usage`. Endpoint and trigger calls reject each other's fixtures; both share the same queue, gates, lazy recording, and fallback controls.
`TestImage`, `TestVideo`, `TestSpeech`, `TestTranscription`, and `TestDecision` follow the same pattern: `testLayer()` provides the modality's client and `Test`, with the same `push`, `always`, `serve`, `requests`, `wait`, and `gate` controls. Script an `ImageResponse` (or the modality's response) or an `AIError`. `stream` expands the response into its events, and `start` returns an already finished generation that `resume` accepts by token. Decision responses are returned as scripted, without checking the answers against the questions.
`TestImage` and `TestDecision` (and `TestVideo`, `TestSpeech`, and `TestTranscription` from `@opencode/ai/experimental/testing`) follow the same pattern: `testLayer()` provides the modality's client and `Test`, with the same `push`, `always`, `serve`, `requests`, `wait`, and `gate` controls. Script an `ImageResponse` (or the modality's response) or an `AIError`. `stream` expands the response into its events, and `start` returns an already finished generation that `resume` accepts by token. Decision responses are returned as scripted, without checking the answers against the questions.
The Promise client accepts the same layers as `AI.make({ clients: TestImage.testLayer() })`; `ai.run(TestImage.Test.use(...))` reaches the controls.
+1
View File
@@ -20,6 +20,7 @@
"./effect": "./src/effect.ts",
"./testing": "./src/testing.ts",
"./experimental": "./src/experimental.ts",
"./experimental/testing": "./src/experimental/testing.ts",
"./route": "./src/route.ts",
"./providers": "./src/providers.ts",
"./providers/alibaba": "./src/providers/alibaba.ts",
-57
View File
@@ -116,63 +116,6 @@ export type {
ImageRoute,
} from "./image.js"
export { Image } from "./image.js"
export { VideoClient } from "./video-client.js"
export {
VideoAspectRatio,
VideoEvent,
VideoFrames,
VideoModel,
VideoModelSchema,
VideoRequest,
VideoResponse,
} from "./video.js"
export type {
VideoModelOptions,
VideoOptions,
VideoRequestFor,
VideoRequestInput,
VideoResolution,
VideoRoute,
} from "./video.js"
export { Video } from "./video.js"
export { SpeechClient } from "./speech-client.js"
export {
SpeechEvent,
SpeechModel,
SpeechModelSchema,
SpeechRequest,
SpeechResponse,
SpeechTimestamp,
SpeechVoice,
} from "./speech.js"
export type {
SpeechFormat,
SpeechModelOptions,
SpeechOptions,
SpeechRequestFor,
SpeechRequestInput,
SpeechRoute,
} from "./speech.js"
export { Speech } from "./speech.js"
export { TranscriptionClient } from "./transcription-client.js"
export {
TranscriptionEvent,
TranscriptionModel,
TranscriptionModelSchema,
TranscriptionRequest,
TranscriptionResponse,
TranscriptionSegment,
TranscriptionTimestamps,
TranscriptionWord,
} from "./transcription.js"
export type {
TranscriptionModelOptions,
TranscriptionOptions,
TranscriptionRequestFor,
TranscriptionRequestInput,
TranscriptionRoute,
} from "./transcription.js"
export { Transcription } from "./transcription.js"
export { DecisionClient } from "./decision-client.js"
export {
ChoiceAnswer,
+57
View File
@@ -1,3 +1,60 @@
export { VideoClient } from "./video-client.js"
export {
VideoAspectRatio,
VideoEvent,
VideoFrames,
VideoModel,
VideoModelSchema,
VideoRequest,
VideoResponse,
} from "./video.js"
export type {
VideoModelOptions,
VideoOptions,
VideoRequestFor,
VideoRequestInput,
VideoResolution,
VideoRoute,
} from "./video.js"
export { Video } from "./video.js"
export { SpeechClient } from "./speech-client.js"
export {
SpeechEvent,
SpeechModel,
SpeechModelSchema,
SpeechRequest,
SpeechResponse,
SpeechTimestamp,
SpeechVoice,
} from "./speech.js"
export type {
SpeechFormat,
SpeechModelOptions,
SpeechOptions,
SpeechRequestFor,
SpeechRequestInput,
SpeechRoute,
} from "./speech.js"
export { Speech } from "./speech.js"
export { TranscriptionClient } from "./transcription-client.js"
export {
TranscriptionEvent,
TranscriptionModel,
TranscriptionModelSchema,
TranscriptionRequest,
TranscriptionResponse,
TranscriptionSegment,
TranscriptionTimestamps,
TranscriptionWord,
} from "./transcription.js"
export type {
TranscriptionModelOptions,
TranscriptionOptions,
TranscriptionRequestFor,
TranscriptionRequestInput,
TranscriptionRoute,
} from "./transcription.js"
export { Transcription } from "./transcription.js"
export { Extraction, ExtractionModel, ExtractionRequest, ExtractionResponse } from "./experimental/extraction.js"
export type {
ExtractionAnswerFor,
+3
View File
@@ -0,0 +1,3 @@
export { TestSpeech } from "../testing/speech.js"
export { TestTranscription } from "../testing/transcription.js"
export { TestVideo } from "../testing/video.js"
+46 -43
View File
@@ -230,49 +230,6 @@ export const make = <R = never>(options: Options<R> = {}) => {
resume: (model: ImageModel, token: unknown, options?: RunOptions) =>
run(Image.resume(model, token), options).then(handle),
},
video: {
request: Video.request,
start: <const Model extends VideoModel>(input: VideoRequestInput<Model> | VideoRequest, options?: RunOptions) =>
run(Video.start(input), options).then(handle),
generate: <const Model extends VideoModel>(
input: VideoRequestInput<Model> | VideoRequest,
options?: AwaitOptions & RunOptions,
) => run(Video.generate(input, { poll: options?.poll }), options),
resume: (model: VideoModel, token: unknown, options?: RunOptions) =>
run(Video.resume(model, token), options).then(handle),
stream: <const Model extends VideoModel>(
input: VideoRequestInput<Model> | VideoRequest,
options?: AwaitOptions & RunOptions,
) => iterate(Video.stream(input, { poll: options?.poll }), options),
},
speech: {
request: Speech.request,
generate: <const Model extends SpeechModel>(
input: SpeechRequestInput<Model> | SpeechRequest,
options?: RunOptions,
) => run(Speech.generate(input), options),
stream: <const Model extends SpeechModel>(
input: SpeechRequestInput<Model> | SpeechRequest,
options?: RunOptions,
) => iterate(Speech.stream(input), options),
},
transcription: {
request: Transcription.request,
generate: <const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model> | TranscriptionRequest,
options?: AwaitOptions & RunOptions,
) => run(Transcription.generate(input, { poll: options?.poll }), options),
stream: <const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model> | TranscriptionRequest,
options?: AwaitOptions & RunOptions,
) => iterate(Transcription.stream(input, { poll: options?.poll }), options),
start: <const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model> | TranscriptionRequest,
options?: RunOptions,
) => run(Transcription.start(input), options).then(handle),
resume: (model: TranscriptionModel, token: unknown, options?: RunOptions) =>
run(Transcription.resume(model, token), options).then(handle),
},
decision: {
request: Decision.request,
generate: <const Model extends DecisionModel, const Questions extends DecisionQuestions>(
@@ -280,6 +237,52 @@ export const make = <R = never>(options: Options<R> = {}) => {
options?: RunOptions,
) => run(Decision.generate(input), options),
},
/** Experimental: may change without compatibility guarantees. */
experimental: {
speech: {
request: Speech.request,
generate: <const Model extends SpeechModel>(
input: SpeechRequestInput<Model> | SpeechRequest,
options?: RunOptions,
) => run(Speech.generate(input), options),
stream: <const Model extends SpeechModel>(
input: SpeechRequestInput<Model> | SpeechRequest,
options?: RunOptions,
) => iterate(Speech.stream(input), options),
},
transcription: {
request: Transcription.request,
generate: <const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model> | TranscriptionRequest,
options?: AwaitOptions & RunOptions,
) => run(Transcription.generate(input, { poll: options?.poll }), options),
stream: <const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model> | TranscriptionRequest,
options?: AwaitOptions & RunOptions,
) => iterate(Transcription.stream(input, { poll: options?.poll }), options),
start: <const Model extends TranscriptionModel>(
input: TranscriptionRequestInput<Model> | TranscriptionRequest,
options?: RunOptions,
) => run(Transcription.start(input), options).then(handle),
resume: (model: TranscriptionModel, token: unknown, options?: RunOptions) =>
run(Transcription.resume(model, token), options).then(handle),
},
video: {
request: Video.request,
start: <const Model extends VideoModel>(input: VideoRequestInput<Model> | VideoRequest, options?: RunOptions) =>
run(Video.start(input), options).then(handle),
generate: <const Model extends VideoModel>(
input: VideoRequestInput<Model> | VideoRequest,
options?: AwaitOptions & RunOptions,
) => run(Video.generate(input, { poll: options?.poll }), options),
resume: (model: VideoModel, token: unknown, options?: RunOptions) =>
run(Video.resume(model, token), options).then(handle),
stream: <const Model extends VideoModel>(
input: VideoRequestInput<Model> | VideoRequest,
options?: AwaitOptions & RunOptions,
) => iterate(Video.stream(input, { poll: options?.poll }), options),
},
},
dispose: () => runtime.dispose(),
}
}
-3
View File
@@ -2,9 +2,6 @@ export * as TestLLM from "./testing.js"
export { compileRequest } from "./route/client.js"
export { TestDecision } from "./testing/decision.js"
export { TestImage } from "./testing/image.js"
export { TestSpeech } from "./testing/speech.js"
export { TestTranscription } from "./testing/transcription.js"
export { TestVideo } from "./testing/video.js"
import {
LLMClient,
+1 -6
View File
@@ -12,13 +12,8 @@ import {
LLM,
LLMClient,
Media,
Speech,
SpeechClient,
SpeechEvent,
TranscriptionClient,
Video,
VideoClient,
} from "@opencode/ai/effect"
import { Speech, SpeechClient, SpeechEvent, TranscriptionClient, Video, VideoClient } from "@opencode/ai/experimental"
import { Route, Protocol, RequestExecutor, WebSocketTransport } from "@opencode/ai/route"
import {
Alibaba,
+7
View File
@@ -1,6 +1,7 @@
import type { Effect, Stream } from "effect"
import type { Client, GenerationHandle } from "../src/index.js"
import type { Decision, Generation, Image, LLM, LLMClient } from "../src/effect.js"
import type { Speech, Transcription, Video } from "../src/experimental.js"
// The Promise client mirrors the Effect API: same methods, same arguments, same results. The only allowed differences
// are `signal` on the options bag, Promise for Effect, AsyncIterable for Stream and GenerationHandle for Generation.
@@ -13,6 +14,9 @@ export type Surface = {
readonly llm: Operations<typeof LLM> & Pick<typeof LLMClient, "compact" | "canCompact">
readonly image: typeof Image
readonly decision: typeof Decision
readonly speech: typeof Speech
readonly transcription: typeof Transcription
readonly video: typeof Video
}
type Operations<M> = { [K in keyof M as M[K] extends (...args: never) => unknown ? K : never]: M[K] }
@@ -68,4 +72,7 @@ export type Checks = [
Assert<Drift<Client["llm"], Surface["llm"]>>,
Assert<Drift<Client["image"], Surface["image"]>>,
Assert<Drift<Client["decision"], Surface["decision"]>>,
Assert<Drift<Client["experimental"]["speech"], Surface["speech"]>>,
Assert<Drift<Client["experimental"]["transcription"], Surface["transcription"]>>,
Assert<Drift<Client["experimental"]["video"], Surface["video"]>>,
]
+25 -15
View File
@@ -4,7 +4,8 @@ import { tmpdir } from "node:os"
import { join } from "node:path"
import { Effect, Layer, Schema } from "effect"
import { HttpClientRequest } from "effect/http"
import { AIError, LLMEvent, Media, SpeechEvent, TranscriptionEvent } from "../src/effect.js"
import { AIError, LLMEvent, Media } from "../src/effect.js"
import { SpeechEvent, TranscriptionEvent } from "../src/experimental.js"
import { RequestExecutor, type StreamOptions } from "../src/route.js"
import { AI } from "../src/index.js"
import { AssemblyAI, OpenAI, Runway } from "../src/providers.js"
@@ -174,7 +175,7 @@ describe("AI promise client", () => {
const ai = AI.make({ layer: executor(seen) })
const model = Runway.configure({ apiKey: "test", baseURL: "https://runway.test/v1" }).video("gen4.5")
const generation = await ai.video.start({ model, prompt: "A kite" })
const generation = await ai.experimental.video.start({ model, prompt: "A kite" })
expect(generation.id).toBe("task_1")
expect(generation.status).toBe("queued")
expect(generation.token).toEqual({ taskID: "task_1" })
@@ -188,12 +189,12 @@ describe("AI promise client", () => {
expect(response.video.source).toMatchObject({ type: "url", url: "https://runway.test/out.mp4" })
expect(response.usage).toEqual({ type: "credits", credits: 5 })
const resumed = await ai.video.resume(model, JSON.parse(JSON.stringify(generation.token)))
const resumed = await ai.experimental.video.resume(model, JSON.parse(JSON.stringify(generation.token)))
expect(resumed.status).toBe("completed")
expect((await resumed.await()).videos).toHaveLength(1)
const events: Array<string> = []
for await (const event of ai.video.stream({ model, prompt: "A kite" }, { poll: { interval: 10 } })) {
for await (const event of ai.experimental.video.stream({ model, prompt: "A kite" }, { poll: { interval: 10 } })) {
events.push(event.type)
}
expect(events).toEqual(["video", "finish"])
@@ -229,7 +230,7 @@ describe("AI promise client", () => {
test("exposes a generation's snapshot and terminal state", async () => {
const ai = AI.make({ layer: executor([]) })
const model = Runway.configure({ apiKey: "test", baseURL: "https://runway.test/v1" }).video("gen4.5")
const generation = await ai.video.start({ model, prompt: "A kite" })
const generation = await ai.experimental.video.start({ model, prompt: "A kite" })
expect(generation.snapshot).toEqual({ id: "task_1", status: "queued", progress: undefined, position: undefined })
expect(generation.terminal).toBe(false)
const refreshed = await generation.refresh()
@@ -241,7 +242,7 @@ describe("AI promise client", () => {
test("observes a started generation's events and fetches its result", async () => {
const ai = AI.make({ layer: executor([]) })
const model = Runway.configure({ apiKey: "test", baseURL: "https://runway.test/v1" }).video("gen4.5")
const generation = await ai.video.start({ model, prompt: "A kite" })
const generation = await ai.experimental.video.start({ model, prompt: "A kite" })
const events: Array<string> = []
for await (const event of generation.events({ poll: { interval: 10 } })) events.push(event.type)
@@ -274,11 +275,16 @@ describe("AI promise client", () => {
const bytes = Media.bytes(Uint8Array.from([0x49, 0x44, 0x33]), "audio/mpeg")
expect(
(await ai.transcription.generate({ model: openai.transcription("gpt-4o-mini-transcribe"), audio: bytes })).text,
(
await ai.experimental.transcription.generate({
model: openai.transcription("gpt-4o-mini-transcribe"),
audio: bytes,
})
).text,
).toBe("Hello there.")
const deltas: Array<string> = []
for await (const event of ai.transcription.stream({
for await (const event of ai.experimental.transcription.stream({
model: openai.transcription("gpt-4o-mini-transcribe"),
audio: bytes,
}))
@@ -288,7 +294,7 @@ describe("AI promise client", () => {
const model = AssemblyAI.configure({ apiKey: "test", baseURL: "https://assemblyai.test" }).transcription(
"universal-2",
)
const generation = await ai.transcription.start({ model, audio })
const generation = await ai.experimental.transcription.start({ model, audio })
expect(generation.token).toEqual({ transcriptID: "tr_1" })
expect((await generation.await({ poll: { interval: 10 } })).text).toBe("Hello there.")
await ai.dispose()
@@ -298,11 +304,12 @@ describe("AI promise client", () => {
const ai = AI.make({ layer: executor([]) })
const model = openai.speech("gpt-4o-mini-tts")
const response = await ai.speech.generate({ model, text: "Hello", voice: "coral", format: "pcm" })
const response = await ai.experimental.speech.generate({ model, text: "Hello", voice: "coral", format: "pcm" })
expect(await ai.run(response.audio.bytes())).toEqual(Uint8Array.from([1, 2, 3]))
const events: Array<SpeechEvent> = []
for await (const event of ai.speech.stream({ model, text: "Hello", voice: "coral" })) events.push(event)
for await (const event of ai.experimental.speech.stream({ model, text: "Hello", voice: "coral" }))
events.push(event)
expect(events.map((event) => event.type)).toEqual(["audio-delta", "audio-delta", "finish"])
const finish = events.find(SpeechEvent.is.finish)
expect(await ai.run(finish!.audio.bytes())).toEqual(Uint8Array.from([1, 2, 3]))
@@ -361,7 +368,10 @@ describe("AI promise client", () => {
).toBe(reason)
const preAborted = await Array.fromAsync(
ai.speech.stream({ model: openai.speech("gpt-4o-mini-tts"), text: "Hello" }, { signal: aborted.signal }),
ai.experimental.speech.stream(
{ model: openai.speech("gpt-4o-mini-tts"), text: "Hello" },
{ signal: aborted.signal },
),
).catch((error: unknown) => error)
expect(preAborted).toBe(aborted.signal.reason)
expect(preAborted).toMatchObject({ name: "AbortError" })
@@ -381,7 +391,7 @@ describe("AI promise client", () => {
expect(midStreamFailure).toMatchObject({ name: "AbortError" })
const model = Runway.configure({ apiKey: "test", baseURL: "https://runway.test/v1" }).video("gen4.5")
const generation = await ai.video.start({ model, prompt: "A kite" })
const generation = await ai.experimental.video.start({ model, prompt: "A kite" })
const polling = new AbortController()
const events: Array<string> = []
const eventsFailure = await Array.fromAsync(
@@ -544,8 +554,8 @@ describe("AI promise client", () => {
test("the default client is created lazily and can be disposed", async () => {
expect(typeof AI.ai.llm.generate).toBe("function")
expect(typeof AI.ai.image.generate).toBe("function")
expect(typeof AI.ai.video.start).toBe("function")
expect(typeof AI.ai.speech.generate).toBe("function")
expect(typeof AI.ai.experimental.video.start).toBe("function")
expect(typeof AI.ai.experimental.speech.generate).toBe("function")
await AI.ai.dispose()
})
})
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { Transcription } from "../../src/effect.js"
import { Transcription } from "../../src/experimental.js"
import { AssemblyAI } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TRANSCRIPT, audio, audioRecording } from "./transcription-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { Speech, SpeechEvent } from "../../src/effect.js"
import { Speech, SpeechEvent } from "../../src/experimental.js"
import { Cartesia } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TEXT, collectSpeech } from "./speech-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { Speech } from "../../src/effect.js"
import { Speech } from "../../src/experimental.js"
import { Deepgram } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TEXT, collectSpeech } from "./speech-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect, Stream } from "effect"
import { Transcription } from "../../src/effect.js"
import { Transcription } from "../../src/experimental.js"
import { Deepgram } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TRANSCRIPT, audio, audioRecording, dialog } from "./transcription-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { Speech, SpeechEvent } from "../../src/effect.js"
import { Speech, SpeechEvent } from "../../src/experimental.js"
import { ElevenLabs } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TEXT, collectSpeech } from "./speech-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect, Stream } from "effect"
import { Transcription } from "../../src/effect.js"
import { Transcription } from "../../src/experimental.js"
import { ElevenLabs } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TRANSCRIPT, audio, audioRecording, dialog } from "./transcription-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { Speech } from "../../src/effect.js"
import { Speech } from "../../src/experimental.js"
import { Google } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TEXT, collectSpeech } from "./speech-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect, Stream } from "effect"
import { Transcription, TranscriptionEvent } from "../../src/effect.js"
import { Transcription, TranscriptionEvent } from "../../src/experimental.js"
import { Google } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TRANSCRIPT, audio, audioRecording, dialog } from "./transcription-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { Video } from "../../src/effect.js"
import { Video } from "../../src/experimental.js"
import { Google } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { queuedPoll } from "./queued-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { Speech } from "../../src/effect.js"
import { Speech } from "../../src/experimental.js"
import { OpenAI } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TEXT, collectSpeech } from "./speech-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect, Stream } from "effect"
import { Transcription, TranscriptionEvent } from "../../src/effect.js"
import { Transcription, TranscriptionEvent } from "../../src/experimental.js"
import { OpenAI } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { TRANSCRIPT, audio, audioRecording } from "./transcription-recording.js"
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { Video } from "../../src/effect.js"
import { Video } from "../../src/experimental.js"
import { Runway } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { queuedPoll } from "./queued-recording.js"
@@ -1,6 +1,6 @@
import { expect } from "bun:test"
import { Effect, Stream } from "effect"
import { SpeechEvent } from "../../src/effect.js"
import { SpeechEvent } from "../../src/experimental.js"
export const TEXT = "Hello from OpenCode."
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect, Stream } from "effect"
import { Video } from "../../src/effect.js"
import { Video } from "../../src/experimental.js"
import { XAI } from "../../src/providers.js"
import { recordedTests } from "../recorded-test.js"
import { queuedPoll } from "./queued-recording.js"
+1 -1
View File
@@ -1,7 +1,7 @@
import { describe, expect } from "bun:test"
import { Effect, Layer, Stream } from "effect"
import { Base64 } from "effect/encoding"
import { Speech, SpeechClient, SpeechEvent } from "../src/effect.js"
import { Speech, SpeechClient, SpeechEvent } from "../src/experimental.js"
import { Cartesia, Deepgram, ElevenLabs, Google, OpenAI } from "../src/providers.js"
import { it } from "./lib/effect.js"
import { dynamicResponse } from "./lib/http.js"
+1 -1
View File
@@ -1,5 +1,5 @@
import type { Stream } from "effect"
import { Speech, SpeechModel, type SpeechEvent, type SpeechOptions } from "../src/effect.js"
import { Speech, SpeechModel, type SpeechEvent, type SpeechOptions } from "../src/experimental.js"
import { ElevenLabs, OpenAI, Runway } from "../src/providers.js"
type StreamItem<T> = T extends Stream.Stream<infer A, infer _E, infer _R> ? A : never
+12 -13
View File
@@ -1,23 +1,18 @@
import { describe, expect, test } from "bun:test"
import { Effect, Layer, Stream } from "effect"
import { AIError, Decision, DecisionResponse, Image, ImageResponse, Media, RateLimitError } from "../src/effect.js"
import {
AIError,
Decision,
DecisionResponse,
Image,
ImageResponse,
Media,
RateLimitError,
Speech,
SpeechResponse,
Transcription,
TranscriptionResponse,
Video,
VideoResponse,
} from "../src/effect.js"
} from "../src/experimental.js"
import { AI } from "../src/index.js"
import { OpenAI, Runway } from "../src/providers.js"
import { TestDecision, TestImage, TestSpeech, TestTranscription, TestVideo } from "../src/testing.js"
import { TestDecision, TestImage } from "../src/testing.js"
import { TestSpeech, TestTranscription, TestVideo } from "../src/experimental/testing.js"
import { testEffect } from "./lib/effect.js"
const openai = OpenAI.configure({ apiKey: "test" })
@@ -113,13 +108,17 @@ describe("media test clients", () => {
)
expect((await ai.image.generate({ model: openai.image("gpt-image-2"), prompt: "A lighthouse" })).image).toBe(png)
const generation = await ai.video.start({ model: runway, prompt: "A kite" })
expect((await ai.video.resume(runway, generation.token).then((resumed) => resumed.await())).video).toBe(mp4)
const generation = await ai.experimental.video.start({ model: runway, prompt: "A kite" })
expect(
await ai.bytes((await ai.speech.generate({ model: openai.speech("gpt-4o-mini-tts"), text: "Hi" })).audio),
(await ai.experimental.video.resume(runway, generation.token).then((resumed) => resumed.await())).video,
).toBe(mp4)
expect(
await ai.bytes(
(await ai.experimental.speech.generate({ model: openai.speech("gpt-4o-mini-tts"), text: "Hi" })).audio,
),
).toEqual(Uint8Array.from([4, 5, 6]))
const deltas = await Array.fromAsync(
ai.transcription.stream({ model: openai.transcription("gpt-4o-mini-transcribe"), audio }),
ai.experimental.transcription.stream({ model: openai.transcription("gpt-4o-mini-transcribe"), audio }),
)
expect(deltas.map((event) => event.type)).toEqual(["finish"])
const decided = await ai.decision.generate({
+2 -1
View File
@@ -2,7 +2,8 @@ import { describe, expect } from "bun:test"
import { Effect, Fiber, Layer, Stream } from "effect"
import * as TestClock from "effect/testing/TestClock"
import { HttpClientRequest } from "effect/http"
import { Media, Transcription, TranscriptionClient, type TranscriptionEvent } from "../src/effect.js"
import { Media } from "../src/effect.js"
import { Transcription, TranscriptionClient, type TranscriptionEvent } from "../src/experimental.js"
import { AssemblyAI, Deepgram, ElevenLabs, Google, OpenAI } from "../src/providers.js"
import { it } from "./lib/effect.js"
import { dynamicResponse, json, observe, type Call } from "./lib/http.js"
+2 -2
View File
@@ -1,11 +1,11 @@
import type { Stream } from "effect"
import { Media } from "../src/effect.js"
import {
Media,
Transcription,
TranscriptionModel,
type TranscriptionEvent,
type TranscriptionOptions,
} from "../src/effect.js"
} from "../src/experimental.js"
import { AssemblyAI, Deepgram, OpenAI } from "../src/providers.js"
type StreamItem<T> = T extends Stream.Stream<infer A, infer _E, infer _R> ? A : never
+2 -1
View File
@@ -1,7 +1,8 @@
import { describe, expect } from "bun:test"
import { Effect, Fiber, Layer, Stream } from "effect"
import * as TestClock from "effect/testing/TestClock"
import { Media, Video, VideoClient, type GenerationEvent, type VideoEvent } from "../src/effect.js"
import { Media, type GenerationEvent } from "../src/effect.js"
import { Video, VideoClient, type VideoEvent } from "../src/experimental.js"
import { Google, Runway, XAI } from "../src/providers.js"
import { it } from "./lib/effect.js"
import { dynamicResponse, json, observe, settle, type Call, type HandlerInput } from "./lib/http.js"
+2 -3
View File
@@ -1,7 +1,6 @@
import { Effect } from "effect"
import { Generation, Media } from "../src/effect.js"
import {
Generation,
Media,
Video,
VideoClient,
VideoModel,
@@ -9,7 +8,7 @@ import {
type VideoModelOptions,
type VideoOptions,
type VideoRequestFor,
} from "../src/effect.js"
} from "../src/experimental.js"
import type { Service } from "../src/video-client.js"
import { Anthropic, Google, OpenAI, Runway, XAI } from "../src/providers.js"