diff --git a/packages/ai/README.md b/packages/ai/README.md index ff1baed737c..ea346383c32 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -897,17 +897,17 @@ Everything below is exported from `@opencode/ai/effect` unless noted. - **`Generation`** — provider-neutral handle for an in-flight media generation (`await`, `refresh`, `cancel`, `events`) used by queued media routes. - **`ai.decision.request` / `ai.decision.generate`** — typed predicate, choice, and score questions over text, JSON, and media input; `DecisionClient` is its Effect service and layer. - **`AIClient.layer` / `AIClient.layerWith(executor)`** — every modality client plus the request executor in one layer. -- **`@opencode/ai`** (Promise root) — a default `ai` with the text calls (`request`, `generate`, `stream`, `loop`, `compact`, `canCompact`) at the top level plus `image` and `decision` (plus `experimental.video`, `experimental.speech`, and `experimental.transcription`) as Promise / `AsyncIterable` APIs, `run(effect)` for any package Effect, plus `file`, `write`, `bytes`, `base64`, and `materialize` for assets. `AI.make({ layer?, clients? })` builds a client over a custom executor or test layers. The same `ai` from `@opencode/ai/effect` returns `Effect` and `Stream`. +- **`@opencode/ai`** (Promise root) — a default `ai` with the text calls (`request`, `generate`, `stream`, `loop`, `compact`, `canCompact`) at the top level plus `image` and `decision` (plus `experimental.video`, `experimental.speech`, and `experimental.transcription`) as Promise / `AsyncIterable` APIs, `run(effect)` for any package Effect, plus `file`, `write`, `bytes`, `base64`, and `materialize` for assets. `AI.make({ layer? })` builds a client over a custom executor. The same `ai` from `@opencode/ai/effect` returns `Effect` and `Stream`. - **`@opencode/ai/experimental`** — APIs that may change without compatibility guarantees: `Video`, `Speech`, `Transcription`, `Extraction`, and `Realtime`. See [Experimental](#experimental). ## Experimental These APIs may change without compatibility guarantees. The Effect API is exported from `@opencode/ai/experimental`, -the Promise client exposes it under `ai.experimental`, and test layers come from `@opencode/ai/experimental/testing`. +and the Promise client exposes it under `ai.experimental`. - **`Video`**, **`Speech`**, and **`Transcription`** — media generation and speech-to-text, with `VideoClient`, `SpeechClient`, and `TranscriptionClient` as their Effect services. Promise: `ai.experimental.video`, `.speech`, and - `.transcription`. Testing: `TestVideo`, `TestSpeech`, and `TestTranscription`. + `.transcription`. - **`Extraction`** — typed values read out of text. - **`Realtime`** — `Realtime.connect` starts an OpenAI, xAI, or Gemini voice session and returns a WebRTC answer SDP or a short-lived WebSocket URL and token. @@ -1236,13 +1236,16 @@ For explicit compaction, script a `CompactionResponse` through `push`, `always`, For `compact(request, { mechanism: "trigger" })`, script a `CompactionCheckpointResponse` instead. It carries `checkpoint`, `responseID`, and optional `usage`. Endpoint and trigger calls reject each other's fixtures; both share the same queue, gates, lazy recording, and fallback controls. -`TestImage` and `TestDecision` (and `TestVideo`, `TestSpeech`, and `TestTranscription` from `@opencode/ai/experimental/testing`) follow the same pattern: `testLayer()` provides the modality's client and `Test`, with the same `push`, `always`, `serve`, `requests`, `wait`, and `gate` controls. Script an `ImageResponse` (or the modality's response) or an `AIError`. `stream` expands the response into its events, and `start` returns an already finished generation that `resume` accepts by token. Decision responses are returned as scripted, without checking the answers against the questions. +Response helpers build common event arrays: `TestLLM.text(value, id)`, `textWithUsage(value, id, inputTokens)`, +`toolCall(id, name, input)`, `stop(...events)`, `toolCalls(...events)`, and +`complete({ reason, usage?, providerMetadata? }, ...events)`. `failAfter(error, ...events)` and `hangAfter(...events)` +return streams that fail or never finish after the given events. -The Promise client accepts the same layers as `AI.make({ clients: TestImage.testLayer() })`; `ai.run(TestImage.Test.use(...))` reaches the controls. +`compileRequest`, also from `@opencode/ai/testing`, compiles an `LLMRequest` through its route without sending it, so +tests can assert the provider payload. -The published legacy `Service`, `layer`, `clientLayer`, and module-level controls remain available as adapters -over the same implementation, including the legacy live `requests` array. New tests should use `Test` and -`testLayer`. +Other modalities and the Promise client have no test double. Test them over a custom executor layer, such as +`AI.make({ layer })`, that answers the HTTP requests. ## Provider compaction diff --git a/packages/ai/package.json b/packages/ai/package.json index fd8bf980601..72d24d9af25 100644 --- a/packages/ai/package.json +++ b/packages/ai/package.json @@ -20,7 +20,6 @@ "./effect": "./src/effect.ts", "./testing": "./src/testing.ts", "./experimental": "./src/experimental.ts", - "./experimental/testing": "./src/experimental/testing.ts", "./route": "./src/route.ts", "./providers": "./src/providers.ts", "./providers/alibaba": "./src/providers/alibaba.ts", diff --git a/packages/ai/src/experimental/testing.ts b/packages/ai/src/experimental/testing.ts deleted file mode 100644 index 44cbc9dfb30..00000000000 --- a/packages/ai/src/experimental/testing.ts +++ /dev/null @@ -1,3 +0,0 @@ -export { TestSpeech } from "../testing/speech.js" -export { TestTranscription } from "../testing/transcription.js" -export { TestVideo } from "../testing/video.js" diff --git a/packages/ai/src/index.ts b/packages/ai/src/index.ts index 65e51b32295..a4513637e0b 100644 --- a/packages/ai/src/index.ts +++ b/packages/ai/src/index.ts @@ -50,11 +50,9 @@ import { Video, type VideoModel, type VideoRequest, type VideoRequestInput } fro * transcription and decision clients over a request executor; every method runs the corresponding Effect API and rethrows * `AIError` unchanged. `file` and `write` load `node:fs/promises` on first use, so importing this module does not. */ -export interface Options { +export interface Options { /** Executor layer; defaults to `RequestExecutor.fetchLayer`. Inject a recorder or `RequestExecutor.middleware(fn)` here. */ readonly layer?: Layer.Layer - /** Replaces default clients, for example with `TestImage.testLayer()`; its other services are available to `run`. */ - readonly clients?: Layer.Layer } export interface RunOptions { @@ -95,20 +93,14 @@ const abortEffect = (signal: AbortSignal | undefined) => return Effect.sync(() => signal.removeEventListener("abort", onAbort)) }) -export const make = (options: Options = {}) => { - const runtime = ManagedRuntime.make( - Layer.merge( - AIClient.layerWith(options.layer ?? RequestExecutor.fetchLayer), - // `R` is only inferred from `clients`, so it is `never` whenever `clients` is omitted. - (options.clients ?? Layer.empty) as Layer.Layer, - ), - ) +export const make = (options: Options = {}) => { + const runtime = ManagedRuntime.make(AIClient.layerWith(options.layer ?? RequestExecutor.fetchLayer)) /** Run any package Effect (for example `ai.compact(...)` from `@opencode/ai/effect`) inside this runtime. */ - const run = (effect: Effect.Effect, options?: RunOptions) => + const run = (effect: Effect.Effect, options?: RunOptions) => runtime.runPromise(Effect.raceFirst(effect, abortEffect(options?.signal))) - const iterate = (stream: Stream.Stream, options?: RunOptions): AsyncIterable => + const iterate = (stream: Stream.Stream, options?: RunOptions): AsyncIterable => Stream.toAsyncIterable( Stream.unwrap( runtime.contextEffect.pipe( @@ -307,7 +299,7 @@ export const make = (options: Options = {}) => { } } -export type Client = ReturnType> +export type Client = ReturnType const fileError = (message: string, cause: unknown) => new AIError({ reason: new InvalidRequestError({ message, cause }) }) diff --git a/packages/ai/src/testing.ts b/packages/ai/src/testing.ts index ef8a5ce97b3..dbf6dc2a0c9 100644 --- a/packages/ai/src/testing.ts +++ b/packages/ai/src/testing.ts @@ -1,7 +1,5 @@ export * as TestLLM from "./testing.js" export { compileRequest } from "./route/client.js" -export { TestDecision } from "./testing/decision.js" -export { TestImage } from "./testing/image.js" import { LLMClient, @@ -21,8 +19,7 @@ import { type ProviderMetadata, type UsageInput, } from "./schema/index.js" -import { Context, Effect, Layer, Stream } from "effect" -import { TestScript } from "./testing/script.js" +import { Context, Deferred, Effect, Latch, Layer, Queue, Scope, Stream } from "effect" export type Response = | readonly LLMEvent[] @@ -30,17 +27,34 @@ export type Response = | CompactionResponse | CompactionCheckpointResponse -export type Gate = TestScript.Gate +export type Gate = Readonly<{ started: Effect.Effect; release: Effect.Effect }> type ClientInterface = Context.Service.Shape export type Responder = (request: LLMRequest) => Response -export interface TestInterface extends ClientInterface, TestScript.Controls {} +interface Controls { + /** Returns a snapshot of requests observed at execution time. */ + readonly requests: () => Effect.Effect + readonly push: (...responses: readonly Response[]) => Effect.Effect + /** Replaces the fallback without changing queued responses. */ + readonly always: (response: Response) => Effect.Effect + /** Answers requests after the one-shot queue is exhausted; receives the original request. */ + readonly serve: (responder: Responder) => Effect.Effect + /** Waits for request arrivals, not output or completion. */ + readonly wait: (count: number) => Effect.Effect + readonly gate: () => Effect.Effect +} + +export interface TestInterface extends ClientInterface, Controls {} export class Test extends Context.Service()("@opencode/ai/TestLLM/Test") {} -export type LayerOptions = TestScript.Options +export interface LayerOptions { + readonly transformRequest?: (request: LLMRequest) => LLMRequest + /** Used after the one-shot response queue is exhausted. Omit to defect on unexpected requests. */ + readonly fallback?: Response +} export const complete = ( options: { @@ -89,7 +103,7 @@ export const hangAfter = (...events: readonly LLMEvent[]) => Stream.concat(Strea const make = (options: LayerOptions) => Effect.sync(() => { - const script = TestScript.make("TestLLM", options) + const script = makeScript(options) const stream: ClientInterface["stream"] = (request) => Stream.unwrap( script.take(request).pipe( @@ -137,3 +151,68 @@ export const testLayer = (options: LayerOptions = {}) => Layer.effectContext( Effect.map(make(options), (test) => Context.make(LLMClient.Service, test).pipe(Context.add(Test, test))), ) + +/** The response queue, fallback, request log, and gates behind the test controls. */ +const makeScript = (options: LayerOptions) => { + const requests: LLMRequest[] = [] + const responses: Response[] = [] + let started = Deferred.makeUnsafe() + const initial = options.fallback + let fallback: Responder | undefined = initial === undefined ? undefined : () => initial + let activeGate: { readonly started: Queue.Queue; readonly release: Latch.Latch } | undefined + const wait = (count: number): Effect.Effect => + Effect.suspend(() => + requests.length >= count ? Effect.void : Deferred.await(started).pipe(Effect.andThen(wait(count))), + ) + + const take = (request: LLMRequest) => + Effect.suspend(() => { + const count = requests.push(options.transformRequest?.(request) ?? request) + const waiting = started + started = Deferred.makeUnsafe() + const gate = activeGate + try { + const response = responses.shift() ?? fallback?.(request) + if (!response) return Effect.die(new Error(`TestLLM has no response for request ${count}`)) + if (!gate) return Effect.succeed(response) + return Queue.offer(gate.started, undefined).pipe(Effect.andThen(gate.release.await), Effect.as(response)) + } finally { + // Waiters can resume synchronously; assign the reply and gate before notifying them. + Deferred.doneUnsafe(waiting, Effect.void) + } + }) + + const controls: Controls = { + requests: () => Effect.sync(() => [...requests]), + push: (...input) => + Effect.sync(() => { + responses.push(...input) + }), + always: (response) => + Effect.sync(() => { + fallback = () => response + }), + serve: (responder) => + Effect.sync(() => { + fallback = responder + }), + wait, + gate: () => + Effect.gen(function* () { + const gate = { + started: yield* Effect.acquireRelease(Queue.unbounded(), Queue.shutdown), + release: yield* Latch.make(), + } + activeGate = gate + const release = Effect.sync(() => { + if (activeGate === gate) activeGate = undefined + }).pipe(Effect.andThen(gate.release.open), Effect.asVoid) + yield* Effect.addFinalizer(() => release) + return { + started: Queue.take(gate.started), + release, + } + }), + } + return { take, controls } +} diff --git a/packages/ai/src/testing/decision.ts b/packages/ai/src/testing/decision.ts deleted file mode 100644 index b44681b6db6..00000000000 --- a/packages/ai/src/testing/decision.ts +++ /dev/null @@ -1,38 +0,0 @@ -import { Context, Effect, Layer } from "effect" -import { DecisionClient, type Interface } from "../decision-client.js" -import type { DecisionRequestFor, DecisionResponse, DecisionResponseFor } from "../decision.js" -import { AIError } from "../schema/index.js" -import { TestScript } from "./script.js" - -/** Returned as scripted, without checking its answers against the request's questions. */ -export type Response = DecisionResponse | AIError - -export interface TestInterface extends Interface, TestScript.Controls {} - -export class Test extends Context.Service()("@opencode/ai/TestDecision/Test") {} - -export type LayerOptions = TestScript.Options - -/** Provides one shared implementation under `DecisionClient.Service` and `TestDecision.Test`. */ -export const testLayer = (options: LayerOptions = {}) => - Layer.effectContext( - Effect.sync(() => { - const script = TestScript.make("TestDecision", options) - const client = Test.of({ - ...script.controls, - generate: (request) => - script - .take(request) - .pipe( - Effect.flatMap((response) => - response instanceof AIError - ? Effect.fail(response) - : Effect.succeed(response as DecisionResponseFor), - ), - ), - }) - return Context.make(DecisionClient.Service, client).pipe(Context.add(Test, client)) - }), - ) - -export * as TestDecision from "./decision.js" diff --git a/packages/ai/src/testing/image.ts b/packages/ai/src/testing/image.ts deleted file mode 100644 index c557df8988d..00000000000 --- a/packages/ai/src/testing/image.ts +++ /dev/null @@ -1,23 +0,0 @@ -import { Context } from "effect" -import { ImageClient } from "../image-client.js" -import type { ImageEvent, ImageRequestFor, ImageResponse } from "../image.js" -import { TestMedia } from "./media.js" -import type { TestScript } from "./script.js" - -export type Response = TestMedia.Response - -export interface TestInterface extends TestMedia.TestInterface {} - -export class Test extends Context.Service()("@opencode/ai/TestImage/Test") {} - -export type LayerOptions = TestScript.Options - -/** Provides one shared implementation under `ImageClient.Service` and `TestImage.Test`. */ -export const testLayer = (options: LayerOptions = {}) => - TestMedia.testLayer(ImageClient.Service, Test, { - name: "TestImage", - responseEvents: ImageClient.responseEvents, - options, - }) - -export * as TestImage from "./image.js" diff --git a/packages/ai/src/testing/media.ts b/packages/ai/src/testing/media.ts deleted file mode 100644 index 5fadc319a71..00000000000 --- a/packages/ai/src/testing/media.ts +++ /dev/null @@ -1,60 +0,0 @@ -import { Context, Effect, Layer, Stream } from "effect" -import { Generation } from "../generation.js" -import type { MediaClient } from "../media-client.js" -import { AIError } from "../schema/index.js" -import { TestScript } from "./script.js" - -/** A scripted reply: the response every operation returns, or the error it fails with. */ -export type Response = Value | AIError - -export interface TestInterface, Event, Value> - extends MediaClient.Interface, - TestScript.Controls> {} - -/** - * Builds one media test client. `stream` expands the response like an inline route, and `start` returns an already - * finished generation whose token `resume` accepts. - */ -export const testLayer = , Event, Value>( - service: Context.Service>, - test: Context.Service>, - input: { - readonly name: string - readonly responseEvents: (response: Value) => ReadonlyArray - readonly options: TestScript.Options> - }, -) => - Layer.effectContext( - Effect.sync(() => { - const script = TestScript.make(input.name, input.options) - const generations = new Map>() - const result = (response: Response) => - response instanceof AIError ? Effect.fail(response) : Effect.succeed(response) - const completed = (id: string, response: Response) => { - const snapshot = { id, status: response instanceof AIError ? "failed" : "completed" } as const - return new Generation({ status: Effect.succeed(snapshot), result: result(response) }, id, snapshot) - } - const generate = (request: Req) => script.take(request).pipe(Effect.flatMap(result)) - const client = test.of({ - ...script.controls, - generate, - stream: (request) => Stream.fromIterableEffect(Effect.map(generate(request), input.responseEvents)), - start: (request) => - script.take(request).pipe( - Effect.map((response) => { - const id = `${input.name}-${generations.size + 1}` - generations.set(id, response) - return completed(id, response) - }), - ), - resume: (_model, token) => { - const response = typeof token === "string" ? generations.get(token) : undefined - if (response === undefined) return Effect.die(new Error(`${input.name} has no generation ${String(token)}`)) - return Effect.succeed(completed(String(token), response)) - }, - }) - return Context.make(service, client).pipe(Context.add(test, client)) - }), - ) - -export * as TestMedia from "./media.js" diff --git a/packages/ai/src/testing/script.ts b/packages/ai/src/testing/script.ts deleted file mode 100644 index cea342beb8b..00000000000 --- a/packages/ai/src/testing/script.ts +++ /dev/null @@ -1,89 +0,0 @@ -import { Deferred, Effect, Latch, Queue, Scope } from "effect" - -export type Gate = Readonly<{ started: Effect.Effect; release: Effect.Effect }> - -export interface Controls { - /** Returns a snapshot of requests observed at execution time. */ - readonly requests: () => Effect.Effect - readonly push: (...responses: readonly Response[]) => Effect.Effect - /** Replaces the fallback without changing queued responses. */ - readonly always: (response: Response) => Effect.Effect - /** Answers requests after the one-shot queue is exhausted; receives the original request. */ - readonly serve: (responder: (request: Request) => Response) => Effect.Effect - /** Waits for request arrivals, not output or completion. */ - readonly wait: (count: number) => Effect.Effect - readonly gate: () => Effect.Effect -} - -export interface Options { - readonly transformRequest?: (request: Request) => Request - /** Used after the one-shot response queue is exhausted. Omit to defect on unexpected requests. */ - readonly fallback?: Response -} - -/** The response queue, fallback, request log, and gates shared by every test client. */ -export const make = (name: string, options: Options) => { - const requests: Request[] = [] - const responses: Response[] = [] - let started = Deferred.makeUnsafe() - const initial = options.fallback - let fallback: ((request: Request) => Response) | undefined = initial === undefined ? undefined : () => initial - let activeGate: { readonly started: Queue.Queue; readonly release: Latch.Latch } | undefined - const wait = (count: number): Effect.Effect => - Effect.suspend(() => - requests.length >= count ? Effect.void : Deferred.await(started).pipe(Effect.andThen(wait(count))), - ) - - const take = (request: Request) => - Effect.suspend(() => { - const count = requests.push(options.transformRequest?.(request) ?? request) - const waiting = started - started = Deferred.makeUnsafe() - const gate = activeGate - try { - const response = responses.shift() ?? fallback?.(request) - if (!response) return Effect.die(new Error(`${name} has no response for request ${count}`)) - if (!gate) return Effect.succeed(response) - return Queue.offer(gate.started, undefined).pipe(Effect.andThen(gate.release.await), Effect.as(response)) - } finally { - // Waiters can resume synchronously; assign the reply and gate before notifying them. - Deferred.doneUnsafe(waiting, Effect.void) - } - }) - - const controls: Controls = { - requests: () => Effect.sync(() => [...requests]), - push: (...input) => - Effect.sync(() => { - responses.push(...input) - }), - always: (response) => - Effect.sync(() => { - fallback = () => response - }), - serve: (responder) => - Effect.sync(() => { - fallback = responder - }), - wait, - gate: () => - Effect.gen(function* () { - const gate = { - started: yield* Effect.acquireRelease(Queue.unbounded(), Queue.shutdown), - release: yield* Latch.make(), - } - activeGate = gate - const release = Effect.sync(() => { - if (activeGate === gate) activeGate = undefined - }).pipe(Effect.andThen(gate.release.open), Effect.asVoid) - yield* Effect.addFinalizer(() => release) - return { - started: Queue.take(gate.started), - release, - } - }), - } - return { take, controls } -} - -export * as TestScript from "./script.js" diff --git a/packages/ai/src/testing/speech.ts b/packages/ai/src/testing/speech.ts deleted file mode 100644 index 0c8e6af8a77..00000000000 --- a/packages/ai/src/testing/speech.ts +++ /dev/null @@ -1,23 +0,0 @@ -import { Context } from "effect" -import { SpeechClient } from "../speech-client.js" -import type { SpeechEvent, SpeechRequestFor, SpeechResponse } from "../speech.js" -import { TestMedia } from "./media.js" -import type { TestScript } from "./script.js" - -export type Response = TestMedia.Response - -export interface TestInterface extends TestMedia.TestInterface {} - -export class Test extends Context.Service()("@opencode/ai/TestSpeech/Test") {} - -export type LayerOptions = TestScript.Options - -/** Provides one shared implementation under `SpeechClient.Service` and `TestSpeech.Test`. */ -export const testLayer = (options: LayerOptions = {}) => - TestMedia.testLayer(SpeechClient.Service, Test, { - name: "TestSpeech", - responseEvents: SpeechClient.responseEvents, - options, - }) - -export * as TestSpeech from "./speech.js" diff --git a/packages/ai/src/testing/transcription.ts b/packages/ai/src/testing/transcription.ts deleted file mode 100644 index 1f2cf3efd16..00000000000 --- a/packages/ai/src/testing/transcription.ts +++ /dev/null @@ -1,24 +0,0 @@ -import { Context } from "effect" -import { TranscriptionClient } from "../transcription-client.js" -import type { TranscriptionEvent, TranscriptionRequestFor, TranscriptionResponse } from "../transcription.js" -import { TestMedia } from "./media.js" -import type { TestScript } from "./script.js" - -export type Response = TestMedia.Response - -export interface TestInterface - extends TestMedia.TestInterface {} - -export class Test extends Context.Service()("@opencode/ai/TestTranscription/Test") {} - -export type LayerOptions = TestScript.Options - -/** Provides one shared implementation under `TranscriptionClient.Service` and `TestTranscription.Test`. */ -export const testLayer = (options: LayerOptions = {}) => - TestMedia.testLayer(TranscriptionClient.Service, Test, { - name: "TestTranscription", - responseEvents: TranscriptionClient.responseEvents, - options, - }) - -export * as TestTranscription from "./transcription.js" diff --git a/packages/ai/src/testing/video.ts b/packages/ai/src/testing/video.ts deleted file mode 100644 index a6bd73fa119..00000000000 --- a/packages/ai/src/testing/video.ts +++ /dev/null @@ -1,23 +0,0 @@ -import { Context } from "effect" -import { VideoClient } from "../video-client.js" -import type { VideoEvent, VideoRequestFor, VideoResponse } from "../video.js" -import { TestMedia } from "./media.js" -import type { TestScript } from "./script.js" - -export type Response = TestMedia.Response - -export interface TestInterface extends TestMedia.TestInterface {} - -export class Test extends Context.Service()("@opencode/ai/TestVideo/Test") {} - -export type LayerOptions = TestScript.Options - -/** Provides one shared implementation under `VideoClient.Service` and `TestVideo.Test`. */ -export const testLayer = (options: LayerOptions = {}) => - TestMedia.testLayer(VideoClient.Service, Test, { - name: "TestVideo", - responseEvents: VideoClient.responseEvents, - options, - }) - -export * as TestVideo from "./video.js" diff --git a/packages/ai/test/error-partial.test.ts b/packages/ai/test/error-partial.test.ts index b006204722b..79d42de69f1 100644 --- a/packages/ai/test/error-partial.test.ts +++ b/packages/ai/test/error-partial.test.ts @@ -1,10 +1,14 @@ import { describe, expect, test } from "bun:test" -import { Effect, Schema, Stream } from "effect" +import { Effect, Layer, Schema, Stream } from "effect" import { ai, LanguageModel, LLMEvent, ProviderInternalError, tool } from "../src/effect.js" import { AI, AIError } from "../src/index.js" import { OpenAIChat } from "../src/protocols/openai-chat.js" +import { OpenAI } from "../src/providers.js" +import { RequestExecutor } from "../src/route.js" import { TestLLM } from "../src/testing.js" import { testEffect } from "./lib/effect.js" +import { handlerLayer } from "./lib/http.js" +import { sseEvents } from "./lib/sse.js" const model = LanguageModel.make({ id: "fictional-model", provider: "fixture", route: OpenAIChat.route }) const it = testEffect(TestLLM.testLayer()) @@ -95,9 +99,27 @@ describe("AIError.partial", () => { describe("AIError.partial from the promise client", () => { test("rejects with the root-exported AIError carrying partial", async () => { - const client = AI.make({ clients: TestLLM.testLayer({ fallback: textThenFail }) }) + const client = AI.make({ + layer: RequestExecutor.layer.pipe( + Layer.provide( + handlerLayer((input) => + Effect.succeed( + input.respond( + sseEvents( + { choices: [{ delta: { content: "Partial answer" } }] }, + { error: { message: "Overloaded" } }, + ), + { headers: { "content-type": "text/event-stream" } }, + ), + ), + ), + ), + ), + }) - const error = await client.generate({ model, prompt: "Hello" }).catch((error: unknown) => error) + const error = await client + .generate({ model: OpenAI.configure({ apiKey: "test" }).chat("gpt-4o-mini"), prompt: "Hello" }) + .catch((error: unknown) => error) expect(error).toBeInstanceOf(AIError) expect(error instanceof AIError && error.partial?.text).toBe("Partial answer") diff --git a/packages/ai/test/testing-media.test.ts b/packages/ai/test/testing-media.test.ts deleted file mode 100644 index 29be0ce84d9..00000000000 --- a/packages/ai/test/testing-media.test.ts +++ /dev/null @@ -1,135 +0,0 @@ -import { describe, expect, test } from "bun:test" -import { Effect, Layer, Stream } from "effect" -import { ai, AIError, DecisionResponse, ImageResponse, Media, RateLimitError } from "../src/effect.js" -import { - Speech, - SpeechResponse, - Transcription, - TranscriptionResponse, - Video, - VideoResponse, -} from "../src/experimental.js" -import { AI } from "../src/index.js" -import { OpenAI, Runway } from "../src/providers.js" -import { TestDecision, TestImage } from "../src/testing.js" -import { TestSpeech, TestTranscription, TestVideo } from "../src/experimental/testing.js" -import { testEffect } from "./lib/effect.js" - -const openai = OpenAI.configure({ apiKey: "test" }) -const runway = Runway.configure({ apiKey: "test" }).video("gen4.5") -const png = Media.bytes(Uint8Array.from([1, 2, 3]), "image/png") -const audio = Media.bytes(Uint8Array.from([4, 5, 6]), "audio/pcm") -const mp4 = Media.url("https://video.test/out.mp4") -const question = { name: "red", type: "predicate", instructions: "Is it red?" } as const -const answer = new DecisionResponse({ answers: [{ name: "red", type: "predicate", probability: 0.9 }] }) - -const doubles = () => - Layer.mergeAll( - TestImage.testLayer(), - TestVideo.testLayer(), - TestSpeech.testLayer(), - TestTranscription.testLayer(), - TestDecision.testLayer(), - ) -const it = testEffect(doubles()) - -describe("media test clients", () => { - it.effect("script image responses and failures and record requests", () => - Effect.gen(function* () { - const test = yield* TestImage.Test - const failure = new AIError({ reason: new RateLimitError({ message: "Try later" }) }) - yield* test.push(new ImageResponse({ images: [png] }), failure) - yield* test.always(new ImageResponse({ images: [png, png] })) - - expect((yield* ai.image.generate({ model: openai.image("gpt-image-2"), prompt: "A lighthouse" })).image).toBe(png) - expect(yield* ai.image.generate({ model: openai.image("gpt-image-2"), prompt: "A kite" }).pipe(Effect.flip)).toBe( - failure, - ) - const events = yield* Stream.runCollect(ai.image.stream({ model: openai.image("gpt-image-2"), prompt: "A boat" })) - expect(Array.from(events, (event) => event.type)).toEqual(["image", "image", "finish"]) - expect((yield* test.requests()).map((request) => request.prompt)).toEqual(["A lighthouse", "A kite", "A boat"]) - }), - ) - - it.effect("starts completed video generations that resume from their token", () => - Effect.gen(function* () { - const test = yield* TestVideo.Test - yield* test.serve((request) => new VideoResponse({ videos: request.prompt === "A kite" ? [mp4] : [] })) - - const generation = yield* Video.start({ model: runway, prompt: "A kite" }) - expect(generation.status).toBe("completed") - expect((yield* generation.await()).video).toBe(mp4) - const resumed = yield* Video.resume(runway, JSON.parse(JSON.stringify(generation.token))) - expect((yield* resumed.result()).video).toBe(mp4) - expect(yield* test.requests()).toHaveLength(1) - expect(yield* Video.resume(runway, "unknown").pipe(Effect.catchDefect(Effect.succeed))).toMatchObject({ - message: "TestVideo has no generation unknown", - }) - }), - ) - - it.effect("script speech, transcription, and decision responses", () => - Effect.gen(function* () { - yield* (yield* TestSpeech.Test).push(new SpeechResponse({ audio })) - yield* (yield* TestTranscription.Test).push(new TranscriptionResponse({ text: "Hello there." })) - const decision = yield* TestDecision.Test - yield* decision.push(answer) - - const speech = yield* Stream.runCollect(Speech.stream({ model: openai.speech("gpt-4o-mini-tts"), text: "Hi" })) - expect(Array.from(speech, (event) => event.type)).toEqual(["finish"]) - expect( - (yield* Transcription.generate({ model: openai.transcription("gpt-4o-mini-transcribe"), audio })).text, - ).toBe("Hello there.") - const response = yield* ai.decision.generate({ - model: openai.decision("decision-model"), - input: [{ type: "text", text: "A red apple" }], - questions: [question], - }) - expect(response.answers).toEqual([{ name: "red", type: "predicate", probability: 0.9 }]) - expect((yield* decision.requests())[0].questions).toEqual([question]) - expect( - yield* Speech.generate({ model: openai.speech("gpt-4o-mini-tts"), text: "Hi" }).pipe( - Effect.catchDefect(Effect.succeed), - ), - ).toMatchObject({ message: "TestSpeech has no response for request 2" }) - }), - ) - - test("replace the Promise client's defaults and expose their controls to run", async () => { - const client = AI.make({ clients: doubles() }) - await client.run( - Effect.gen(function* () { - yield* (yield* TestImage.Test).push(new ImageResponse({ images: [png] })) - yield* (yield* TestVideo.Test).push(new VideoResponse({ videos: [mp4] })) - yield* (yield* TestSpeech.Test).push(new SpeechResponse({ audio })) - yield* (yield* TestTranscription.Test).push(new TranscriptionResponse({ text: "Hello there." })) - yield* (yield* TestDecision.Test).push(answer) - }), - ) - - expect((await client.image.generate({ model: openai.image("gpt-image-2"), prompt: "A lighthouse" })).image).toBe( - png, - ) - const generation = await client.experimental.video.start({ model: runway, prompt: "A kite" }) - expect( - (await client.experimental.video.resume(runway, generation.token).then((resumed) => resumed.await())).video, - ).toBe(mp4) - expect( - await client.bytes( - (await client.experimental.speech.generate({ model: openai.speech("gpt-4o-mini-tts"), text: "Hi" })).audio, - ), - ).toEqual(Uint8Array.from([4, 5, 6])) - const deltas = await Array.fromAsync( - client.experimental.transcription.stream({ model: openai.transcription("gpt-4o-mini-transcribe"), audio }), - ) - expect(deltas.map((event) => event.type)).toEqual(["finish"]) - const decided = await client.decision.generate({ - model: openai.decision("decision-model"), - input: [{ type: "text", text: "A red apple" }], - questions: [question], - }) - expect(decided.answers).toEqual([{ name: "red", type: "predicate", probability: 0.9 }]) - expect((await client.run(TestImage.Test.use((test) => test.requests())))[0].prompt).toBe("A lighthouse") - await client.dispose() - }) -})