mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-17 03:12:19 +03:00
49165bcaff
The auxiliary client decided every recovery step (same-provider retry, credential refresh, pool rotation, unhealthy mark, provider fallback) with private substring predicates and consulted classify_api_error only to label telemetry. The logged reason and the action taken could disagree, plugin transform_api_error_classification hooks and ProviderProfile.classify_api_error overrides never steered an aux decision, and the tables had drifted from the main loop's. What changed: - Each failed request is classified once, on the backend, model, endpoint and credential it was sent with (_classify_failure): the primary, a same-provider retry (its rebuilt client and model), a refreshed Nous client or healed model, a parameter-rung or backoff re-send, and a fallback candidate (its effective provider or destination; an async candidate classifies on the sync client it was converted from). The streamed-negotiation and affordable-402 checks classify on the same attempted provider, passed in by the caller. Every later rung reads the resulting Failure (error, verdict, whether a parameter rung's detector names the rejected field). A request boundary that already classified (a rebuilt same-provider retry, a judged stream negotiation) raises the typed _ClassifiedAttempt carrying its Failure, and recovery reads that instead of classifying again. The relay logical call records each failed attempt with the ladder's Failure for it and reports that verdict at close (one classification only when no verdict exists): classifier hooks fire once per failed request. - The primary's retries, the candidates' parameter rungs and the ladder driver handle a failure after its except block ends, so a retry's own failure is classified as itself instead of through the previous error's context. The stream watchdogs raise their timeout from None. - agent/auxiliary_error_policy.py (new): small pure functions over the verdict (same_provider_retry, auth_refresh, same_key_retry, benches_provider, fallback_label, ladder_accepts, stream_negotiation_reraises). - The sync and async fallback-candidate helpers share one generator. - context_compressor classifies a summary failure once; its session-preserving abort keys on the verdict's transport fact. - The parameter rungs return a _ParameterRepair NamedTuple whose repaired flag tells provider fallback a rung changed the request. - Deleted: the private _is_*_error predicates, _credential_rung_accepts, _FALLBACK_REASONS, the keyword tables, fallback_candidate_unavailable_ reason and agent/auxiliary_error_shapes.py. Behaviour changes: - Transient failures (5xx, 408, overload, timeouts, connection failures incl. DNS/ECONNREFUSED and SDK transport classes) retry the same provider, then fall back. - A missing model falls back on every route; a 404 free-tier body falls back; 400-coded billing and rate-limit bodies bench / rotate and fall back. - RESOURCE_EXHAUSTED on a 429 is a rate limit unless it names a daily or weekly window, which is billing and benches the provider. - An OpenRouter-wrapped upstream 429 and a 429 "overloaded" do not rotate a healthy key; a 429 naming a rate limit and a billing phrase stays a rate limit. - A generic 403 falls back (auto routes; an explicit provider only with its task fallback_chain) without a credential refresh. - Account gating of the model rotates (model-scoped) before falling back; a feature restriction falls back without touching the key. - A rejection naming a request parameter with no applicable rung re-raises (primary and candidates); after a spent rung it falls back. - A verdict with retryable=False skips the same-provider retry and the same-key rate-limit backoff; a plugin or profile verdict with should_fallback=False never reaches provider fallback. A verdict that is neither retryable nor should_fallback (data_policy_denied, the model's malformed tool-call arguments) re-raises even after a parameter rung changed the request: another provider would get the same content. - An InterruptedError (a cancelled Codex stream) stops the call: no retry, no fallback. - A missing or route-refused model on a fallback candidate quarantines that model only (model-scoped unhealthy key); siblings on the same provider stay eligible. The key holds the provider-normalized model, so a configured name and its resolved form (gpt-x / openai/gpt-x) match, and the discovery chain skips a hidden resolved model too. - Aux pool selection (_select_pool_entry, _peek_pool_entry and the OpenRouter, Codex, Anthropic, xAI OAuth, plugin OAuth, named custom and API-key builders) passes the requested model, so a model-scoped cooldown on a key does not hide it from sibling models; CredentialPool.peek takes model= like select. - 429 then a generic 403 on an explicit provider with no task chain re-raises the 403. - Plugin and provider-profile classifiers steer every aux decision. Tests: tests/agent/test_auxiliary_error_policy.py drives call_llm end to end over one table of the behaviour decisions, plus regressions for a failure after a 401 refresh, 429-then-403, retryable=False, OpenCode Go classification (sync and async), a refreshed candidate's bad answer, a Nous SSE error after refresh, a terminal verdict after a spent parameter rung, one hook call per failed request (primary, retry and candidate; streamed and plain, sync and async), TLS failure and a 502 connection reset in compression, plugin overrides, an interrupt, a pinned custom provider's model-scoped rotation, a candidate's model-scoped quarantine under its resolved name, a sibling model keeping a pooled key, and a rebuilt retry classified on its own model. Tests of the removed predicates are deleted; ladder-internals tests are adapted to the Failure tuple. Part of the Code Health plan.
94 lines
4.6 KiB
Python
94 lines
4.6 KiB
Python
"""Auxiliary error classification shares the main classifier's billing table and labels quarantines truthfully.
|
|
|
|
Invariants for #107166 (OpenRouter org "Budget limit exceeded" 403 must be a payment error for the aux
|
|
ladder, like the main loop), #64144 (an unhealthy mark for absent credentials must not claim a payment
|
|
error), and #108349 (a vision fallback never lands on a text-only main model; an upstream-capacity 429
|
|
never benches the credential pool).
|
|
"""
|
|
import logging
|
|
|
|
import pytest
|
|
|
|
from agent import auxiliary_client as ac
|
|
from agent import auxiliary_error_policy as aux_errors
|
|
from agent.error_classifier import FailoverReason, classify_api_error
|
|
|
|
|
|
class _Err(Exception):
|
|
def __init__(self, msg, status):
|
|
super().__init__(msg)
|
|
self.status_code = status
|
|
|
|
|
|
@pytest.mark.parametrize("body", [
|
|
"Error code: 403 - {'error': {'message': 'Budget limit exceeded (monthly limit). Contact your org admin.', 'code': 403}}",
|
|
"Error code: 403 - {'error': {'message': 'Key limit exceeded (total limit). Contact your org admin.', 'code': 403}}",
|
|
])
|
|
def test_openrouter_limit_403_is_billing_for_main_loop_and_aux_ladder(body):
|
|
exc = _Err(body, 403)
|
|
assert classify_api_error(exc).reason is FailoverReason.billing
|
|
# The aux ladder decides from the same verdict (one classifier, no drift).
|
|
verdict = aux_errors.classify_attempt(exc, "openrouter")
|
|
assert aux_errors.benches_provider(verdict)
|
|
assert aux_errors.fallback_label(aux_errors.Failure(exc, verdict)) == aux_errors.PAYMENT_ERROR
|
|
|
|
|
|
def test_absent_credentials_quarantine_is_debug_and_names_the_real_reason(caplog):
|
|
ac._reset_aux_unhealthy_cache()
|
|
with caplog.at_level(logging.DEBUG, logger="agent.auxiliary_client"):
|
|
ac._mark_provider_unhealthy("openrouter", ttl=60, reason="OPENROUTER_API_KEY not set", level=logging.DEBUG)
|
|
ac._log_skip_unhealthy("openrouter", "title_generation")
|
|
ac._mark_provider_unhealthy("nous") # confirmed 402 path keeps the payment wording at WARNING
|
|
marks = [r for r in caplog.records if "marking" in r.getMessage()]
|
|
assert [(r.levelno, "payment" in r.getMessage()) for r in marks] == [(logging.DEBUG, False), (logging.WARNING, True)]
|
|
assert "OPENROUTER_API_KEY not set" in marks[0].getMessage()
|
|
skip = next(r for r in caplog.records if "skipping" in r.getMessage())
|
|
assert "OPENROUTER_API_KEY not set" in skip.getMessage() and "payment" not in skip.getMessage()
|
|
ac._reset_aux_unhealthy_cache()
|
|
|
|
|
|
def test_vision_fallback_skips_text_only_main_model(monkeypatch):
|
|
monkeypatch.setattr(ac, "_read_main_provider", lambda: "zai")
|
|
monkeypatch.setattr(ac, "_read_main_model", lambda: "glm-5.3")
|
|
monkeypatch.setattr(ac, "_main_model_supports_vision", lambda provider, model: False)
|
|
monkeypatch.setattr(ac, "resolve_provider_client", lambda **kw: pytest.fail("must not build a client for a text-only model"))
|
|
assert ac._try_main_agent_model_fallback("nous", "vision", reason="rate limit") == (None, None, "")
|
|
|
|
|
|
@pytest.mark.parametrize("body, benches_pool", [
|
|
("Error code: 429 - {'status': 429, 'message': \"The requested model is temporarily at capacity upstream. "
|
|
"This is not your API key's rate limit — please retry shortly.\"}", False),
|
|
("Error code: 429 - {'error': {'message': 'Rate limit exceeded for this API key', 'code': 429}}", True),
|
|
])
|
|
def test_upstream_capacity_429_is_not_a_credential_to_bench(monkeypatch, body, benches_pool):
|
|
exc = _Err(body, 429)
|
|
assert classify_api_error(exc).reason is (
|
|
FailoverReason.rate_limit if benches_pool else FailoverReason.overloaded)
|
|
assert classify_api_error(exc).should_rotate_credential is benches_pool
|
|
|
|
# The verdict is only half the contract: _recover_provider_pool must not bench a credential
|
|
# for an upstream-capacity 429, while a real per-key rate limit still rotates the pool.
|
|
class _StubPool:
|
|
def __init__(self):
|
|
self.rotate_calls = []
|
|
|
|
def has_credentials(self):
|
|
return True
|
|
|
|
def try_refresh_current(self):
|
|
return None
|
|
|
|
def mark_exhausted_and_rotate(self, **kwargs):
|
|
self.rotate_calls.append(kwargs)
|
|
return object() # a next entry exists
|
|
|
|
pool = _StubPool()
|
|
monkeypatch.setattr(ac, "load_pool", lambda provider: pool)
|
|
monkeypatch.setattr(ac, "_evict_cached_clients", lambda provider: None)
|
|
recovered = ac._recover_provider_pool("openrouter", exc, failed_api_key="sk-failed")
|
|
assert recovered is benches_pool
|
|
assert len(pool.rotate_calls) == (1 if benches_pool else 0)
|
|
if benches_pool:
|
|
assert pool.rotate_calls[0]["status_code"] == 429
|
|
assert pool.rotate_calls[0]["api_key_hint"] == "sk-failed"
|