Files
hermes-agent/tests/agent/test_auxiliary_error_classification.py
kshitijk4poor 49165bcaff fix(aux): retry, fallback and refresh decisions come from classify_api_error
The auxiliary client decided every recovery step (same-provider retry,
credential refresh, pool rotation, unhealthy mark, provider fallback)
with private substring predicates and consulted classify_api_error only
to label telemetry. The logged reason and the action taken could
disagree, plugin transform_api_error_classification hooks and
ProviderProfile.classify_api_error overrides never steered an aux
decision, and the tables had drifted from the main loop's.

What changed:
- Each failed request is classified once, on the backend, model,
  endpoint and credential it was sent with (_classify_failure): the
  primary, a same-provider retry (its rebuilt client and model), a
  refreshed Nous client or healed
  model, a parameter-rung or backoff re-send, and a fallback candidate
  (its effective provider or destination; an async candidate classifies
  on the sync client it was converted from). The streamed-negotiation
  and affordable-402 checks classify on the same attempted provider,
  passed in by the caller. Every later rung reads the
  resulting Failure (error, verdict, whether a parameter rung's detector
  names the rejected field). A request boundary that already classified
  (a rebuilt same-provider retry, a judged stream negotiation) raises the
  typed _ClassifiedAttempt carrying its Failure, and recovery reads that
  instead of classifying again. The relay logical call records each
  failed attempt with the ladder's Failure for it and reports that
  verdict at close (one classification only when no verdict exists):
  classifier hooks fire once per failed request.
- The primary's retries, the candidates' parameter rungs and the ladder
  driver handle a failure after its except block ends, so a retry's own
  failure is classified as itself instead of through the previous
  error's context. The stream watchdogs raise their timeout from None.
- agent/auxiliary_error_policy.py (new): small pure functions over the
  verdict (same_provider_retry, auth_refresh, same_key_retry,
  benches_provider, fallback_label, ladder_accepts,
  stream_negotiation_reraises).
- The sync and async fallback-candidate helpers share one generator.
- context_compressor classifies a summary failure once; its
  session-preserving abort keys on the verdict's transport fact.
- The parameter rungs return a _ParameterRepair NamedTuple whose
  repaired flag tells provider fallback a rung changed the request.
- Deleted: the private _is_*_error predicates, _credential_rung_accepts,
  _FALLBACK_REASONS, the keyword tables, fallback_candidate_unavailable_
  reason and agent/auxiliary_error_shapes.py.

Behaviour changes:
- Transient failures (5xx, 408, overload, timeouts, connection failures
  incl. DNS/ECONNREFUSED and SDK transport classes) retry the same
  provider, then fall back.
- A missing model falls back on every route; a 404 free-tier body falls
  back; 400-coded billing and rate-limit bodies bench / rotate and fall
  back.
- RESOURCE_EXHAUSTED on a 429 is a rate limit unless it names a daily or
  weekly window, which is billing and benches the provider.
- An OpenRouter-wrapped upstream 429 and a 429 "overloaded" do not rotate
  a healthy key; a 429 naming a rate limit and a billing phrase stays a
  rate limit.
- A generic 403 falls back (auto routes; an explicit provider only with
  its task fallback_chain) without a credential refresh.
- Account gating of the model rotates (model-scoped) before falling back;
  a feature restriction falls back without touching the key.
- A rejection naming a request parameter with no applicable rung
  re-raises (primary and candidates); after a spent rung it falls back.
- A verdict with retryable=False skips the same-provider retry and the
  same-key rate-limit backoff; a plugin or profile verdict with
  should_fallback=False never reaches provider fallback. A verdict that
  is neither retryable nor should_fallback (data_policy_denied, the
  model's malformed tool-call arguments) re-raises even after a
  parameter rung changed the request: another provider would get the
  same content.
- An InterruptedError (a cancelled Codex stream) stops the call: no
  retry, no fallback.
- A missing or route-refused model on a fallback candidate quarantines
  that model only (model-scoped unhealthy key); siblings on the same
  provider stay eligible. The key holds the provider-normalized model,
  so a configured name and its resolved form (gpt-x / openai/gpt-x)
  match, and the discovery chain skips a hidden resolved model too.
- Aux pool selection (_select_pool_entry, _peek_pool_entry and the
  OpenRouter, Codex, Anthropic, xAI OAuth, plugin OAuth, named custom
  and API-key builders) passes the requested model, so a model-scoped cooldown on a key does not hide
  it from sibling models; CredentialPool.peek takes model= like
  select.
- 429 then a generic 403 on an explicit provider with no task chain
  re-raises the 403.
- Plugin and provider-profile classifiers steer every aux decision.

Tests: tests/agent/test_auxiliary_error_policy.py drives call_llm end to
end over one table of the behaviour decisions, plus regressions for a
failure after a 401 refresh, 429-then-403, retryable=False, OpenCode Go
classification (sync and async), a refreshed candidate's bad answer, a
Nous SSE error after refresh, a terminal verdict after a spent
parameter rung, one hook call per failed request (primary, retry and
candidate; streamed and plain, sync and async), TLS failure and a 502 connection reset in
compression, plugin overrides, an interrupt, a pinned custom provider's
model-scoped rotation, a candidate's model-scoped quarantine under its
resolved name, a sibling model keeping a pooled key, and a rebuilt
retry classified on its own model. Tests of
the removed predicates are deleted; ladder-internals tests are adapted
to the Failure tuple.

Part of the Code Health plan.
2026-11-08 13:18:54 +05:30

94 lines
4.6 KiB
Python

"""Auxiliary error classification shares the main classifier's billing table and labels quarantines truthfully.
Invariants for #107166 (OpenRouter org "Budget limit exceeded" 403 must be a payment error for the aux
ladder, like the main loop), #64144 (an unhealthy mark for absent credentials must not claim a payment
error), and #108349 (a vision fallback never lands on a text-only main model; an upstream-capacity 429
never benches the credential pool).
"""
import logging
import pytest
from agent import auxiliary_client as ac
from agent import auxiliary_error_policy as aux_errors
from agent.error_classifier import FailoverReason, classify_api_error
class _Err(Exception):
def __init__(self, msg, status):
super().__init__(msg)
self.status_code = status
@pytest.mark.parametrize("body", [
"Error code: 403 - {'error': {'message': 'Budget limit exceeded (monthly limit). Contact your org admin.', 'code': 403}}",
"Error code: 403 - {'error': {'message': 'Key limit exceeded (total limit). Contact your org admin.', 'code': 403}}",
])
def test_openrouter_limit_403_is_billing_for_main_loop_and_aux_ladder(body):
exc = _Err(body, 403)
assert classify_api_error(exc).reason is FailoverReason.billing
# The aux ladder decides from the same verdict (one classifier, no drift).
verdict = aux_errors.classify_attempt(exc, "openrouter")
assert aux_errors.benches_provider(verdict)
assert aux_errors.fallback_label(aux_errors.Failure(exc, verdict)) == aux_errors.PAYMENT_ERROR
def test_absent_credentials_quarantine_is_debug_and_names_the_real_reason(caplog):
ac._reset_aux_unhealthy_cache()
with caplog.at_level(logging.DEBUG, logger="agent.auxiliary_client"):
ac._mark_provider_unhealthy("openrouter", ttl=60, reason="OPENROUTER_API_KEY not set", level=logging.DEBUG)
ac._log_skip_unhealthy("openrouter", "title_generation")
ac._mark_provider_unhealthy("nous") # confirmed 402 path keeps the payment wording at WARNING
marks = [r for r in caplog.records if "marking" in r.getMessage()]
assert [(r.levelno, "payment" in r.getMessage()) for r in marks] == [(logging.DEBUG, False), (logging.WARNING, True)]
assert "OPENROUTER_API_KEY not set" in marks[0].getMessage()
skip = next(r for r in caplog.records if "skipping" in r.getMessage())
assert "OPENROUTER_API_KEY not set" in skip.getMessage() and "payment" not in skip.getMessage()
ac._reset_aux_unhealthy_cache()
def test_vision_fallback_skips_text_only_main_model(monkeypatch):
monkeypatch.setattr(ac, "_read_main_provider", lambda: "zai")
monkeypatch.setattr(ac, "_read_main_model", lambda: "glm-5.3")
monkeypatch.setattr(ac, "_main_model_supports_vision", lambda provider, model: False)
monkeypatch.setattr(ac, "resolve_provider_client", lambda **kw: pytest.fail("must not build a client for a text-only model"))
assert ac._try_main_agent_model_fallback("nous", "vision", reason="rate limit") == (None, None, "")
@pytest.mark.parametrize("body, benches_pool", [
("Error code: 429 - {'status': 429, 'message': \"The requested model is temporarily at capacity upstream. "
"This is not your API key's rate limit — please retry shortly.\"}", False),
("Error code: 429 - {'error': {'message': 'Rate limit exceeded for this API key', 'code': 429}}", True),
])
def test_upstream_capacity_429_is_not_a_credential_to_bench(monkeypatch, body, benches_pool):
exc = _Err(body, 429)
assert classify_api_error(exc).reason is (
FailoverReason.rate_limit if benches_pool else FailoverReason.overloaded)
assert classify_api_error(exc).should_rotate_credential is benches_pool
# The verdict is only half the contract: _recover_provider_pool must not bench a credential
# for an upstream-capacity 429, while a real per-key rate limit still rotates the pool.
class _StubPool:
def __init__(self):
self.rotate_calls = []
def has_credentials(self):
return True
def try_refresh_current(self):
return None
def mark_exhausted_and_rotate(self, **kwargs):
self.rotate_calls.append(kwargs)
return object() # a next entry exists
pool = _StubPool()
monkeypatch.setattr(ac, "load_pool", lambda provider: pool)
monkeypatch.setattr(ac, "_evict_cached_clients", lambda provider: None)
recovered = ac._recover_provider_pool("openrouter", exc, failed_api_key="sk-failed")
assert recovered is benches_pool
assert len(pool.rotate_calls) == (1 if benches_pool else 0)
if benches_pool:
assert pool.rotate_calls[0]["status_code"] == 429
assert pool.rotate_calls[0]["api_key_hint"] == "sk-failed"