Luna/Documentation/References/ZeroClaw/ReliableProvider.md

ReliableProvider

The ReliableProvider is a resilient decorator in ZeroClaw that wraps one or more standard providers to implement a three-level failover strategy. It ensures high availability by managing model fallbacks, provider redundancy, and intelligent retry logic with backoff.

---

Provider Trait

All LLM integrations in ZeroClaw implement the Provider trait. This trait defines a unified interface for chat interactions, tool calling, and capability discovery.

#[async_trait]
pub trait Provider: Send + Sync {
    fn capabilities(&self) -> ProviderCapabilities {
        ProviderCapabilities::default()
    }

    async fn chat_with_system(
        &self,
        system_prompt: Option<&str>,
        message: &str,
        model: &str,
        temperature: f64,
    ) -> anyhow::Result<String>;

    async fn chat_with_history(
        &self,
        messages: &[ChatMessage],
        model: &str,
        temperature: f64,
    ) -> anyhow::Result<String>;

    async fn chat(
        &self,
        request: ChatRequest<'_>,
        model: &str,
        temperature: f64,
    ) -> anyhow::Result<ChatResponse>;

    async fn chat_with_tools(
        &self,
        messages: &[ChatMessage],
        _tools: &[serde_json::Value],
        model: &str,
        temperature: f64,
    ) -> anyhow::Result<ChatResponse>;
}

ProviderCapabilities

The ProviderCapabilities struct allows providers to declare their feature set, which ReliableProvider uses to route requests correctly:

• native_tool_calling: Support for API-native function calling (Gemini, Anthropic, OpenAI).
• vision: Support for multimodal image inputs.

ToolsPayload

When tools are used, Provider::convert_tools returns a ToolsPayload enum to handle varying API requirements:

• Gemini { function_declarations }
• Anthropic { tools }
• OpenAI { tools }
• PromptGuided { instructions }: Textual fallback for providers without native support.

---

Three-Level Failover

The ReliableProvider implements a nested loop structure to exhaust all possibilities before returning an error.

pub struct ReliableProvider {
    providers: Vec<(String, Box<dyn Provider>)>,
    max_retries: u32,
    base_backoff_ms: u64,
    api_keys: Vec<String>,
    key_index: AtomicUsize,
    model_fallbacks: HashMap<String, Vec<String>>,
}

1. Level 1: Model Chain: Iterates through the primary model and its configured fallbacks (e.g., try claude-3-5-sonnet, fallback to claude-3-haiku).
2. Level 2: Provider Chain: For each model, iterates through registered providers in priority order (e.g., try Anthropic direct, fallback to OpenRouter).
3. Level 3: Retry Loop: For a specific (provider, model) pair, retries transient failures with exponential backoff.

---

Error Classification

Intelligent failure handling depends on distinguishing transient issues from permanent ones. ZeroClaw uses heuristics and status codes for this classification.

Non-Retryable Errors

The is_non_retryable() function identifies errors that won't resolve with retries, such as:

• Client Errors: HTTP 4xx (except 429/408).
• Authentication: Key-word matching for "invalid api key", "unauthorized", or "permission denied".
• Model Availability: Heuristics for "model not found", "unknown model", or "unsupported".

Rate Limiting and Quota

• is_rate_limited(): Specifically detects HTTP 429 errors.
• is_non_retryable_rate_limit(): Detects business-level failures returned as 429s, such as "insufficient balance", "quota exhausted", or "plan does not include requested model". These trigger a provider fallback immediately.

Context Window Exceeded

The system short-circuits when the context window is exceeded to avoid useless retries:

fn is_context_window_exceeded(err: &anyhow::Error) -> bool {
    let hints = [
        "exceeds the context window",
        "maximum context length",
        "token limit exceeded",
        "prompt is too long",
    ];
    // ...
}

---

Exponential Backoff

Retries use an exponential backoff strategy starting from base_backoff_ms.

• Retry-After Parsing: The system parses the Retry-After header or error body.
• Capping: Backoff is capped at 30 seconds to prevent indefinite stalls.
• Jitter: While the core logic doubles the wait time, it ensures the wait respects provider-suggested intervals.

---

API Key Rotation

ZeroClaw supports round-robin API key rotation for the same provider to maximize throughput:

fn rotate_key(&self) -> Option<&str> {
    if self.api_keys.is_empty() { return None; }
    let idx = self.key_index.fetch_add(1, Ordering::Relaxed) % self.api_keys.len();
    Some(&self.api_keys[idx])
}

When a 429 is encountered (and it's not a quota error), the system cycles to the next key for the subsequent retry attempt.

---

Relevance to Luna

Currently, Luna's Core implementation of IProvider is simple and lacks resilience. If a provider is down, the request fails.

By implementing a ReliableProvider decorator for Luna's IProvider, we can:

• Wrap multiple IChatClient instances.
• Implement per-sender model overrides to allow specific users to access higher-tier models with fallback to cheaper ones.
• Integrate the ZeroClaw error classification logic to handle common LLM API failures gracefully.

This pattern is essential for moving from a prototype to a production-grade system where provider stability cannot be guaranteed.

---

Core Configuration