Custom Providers
================

Learn how to integrate custom API providers with SpecAlign.

Overview
--------

In this tutorial, you will learn:

- How the provider system works
- How to configure different API providers
- How to add custom providers
- How to use local models

Prerequisites
-------------

- SpecAlign installed
- API credentials for your provider
- Basic understanding of OpenAI-compatible APIs

Provider System Architecture
----------------------------

SpecAlign uses a flexible provider system:

.. code-block:: text

   ┌─────────────────────────────────────────────────┐
   │                  providers.json                  │
   │  ┌─────────────────────────────────────────┐    │
   │  │ "openai": { base_url, api_key, models } │    │
   │  │ "anthropic": { ... }                    │    │
   │  │ "local": { ... }                        │    │
   │  └─────────────────────────────────────────┘    │
   └─────────────────────────────────────────────────┘
                          │
                          ▼
   ┌─────────────────────────────────────────────────┐
   │              ProviderLoader                      │
   │  - Validates configuration                       │
   │  - Initializes API clients                       │
   │  - Manages rate limiting                         │
   └─────────────────────────────────────────────────┘
                          │
                          ▼
   ┌─────────────────────────────────────────────────┐
   │              API Client                          │
   │  - OpenAI SDK (for compatible APIs)              │
   │  - Custom clients for non-standard APIs          │
   └─────────────────────────────────────────────────┘

Complete Example
----------------

.. code-block:: python

   """
   Custom Providers Example

   This script demonstrates how to configure and use
   different API providers with SpecAlign.
   """

   import json
   from pathlib import Path

   from specalign.providers import ProviderLoader
   from specalign.utils.api import APIClient
   from specalign.config import load_config


   def create_providers_config():
       """Create a multi-provider configuration."""
       print("Step 1: Creating providers configuration...")

       providers_config = {
           "providers": {
               # OpenAI (default)
               "openai": {
                   "base_url": "https://api.openai.com/v1",
                   "api_key": "${OPENAI_API_KEY}",  # Environment variable
                   "models": {
                       "default": "gpt-4o",
                       "embedding": "text-embedding-3-small"
                   }
               },

               # Anthropic Claude
               "anthropic": {
                   "base_url": "https://api.anthropic.com/v1",
                   "api_key": "${ANTHROPIC_API_KEY}",
                   "models": {
                       "default": "claude-3-opus-20240229"
                   },
                   "extra_headers": {
                       "anthropic-version": "2024-01-01"
                   }
               },

               # Azure OpenAI
               "azure": {
                   "base_url": "https://your-resource.openai.azure.com/",
                   "api_key": "${AZURE_OPENAI_KEY}",
                   "api_version": "2024-02-15-preview",
                   "models": {
                       "default": "gpt-4-deployment"
                   }
               },

               # Local model (Ollama, vLLM, etc.)
               "local": {
                   "base_url": "http://localhost:8000/v1",
                   "api_key": "not-needed",
                   "models": {
                       "default": "llama-3.1-70b"
                   },
                   "timeout": 120
               },

               # Together AI
               "together": {
                   "base_url": "https://api.together.xyz/v1",
                   "api_key": "${TOGETHER_API_KEY}",
                   "models": {
                       "default": "meta-llama/Llama-3-70b-chat-hf"
                   }
               }
           },

           "default_provider": "openai"
       }

       # Save configuration
       with open("providers.json", 'w') as f:
           json.dump(providers_config, f, indent=2)

       print("✓ Created providers.json with 5 providers")
       return providers_config


   def test_provider(provider_name: str, loader: ProviderLoader):
       """Test a specific provider."""
       print(f"\nTesting provider: {provider_name}")

       try:
           client = loader.get_client(provider_name)

           response = client.chat.completions.create(
               model=loader.get_model(provider_name, "default"),
               messages=[{"role": "user", "content": "Say 'Hello, SpecAlign!'"}],
               max_tokens=50
           )

           result = response.choices[0].message.content
           print(f"✓ {provider_name}: {result}")
           return True

       except Exception as e:
           print(f"✗ {provider_name}: {str(e)[:50]}...")
           return False


   def configure_per_component(config: dict):
       """
       Configure different providers for different components.

       This allows using cheaper/faster models for some tasks
       and more capable models for others.
       """
       print("\nStep 3: Configuring per-component providers...")

       config['api'] = {
           # Main agent uses GPT-4
           "provider": "openai",
           "model": "gpt-4o",

           # Embeddings use OpenAI
           "embedding_provider": "openai",
           "embedding_model": "text-embedding-3-small",
       }

       config['redteam']['judges'] = {
           # Safety judge uses more capable model
           "provider": "openai",
           "model": "gpt-4o",
       }

       config['redteam']['planner'] = {
           # Planner can use faster model
           "provider": "openai",
           "model": "gpt-4o-mini",
       }

       print("✓ Configured component-specific providers:")
       print("  - Main agent: openai/gpt-4o")
       print("  - Embeddings: openai/text-embedding-3-small")
       print("  - Safety judge: openai/gpt-4o")
       print("  - Planner: openai/gpt-4o-mini")

       return config


   def add_custom_provider(providers_config: dict):
       """Add a custom provider with special configuration."""
       print("\nStep 4: Adding custom provider...")

       # Example: Custom fine-tuned model endpoint
       providers_config['providers']['custom_finetuned'] = {
           "base_url": "https://your-custom-endpoint.com/v1",
           "api_key": "${CUSTOM_API_KEY}",
           "models": {
               "default": "your-finetuned-model-v1",
               "safety": "your-safety-model-v1"
           },
           "rate_limit": {
               "requests_per_minute": 60,
               "tokens_per_minute": 100000
           },
           "retry": {
               "max_retries": 3,
               "retry_delay": 1.0
           }
       }

       with open("providers.json", 'w') as f:
           json.dump(providers_config, f, indent=2)

       print("✓ Added custom_finetuned provider")
       return providers_config


   def setup_local_model():
       """Guide for setting up a local model."""
       print("\nStep 5: Local model setup guide...")

       guide = """
   Local Model Setup
   =================

   Option A: Ollama
   ----------------
   1. Install Ollama: https://ollama.ai
   2. Pull a model: ollama pull llama3.1
   3. Start server: ollama serve
   4. Configure provider:
      {
        "base_url": "http://localhost:11434/v1",
        "api_key": "not-needed",
        "models": {"default": "llama3.1"}
      }

   Option B: vLLM
   --------------
   1. Install: pip install vllm
   2. Start server:
      python -m vllm.entrypoints.openai.api_server \\
        --model meta-llama/Llama-3.1-70B-Instruct \\
        --port 8000
   3. Configure provider:
      {
        "base_url": "http://localhost:8000/v1",
        "api_key": "not-needed",
        "models": {"default": "meta-llama/Llama-3.1-70B-Instruct"}
      }

   Option C: Text Generation Inference (TGI)
   -----------------------------------------
   1. Run with Docker:
      docker run -p 8080:80 \\
        ghcr.io/huggingface/text-generation-inference:latest \\
        --model-id meta-llama/Llama-3.1-70B-Instruct
   2. Configure provider:
      {
        "base_url": "http://localhost:8080/v1",
        "api_key": "not-needed",
        "models": {"default": "meta-llama/Llama-3.1-70B-Instruct"}
      }
   """
       print(guide)


   def main():
       """Run the complete custom providers example."""
       print("=" * 50)
       print("Custom Providers Example")
       print("=" * 50)

       # Create providers configuration
       providers_config = create_providers_config()

       # Load and test providers
       print("\nStep 2: Testing providers...")
       loader = ProviderLoader("providers.json")

       # Test available providers
       for provider_name in providers_config['providers']:
           test_provider(provider_name, loader)

       # Configure per-component
       config = load_config("config.json")
       config = configure_per_component(config)

       # Add custom provider
       providers_config = add_custom_provider(providers_config)

       # Show local model setup guide
       setup_local_model()

       print("\n" + "=" * 50)
       print("✓ Custom providers configuration complete!")
       print("=" * 50)


   if __name__ == "__main__":
       main()

CLI Commands
------------

.. code-block:: bash

   # List configured providers
   specalign providers list

   # Add a new provider
   specalign providers add mycloud \
       --base-url https://api.mycloud.com/v1 \
       --api-key sk-xxx \
       --model gpt-4-equivalent

   # Test a provider
   specalign providers test mycloud

   # Set default provider
   specalign providers set-default mycloud

   # Remove a provider
   specalign providers remove mycloud

Expected Output
---------------

.. code-block:: text

   ==================================================
   Custom Providers Example
   ==================================================
   Step 1: Creating providers configuration...
   ✓ Created providers.json with 5 providers

   Step 2: Testing providers...

   Testing provider: openai
   ✓ openai: Hello, SpecAlign!

   Testing provider: anthropic
   ✗ anthropic: API key not configured...

   Testing provider: azure
   ✗ azure: Resource not found...

   Testing provider: local
   ✗ local: Connection refused...

   Testing provider: together
   ✗ together: API key not configured...

   Step 3: Configuring per-component providers...
   ✓ Configured component-specific providers:
     - Main agent: openai/gpt-4o
     - Embeddings: openai/text-embedding-3-small
     - Safety judge: openai/gpt-4o
     - Planner: openai/gpt-4o-mini

   Step 4: Adding custom provider...
   ✓ Added custom_finetuned provider

   ==================================================
   ✓ Custom providers configuration complete!
   ==================================================

Key Takeaways
-------------

1. **OpenAI-compatible APIs** work out of the box with the standard provider format
2. **Environment variables** can be used for API keys with ``${VAR_NAME}`` syntax
3. **Per-component configuration** allows optimizing cost vs capability tradeoffs
4. **Local models** can be integrated via Ollama, vLLM, or TGI
5. **Rate limiting** and retry logic can be configured per-provider

Next Steps
----------

- :doc:`../user_guide/configuration` - Full configuration reference
- :doc:`../api_reference/utils` - API client implementation details