Custom Providers
Learn how to integrate custom API providers with SpecAlign.
Overview
In this tutorial, you will learn:
How the provider system works
How to configure different API providers
How to add custom providers
How to use local models
Prerequisites
SpecAlign installed
API credentials for your provider
Basic understanding of OpenAI-compatible APIs
Provider System Architecture
SpecAlign uses a flexible provider system:
┌─────────────────────────────────────────────────┐
│ providers.json │
│ ┌─────────────────────────────────────────┐ │
│ │ "openai": { base_url, api_key, models } │ │
│ │ "anthropic": { ... } │ │
│ │ "local": { ... } │ │
│ └─────────────────────────────────────────┘ │
└─────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ ProviderLoader │
│ - Validates configuration │
│ - Initializes API clients │
│ - Manages rate limiting │
└─────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ API Client │
│ - OpenAI SDK (for compatible APIs) │
│ - Custom clients for non-standard APIs │
└─────────────────────────────────────────────────┘
Complete Example
"""
Custom Providers Example
This script demonstrates how to configure and use
different API providers with SpecAlign.
"""
import json
from pathlib import Path
from specalign.providers import ProviderLoader
from specalign.utils.api import APIClient
from specalign.config import load_config
def create_providers_config():
"""Create a multi-provider configuration."""
print("Step 1: Creating providers configuration...")
providers_config = {
"providers": {
# OpenAI (default)
"openai": {
"base_url": "https://api.openai.com/v1",
"api_key": "${OPENAI_API_KEY}", # Environment variable
"models": {
"default": "gpt-4o",
"embedding": "text-embedding-3-small"
}
},
# Anthropic Claude
"anthropic": {
"base_url": "https://api.anthropic.com/v1",
"api_key": "${ANTHROPIC_API_KEY}",
"models": {
"default": "claude-3-opus-20240229"
},
"extra_headers": {
"anthropic-version": "2024-01-01"
}
},
# Azure OpenAI
"azure": {
"base_url": "https://your-resource.openai.azure.com/",
"api_key": "${AZURE_OPENAI_KEY}",
"api_version": "2024-02-15-preview",
"models": {
"default": "gpt-4-deployment"
}
},
# Local model (Ollama, vLLM, etc.)
"local": {
"base_url": "http://localhost:8000/v1",
"api_key": "not-needed",
"models": {
"default": "llama-3.1-70b"
},
"timeout": 120
},
# Together AI
"together": {
"base_url": "https://api.together.xyz/v1",
"api_key": "${TOGETHER_API_KEY}",
"models": {
"default": "meta-llama/Llama-3-70b-chat-hf"
}
}
},
"default_provider": "openai"
}
# Save configuration
with open("providers.json", 'w') as f:
json.dump(providers_config, f, indent=2)
print("✓ Created providers.json with 5 providers")
return providers_config
def test_provider(provider_name: str, loader: ProviderLoader):
"""Test a specific provider."""
print(f"\nTesting provider: {provider_name}")
try:
client = loader.get_client(provider_name)
response = client.chat.completions.create(
model=loader.get_model(provider_name, "default"),
messages=[{"role": "user", "content": "Say 'Hello, SpecAlign!'"}],
max_tokens=50
)
result = response.choices[0].message.content
print(f"✓ {provider_name}: {result}")
return True
except Exception as e:
print(f"✗ {provider_name}: {str(e)[:50]}...")
return False
def configure_per_component(config: dict):
"""
Configure different providers for different components.
This allows using cheaper/faster models for some tasks
and more capable models for others.
"""
print("\nStep 3: Configuring per-component providers...")
config['api'] = {
# Main agent uses GPT-4
"provider": "openai",
"model": "gpt-4o",
# Embeddings use OpenAI
"embedding_provider": "openai",
"embedding_model": "text-embedding-3-small",
}
config['redteam']['judges'] = {
# Safety judge uses more capable model
"provider": "openai",
"model": "gpt-4o",
}
config['redteam']['planner'] = {
# Planner can use faster model
"provider": "openai",
"model": "gpt-4o-mini",
}
print("✓ Configured component-specific providers:")
print(" - Main agent: openai/gpt-4o")
print(" - Embeddings: openai/text-embedding-3-small")
print(" - Safety judge: openai/gpt-4o")
print(" - Planner: openai/gpt-4o-mini")
return config
def add_custom_provider(providers_config: dict):
"""Add a custom provider with special configuration."""
print("\nStep 4: Adding custom provider...")
# Example: Custom fine-tuned model endpoint
providers_config['providers']['custom_finetuned'] = {
"base_url": "https://your-custom-endpoint.com/v1",
"api_key": "${CUSTOM_API_KEY}",
"models": {
"default": "your-finetuned-model-v1",
"safety": "your-safety-model-v1"
},
"rate_limit": {
"requests_per_minute": 60,
"tokens_per_minute": 100000
},
"retry": {
"max_retries": 3,
"retry_delay": 1.0
}
}
with open("providers.json", 'w') as f:
json.dump(providers_config, f, indent=2)
print("✓ Added custom_finetuned provider")
return providers_config
def setup_local_model():
"""Guide for setting up a local model."""
print("\nStep 5: Local model setup guide...")
guide = """
Local Model Setup
=================
Option A: Ollama
----------------
1. Install Ollama: https://ollama.ai
2. Pull a model: ollama pull llama3.1
3. Start server: ollama serve
4. Configure provider:
{
"base_url": "http://localhost:11434/v1",
"api_key": "not-needed",
"models": {"default": "llama3.1"}
}
Option B: vLLM
--------------
1. Install: pip install vllm
2. Start server:
python -m vllm.entrypoints.openai.api_server \\
--model meta-llama/Llama-3.1-70B-Instruct \\
--port 8000
3. Configure provider:
{
"base_url": "http://localhost:8000/v1",
"api_key": "not-needed",
"models": {"default": "meta-llama/Llama-3.1-70B-Instruct"}
}
Option C: Text Generation Inference (TGI)
-----------------------------------------
1. Run with Docker:
docker run -p 8080:80 \\
ghcr.io/huggingface/text-generation-inference:latest \\
--model-id meta-llama/Llama-3.1-70B-Instruct
2. Configure provider:
{
"base_url": "http://localhost:8080/v1",
"api_key": "not-needed",
"models": {"default": "meta-llama/Llama-3.1-70B-Instruct"}
}
"""
print(guide)
def main():
"""Run the complete custom providers example."""
print("=" * 50)
print("Custom Providers Example")
print("=" * 50)
# Create providers configuration
providers_config = create_providers_config()
# Load and test providers
print("\nStep 2: Testing providers...")
loader = ProviderLoader("providers.json")
# Test available providers
for provider_name in providers_config['providers']:
test_provider(provider_name, loader)
# Configure per-component
config = load_config("config.json")
config = configure_per_component(config)
# Add custom provider
providers_config = add_custom_provider(providers_config)
# Show local model setup guide
setup_local_model()
print("\n" + "=" * 50)
print("✓ Custom providers configuration complete!")
print("=" * 50)
if __name__ == "__main__":
main()
CLI Commands
# List configured providers
specalign providers list
# Add a new provider
specalign providers add mycloud \
--base-url https://api.mycloud.com/v1 \
--api-key sk-xxx \
--model gpt-4-equivalent
# Test a provider
specalign providers test mycloud
# Set default provider
specalign providers set-default mycloud
# Remove a provider
specalign providers remove mycloud
Expected Output
==================================================
Custom Providers Example
==================================================
Step 1: Creating providers configuration...
✓ Created providers.json with 5 providers
Step 2: Testing providers...
Testing provider: openai
✓ openai: Hello, SpecAlign!
Testing provider: anthropic
✗ anthropic: API key not configured...
Testing provider: azure
✗ azure: Resource not found...
Testing provider: local
✗ local: Connection refused...
Testing provider: together
✗ together: API key not configured...
Step 3: Configuring per-component providers...
✓ Configured component-specific providers:
- Main agent: openai/gpt-4o
- Embeddings: openai/text-embedding-3-small
- Safety judge: openai/gpt-4o
- Planner: openai/gpt-4o-mini
Step 4: Adding custom provider...
✓ Added custom_finetuned provider
==================================================
✓ Custom providers configuration complete!
==================================================
Key Takeaways
OpenAI-compatible APIs work out of the box with the standard provider format
Environment variables can be used for API keys with
${VAR_NAME}syntaxPer-component configuration allows optimizing cost vs capability tradeoffs
Local models can be integrated via Ollama, vLLM, or TGI
Rate limiting and retry logic can be configured per-provider
Next Steps
Configuration Reference - Full configuration reference
Utils Module - API client implementation details