Webhooks

Receive real-time notifications for budget alerts, provider status changes, and daily summaries via Slack, Discord, or custom webhooks.

Features

Multiple formats — Slack, Discord, or generic JSON
Event filtering — Subscribe to specific event types
Custom headers — Add authentication or custom headers
Async dispatch — Non-blocking webhook delivery
Automatic formatting — Rich messages with emojis and colors
Test functionality — Verify webhook configuration before enabling

Configuration

{
  "webhooks": [
    {
      "enabled": true,
      "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
      "events": [
        "budget_warning",
        "budget_exceeded",
        "provider_down",
        "provider_up",
        "failover",
        "daily_summary"
      ],
      "headers": {
        "Authorization": "Bearer YOUR_TOKEN"
      }
    }
  ]
}

Event Types

Event	Description	When Triggered
`budget_warning`	Budget threshold reached	When spending reaches 80% of limit
`budget_exceeded`	Budget limit exceeded	When spending exceeds configured limit
`provider_down`	Provider becomes unhealthy	When success rate drops below 70%
`provider_up`	Provider recovers	When unhealthy provider becomes healthy again
`failover`	Request failed over	When request switches to backup provider
`daily_summary`	Daily usage summary	Once per day at midnight UTC

Webhook Formats

Slack

Automatically detected when URL contains slack.com.

Example message:

⚠️ Budget Warning: daily budget at 85.0% ($8.50 / $10.00)

Format:

{
  "text": "⚠️ Budget Warning: daily budget at 85.0% ($8.50 / $10.00)",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "⚠️ Budget Warning: daily budget at 85.0% ($8.50 / $10.00)"
      }
    }
  ]
}

Discord

Automatically detected when URL contains discord.com.

Example embed:

Title: budget_warning
Description: ⚠️ Budget Warning: daily budget at 85.0% ($8.50 / $10.00)
Color: Amber (#FBBF24)
Timestamp: 2026-03-05T10:30:00Z

Format:

{
  "content": "⚠️ Budget Warning: daily budget at 85.0% ($8.50 / $10.00)",
  "embeds": [
    {
      "title": "budget_warning",
      "description": "⚠️ Budget Warning: daily budget at 85.0% ($8.50 / $10.00)",
      "timestamp": "2026-03-05T10:30:00Z",
      "color": 16432932
    }
  ]
}

Generic JSON

Used for all other URLs.

Format:

{
  "event": "budget_warning",
  "timestamp": "2026-03-05T10:30:00Z",
  "data": {
    "period": "daily",
    "spent": 8.5,
    "limit": 10.0,
    "percent": 85.0,
    "project": ""
  }
}

Event Data Structures

Budget Warning / Exceeded

{
  "event": "budget_warning",
  "timestamp": "2026-03-05T10:30:00Z",
  "data": {
    "period": "daily",
    "spent": 8.5,
    "limit": 10.0,
    "percent": 85.0,
    "action": "warn",
    "project": "my-project"
  }
}

Provider Down / Up

{
  "event": "provider_down",
  "timestamp": "2026-03-05T10:30:00Z",
  "data": {
    "provider": "anthropic-primary",
    "status": "unhealthy",
    "error": "connection timeout",
    "latency_ms": 0
  }
}

Failover

{
  "event": "failover",
  "timestamp": "2026-03-05T10:30:00Z",
  "data": {
    "from_provider": "anthropic-primary",
    "to_provider": "anthropic-backup",
    "reason": "rate limit exceeded",
    "session_id": "sess_abc123"
  }
}

Daily Summary

{
  "event": "daily_summary",
  "timestamp": "2026-03-05T00:00:00Z",
  "data": {
    "date": "2026-03-04",
    "total_cost": 25.50,
    "total_requests": 150,
    "total_input_tokens": 125000,
    "total_output_tokens": 35000,
    "by_provider": {
      "anthropic": 18.20,
      "openai": 7.30
    }
  }
}

Platform Setup

Slack

Go to Slack API
Create a new app or select existing
Enable "Incoming Webhooks"
Add webhook to workspace
Copy webhook URL (starts with https://hooks.slack.com/)

Configuration:

{
  "webhooks": [
    {
      "enabled": true,
      "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXX",
      "events": ["budget_warning", "provider_down"]
    }
  ]
}

Discord

Open Discord server settings
Go to Integrations → Webhooks
Click "New Webhook"
Select channel and copy webhook URL

Configuration:

{
  "webhooks": [
    {
      "enabled": true,
      "url": "https://discord.com/api/webhooks/123456789/XXXXXXXXXXXXXXXXXXXX",
      "events": ["budget_exceeded", "failover"]
    }
  ]
}

Custom Webhook

For custom integrations, use generic JSON format:

{
  "webhooks": [
    {
      "enabled": true,
      "url": "https://your-server.com/webhook",
      "events": ["budget_warning", "daily_summary"],
      "headers": {
        "Authorization": "Bearer YOUR_SECRET_TOKEN",
        "X-Custom-Header": "value"
      }
    }
  ]
}

Web UI Configuration

Access webhook settings at http://localhost:19840/settings:

Navigate to "Webhooks" tab
Click "Add Webhook"
Enter webhook URL
Select events to subscribe
(Optional) Add custom headers
Click "Test" to verify configuration
Click "Save"

API Endpoints

List Webhooks

GET /api/v1/webhooks

Add Webhook

POST /api/v1/webhooks
Content-Type: application/json

{
  "enabled": true,
  "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
  "events": ["budget_warning", "provider_down"]
}

Update Webhook

PUT /api/v1/webhooks/{id}
Content-Type: application/json

{
  "enabled": false
}

Delete Webhook

DELETE /api/v1/webhooks/{id}

Test Webhook

POST /api/v1/webhooks/{id}/test

Sends a test message to verify configuration.

Message Examples

Budget Warning (Slack)

⚠️ Budget Warning: daily budget at 85.0% ($8.50 / $10.00)

Budget Exceeded (Discord)

🚫 Budget Exceeded: monthly limit of $200.00 reached (spent: $205.50). Action: block

Provider Down (Slack)

🔴 Provider Down: anthropic-primary is unhealthy. Error: connection timeout

Provider Up (Discord)

🟢 Provider Up: anthropic-primary is healthy again (latency: 1250ms)

Failover (Slack)

🔄 Failover: Switched from anthropic-primary to anthropic-backup. Reason: rate limit exceeded

Daily Summary (Discord)

📊 Daily Summary (2026-03-04): 150 requests, $25.50 total cost, 125000 input / 35000 output tokens

Best Practices

Use separate webhooks — Create different webhooks for different event types
Test before enabling — Always test webhook configuration before saving
Secure custom webhooks — Use HTTPS and authentication headers
Monitor webhook failures — Check daemon logs if notifications stop
Avoid sensitive data — Don't include API keys or tokens in webhook URLs
Set up alerts — Subscribe to critical events like budget_exceeded and provider_down

Troubleshooting

Webhook not receiving messages

Verify webhook is enabled in configuration
Check URL is correct (test with curl)
Verify events are configured correctly
Check daemon logs for webhook errors: tail -f ~/.zen/zend.log
Test webhook via API: POST /api/v1/webhooks/{id}/test

Slack webhook failing

Verify webhook URL starts with https://hooks.slack.com/
Check webhook is not revoked in Slack settings
Ensure workspace has not disabled incoming webhooks

Test with curl:

curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-Type: application/json' \
  -d '{"text":"test"}'

Discord webhook failing

Verify webhook URL starts with https://discord.com/api/webhooks/
Check webhook is not deleted in Discord settings
Ensure bot has permission to post in channel

Test with curl:

curl -X POST https://discord.com/api/webhooks/YOUR/WEBHOOK/URL \
  -H 'Content-Type: application/json' \
  -d '{"content":"test"}'

Custom webhook not working

Verify endpoint is accessible (test with curl)
Check authentication headers are correct
Ensure endpoint accepts POST requests
Verify endpoint returns 2xx status code
Check endpoint logs for errors

Security Considerations

Protect webhook URLs — Treat webhook URLs as secrets
Use HTTPS — Always use HTTPS for webhook endpoints
Validate signatures — Implement signature validation for custom webhooks
Rate limiting — Implement rate limiting on webhook endpoints
Don't log sensitive data — Avoid logging full webhook payloads

Advanced Configuration

Conditional Webhooks

Send different events to different webhooks:

{
  "webhooks": [
    {
      "enabled": true,
      "url": "https://hooks.slack.com/services/CRITICAL/ALERTS",
      "events": ["budget_exceeded", "provider_down"]
    },
    {
      "enabled": true,
      "url": "https://hooks.slack.com/services/DAILY/REPORTS",
      "events": ["daily_summary"]
    },
    {
      "enabled": true,
      "url": "https://discord.com/api/webhooks/MONITORING",
      "events": ["failover", "provider_up"]
    }
  ]
}

Custom Headers for Authentication

{
  "webhooks": [
    {
      "enabled": true,
      "url": "https://your-server.com/webhook",
      "events": ["budget_warning"],
      "headers": {
        "Authorization": "Bearer YOUR_SECRET_TOKEN",
        "X-API-Key": "your-api-key",
        "X-Webhook-Source": "gozen"
      }
    }
  ]
}

Features​

Configuration​

Event Types​

Webhook Formats​

Slack​

Discord​

Generic JSON​

Event Data Structures​

Budget Warning / Exceeded​

Provider Down / Up​

Failover​

Daily Summary​

Platform Setup​

Slack​

Discord​

Custom Webhook​

Web UI Configuration​

API Endpoints​

List Webhooks​

Add Webhook​

Update Webhook​

Delete Webhook​

Test Webhook​

Message Examples​

Budget Warning (Slack)​

Budget Exceeded (Discord)​

Provider Down (Slack)​

Provider Up (Discord)​

Failover (Slack)​

Daily Summary (Discord)​

Best Practices​

Troubleshooting​

Webhook not receiving messages​

Slack webhook failing​

Discord webhook failing​

Custom webhook not working​

Security Considerations​

Advanced Configuration​

Conditional Webhooks​

Custom Headers for Authentication​

Features

Configuration

Event Types

Webhook Formats

Slack

Discord

Generic JSON

Event Data Structures

Budget Warning / Exceeded

Provider Down / Up

Failover

Daily Summary

Platform Setup

Slack

Discord

Custom Webhook

Web UI Configuration

API Endpoints

List Webhooks

Add Webhook

Update Webhook

Delete Webhook

Test Webhook

Message Examples

Budget Warning (Slack)

Budget Exceeded (Discord)

Provider Down (Slack)

Provider Up (Discord)

Failover (Slack)

Daily Summary (Discord)

Best Practices

Troubleshooting

Webhook not receiving messages

Slack webhook failing

Discord webhook failing

Custom webhook not working

Security Considerations

Advanced Configuration

Conditional Webhooks

Custom Headers for Authentication