Alerting Guide

Sky Puppy provides flexible alerting capabilities to notify you when your services change status. This guide covers setting up alerts for various platforms and best practices for effective monitoring.

Alert Types

Sky Puppy supports five types of alerts:

• down - Service is completely unavailable (status code -1)
• unhealthy - Service is responding but with issues (status code 0)
• unhealthy_status - Service returned unexpected status code (status code 0)
• unhealthy_response_time - Service is responding too slowly (status code 0)
• healthy - Service has recovered and is working normally (status code 1)

Discord Alerts

Discord is one of the most popular platforms for Sky Puppy alerts due to its rich embed support.

{
  "alerters": {
    "discord_critical": {
      "uri": "https://discord.com/api/webhooks/YOUR_WEBHOOK_URL",
      "json": true,
      "method": "POST",
      "timeout": 60,
      "body": {
        "embeds": [
          {
            "title": "🚨 {{service_name}} is {{alert_type}}!",
            "description": "Service was healthy for {{last_healthy_total_duration}} seconds",
            "color": 14828098,
            "fields": [
              {
                "name": "Service",
                "value": "{{service_name}}",
                "inline": true
              },
              {
                "name": "Status",
                "value": "{{alert_type}}",
                "inline": true
              },
              {
                "name": "Message",
                "value": "{{message}}",
                "inline": false
              }
            ],
            "timestamp": "{{timestamp}}",
            "footer": {
              "text": "Sky Puppy Monitor"
            }
          }
        ],
        "username": "Sky Puppy",
        "avatar_url": "https://i.imgur.com/J5vIVSt.png"
      }
    }
  }
}

Slack Alerts

Slack provides excellent integration capabilities for team notifications.

{
  "alerters": {
    "slack_alert": {
      "uri": "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK",
      "json": true,
      "method": "POST",
      "timeout": 60,
      "body": {
        "text": "🚨 {{service_name}} is {{alert_type}}!",
        "attachments": [
          {
            "color": "#ff0000",
            "fields": [
              {
                "title": "Service",
                "value": "{{service_name}}",
                "short": true
              },
              {
                "title": "Status",
                "value": "{{alert_type}}",
                "short": true
              },
              {
                "title": "Duration",
                "value": "{{last_healthy_total_duration}} seconds",
                "short": false
              },
              {
                "title": "Message",
                "value": "{{message}}",
                "short": false
              }
            ],
            "timestamp": "{{timestamp}}"
          }
        ]
      }
    }
  }
}

Template Variables

Sky Puppy provides several template variables for use in alert messages:

Variable	Description	Example
{{service_name}}	Name of the service	"web-frontend"
{{alert_type}}	Type of alert	"down", "unhealthy", "healthy"
{{message}}	Service status message	"Connection timeout"
{{timestamp}}	Current timestamp	"2024-01-15T10:30:00Z"
{{last_healthy_total_duration}}	Duration service was healthy (seconds)	"3600"
{{last_unhealthy_total_duration}}	Duration service was unhealthy (seconds)	"300"

Alert Configuration

Configure alerts in your service configuration:

{
  "services": {
    "my-website": {
      "interval": 30,
      "expected_response_time": 1000,
      "checker": {
        "name": "request",
        "settings": {
          "uri": "https://myapp.com/health",
          "timeout": 10
        }
      },
      "alerts": [
        {
          "type": "down",
          "alerter": "discord_critical",
          "for": 2
        },
        {
          "type": "unhealthy_response_time",
          "alerter": "discord_warning",
          "for": 3
        },
        {
          "type": "unhealthy_status",
          "alerter": "discord_warning",
          "for": 1
        },
        {
          "type": "unhealthy",
          "alerter": "discord_warning",
          "for": 1
        },
        {
          "type": "healthy",
          "alerter": "discord_recovery"
        }
      ]
    }
  }
}

Alert Options

Option	Type	Default	Description
type	string	required	Alert type: down, unhealthy, unhealthy_status, unhealthy_response_time, healthy
alerter	string	required	Name of the alerter to use
for	number	1	Number of consecutive failures before alerting
overrides	object	{}	Override alerter settings for this specific alert

Best Practices

1. Use Different Alerters for Different Severities

{
  "services": {
    "critical-service": {
      "alerts": [
        {
          "type": "down",
          "alerter": "pagerduty_critical"
        },
        {
          "type": "unhealthy",
          "alerter": "slack_warning"
        },
        {
          "type": "healthy",
          "alerter": "slack_recovery"
        }
      ]
    }
  }
}

2. Include Relevant Information

Always include:

• Service name
• Current status
• Duration information
• Error message
• Timestamp

3. Use Appropriate Colors and Icons

• 🔴 Red for critical/down alerts
• 🟡 Yellow/Orange for warnings/unhealthy
• 🟢 Green for recovery/healthy

4. Set Appropriate Thresholds

Use the for option to prevent false positives:

{
  "type": "down",
  "alerter": "discord_critical",
  "for": 3  // Only alert after 3 consecutive failures
}

Environment Variables

Use environment variables for sensitive data like webhook URLs:

# Set environment variables
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/..."

# Use in configuration
{
  "alerters": {
    "discord_alert": {
      "uri": "${DISCORD_WEBHOOK_URL}",
      // ... rest of configuration
    }
  }
}