Webhooks

TraceStax can POST alert payloads to any HTTPS endpoint you control. Webhooks are useful when you need to integrate TraceStax alerts with an internal system, a custom on-call tool, or any destination not covered by the built-in integrations.

Overview

When an alert fires or resolves, TraceStax sends an HTTP POST request to the configured webhook URL. The payload is a signed JSON document describing the alert, the affected job, and the metrics that triggered it. Your endpoint should respond with a 2xx status code to acknowledge receipt.

Webhook payload

Every webhook request has a Content-Type: application/json header and the following body structure:

interface WebhookAlertPayload {
  tracestax_event: "alert";
  version: "1";
  alert: {
    type: AlertType;
    severity: "info" | "warning" | "critical";
    title: string;
    project: { id: string; name: string };
    entity: { type: string; [key: string]: string };
    timestamp: string;      // ISO 8601
    dashboard_url: string;
  };
}

type AlertType =
  | "dead_worker"
  | "fleet_degraded"
  | "failure_rate"
  | "anomaly_duration"
  | "anomaly_silence"
  | "queue_depth"
  | "drain_time"
  | "retry_storm"
  | "error_surge"
  | "plan_limit_warning"
  | "plan_limit_reached";

Example:

{
  "tracestax_event": "alert",
  "version": "1",
  "alert": {
    "type": "failure_rate",
    "severity": "critical",
    "title": "Failure rate spike on SendWelcomeEmail",
    "project": {
      "id": "proj_01hwx3k9abcdef0123456789",
      "name": "orders-service"
    },
    "entity": {
      "type": "task",
      "task_name": "app.tasks.email.send_welcome_email",
      "queue": "email"
    },
    "timestamp": "2026-03-24T14:22:00.000Z",
    "dashboard_url": "https://app.tracestax.com/proj_01hwx3k9abcdef0123456789/alerts/alrt_01hwx9k3mzp"
  }
}

Field descriptions

Field	Description
tracestax_event	Always "alert" for alert webhook deliveries
version	Payload schema version. Currently "1"
alert.type	The alert type: dead_worker, fleet_degraded, failure_rate, anomaly_duration, anomaly_silence, queue_depth, drain_time, retry_storm, error_surge, plan_limit_warning, plan_limit_reached
alert.severity	info, warning, or critical
alert.title	Human-readable summary of the alert
alert.project.id	The TraceStax project ID this alert belongs to
alert.project.name	The TraceStax project name
alert.entity	The entity the alert relates to. Always has a type field (e.g. "task", "worker", "queue"); additional fields vary by entity type
alert.timestamp	ISO 8601 timestamp of when the alert fired
alert.dashboard_url	Direct link to the alert in the TraceStax dashboard

Configuring a webhook

1. Navigate to your project in the TraceStax dashboard.

2. Open Project Settings from the left sidebar.

3. Select the Integrations tab.

4. Under Webhook, click Add webhook.

5. Enter the Endpoint URL. It must be an https:// URL. TraceStax does not support plain HTTP webhook endpoints.

6. Optionally enter a description to help identify this webhook in the integrations list.

7. Click Save. TraceStax will generate a webhook secret and display it once — copy it now and store it securely. You will use this secret to verify signatures.

8. Configure a routing rule to send alerts to this webhook. See Alert routing for details.

Note

The webhook secret is shown only once, immediately after creation. If you lose it, delete the webhook and create a new one to receive a new secret.

Verifying webhook signatures

TraceStax signs every webhook request using HMAC-SHA256. The signature is included in the X-TraceStax-Signature request header as a hex-encoded digest. A Unix timestamp is included in the X-TraceStax-Timestamp header.

The signature is computed over the timestamp concatenated with the raw request body, separated by a dot:

HMAC-SHA256(key=webhook_secret, message=timestamp + "." + raw_request_body)

You should verify this signature before processing the payload to confirm the request originated from TraceStax and was not modified in transit.

Headers

Header	Description
X-TraceStax-Signature	Hex-encoded HMAC-SHA256 signature
X-TraceStax-Timestamp	Unix timestamp (seconds) of when TraceStax sent the request

Node.js

import crypto from "crypto";

function verifyTraceStaxSignature(
  rawBody,       // Buffer or string — the raw request body
  signature,     // string — value of the X-TraceStax-Signature header
  timestamp,     // string — value of the X-TraceStax-Timestamp header
  webhookSecret  // string — your webhook secret from project settings
) {
  // Replay protection: reject timestamps older than 5 minutes
  const age = Math.floor(Date.now() / 1000) - Number(timestamp);
  if (Math.abs(age) > 300) {
    return false;
  }

  const expected = crypto
    .createHmac("sha256", webhookSecret)
    .update(timestamp + "." + rawBody)
    .digest("hex");

  // Use timingSafeEqual to prevent timing attacks
  const sigBuffer = Buffer.from(signature, "hex");
  const expectedBuffer = Buffer.from(expected, "hex");

  if (sigBuffer.length !== expectedBuffer.length) {
    return false;
  }

  return crypto.timingSafeEqual(sigBuffer, expectedBuffer);
}

// Express example
app.post("/webhooks/tracestax", express.raw({ type: "application/json" }), (req, res) => {
  const sig = req.headers["x-tracestax-signature"];
  const timestamp = req.headers["x-tracestax-timestamp"];

  if (!verifyTraceStaxSignature(req.body, sig, timestamp, process.env.TRACESTAX_WEBHOOK_SECRET)) {
    return res.status(401).json({ error: "Invalid signature" });
  }

  const payload = JSON.parse(req.body);
  // handle payload...

  res.status(200).json({ ok: true });
});

Python

import hashlib
import hmac
import os
import time

def verify_tracestax_signature(
    raw_body: bytes,
    signature: str,
    timestamp: str,
    webhook_secret: str
) -> bool:
    # Replay protection: reject timestamps older than 5 minutes
    age = abs(int(time.time()) - int(timestamp))
    if age > 300:
        return False

    message = timestamp.encode("utf-8") + b"." + raw_body
    expected = hmac.new(
        webhook_secret.encode("utf-8"),
        message,
        hashlib.sha256
    ).hexdigest()

    # Use compare_digest to prevent timing attacks
    return hmac.compare_digest(expected, signature)


# Flask example
from flask import Flask, request, abort
import json

app = Flask(__name__)

@app.route("/webhooks/tracestax", methods=["POST"])
def tracestax_webhook():
    sig = request.headers.get("X-TraceStax-Signature", "")
    timestamp = request.headers.get("X-TraceStax-Timestamp", "")
    secret = os.environ["TRACESTAX_WEBHOOK_SECRET"]

    if not verify_tracestax_signature(request.data, sig, timestamp, secret):
        abort(401)

    payload = json.loads(request.data)
    # handle payload...

    return {"ok": True}, 200

Caution

Always read the raw request body bytes for signature verification — do not parse the JSON first and re-serialize it. Re-serialized JSON may differ in whitespace or key ordering from what TraceStax signed.

Note

Replay protection: We recommend rejecting webhook requests with a timestamp older than 5 minutes. This prevents an attacker who has captured a valid request from replaying it later. Both code examples above include this check.

Inbound webhooks (ack-sync)

TraceStax can receive acknowledgment and resolution events back from your on-call provider. When an engineer acknowledges or resolves an incident in PagerDuty, OpsGenie, incident.io, Grafana OnCall, or Rootly, the corresponding TraceStax alert is automatically updated.

Supported providers

Provider	Signature method	Header
PagerDuty	HMAC-SHA256	X-PagerDuty-Signature
OpsGenie	Token comparison	X-OpsGenie-Token
incident.io	HMAC-SHA256	X-Incident-Signature
Grafana OnCall	Key comparison	X-Grafana-Alertmanager-Key
Rootly	HMAC-SHA256	X-Rootly-Signature

Setup

1. In TraceStax, navigate to Integrations and configure your provider (e.g. PagerDuty).

2. Enter the Inbound Webhook Secret — this is the signing secret or token from your provider’s outbound webhook configuration.

3. After saving, an Inbound Webhook URL appears on the integration card. It looks like:

   https://api.tracestax.com/v1/webhooks/oncall/<provider>/<channelId>

4. Copy this URL and paste it into your provider’s outbound webhook configuration (e.g. PagerDuty → Extensions → Generic Webhook V3).

Each URL is unique to your notification channel — the signing secret stored in TraceStax is used to verify that incoming requests genuinely came from your provider.

Note

The inbound webhook URL only appears after you save an inbound signing secret. If you skip this field, outbound alert notifications still work — you just won’t get automatic ack-sync back from your provider.

Retry policy

If your endpoint returns a non-2xx response, or does not respond within 10 seconds, TraceStax treats the delivery as failed and will retry:

Attempt	Delay before retry
1 (initial)	—
2	30 seconds
3	2 minutes
4	10 minutes
5	30 minutes

After 5 failed attempts TraceStax stops retrying. A delivery failure warning is logged in Project Settings → Integrations → Webhook so you can investigate.

If your endpoint is intermittently slow, returning a 2xx status immediately and processing the payload asynchronously is the recommended pattern. TraceStax will not retry a request that received a 2xx response, regardless of what your endpoint does with the payload afterwards.

Note

TraceStax sends at-least-once. In rare failure scenarios involving network partitions, your endpoint may receive duplicate deliveries for the same alert.id and event combination. Use the alert.id field to deduplicate if your handler is not idempotent.