Webhooks¶

A webhook tells an endpoint you control that something happened on this instance: a push landed, a repository was created or archived, a transfer needs approval, an experiment run finished. The server sends an HTTP POST with a JSON body and a signature you can verify, retries on failure, and keeps a delivery history you can inspect and replay.

Where to configure them¶

Namespace	Screen
Your own	`/settings/webhooks` (http://localhost:3000/settings/webhooks)
An organization	`/orgs/{org}/settings/webhooks`

Both screens are the same manager: the personal one lets you pick from every namespace you administer, the organization one is pinned to that organization.

Webhooks are admin-only. In an organization that means the admin role, not write — a webhook carries a namespace secret to an external URL, which is treated as an administrative act rather than a content change. A write member does not see the namespace offered in the picker, and gets a 403 from listing or creating webhooks for it (GET/POST /api/v1/namespaces/{ns}/webhooks). Every /api/v1/webhooks/{id} route (get, update, delete, deliveries, redeliver) instead answers 404 for a webhook that member may not administer, exactly as it does for one that does not exist — a 403 there would let anyone walk the small sequential ids and read back who owns each webhook. In your own namespace you are the admin, so nothing changes.

Each webhook has a scope: leave it blank and it fires for every repository in the namespace, or pick a single repository and it fires only for that one. It can also be created (or later toggled) inactive, which parks it without deleting it.

Events¶

There are ten events. All ten are offered in the UI and accepted by the API; anything else is rejected as an unknown event rather than silently stored.

Event	Fires when
`repo.push`	Post-push processing for a ref finished (blobs published, file/parquet indexes refreshed). Creating a branch through the API schedules the same work, so it also fires this — there is no separate branch-created event. Creating a tag does not.
`repo.created`	A repository was created
`repo.deleted`	A repository was deleted. Only namespace-wide subscriptions can receive it: a repository-scoped webhook is deleted along with its repository
`repo.moved`	A transfer or rename completed. Delivered to subscriptions on the new namespace
`repo.transfer_requested`	A transfer is waiting for the destination's approval. Delivered to the destination namespace
`repo.archived`	A repository was frozen read-only
`repo.unarchived`	An archived repository was thawed
`repo.ref_deleted`	A branch or tag was removed. Fires for a branch deleted either by `git push --delete` or through the API, but for a tag only through the API (`DELETE .../tag/{tag}`) — `git push --delete` on a tag is invisible to the server the same way creating a tag through a push is, and produces no webhook of any kind
`run.finished`	A tracked experiment run transitioned into `finished`
`run.failed`	A tracked experiment run transitioned into `failed`

run.finished / run.failed fire on the transition into that status, so a run that keeps logging after finishing — or a retried finish call — does not send a fresh delivery each time.

Run events need the live ingest API

Only runs reported through the ingest API — thinkingface.trackio, which calls finish() at the end of a run — produce run.finished / run.failed. Runs that arrive by the other route, where trackio writes its own Parquet files and you push them to a dataset repository, are indexed with a status but fire no webhook. Subscribe to repo.push on the experiment repository if that is how your runs get here. See Tracking Experiments.

What a delivery looks like¶

The request is a POST with a JSON body:

POST /your-endpoint HTTP/1.1
Content-Type: application/json
User-Agent: thinkingface-webhooks/1.0
X-Thinkingface-Event: repo.push
X-Thinkingface-Delivery: 4127
X-Thinkingface-Signature: sha256=9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08

Payloads¶

Each event's body is a flat JSON object. kind is "dataset" or "model"; full_name is "{namespace}/{name}".

repo.push:

{
  "namespace": "acme",
  "repo": "sentiment-base",
  "full_name": "acme/sentiment-base",
  "kind": "model",
  "ref": "main",
  "old_sha": "0a1b2c3…",
  "new_sha": "4d5e6f7…",
  "changed_files": 3
}

repo.created and repo.deleted:

{ "namespace": "acme", "name": "sentiment-base", "kind": "model", "full_name": "acme/sentiment-base" }

repo.archived and repo.unarchived — the same fields plus the resulting state, so a consumer does not have to infer it from the event name:

{
  "namespace": "acme", "name": "sentiment-base", "kind": "model",
  "full_name": "acme/sentiment-base", "archived": true
}

repo.ref_deleted:

{
  "namespace": "acme",
  "repo": "sentiment-base",
  "full_name": "acme/sentiment-base",
  "kind": "model",
  "ref": "old-experiment",
  "ref_type": "branch",
  "old_sha": "0a1b2c3…",
  "new_sha": ""
}

ref_type is "branch" or "tag", since a short ref name can name either. new_sha is always the empty string — there is no new value once a ref is gone.

repo.moved:

{
  "kind": "model",
  "from": { "namespace": "admin", "name": "sentiment-base" },
  "to": { "namespace": "acme", "name": "sentiment-base" },
  "full_name": "acme/sentiment-base"
}

repo.transfer_requested:

{
  "transfer_id": 12,
  "kind": "model",
  "from": { "namespace": "admin", "name": "sentiment-base" },
  "to": { "namespace": "acme", "name": "sentiment-base" },
  "requested_by": "admin",
  "expires_at": "2026-01-31T09:00:00Z"
}

run.finished and run.failed:

{
  "namespace": "acme",
  "repo": "training-metrics",
  "full_name": "acme/training-metrics",
  "project": "sentiment",
  "run": "run-2026-01-30-a",
  "status": "finished"
}

Verifying the signature¶

When you create a webhook the server generates a secret and shows it once, prefixed whsec_. Copy it then — no later response ever returns it. If you lose it, edit the webhook and rotate the secret, which shows a new one once and invalidates the old.

X-Thinkingface-Signature is sha256= followed by the lowercase hex HMAC-SHA256 of the raw request body, keyed by the secret:

X-Thinkingface-Signature: sha256=<hex(HMAC_SHA256(secret, raw_body))>

Two things matter for getting this right:

Hash the raw bytes, before any JSON parse-and-reserialize. Re-encoding changes whitespace and key order, and the signature will not match.
Compare in constant time (hmac.compare_digest in Python, hmac.Equal in Go, and so on), never with ==.

The secret is used as the HMAC key verbatim, including its whsec_ prefix — pass the whole string exactly as it was shown to you.

A minimal receiver, using Flask:

import hashlib
import hmac
import os

from flask import Flask, request

app = Flask(__name__)
SECRET = os.environ["THINKINGFACE_WEBHOOK_SECRET"].encode()  # the whole "whsec_..." string


@app.post("/thinkingface")
def receive():
    body = request.get_data()  # raw bytes, not request.json
    expected = "sha256=" + hmac.new(SECRET, body, hashlib.sha256).hexdigest()
    sent = request.headers.get("X-Thinkingface-Signature", "")
    if not hmac.compare_digest(sent, expected):
        return "bad signature", 401

    event = request.headers["X-Thinkingface-Event"]
    delivery = request.headers["X-Thinkingface-Delivery"]
    payload = request.get_json()
    app.logger.info("%s delivery=%s %s", event, delivery, payload)
    return "", 204  # any 2xx counts as success

Retries and delivery guarantees¶

Firing an event only writes delivery rows; a background worker pool does the actual sending, so nothing on the server waits for your endpoint.

One attempt has a 10-second timeout. Anything outside 2xx, and any transport failure, counts as a failed attempt.
Up to 5 attempts in total (the first plus four retries), so four waits: 30s, then 1m, 2m, 4m — each double the last, and capped at 15 minutes should the limits ever be raised. After the fifth failure the delivery is parked as failed and is not retried again on its own.
A disabled webhook parks its deliveries instead of failing them. While a webhook is inactive its pending deliveries are left untouched — no attempts are burned — and they go out once you re-enable it.
Delivery is at-least-once, and unordered. A worker that dies mid-delivery releases its claim after a couple of minutes and the delivery is retried, which can produce a duplicate of a request your endpoint already processed. Nothing guarantees that two events fired close together arrive in the order they happened, either. Make your handler idempotent and deduplicate on X-Thinkingface-Delivery.
Your response body is recorded. The first 4 KB of it is stored with the delivery, along with the status code (null when the endpoint could not be reached at all), which is what makes the history below useful for debugging.

Delivery history and redelivering¶

Expanding a webhook in the settings screen shows its deliveries, newest first: the event, the payload, the status (pending / success / failed), how many attempts it took, when the last attempt was, and the response the endpoint gave.

Redeliver re-enqueues the same event and payload as a new delivery. The original row is left exactly as it was, so the history keeps showing what really happened, and the replay gets its own delivery id — which is another reason to deduplicate on that header rather than assume one id means one arrival.

Which URLs are allowed¶

Only http and https URLs are accepted, and by default the server refuses to deliver to local or private addresses. That covers localhost and anything under .localhost, 127.0.0.0/8, 10/8, 172.16/12, 192.168/16, link-local ranges such as 169.254/16 (which is how cloud instance metadata is reached), the unspecified address, and ::1.

It also covers several ranges that are not private but are not somewhere a webhook should be delivered either: 100.64.0.0/10 (carrier-grade NAT — this is the range Tailscale hands out to a tailnet, so a receiver on your tailnet is refused), 0.0.0.0/8, 192.0.0.0/24, 198.18.0.0/15, 240.0.0.0/4, and multicast. An IPv4 address written as IPv6 — ::ffff:10.0.0.1, a NAT64 address embedding a private IPv4 — is judged by the IPv4 inside it rather than waved through.

A delivery is not followed across a redirect. A 3xx is recorded as the failed attempt it is, with its status in the delivery history, rather than being chased to a host you did not configure and never had checked.

The address check runs twice, deliberately: once when the webhook is created or edited, and again on the actual TCP connection each delivery opens, against the address it really resolved to. The second check is what stops a hostname that only resolves to a private address later — DNS rebinding, or an operator's DNS changing after the webhook was created.

For local development, where the receiver legitimately lives on localhost, an operator can set TF_WEBHOOKS_ALLOW_PRIVATE_TARGETS=true, which disables both checks. Leave it off in production: it lets anyone with admin on any namespace point a webhook at the instance's own internal network.