# Dog Academy CAPI: QA runbook and go/no-go gate

Version 1.0, 2026-09-28.
- **Companion to** [`DEV-SPEC.md`](DEV-SPEC.md) and [`event-contract.json`](event-contract.json). The contract is canonical.
- **Citations** resolve in [`sources.md`](sources.md).
- **Placeholders** are written `<LIKE_THIS>`.
- **Evidence** (screenshots, `event_id`s, `fbtrace_id`s) goes in a dated QA folder under `brands/dog-academy/research/`, never in this spec folder.
- **Tokens** never appear in evidence, screenshots, tickets or chat.

## 0. Roles, environments, test identity

| Item | Rule |
|---|---|
| Staging | Its own dataset, `<STAGING_DATASET_ID>`, and its own token. `META_TEST_EVENT_CODE` is set. Staging **never** posts to `481578137302100`, because "Events sent with test_event_code are not dropped. They flow into Events Manager and are used for targeting and ads measurement purposes" [D7]. |
| Production | Dataset `481578137302100`. A test code applies only to the labelled test lead, through `META_TEST_LEAD_EMAIL_REGEX` + `META_TEST_EVENT_CODE_PROD`. |
| Test identity | A consenting team member whose email and phone belong to a Meta account. Meta says test events "may get discarded if they don't match a Facebook or Meta account" [D8]. Name `Capi Test`, email `capi-test+<YYYYMMDD>@<TEAM_DOMAIN>`, a zip in a served area, dog `Testdog`. |
| Who runs what | DA's developer runs the staging tests. AdMaker witnesses them and checks every payload against the contract. Matthew stops sales from working the production test lead. |
| Test Events | Browser activity "is only recorded while the test events web page is open", so open the site *from* Test Events [H21]. Test data stays for 24 hours [H21]. |

## 1. Gate 0: run in the Ads Manager UI before building crm mode

Follow DEV-SPEC §0.4 (0-A, 0-B, optional 0-C). Save drafts only and never publish. Record the results in this table:

| Check | Result (fill in) | Screenshot |
|---|---|---|
| 0-A: "Maximize number of qualified leads" is listed for Leads → **Website** and can be selected | yes / no / greyed out (paste tooltip) | |
| 0-A: dataset(s) it accepts (does it take `481578137302100`?) | | |
| 0-A: signal-source options shown | | |
| 0-A: stage or target picker, and the events it lists | | |
| 0-A: requirement or eligibility messages; any stated window | | |
| 0-A: the same checks for Website and calls, and for Website and instant forms (recorded only) | | |
| 0-A: the dataset Settings page shows a CRM integration or "Configure sales funnel"; the data source category | | |
| 0-B: `Schedule` can be selected as the conversion event under Leads → Website | must be **yes** to launch | |
| 0-B: non-website events or their custom conversions appear in that list (record only; don't use them) | | |

**Decision:**
- 0-A positive → crm mode becomes a challenger cell (DEV-SPEC §6.6).
- 0-A negative → stay on `Schedule`.
- 0-B negative → **stop** and investigate the dataset and ad-account connection.

## 2. Static, cookie and pixel-coverage checks (staging, then production)

| ID | Test | Pass condition |
|---|---|---|
| T1 | Load `/`, `/in-person-dog-training/book-training`, the **private-, group- and online-training** pages, one city page, `/register`, `/login` and the scheduler page (signed in). Watch Test Events → browser | Exactly one `PageView` per load, from pixel `481578137302100`. No page initializes the pixel twice. |
| T2 | Load `/pages/psychiatric-service-dog-training-consult`, `/pages/service-dog-training-consult`, `/blog/`, `/adoption/` | **No** request to `facebook.com/tr` and no `fbq` in the page source |
| T3 | Run the domain-tag check below, then open Business Settings → Brand safety → Domains | The tag is present in static HTML; `dogacademy.org` shows **Verified** in DA's Business [H24] |
| T4 | Open `https://dogacademy.org/in-person-dog-training/private-training?fbclid=CAPITEST<YYYYMMDD>&utm_source=facebook` in a clean browser | The response sets `_fbc` (`Set-Cookie`, domain `dogacademy.org`, path `/`, **not HttpOnly**) and the value contains `CAPITEST<YYYYMMDD>`. `_fbp` exists. The first-touch UTMs are in the session or `__ussa`. |
| T5 | Continue to the quiz and reach the email step | The `lead_meta_identities` row stores plain `fb.1.…` `fbc` and `fbp` values (not encrypted blobs), IP, user agent, the quiz page path and UTMs |

The T3 domain-tag check (the tag must be in the static HTML, because Meta's crawler does not run JavaScript [H24]):

```bash
curl -s https://dogacademy.org/ | grep -o 'facebook-domain-verification[^>]*'
```

## 3. Event checks in Test Events (staging first, then the production test lead)

For each event, check both Test Events and the outbox row (`meta_capi_events`).

| ID | Event | Expected in Test Events | Payload assertions (outbox `payload`) |
|---|---|---|---|
| T6 | `QuizStart` (P2, if built) | Browser + server, same Event ID `da:quizstart:<SESSION_ID>` | `action_source=website`. The URL is the quiz page path. `user_data` only has zp/ct/st/country/IP/UA/fbc/fbp. |
| T7 | `Lead` | Browser (fired on the scheduler page after the redirect) + server, same Event ID `da:lead:<LEAD_ID>`, **one of them marked deduplicated** | `action_source=website`. `event_source_url` = `https://dogacademy.org/…book-training` (never `/livewire/message/…`). `user_data` has em, ph, fn, ln, ct, st, zp, country, external_id, client_ip_address, client_user_agent, fbc, fbp. `custom_data` holds only allowlisted keys, identical in browser and server. |
| T8 | `Schedule` (self-booked) | Browser + server, Event ID `da:schedule:<LEAD_ID>`, deduplicated | The same `user_data` rules. The IP and user agent come from the booking request. `custom_data` = `content_name: free_consultation`, `content_category`, `in_service_area`. Rescheduling the slot creates **no** new event. |
| T9 | No legacy events | During the whole quiz walk, no event arrives without an Event ID except `PageView`, and `Lead` arrives once (as a deduplicated pair) | Legacy `send-fb-event` names are gone (DEV-SPEC §5.4) |

**How to check deduplication:**
- Test Events lets you "verify that you've deduplicated events correctly by seeing which events were processed and deduplicated" [D8].
- After production traffic, go to Events Manager → event → **View details → Event deduplication**. The overlap for the `event_id` key should be ≥90% for `Lead` and `Schedule` [H20].
- The rule being tested: the same `event_name` and the same ID, within 48 hours. Meta favours the browser event if both arrive within 5 minutes [D1][D9].

## 4. Stage-feed checks (staging)

Use signed requests. A test helper can build the signature:

```bash
TS=$(date +%s); BODY='<JSON_PAYLOAD>'
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$(cat /run/secrets/da_stage_hmac)" | sed 's/^.* //')
curl -sS -X POST https://<STAGING_HOST>/api/meta/stage-events \
  -H "Content-Type: application/json" -H "X-DA-Timestamp: $TS" -H "X-DA-Signature: sha256=$SIG" \
  --data "$BODY"
```

| ID | Test | Pass condition |
|---|---|---|
| T10 | Missing or wrong signature; a timestamp 10 minutes old | 401, and nothing stored |
| T11 | Send the same `(source, event_id)` twice | The second reply is `duplicate`; exactly one ledger row and one outbox row |
| T12 | `consultation_scheduled` for a lead with **no** website Schedule | Server-only `Schedule` with `action_source=phone_call` and no IP or user agent. For a lead that *has* one: not sent again. |
| T13 | `consultation_completed` with `channel=phone`, then another lead with `channel=video` | `ConsultationCompleted` with `phone_call` and with `other`; Event ID `da:consult:<LEAD_ID>`; no IP or user agent |
| T14 | `purchase` with `payment_channel=phone`, then `payment_link`, then `website` | `Purchase` with `phone_call` and with `other`, each with `value`, `currency=USD`, `order_id` and `content_name=trained_for_life`. The `website` one is skipped (the site checkout sends it). |
| T15 | `purchase` for a lead with no consultation event | `ConsultationCompleted` is emitted first (the consultation time, or the purchase time minus 1 s), then `Purchase` |
| T16 | `refund`, `no_show`, `cancelled`, `disqualified` | `stored_not_sent`; no outbox row |
| T17 | `occurred_at` 8 days ago; also an outbox row whose `event_time` is 8 days old | `too_old`; never sent. The job marks it `dead` without making an HTTP call [D1]. |
| T18 | CSV import `--dry-run`, then a real run, with the same rows as T12–T16 | Identical results to the webhook. Unmatched rows are listed. |

## 5. Privacy, security and failure checks

| ID | Test | Pass condition |
|---|---|---|
| T19 | Request pages with the header `Sec-GPC: 1`, and complete a lead | **optout_suppress:** no pixel, no `_fbc`/`_fbp` set, outbox rows `skipped`. **optout_ldu:** `fbq('dataProcessingOptions',['LDU'],0,0)` runs before `init`, and CAPI rows carry `data_processing_options=["LDU"]` with country and state `0` [D24]. |
| T20 | Submit the PSD consultation form; send a `purchase` with a PSD product line | Nothing is sent to Meta |
| T21 | Unit test: build an event whose `custom_data` contains "service dog" | The guard throws. It is logged without PII. The quiz or booking flow still completes. |
| T22 | Run the log scan below after the staging tests | No matches. The token exists only in the secret store. |
| T23 | Staging with an invalid token | Graph code `190` produces an **alert** and rows go `failed`. Restore the token, run `meta:capi:flush`, and the rows are `sent`. |
| T24 | Force an HTTP 500 (fake) | The job retries on the backoff schedule. After the last try the row is `dead` and an alert fires. |

The T22 log scan: tokens, token-like strings and email addresses must never appear in the Meta log channel.

```bash
grep -RinE 'access_token|EAA[A-Za-z0-9]{20}|@' storage/logs/meta_capi* && echo "FAIL" || echo "PASS"
```

## 6. Meta-side dataset checks (before launch)

- `GET /v25.0/481578137302100?fields=is_consolidated_container` returns `true`. That allows offline events on this dataset [D12].
- Events Manager → dataset → Settings shows **no restricted data source category** [H27]. Diagnostics shows **no prohibited-information notices** [H25]. Custom events are reviewed if Meta asks [H26].
- Manage integrations: **no** Meta-enabled CAPI, Gateway or partner CAPI integration on this dataset [H23].
- Custom conversions **exist before paid traffic starts**, because offline custom conversions only count data received after they are created [H14]. Create each one once its source event has reached `481578137302100` (the §8 test lead sends `Schedule` and `ConsultationCompleted`). The names are neutral:
  - "Schedule – in service area";
  - "Consultation completed";
  - optional "TFL purchase", created after the first real sale.

## 7. Quality targets after 48 hours of production traffic

The site already has organic and Google traffic, so this can finish before any Meta spend.

| Metric | Where | Target |
|---|---|---|
| EMQ `Lead` | Events Manager → event → View details → Event matching [H19][H20] | **≥ 8.0** target; **7.5** is the minimum to launch. Meta publishes no pass mark; EMQ covers website events only [H19][D8]. |
| EMQ `Schedule` | same | ≥ 8.0 target; minimum 7.5 |
| EMQ `PartialLead` / `QuizStart` | same | ≥ 7.0 / informational |
| Match-key coverage for offline events (no EMQ exists) | outbox query, §9 C | em ≥ 99%, ph ≥ 95%, external_id 100%, fbc on ≥ 90% of Meta-click leads |
| Event coverage (CAPI vs pixel) | Event coverage tab [H20] | Meta: aim for ≥ 75%. Ours: ≥ 90% for Lead and Schedule, and server count ≥ browser count |
| Dedup-key overlap | Event deduplication tab [H20] | ≥ 90% for Lead and Schedule |
| Freshness | Data freshness tab [H20]; outbox `sent_at − event_time` | Web p95 < 5 min; stage p95 < 1 h (webhook) or < 24 h (CSV); 0 rows `too_old` |
| Diagnostics | Events Manager Diagnostics | 0 open errors |

To pull these programmatically (read-only; the token needs the dataset permission [D18]):

```bash
curl -sG "https://graph.facebook.com/v25.0/dataset_quality" \
  --data-urlencode "dataset_id=481578137302100" \
  --data-urlencode "fields=web{event_name,event_match_quality,event_coverage,dedup_key_feedback,data_freshness}" \
  --data-urlencode "access_token@/run/secrets/meta_read_token"
```

The field names come from Meta's reference [D18], but the subfields are not listed there. If the default response is thin, expand the nodes using the reference. The documented EMQ example uses `web{event_match_quality{match_key_feedback{identifier,potential_aly_acr_increase{percentage,description}}},event_name}` [D18].

## 8. End-to-end labelled test lead (production; operator-approved, one lead)

**Before you start:**
- Every staging test (T1–T24) passes.
- Matthew has flagged the test identity in the CRM as "TEST – do not work".
- `META_TEST_LEAD_EMAIL_REGEX` and `META_TEST_EVENT_CODE_PROD` are set.
- Test Events is open on `481578137302100`.

**Steps:**
1. In Test Events → website events → **Open website**, enter:
   `https://dogacademy.org/in-person-dog-training/private-training?utm_source=facebook&utm_medium=paid_social&utm_campaign=capi_test&campaign_id=0&adset_id=0&ad_id=0&fbclid=CAPITEST<YYYYMMDD>`

   Expect a `PageView` (the program page now has the pixel) and `_fbc` containing the test fbclid. The fbclid is synthetic, so this only checks the format.
2. Click **Book Training** and complete the quiz with the test identity. Expect `PartialLead` if built, then `Lead` as browser + server with one deduplicated. Write down the `event_id` and `fbtrace_id`.
3. On the scheduler, book a slot. Expect `Schedule` as browser + server, deduplicated.
4. Matthew's team marks the consultation **completed**, or the CRM sandbox posts `consultation_completed` with `"test": true`. Expect a server `ConsultationCompleted` with `phone_call` or `other`.
5. **Do not test `Purchase` in production.** It would put a fake purchase into the dataset. `Purchase` is proven on staging (T14/T15) and then on the first real sale through §9.
6. Clean up:
   - cancel the booking;
   - close the test lead in the CRM;
   - add its `event_id`s to the reconciliation exclusions (test events stay in the dataset);
   - file the evidence.

**Pass:** steps 1–4 all show the expected events with correct payloads, deduplication works, and no alert fired.

## 9. Seven-day CRM ↔ app ↔ Meta reconciliation

Run it daily. The **gate** is the first full 7 days after launch (G10).

The SQL is MySQL 8. Adapt `leads` and `completed_at` to DA's schema.

```sql
SET time_zone = '+00:00';
SET @to   = UTC_DATE();                 -- today 00:00 UTC, exclusive
SET @from = @to - INTERVAL 7 DAY;

-- A. What the app queued and sent, per event and action source
SELECT e.event_name, e.action_source,
       COUNT(*)                                          AS outbox_rows,
       SUM(e.status = 'sent')                            AS sent,
       SUM(e.status = 'sent' AND e.events_received = 1)  AS acked,
       SUM(e.status IN ('failed', 'dead'))               AS failed_or_dead,
       SUM(e.status = 'skipped')                         AS skipped_by_rule,
       ROUND(AVG(CASE WHEN e.status = 'sent'
                      THEN TIMESTAMPDIFF(SECOND, FROM_UNIXTIME(e.event_time), e.sent_at) END) / 60, 1) AS avg_delay_min
FROM meta_capi_events e
WHERE e.event_time >= UNIX_TIMESTAMP(@from) AND e.event_time < UNIX_TIMESTAMP(@to)
  AND e.test_event_code IS NULL
GROUP BY e.event_name, e.action_source
ORDER BY e.event_name, e.action_source;

-- B. Completed leads without a sent Lead event (every row must be explained by opt-out or exclusion)
SELECT l.id AS lead_id, i.meta_opt_out, i.meta_excluded, e.status
FROM leads l
LEFT JOIN lead_meta_identities i ON i.lead_id = l.id
LEFT JOIN meta_capi_events e     ON e.event_id = CONCAT('da:lead:', l.id)
WHERE l.completed_at >= @from AND l.completed_at < @to
  AND (e.id IS NULL OR e.status NOT IN ('sent', 'skipped'));

-- C. Stage-feed ledger outcomes (compare "received" with the CRM/dialer/billing export for the same UTC window)
SELECT s.stage,
       COUNT(*)                            AS received,
       SUM(s.result = 'queued')            AS queued_for_meta,
       SUM(s.result = 'already_sent')      AS already_sent_by_site,
       SUM(s.result = 'ignored_negative')  AS negative_not_sent,
       SUM(s.result = 'too_old')           AS too_old,
       SUM(s.result = 'unmatched')         AS unmatched,
       SUM(s.result = 'excluded')          AS excluded
FROM lead_stage_events s
WHERE s.occurred_at >= @from AND s.occurred_at < @to AND s.is_test = 0
GROUP BY s.stage;

-- D. Stage coverage: queued stage changes that Meta acknowledged
SELECT s.stage,
       SUM(s.result = 'queued') AS queued,
       SUM(s.result = 'queued' AND e.status = 'sent' AND e.events_received = 1) AS acked,
       ROUND(100 * SUM(s.result = 'queued' AND e.status = 'sent' AND e.events_received = 1)
             / NULLIF(SUM(s.result = 'queued'), 0), 1) AS coverage_pct
FROM lead_stage_events s
LEFT JOIN meta_capi_events e ON e.event_id = CASE s.stage
        WHEN 'consultation_scheduled' THEN CONCAT('da:schedule:', s.lead_id)
        WHEN 'consultation_completed' THEN CONCAT('da:consult:',  s.lead_id)
        WHEN 'purchase'               THEN CONCAT('da:purchase:', s.order_id)
     END
WHERE s.occurred_at >= @from AND s.occurred_at < @to AND s.is_test = 0
GROUP BY s.stage;

-- E. Match-key coverage for offline events (EMQ does not exist for them)
SELECT e.event_name,
       COUNT(*) AS sent,
       ROUND(100 * AVG(JSON_CONTAINS_PATH(e.payload, 'one', '$.user_data.em')), 1)          AS em_pct,
       ROUND(100 * AVG(JSON_CONTAINS_PATH(e.payload, 'one', '$.user_data.ph')), 1)          AS ph_pct,
       ROUND(100 * AVG(JSON_CONTAINS_PATH(e.payload, 'one', '$.user_data.external_id')), 1) AS external_id_pct,
       ROUND(100 * AVG(JSON_CONTAINS_PATH(e.payload, 'one', '$.user_data.fbc')), 1)         AS fbc_pct
FROM meta_capi_events e
WHERE e.status = 'sent' AND e.action_source <> 'website'
  AND e.event_time >= UNIX_TIMESTAMP(@from) AND e.event_time < UNIX_TIMESTAMP(@to)
GROUP BY e.event_name;
```

**Meta side.** Read-only. The stats endpoint returns data "up to seven days from the request time" [D19], so pull it daily and keep the results.

```bash
curl -sG "https://graph.facebook.com/v25.0/481578137302100/stats" \
  --data-urlencode "aggregation=event" \
  --data-urlencode "event_source=SERVER_ONLY" \
  --data-urlencode "start_time=<UNIX_FROM>" \
  --data-urlencode "end_time=<UNIX_TO>" \
  --data-urlencode "access_token@/run/secrets/meta_read_token"
```

**CRM side.** Matthew's export of stage counts for the same UTC window, per stage, taken from the CRM, dialer and billing systems.

| Comparison | Tolerance |
|---|---|
| CRM export vs `received` (C) | Equal, apart from explained edge-of-window timing |
| `failed_or_dead` (A) | 0 open. Each one is investigated, fixed and re-flushed within 6 days. |
| Leads without a Lead event (B) | 0 unexplained |
| Stage coverage (D) | ≥ 98% within 24 hours; 100% of eligible rows within 6 days |
| `unmatched` (C) | ≤ 1%, and each one is resolved |
| `acked` (A) vs Meta `/stats` SERVER_ONLY per event | Within ±2% (exclude test events) |
| Match keys (E) | em ≥ 99%, ph ≥ 95%, external_id 100% |

**Reporting rules:**
- Judge performance on **CRM cohorts**: cost per completed consultation and per sale, by lead week, using the stored `campaign_id`/`adset_id`/`ad_id`.
- Meta's Schedule results are capped by the 7-day window [H7].
- Never add CRM-mode `Sale Closed` to `Purchase`.

## 10. Go/no-go gate

| Gate | Check | Pass | Owner | Evidence |
|---|---|---|---|---|
| G0 | Access | The operator is admin on the Business, dataset, ad account and Page, and our connector can read dataset details and stats | Operator + DA admin | Readback |
| G1 | Dataset health | Not consolidated → fail. Any restricted category or prohibited-info notice → fail. More than one server sender → fail. (§6) | AdMaker | Screenshots + API output |
| G2 | Domain | `dogacademy.org` verified in DA's Business | DA admin | Screenshot |
| G3 | Pixel coverage | T1–T5 pass | AdMaker | Test Events + curl |
| G4 | Web events | T6–T9 pass on staging and production; deduplication confirmed | AdMaker | Test Events screenshots + outbox IDs |
| G5 | Stage feed | T10–T18 pass on staging. The production webhook has received real stage changes for 3 business days with 0 `dead`. | DA dev + AdMaker | Outbox report |
| G6 | Privacy, security, failure | T19–T24 pass; the secret is in the secret store; alerts have been tested | DA dev | Log grep + alert screenshot |
| G7 | Ads Manager | Gate 0-B: `Schedule` can be selected on dataset `481578137302100`; Gate 0-A recorded; custom conversions "Schedule – in service area" and "Consultation completed" exist (§6) | AdMaker | Screenshots |
| G8 | Quality | After 48 hours: EMQ Lead and Schedule ≥ 7.5, coverage ≥ 75% (target 90%), dedup ≥ 90%, freshness in target | AdMaker | Events Manager / Dataset Quality API |
| G9 | Test lead | §8 passes | AdMaker + Matthew | QA log |
| **Launch** | **G0–G9 all green** | Operator approves | Operator | — |
| G10 | Day-7 reconciliation | Every §9 tolerance met **before any budget increase** | AdMaker + DA dev | Reconciliation output |

## 11. Ongoing cadence

| When | Check |
|---|---|
| Real time | Alerts: `dead`, 190/200/10, failure rate > 2% per hour, no `Lead` sends for 3 business hours |
| Daily | `meta:capi:reconcile`; pull `/stats`; confirm no row is close to the 7-day limit |
| Weekly | Dataset Quality API: EMQ, coverage, dedup, freshness. §9 in full. Custom conversions still Active. CRM cohort report (cost per consultation and per sale). |
| Monthly | Lag distribution (share of consultations and sales within 7 and 28 days of the lead); stage rates; rerun Gate 0-A (Meta's qualified-leads support may change); rotate or review the token holder list |