# Provisioning Runbook (Salesforce → Portal → WHMCS)

This runbook helps operators diagnose issues in the order fulfillment path.

## Endpoints & Paths

- Salesforce Quick Action: POST `.../orders/{sfOrderId}/fulfill`
- Backend health: GET `/health`

## Required Env (Backend)

- `SF_LOGIN_URL`, `SF_CLIENT_ID`, `SF_USERNAME`
- `SF_PRIVATE_KEY_PATH` (prod: `/app/secrets/sf-private.key`)
- `SF_WEBHOOK_SECRET`
- `PORTAL_PRICEBOOK_ID`
- Optional: `WEBHOOK_TIMESTAMP_TOLERANCE` (ms)

## Common Symptoms and Fixes

- 401 Invalid signature
  - Verify `SF_WEBHOOK_SECRET` matches Salesforce Named/External Credential
  - Confirm Apex computes HMAC-SHA256 over raw JSON body
  - Clocks skewed: adjust `WEBHOOK_TIMESTAMP_TOLERANCE` or fix server time

- 401 Nonce already used
  - Replay blocked by Redis-backed nonce store. Ensure the Quick Action does not retry with identical nonce.
  - If Redis is down, the system falls back to in-memory (dev only); restore Redis for cluster safety.

- 400 Missing fields (orderId/timestamp/nonce)
  - Inspect Apex payload construction and headers
  - Ensure `Idempotency-Key` header is unique per attempt

- 409 Payment method missing
  - Customer has no WHMCS payment method
  - Ask customer to add a payment method; retry fulfill

- WHMCS Add/Accept errors
  - Check product mappings: `Product2.WH_Product_ID__c` and `Billing_Cycle__c`
  - Backend logs show the item mapping report; fix missing mappings

- Salesforce status not updated
  - Backend updates `Provisioning_Status__c` and `WHMCS_Order_ID__c` on success, `Provisioning_Error_*` on failure
  - Verify connected app JWT config and that the API user has Order update permissions

## Verification Steps

1. In SF, create an Order with OrderItems
2. Trigger Quick Action; note `Idempotency-Key`
3. Check `/health`: database connected, environment correct
4. Tail logs; confirm steps: Activating → WHMCS add → WHMCS accept → Provisioned
5. Verify SF fields updated and WHMCS order/service IDs exist

## Logging Cheatsheet

- "Salesforce order fulfillment request received" — controller entry
- "Starting fulfillment orchestration" — orchestrator start
- Step logs: `validation`, `sf_status_update`, `order_details`, `mapping`, `whmcs_create`, `whmcs_accept`, `sf_success_update`
- On error: orchestrator updates SF with `Provisioning_Status__c='Failed'` and error code

## Security Notes

- HMAC and headers
  - All inbound calls must include an HMAC signature.
  - Salesforce must include `X-SF-Timestamp` and `X-SF-Nonce` headers.
  - WHMCS timestamp/nonce are optional (validated if present).

- Env variables (backend)
  - `SF_WEBHOOK_SECRET` (required)
  - `WHMCS_WEBHOOK_SECRET` (required if WHMCS webhooks are enabled)
  - `WEBHOOK_TIMESTAMP_TOLERANCE` (ms; default 300000)
  - `SF_WEBHOOK_IP_ALLOWLIST` (CSV of IP/CIDR; optional)
  - `WHMCS_WEBHOOK_IP_ALLOWLIST` (CSV of IP/CIDR; optional)

- Replay protection
  - Redis-backed nonce store blocks replays (Salesforce required; WHMCS optional).
  - If Redis is down, a local in-memory fallback is used (dev only). Restore Redis in prod.

- Health endpoint
  - `/health` includes `integrations.redis` probe to confirm nonce store availability.