Assist_Design/docs/ORDER_CDC_SETUP.md

# Order CDC Setup Guide

## Overview

This guide explains how to use Change Data Capture (CDC) for **Order cache invalidation** while keeping the Platform Event-based **fulfillment flow** intact.

---

## 🔑 Key Concept: Dual Approach

Your order system uses **TWO separate mechanisms** for different purposes:

| Mechanism | Purpose | Channel Type | Trigger |
|-----------|---------|--------------|---------|
| **Platform Events** | Order provisioning/fulfillment | `/event/Order_Fulfilment_Requested__e` | Salesforce Flow when Status = Approved |
| **CDC** | Order cache invalidation | `/data/OrderChangeEvent` | ANY Order field change in Salesforce |

---

## 🎯 Problem Statement

**Challenge:** Orders have both:
1. **Customer-facing fields** (Status, TotalAmount, BillingAddress) - changes should invalidate cache
2. **Internal system fields** (Activation_Status__c, WHMCS_Order_ID__c) - updated by fulfillment, should NOT invalidate cache

**Why it matters:**
- Fulfillment process updates internal fields **every time it runs**
- Without filtering, CDC would trigger unnecessary cache invalidation
- This would cause cache thrashing and wasted API calls

---

## 🧠 Smart Filtering Strategy

The `OrderCdcSubscriber` implements **intelligent field filtering** to solve this problem:

### Customer-Facing Fields (INVALIDATE cache)

Changes to these fields invalidate cache because customers need to see updates:

```typescript
// Order fields
- Status                    // Draft → Pending Review → Completed
- TotalAmount              // Order total
- EffectiveDate            // Order date
- BillingStreet            // Billing address
- BillingCity
- BillingState
- BillingPostalCode
- BillingCountry
- Type                     // Internet, SIM, VPN
- Activation_Type__c       // Immediate, Scheduled
- Installation_Type__c
- Access_Mode__c
- Hikari_Denwa__c
- VPN_Region__c
- SIM_Type__c
- EID__c
- Address_Changed__c

// OrderItem fields
- Quantity
- UnitPrice
- Description
- Product2Id
- Billing_Cycle__c
```

### Internal System Fields (IGNORE - don't invalidate cache)

Changes to these fields are ignored because they're updated by the fulfillment process:

```typescript
// Order fields
private readonly INTERNAL_FIELDS = new Set([
  "Activation_Status__c",         // Activating → Activated/Failed
  "WHMCS_Order_ID__c",            // Set during fulfillment
  "Activation_Error_Code__c",     // Error tracking
  "Activation_Error_Message__c",  // Error messages
  "Activation_Last_Attempt_At__c",// Timestamp
  "ActivatedDate",                // Activation timestamp
]);

// OrderItem fields
private readonly INTERNAL_ORDER_ITEM_FIELDS = new Set([
  "WHMCS_Service_ID__c",          // Set during fulfillment
]);
```

---

## 🔄 Order Lifecycle & Cache Invalidation

### Scenario 1: Order Creation (Portal → Salesforce)

```
1. Customer creates order in Portal
2. Portal creates Order in Salesforce (Status: "Pending Review")
3. CDC fires → Cache invalidation NOT needed (order just created, not in cache)
4. Customer sees "Pending Review" status
```

**Cache invalidation:** ❌ Not needed (new order)

---

### Scenario 2: Order Approval (Salesforce → Fulfillment)

```
1. Admin approves Order in Salesforce (Status: "Pending Review" → "Approved")
2. CDC fires → CUSTOMER-FACING field changed (Status)
3. Cache invalidated ✅
4. Flow publishes Order_Fulfilment_Requested__e Platform Event
5. Portal subscriber enqueues provisioning job
6. Fulfillment process updates:
   - Activation_Status__c: "Activating"
   - CDC fires → INTERNAL field changed
   - Cache invalidation SKIPPED ❌ (internal field only)
7. Fulfillment completes, updates:
   - Status: "Completed"
   - Activation_Status__c: "Activated"
   - WHMCS_Order_ID__c: "12345"
   - CDC fires → CUSTOMER-FACING field changed (Status)
   - Cache invalidated ✅
8. Customer polls for updates, sees "Completed" status
```

**Cache invalidations:**
- Step 2: ✅ YES (Status changed - customer-facing)
- Step 6: ❌ NO (Only internal fields changed)
- Step 7: ✅ YES (Status changed - customer-facing)

**Why this is smart:**
- Step 6 doesn't invalidate cache even though CDC fired
- Prevents unnecessary cache invalidation during fulfillment
- Cache is only invalidated when customer-visible data changes

---

### Scenario 3: Admin Updates Order Details (Salesforce UI)

```
1. Admin updates BillingAddress in Salesforce UI
2. CDC fires → CUSTOMER-FACING field changed
3. Cache invalidated ✅
4. Customer sees updated billing address on next page load
```

**Cache invalidation:** ✅ YES (customer-facing field)

---

### Scenario 4: Fulfillment Retry After Failure

```
1. Order in "Failed" state (Activation_Status__c: "Failed")
2. Customer adds payment method
3. Admin clicks "Retry Fulfillment" → Activation_Status__c: "Activating"
4. CDC fires → INTERNAL field changed
5. Cache invalidation SKIPPED ❌
6. Platform Event triggers fulfillment
7. Fulfillment completes → Status: "Completed"
8. CDC fires → CUSTOMER-FACING field changed
9. Cache invalidated ✅
```

**Cache invalidations:**
- Step 4: ❌ NO (internal field)
- Step 8: ✅ YES (customer-facing field)

---

## 🔧 Implementation Details

### How Field Filtering Works

```typescript
private async handleOrderEvent(
  channel: string,
  subscription: { topicName?: string },
  callbackType: string,
  data: unknown
): Promise<void> {
  const payload = this.extractPayload(data);
  const changedFields = this.extractChangedFields(payload);
  
  // Filter: Only invalidate if customer-facing fields changed
  const hasCustomerFacingChange = this.hasCustomerFacingChanges(changedFields);
  
  if (!hasCustomerFacingChange) {
    this.logger.debug("Order CDC event contains only internal field changes; skipping", {
      orderId,
      changedFields: Array.from(changedFields),
    });
    return; // ❌ Don't invalidate cache
  }
  
  // ✅ Invalidate cache
  await this.ordersCache.invalidateOrder(orderId);
  await this.ordersCache.invalidateAccountOrders(accountId);
}

private hasCustomerFacingChanges(changedFields: Set<string>): boolean {
  if (changedFields.size === 0) {
    return true; // Safe default: assume customer-facing if no info
  }
  
  // Remove internal fields
  const customerFacingChanges = Array.from(changedFields).filter(
    (field) => !this.INTERNAL_FIELDS.has(field)
  );
  
  return customerFacingChanges.length > 0;
}
```

### CDC Payload Structure

Salesforce CDC events include information about which fields changed:

```json
{
  "payload": {
    "Id": "801xxx",
    "Status": "Completed",
    "Activation_Status__c": "Activated",
    "changeType": "UPDATE",
    "changedFields": [
      "Status",
      "Activation_Status__c"
    ],
    "changeOrigin": {
      "changedFields": ["Status", "Activation_Status__c"]
    }
  }
}
```

The subscriber extracts `changedFields` and determines if ANY customer-facing field was changed.

---

## 📊 Benefits

### Before (No Filtering)

```
Fulfillment Process:
1. Update Activation_Status__c = "Activating"
   → CDC fires → Cache invalidated
2. Update WHMCS_Order_ID__c = "12345"
   → CDC fires → Cache invalidated
3. Update Activation_Status__c = "Activated"
   → CDC fires → Cache invalidated
4. Update Status = "Completed"
   → CDC fires → Cache invalidated

Result: 4 cache invalidations, 4 Salesforce API calls to refetch order
```

### After (With Smart Filtering)

```
Fulfillment Process:
1. Update Activation_Status__c = "Activating"
   → CDC fires → Skipped (internal field)
2. Update WHMCS_Order_ID__c = "12345"
   → CDC fires → Skipped (internal field)
3. Update Activation_Status__c = "Activated", Status = "Completed"
   → CDC fires → Cache invalidated (Status is customer-facing)

Result: 1 cache invalidation, 1 Salesforce API call to refetch order
```

**Savings:** 75% fewer cache invalidations during fulfillment!

---

## 🔧 Salesforce Setup

### Enable CDC on Order Objects

1. Go to **Setup → Integrations → Change Data Capture**
2. Select objects:
   - ✅ **Order**
   - ✅ **OrderItem**
3. Click **Save**

**That's it!** CDC is built into Salesforce - no custom Platform Events needed.

### Permissions

Ensure your Salesforce integration user has:
- **View Change Data Capture Events** permission
- **Read** access to Order and OrderItem objects

---

## ✅ Verification Steps

### 1. Check Logs on Application Start

```bash
tail -f logs/app.log | grep -i "order cdc\|OrderChangeEvent"
```

**Expected output:**
```
Subscribed to Order CDC channel {"orderChannel":"/data/OrderChangeEvent"}
Subscribed to OrderItem CDC channel {"orderItemChannel":"/data/OrderItemChangeEvent"}
```

### 2. Test Fulfillment (Internal Field Changes)

1. Trigger order fulfillment
2. **Check logs for CDC events:**
   ```
   Order CDC event contains only internal field changes; skipping cache invalidation
   {"orderId":"801xxx","changedFields":["Activation_Status__c"]}
   ```
3. **Verify cache NOT invalidated** (logs show "skipping")

### 3. Test Admin Update (Customer-Facing Field Changes)

1. In Salesforce, update Order Status from "Pending Review" to "Cancelled"
2. **Check logs for CDC event:**
   ```
   Order CDC event received with customer-facing changes, invalidating cache
   {"orderId":"801xxx","changedFields":["Status"]}
   ```
3. **Verify cache invalidated** (logs show "invalidating cache")

### 4. Monitor Cache Metrics

```bash
curl http://localhost:4000/health/orders
```

**Response:**
```json
{
  "status": "ok",
  "cdc": {
    "orderChannel": "/data/OrderChangeEvent",
    "orderItemChannel": "/data/OrderItemChangeEvent",
    "status": "connected"
  },
  "cache": {
    "ttl": {
      "summarySeconds": null,
      "detailSeconds": null
    },
    "metrics": {
      "invalidations": 45,
      "skippedInternal": 120
    }
  }
}
```

**Key indicators:**
- `status: "connected"` = CDC is active
- `invalidations: 45` = Cache invalidated 45 times for customer-facing changes
- `skippedInternal: 120` = Skipped 120 internal field changes (smart filtering working!)

---

## 🚨 Troubleshooting

### Problem: Cache thrashing during fulfillment

**Symptom:** Logs show cache invalidation every time fulfillment updates internal fields

**Solution:** Check `INTERNAL_FIELDS` set includes all system fields:

```typescript
private readonly INTERNAL_FIELDS = new Set([
  "Activation_Status__c",
  "WHMCS_Order_ID__c",
  "Activation_Error_Code__c",
  "Activation_Error_Message__c",
  "Activation_Last_Attempt_At__c",
  "ActivatedDate",
]);
```

### Problem: Cache not invalidating when admin updates order

**Symptom:** Admin changes order in Salesforce, but customer doesn't see updates

**Check:**
1. CDC is enabled for Order object in Salesforce
2. Logs show CDC event received
3. Changed field is NOT in `INTERNAL_FIELDS` set

### Problem: Too aggressive invalidation

**Symptom:** Cache is invalidated even for non-customer-facing fields

**Solution:** Add field to `INTERNAL_FIELDS` set if it's updated by system processes.

---

## 📝 Adding New Internal Fields

If you add new system fields that are updated by fulfillment or background processes:

```typescript
// In order-cdc.subscriber.ts
private readonly INTERNAL_FIELDS = new Set([
  "Activation_Status__c",
  "WHMCS_Order_ID__c",
  "Activation_Error_Code__c",
  "Activation_Error_Message__c",
  "Activation_Last_Attempt_At__c",
  "ActivatedDate",
  
  // ✅ Add your new internal fields here
  "Your_New_System_Field__c",
]);
```

**Rule of thumb:**
- Field updated by **system/fulfillment** → Add to `INTERNAL_FIELDS`
- Field updated by **admins/users** → DON'T add (customer-facing)

---

## 🎯 Summary

Your Order CDC setup provides:

✅ **Smart filtering** - Only invalidates cache for customer-facing field changes  
✅ **Fulfillment-aware** - Doesn't interfere with Platform Event-based provisioning  
✅ **Cache efficiency** - 75% fewer invalidations during fulfillment  
✅ **Real-time updates** - Admin changes reflected within seconds  
✅ **No manual invalidation** - System handles it automatically  

**Next Steps:**
1. Enable CDC for Order and OrderItem in Salesforce
2. Restart your application
3. Monitor logs for successful CDC subscriptions
4. Test by updating an order in Salesforce and verifying cache invalidation

---

## 📚 Related Documentation

- [CDC_SETUP_VERIFICATION.md](./CDC_SETUP_VERIFICATION.md) - Catalog CDC setup
- [SALESFORCE-ORDER-COMMUNICATION.md](./salesforce/SALESFORCE-ORDER-COMMUNICATION.md) - Platform Events for fulfillment
- [ORDER-FULFILLMENT-COMPLETE-GUIDE.md](./orders/ORDER-FULFILLMENT-COMPLETE-GUIDE.md) - Fulfillment workflow