Assist_Design/docs/ORDER_CDC_SETUP.md

Order CDC Setup Guide

Overview

This guide explains how to use Change Data Capture (CDC) for Order cache invalidation while keeping the Platform Event-based fulfillment flow intact.

---

🔑 Key Concept: Dual Approach

Your order system uses TWO separate mechanisms for different purposes:

Mechanism	Purpose	Channel Type	Trigger
Platform Events	Order provisioning/fulfillment	/event/Order_Fulfilment_Requested__e	Salesforce Flow when Status = Approved
CDC	Order cache invalidation	/data/OrderChangeEvent	ANY Order field change in Salesforce

---

🎯 Problem Statement

Challenge: Orders have both:

1. Customer-facing fields (Status, TotalAmount, BillingAddress) - changes should invalidate cache
2. Internal system fields (Activation_Status__c, WHMCS_Order_ID__c) - updated by fulfillment, should NOT invalidate cache

Why it matters:

• Fulfillment process updates internal fields every time it runs
• Without filtering, CDC would trigger unnecessary cache invalidation
• This would cause cache thrashing and wasted API calls

---

🧠 Smart Filtering Strategy

The OrderCdcSubscriber implements intelligent field filtering to solve this problem:

Customer-Facing Fields (INVALIDATE cache)

Changes to these fields invalidate cache because customers need to see updates:

// Order fields
- Status                    // Draft → Pending Review → Completed
- TotalAmount              // Order total
- EffectiveDate            // Order date
- BillingStreet            // Billing address
- BillingCity
- BillingState
- BillingPostalCode
- BillingCountry
- Type                     // Internet, SIM, VPN
- Activation_Type__c       // Immediate, Scheduled
- Installation_Type__c
- Access_Mode__c
- Hikari_Denwa__c
- VPN_Region__c
- SIM_Type__c
- EID__c
- Address_Changed__c

// OrderItem fields
- Quantity
- UnitPrice
- Description
- Product2Id
- Billing_Cycle__c

Internal System Fields (IGNORE - don't invalidate cache)

Changes to these fields are ignored because they're updated by the fulfillment process:

// Order fields
private readonly INTERNAL_FIELDS = new Set([
  "Activation_Status__c",         // Activating → Activated/Failed
  "WHMCS_Order_ID__c",            // Set during fulfillment
  "Activation_Error_Code__c",     // Error tracking
  "Activation_Error_Message__c",  // Error messages
  "Activation_Last_Attempt_At__c",// Timestamp
  "ActivatedDate",                // Activation timestamp
]);

// OrderItem fields
private readonly INTERNAL_ORDER_ITEM_FIELDS = new Set([
  "WHMCS_Service_ID__c",          // Set during fulfillment
]);

---

🔄 Order Lifecycle & Cache Invalidation

Scenario 1: Order Creation (Portal → Salesforce)

1. Customer creates order in Portal
2. Portal creates Order in Salesforce (Status: "Pending Review")
3. CDC fires → Cache invalidation NOT needed (order just created, not in cache)
4. Customer sees "Pending Review" status

Cache invalidation: ❌ Not needed (new order)

---

Scenario 2: Order Approval (Salesforce → Fulfillment)

1. Admin approves Order in Salesforce (Status: "Pending Review" → "Approved")
2. CDC fires → CUSTOMER-FACING field changed (Status)
3. Cache invalidated ✅
4. Flow publishes Order_Fulfilment_Requested__e Platform Event
5. Portal subscriber enqueues provisioning job
6. Fulfillment process updates:
   - Activation_Status__c: "Activating"
   - CDC fires → INTERNAL field changed
   - Cache invalidation SKIPPED ❌ (internal field only)
7. Fulfillment completes, updates:
   - Status: "Completed"
   - Activation_Status__c: "Activated"
   - WHMCS_Order_ID__c: "12345"
   - CDC fires → CUSTOMER-FACING field changed (Status)
   - Cache invalidated ✅
8. Customer polls for updates, sees "Completed" status

Cache invalidations:

• Step 2: ✅ YES (Status changed - customer-facing)
• Step 6: ❌ NO (Only internal fields changed)
• Step 7: ✅ YES (Status changed - customer-facing)

Why this is smart:

• Step 6 doesn't invalidate cache even though CDC fired
• Prevents unnecessary cache invalidation during fulfillment
• Cache is only invalidated when customer-visible data changes

---

Scenario 3: Admin Updates Order Details (Salesforce UI)

1. Admin updates BillingAddress in Salesforce UI
2. CDC fires → CUSTOMER-FACING field changed
3. Cache invalidated ✅
4. Customer sees updated billing address on next page load

Cache invalidation: ✅ YES (customer-facing field)

---

Scenario 4: Fulfillment Retry After Failure

1. Order in "Failed" state (Activation_Status__c: "Failed")
2. Customer adds payment method
3. Admin clicks "Retry Fulfillment" → Activation_Status__c: "Activating"
4. CDC fires → INTERNAL field changed
5. Cache invalidation SKIPPED ❌
6. Platform Event triggers fulfillment
7. Fulfillment completes → Status: "Completed"
8. CDC fires → CUSTOMER-FACING field changed
9. Cache invalidated ✅

Cache invalidations:

• Step 4: ❌ NO (internal field)
• Step 8: ✅ YES (customer-facing field)

---

🔧 Implementation Details

How Field Filtering Works

private async handleOrderEvent(
  channel: string,
  subscription: { topicName?: string },
  callbackType: string,
  data: unknown
): Promise<void> {
  const payload = this.extractPayload(data);
  const changedFields = this.extractChangedFields(payload);
  
  // Filter: Only invalidate if customer-facing fields changed
  const hasCustomerFacingChange = this.hasCustomerFacingChanges(changedFields);
  
  if (!hasCustomerFacingChange) {
    this.logger.debug("Order CDC event contains only internal field changes; skipping", {
      orderId,
      changedFields: Array.from(changedFields),
    });
    return; // ❌ Don't invalidate cache
  }
  
  // ✅ Invalidate cache
  await this.ordersCache.invalidateOrder(orderId);
  await this.ordersCache.invalidateAccountOrders(accountId);
}

private hasCustomerFacingChanges(changedFields: Set<string>): boolean {
  if (changedFields.size === 0) {
    return true; // Safe default: assume customer-facing if no info
  }
  
  // Remove internal fields
  const customerFacingChanges = Array.from(changedFields).filter(
    (field) => !this.INTERNAL_FIELDS.has(field)
  );
  
  return customerFacingChanges.length > 0;
}

CDC Payload Structure

Salesforce CDC events include information about which fields changed:

{
  "payload": {
    "Id": "801xxx",
    "Status": "Completed",
    "Activation_Status__c": "Activated",
    "changeType": "UPDATE",
    "changedFields": [
      "Status",
      "Activation_Status__c"
    ],
    "changeOrigin": {
      "changedFields": ["Status", "Activation_Status__c"]
    }
  }
}

The subscriber extracts changedFields and determines if ANY customer-facing field was changed.

---

📊 Benefits

Before (No Filtering)

Fulfillment Process:
1. Update Activation_Status__c = "Activating"
   → CDC fires → Cache invalidated
2. Update WHMCS_Order_ID__c = "12345"
   → CDC fires → Cache invalidated
3. Update Activation_Status__c = "Activated"
   → CDC fires → Cache invalidated
4. Update Status = "Completed"
   → CDC fires → Cache invalidated

Result: 4 cache invalidations, 4 Salesforce API calls to refetch order

After (With Smart Filtering)

Fulfillment Process:
1. Update Activation_Status__c = "Activating"
   → CDC fires → Skipped (internal field)
2. Update WHMCS_Order_ID__c = "12345"
   → CDC fires → Skipped (internal field)
3. Update Activation_Status__c = "Activated", Status = "Completed"
   → CDC fires → Cache invalidated (Status is customer-facing)

Result: 1 cache invalidation, 1 Salesforce API call to refetch order

Savings: 75% fewer cache invalidations during fulfillment!

---

🔧 Salesforce Setup

Enable CDC on Order Objects

1. Go to Setup → Integrations → Change Data Capture
2. Select objects:
   • ✅ Order
   • ✅ OrderItem
3. Click Save

That's it! CDC is built into Salesforce - no custom Platform Events needed.

Permissions

Ensure your Salesforce integration user has:

• View Change Data Capture Events permission
• Read access to Order and OrderItem objects

---

✅ Verification Steps

1. Check Logs on Application Start

tail -f logs/app.log | grep -i "order cdc\|OrderChangeEvent"

Expected output:

Subscribed to Order CDC channel {"orderChannel":"/data/OrderChangeEvent"}
Subscribed to OrderItem CDC channel {"orderItemChannel":"/data/OrderItemChangeEvent"}

2. Test Fulfillment (Internal Field Changes)

1. Trigger order fulfillment
2. Check logs for CDC events:

   Order CDC event contains only internal field changes; skipping cache invalidation
   {"orderId":"801xxx","changedFields":["Activation_Status__c"]}
   

3. Verify cache NOT invalidated (logs show "skipping")

3. Test Admin Update (Customer-Facing Field Changes)

1. In Salesforce, update Order Status from "Pending Review" to "Cancelled"
2. Check logs for CDC event:

   Order CDC event received with customer-facing changes, invalidating cache
   {"orderId":"801xxx","changedFields":["Status"]}
   

3. Verify cache invalidated (logs show "invalidating cache")

4. Monitor Cache Metrics

curl http://localhost:4000/health/orders

Response:

{
  "status": "ok",
  "cdc": {
    "orderChannel": "/data/OrderChangeEvent",
    "orderItemChannel": "/data/OrderItemChangeEvent",
    "status": "connected"
  },
  "cache": {
    "ttl": {
      "summarySeconds": null,
      "detailSeconds": null
    },
    "metrics": {
      "invalidations": 45,
      "skippedInternal": 120
    }
  }
}

Key indicators:

• status: "connected" = CDC is active
• invalidations: 45 = Cache invalidated 45 times for customer-facing changes
• skippedInternal: 120 = Skipped 120 internal field changes (smart filtering working!)

---

🚨 Troubleshooting

Problem: Cache thrashing during fulfillment

Symptom: Logs show cache invalidation every time fulfillment updates internal fields

Solution: Check INTERNAL_FIELDS set includes all system fields:

private readonly INTERNAL_FIELDS = new Set([
  "Activation_Status__c",
  "WHMCS_Order_ID__c",
  "Activation_Error_Code__c",
  "Activation_Error_Message__c",
  "Activation_Last_Attempt_At__c",
  "ActivatedDate",
]);

Problem: Cache not invalidating when admin updates order

Symptom: Admin changes order in Salesforce, but customer doesn't see updates

Check:

1. CDC is enabled for Order object in Salesforce
2. Logs show CDC event received
3. Changed field is NOT in INTERNAL_FIELDS set

Problem: Too aggressive invalidation

Symptom: Cache is invalidated even for non-customer-facing fields

Solution: Add field to INTERNAL_FIELDS set if it's updated by system processes.

---

📝 Adding New Internal Fields

If you add new system fields that are updated by fulfillment or background processes:

// In order-cdc.subscriber.ts
private readonly INTERNAL_FIELDS = new Set([
  "Activation_Status__c",
  "WHMCS_Order_ID__c",
  "Activation_Error_Code__c",
  "Activation_Error_Message__c",
  "Activation_Last_Attempt_At__c",
  "ActivatedDate",
  
  // ✅ Add your new internal fields here
  "Your_New_System_Field__c",
]);

Rule of thumb:

• Field updated by system/fulfillment → Add to INTERNAL_FIELDS
• Field updated by admins/users → DON'T add (customer-facing)

---

🎯 Summary

Your Order CDC setup provides:

✅ Smart filtering - Only invalidates cache for customer-facing field changes
✅ Fulfillment-aware - Doesn't interfere with Platform Event-based provisioning
✅ Cache efficiency - 75% fewer invalidations during fulfillment
✅ Real-time updates - Admin changes reflected within seconds
✅ No manual invalidation - System handles it automatically

Next Steps:

1. Enable CDC for Order and OrderItem in Salesforce
2. Restart your application
3. Monitor logs for successful CDC subscriptions
4. Test by updating an order in Salesforce and verifying cache invalidation

---

📚 Related Documentation

• CDC_SETUP_VERIFICATION.md - Catalog CDC setup
• SALESFORCE-ORDER-COMMUNICATION.md - Platform Events for fulfillment
• ORDER-FULFILLMENT-COMPLETE-GUIDE.md - Fulfillment workflow