Assist_Design/docs/operations/external-dependencies.md

External Dependencies Runbook

This document covers health checking, monitoring, and troubleshooting for external systems integrated with the Customer Portal.

---

System Overview

System	Purpose	Integration	Health Check
Salesforce	CRM, Orders, Catalog	REST API + Platform Events	JWT auth test
WHMCS	Billing, Payments, Subscriptions	REST API	API action test
Freebit	SIM Management	REST API	OEM auth test
SFTP (fs.mvno.net)	Call/SMS Records	SFTP	Connection test
Redis	Cache, Sessions, Queues	Direct connection	PING command
PostgreSQL	User data, Mappings	Direct connection	Query test

---

Salesforce

Configuration

Variable	Description
SF_LOGIN_URL	Login URL (login.salesforce.com or test.salesforce.com)
SF_CLIENT_ID	Connected App Consumer Key
SF_USERNAME	Integration user username
SF_PRIVATE_KEY_PATH	Path to JWT private key
SF_EVENTS_ENABLED	Enable Platform Event subscription
SF_PROVISION_EVENT_CHANNEL	Platform Event channel for provisioning
PORTAL_PRICEBOOK_ID	Salesforce Pricebook ID for catalog

Health Check

# Check Salesforce connectivity via BFF health endpoint
curl http://localhost:4000/health | jq '.'

# Test JWT authentication manually
# The BFF authenticates automatically; check logs for auth errors
grep "Salesforce" /var/log/bff/combined.log | tail -20

Common Issues

JWT Authentication Failure

• Verify private key file exists and is readable
• Check Connected App settings in Salesforce
• Ensure integration user is pre-authorized for Connected App
• Verify SF_USERNAME matches the user assigned to Connected App

Platform Events Not Receiving

• Check SF_EVENTS_ENABLED=true
• Verify Platform Event permissions for integration user
• Check Redis for replay ID: redis-cli GET "sf:pe:replay:/event/OrderProvisionRequested__e"
• Set SF_EVENTS_REPLAY=ALL temporarily to catch up on missed events

API Limits

• Salesforce has daily API call limits
• Monitor usage in Salesforce Setup > API Usage
• Consider caching frequently accessed data

Expected Response Times

Operation	Expected	Alert Threshold
Query	<500ms	>2s
Update	<1s	>3s
Platform Event	Real-time	>5s delay

---

WHMCS

Configuration

Variable	Description
WHMCS_API_URL	WHMCS API endpoint URL
WHMCS_API_IDENTIFIER	API credentials identifier
WHMCS_API_SECRET	API credentials secret
WHMCS_CUSTOMER_NUMBER_FIELD_ID	Custom field ID for Customer Number

Health Check

# Test WHMCS API directly
curl -X POST "$WHMCS_API_URL" \
  -d "identifier=$WHMCS_API_IDENTIFIER" \
  -d "secret=$WHMCS_API_SECRET" \
  -d "action=GetClients" \
  -d "responsetype=json" \
  -d "limitnum=1"

# Should return: {"result":"success","totalresults":...}

Common Issues

Authentication Failure

• Verify API credentials in WHMCS Admin > Setup > Staff Management > API Credentials
• Check IP whitelist settings (if configured)
• Ensure API credentials have required permissions

Rate Limiting

• WHMCS may rate limit excessive requests
• Check for 429 responses in logs
• Implement request queuing if needed

Field Mapping Issues

• Payment method fields may use different names between WHMCS versions
• Check WHMCS Troubleshooting for field mapping

Expected Response Times

Operation	Expected	Alert Threshold
GetInvoices	<500ms	>2s
AddOrder	<1s	>3s
AcceptOrder	<1s	>3s
SSO Token	<500ms	>2s

---

Freebit

Configuration

Variable	Description
FREEBIT_BASE_URL	Freebit API base URL
FREEBIT_OEM_ID	OEM identifier
FREEBIT_OEM_KEY	OEM authentication key
FREEBIT_TIMEOUT	Request timeout (ms)

Health Check

# Check Freebit OEM authentication
# The BFF handles auth automatically; check logs for auth errors
grep "Freebit" /var/log/bff/combined.log | tail -20

# Check for auth token in cache
redis-cli GET "freebit:auth:token"

Common Issues

OEM Authentication Failure

• Verify FREEBIT_OEM_ID and FREEBIT_OEM_KEY
• Check Freebit API endpoint accessibility
• Auth tokens are cached; clear cache if credentials changed

SIM Operations Failing

• Verify SIM account identifier (phone number) format
• Check 30-minute operation gap requirements
• See Freebit SIM Management for operation constraints

Network Type Changes Delayed

• Network type changes are queued with 30-minute delay
• Check BullMQ queue for pending jobs

Expected Response Times

Operation	Expected	Alert Threshold
Auth (cached)	<100ms	>500ms
GetDetail	<1s	>3s
Plan Change	<2s	>5s
Top-up	<2s	>5s

---

SFTP (fs.mvno.net)

Configuration

Variable	Description
SFTP_HOST	SFTP server hostname
SFTP_PORT	SFTP port (default: 22)
SFTP_USERNAME	SFTP username
SFTP_PRIVATE_KEY_PATH	Path to SSH private key

Health Check

# Test SFTP connectivity
sftp -i $SFTP_PRIVATE_KEY_PATH $SFTP_USERNAME@$SFTP_HOST << EOF
ls
exit
EOF

Common Issues

Connection Refused

• Verify SFTP server is accessible
• Check firewall rules
• Verify SSH key fingerprint

Authentication Failure

• Verify SSH private key is correct
• Check key permissions (should be 600)
• Ensure public key is authorized on SFTP server

Files Not Found

• Call/SMS records are available 2 months behind current date
• File naming: PASI_talk-detail-YYYYMM.csv, PASI_sms-detail-YYYYMM.csv

Data Availability

Record Type	Availability	File Pattern
Call Details	2 months behind	PASI_talk-detail-YYYYMM.csv
SMS Details	2 months behind	PASI_sms-detail-YYYYMM.csv

---

Credential Rotation

Salesforce JWT Key Rotation

1. Generate new key pair
2. Upload new public key to Connected App
3. Update SF_PRIVATE_KEY_PATH or SF_PRIVATE_KEY_BASE64
4. Deploy and verify authentication
5. Remove old key from Connected App after verification

WHMCS API Credentials Rotation

1. Create new API credentials in WHMCS Admin
2. Update WHMCS_API_IDENTIFIER and WHMCS_API_SECRET
3. Deploy and verify API calls work
4. Disable old API credentials

Freebit Key Rotation

1. Request new OEM key from Freebit
2. Update FREEBIT_OEM_KEY
3. Clear cached auth token: redis-cli DEL "freebit:auth:token"
4. Deploy and verify authentication

SSH Key Rotation (SFTP)

1. Generate new SSH key pair
2. Provide public key to SFTP administrator
3. Wait for key to be authorized
4. Update SFTP_PRIVATE_KEY_PATH
5. Test connectivity
6. Request old key removal from SFTP server

---

Monitoring Recommendations

Alerting Thresholds

System	Metric	Warning	Critical
Salesforce	Response time	>2s	>5s
Salesforce	Error rate	>1%	>5%
WHMCS	Response time	>2s	>5s
WHMCS	Error rate	>1%	>5%
Freebit	Response time	>3s	>10s
Redis	Response time	>100ms	>500ms
PostgreSQL	Response time	>500ms	>2s

Key Metrics to Monitor

• External API response times
• Error rates per integration
• Authentication success/failure rates
• Cache hit rates
• Queue depths (for async operations)

Health Check Schedule

System	Check Frequency	Method
Salesforce	Every 5 minutes	Query test
WHMCS	Every 5 minutes	GetClients call
Freebit	Every 15 minutes	Auth token refresh
Redis	Every 1 minute	PING
PostgreSQL	Every 1 minute	SELECT 1
SFTP	Every 1 hour	Connection test

---

Fallback Behaviors

System Down	User Impact	Fallback
Salesforce	No orders, no catalog	Show cached catalog, queue orders
WHMCS	No billing, no payments	Show cached invoices, block checkout
Freebit	No SIM management	Show cached data, disable actions
Redis	Slow performance	Direct API calls (no cache)
PostgreSQL	Portal unusable	Display maintenance message

---

Related Documents

• Incident Response
• Provisioning Runbook
• Salesforce Requirements
• WHMCS Troubleshooting
• Freebit SIM Management

---

Last Updated: December 2025