Assist_Design/docs/INTEGRATION-DATAFLOW.md

732 lines
23 KiB
Markdown

# Integration & Data Flow Design
**Customer Portal - External System Integration Architecture**
---
## Table of Contents
1. [Integration Overview](#integration-overview)
2. [Integration Patterns](#integration-patterns)
3. [Salesforce Integration](#salesforce-integration)
4. [WHMCS Integration](#whmcs-integration)
5. [Freebit SIM Management](#freebit-sim-management)
6. [Data Transformation](#data-transformation)
7. [Error Handling](#error-handling)
8. [Caching Strategy](#caching-strategy)
---
## Integration Overview
The BFF acts as an integration layer between the portal and external systems, following clean architecture principles with clear separation between infrastructure and domain concerns.
### Integration Architecture
```
┌────────────────────────────────────────────────────────────┐
│ BFF Integration Layer │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Salesforce │ │ WHMCS │ │ Freebit │ │
│ │ Integration │ │ Integration │ │ Integration │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ ┌──────▼──────────────────▼──────────────────▼───────┐ │
│ │ Domain Mappers │ │
│ │ (Transform raw data → domain types) │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Application Services (Orchestrators) │ │
│ │ (Business workflows, multi-system coordination) │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
### Core Principle: "Map Once, Use Everywhere"
**Domain mappers are the ONLY place that transforms raw provider data to domain types.**
```
External API → Integration Service → Domain Mapper → Domain Type → Use Everywhere
SINGLE transformation!
```
---
## Integration Patterns
### Integration Service Structure
Each external system integration follows a consistent pattern:
```
apps/bff/src/integrations/{provider}/
├── services/
│ ├── {provider}-connection.service.ts # HTTP client, authentication
│ ├── {provider}-{entity}.service.ts # Entity-specific operations
│ └── {provider}-orchestrator.service.ts # Coordinates multiple operations
├── utils/
│ └── {entity}-query-builder.ts # Query construction (SOQL, etc.)
└── {provider}.module.ts # NestJS module
```
### Integration Service Responsibilities
**DO:**
1. ✅ Build queries (SOQL, API parameters)
2. ✅ Execute API calls (HTTP, authentication)
3. ✅ Use domain mappers to transform responses
4. ✅ Return domain types
5. ✅ Handle errors gracefully
**DON'T:**
6. ❌ Add additional mapping beyond domain mappers
7. ❌ Include business logic
8. ❌ Expose raw provider types
9. ❌ Create wrapper services around domain mappers
### Example Integration Service
```typescript
@Injectable()
export class SalesforceOrderService {
constructor(private readonly sf: SalesforceConnection) {}
async getOrderById(orderId: string): Promise<OrderDetails | null> {
// 1. Build query (infrastructure concern)
const fields = buildOrderSelectFields(["Account.Name"]).join(", ");
const soql = `
SELECT ${fields}
FROM Order
WHERE Id = '${orderId}'
LIMIT 1
`;
// 2. Execute query
const result = await this.sf.query(soql);
const order = result.records?.[0];
if (!order) return null;
// 3. Use domain mapper (SINGLE transformation!)
return OrderProviders.Salesforce.transformSalesforceOrderDetails(order, []);
}
}
```
---
## Salesforce Integration
### Connection Architecture
```
┌──────────────────────────────────────────────────────────┐
│ Salesforce Integration Services │
│ │
│ ┌─────────────────┐ ┌────────────────────────┐ │
│ │ REST API │ │ Platform Events │ │
│ │ (JSForce) │ │ (Pub/Sub gRPC) │ │
│ └────────┬────────┘ └──────────┬─────────────┘ │
│ │ │ │
│ ┌────────▼────────────────────────────▼─────────────┐ │
│ │ SalesforceConnection Service │ │
│ │ • OAuth2 JWT authentication │ │
│ │ • Connection pooling │ │
│ │ • Retry logic │ │
│ └────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘
```
### REST API Integration
**Authentication**: OAuth 2.0 JWT Bearer Flow
```typescript
@Injectable()
export class SalesforceConnection {
private connection: jsforce.Connection;
async connect() {
this.connection = new jsforce.Connection({
instanceUrl: process.env.SF_INSTANCE_URL,
accessToken: await this.getAccessToken()
});
}
private async getAccessToken(): Promise<string> {
// JWT Bearer Token Flow
const jwt = this.createJWT({
iss: process.env.SF_CLIENT_ID,
sub: process.env.SF_USERNAME,
aud: process.env.SF_LOGIN_URL,
exp: Math.floor(Date.now() / 1000) + 300
});
const response = await fetch(`${process.env.SF_LOGIN_URL}/services/oauth2/token`, {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: new URLSearchParams({
grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer',
assertion: jwt
})
});
const { access_token } = await response.json();
return access_token;
}
async query<T>(soql: string): Promise<QueryResult<T>> {
return this.connection.query<T>(soql);
}
}
```
### Platform Events (Event-Driven)
**Purpose**: Receive order provisioning requests from Salesforce without inbound webhooks.
```typescript
@Injectable()
export class PlatformEventsSubscriber implements OnModuleInit {
private client: avro.PublishableClient;
async onModuleInit() {
await this.subscribe();
}
private async subscribe() {
const client = await this.createPubSubClient();
// Subscribe to OrderProvisionRequested__e
const subscription = await client.subscribe({
topicName: '/event/OrderProvisionRequested__e',
numRequested: 100,
replayPreset: grpc.ReplayPreset.LATEST
});
for await (const event of subscription) {
await this.handleProvisioningEvent(event);
}
}
private async handleProvisioningEvent(event: any) {
const { orderId__c, idemKey__c } = event.payload;
// Enqueue provisioning job
await this.provisioningQueue.add('provision-order', {
orderId: orderId__c,
idempotencyKey: idemKey__c || orderId__c
});
this.logger.log({ orderId: orderId__c }, 'Provisioning job enqueued');
}
}
```
**Benefits:**
- ✅ No inbound webhooks (security)
- ✅ Durable replay (reliability)
- ✅ Real-time processing (performance)
- ✅ Scalable (high volume)
### Salesforce Query Builders
Query builders are infrastructure concerns and belong in the BFF integration layer.
```typescript
// apps/bff/src/integrations/salesforce/utils/order-query-builder.ts
export function buildOrderSelectFields(additional: string[] = []): string[] {
const fields = [
'Id',
'AccountId',
'Status',
'Type',
'EffectiveDate',
'OrderNumber',
'TotalAmount',
'CreatedDate',
'Account.Name',
'Account.Email__c'
];
return UNIQUE([...fields, ...additional]);
}
export function buildOrderItemsQuery(orderId: string): string {
return `
SELECT Id, OrderId, Product2Id, Quantity, UnitPrice, TotalPrice,
Product2.Name, Product2.StockKeepingUnit,
Product2.Item_Class__c
FROM OrderItem
WHERE OrderId = '${orderId}'
`;
}
```
### Salesforce Data Entities
**Orders**:
- Read orders: `GET /orders``SELECT ... FROM Order`
- Create orders: `POST /orders``sobject('Order').create()`
- OrderItems created separately
**Products (Catalog)**:
- Read catalog: `GET /catalog``SELECT ... FROM Product2 WHERE Portal_Catalog__c = true`
- Includes related PricebookEntries
**Accounts (Customers)**:
- Customer profile linked to Account
---
## WHMCS Integration
### Connection Architecture
```
┌──────────────────────────────────────────────────────────┐
│ WHMCS Integration Services │
│ │
│ ┌────────────────────┐ ┌────────────────────────┐ │
│ │ REST API │ │ Webhooks (Inbound) │ │
│ │ (API Key Auth) │ │ (HMAC Validation) │ │
│ └──────────┬─────────┘ └─────────┬──────────────┘ │
│ │ │ │
│ ┌──────────▼───────────────────────────▼──────────────┐ │
│ │ WhmcsConnection Service │ │
│ │ • API key authentication │ │
│ │ • Request signing │ │
│ │ • Response validation │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘
```
### REST API Integration
**Authentication**: API Key + Secret
```typescript
@Injectable()
export class WhmcsConnection {
private readonly apiUrl = process.env.WHMCS_API_URL;
private readonly apiKey = process.env.WHMCS_API_KEY;
private readonly apiSecret = process.env.WHMCS_API_SECRET;
async call<T>(action: string, params: Record<string, any> = {}): Promise<T> {
const response = await fetch(this.apiUrl, {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: new URLSearchParams({
action,
identifier: this.apiKey,
secret: this.apiSecret,
responsetype: 'json',
...params
})
});
const data = await response.json();
if (data.result === 'error') {
throw new WhmcsApiException(data.message);
}
return data;
}
}
```
### WHMCS API Operations
**Invoices**:
```typescript
@Injectable()
export class WhmcsInvoiceService {
constructor(
private readonly whmcs: WhmcsConnection,
private readonly currencyService: CurrencyService
) {}
async getInvoices(clientId: number): Promise<Invoice[]> {
const response = await this.whmcs.call('GetInvoices', { clientid: clientId });
if (!response.invoices?.invoice) return [];
const defaultCurrency = this.currencyService.getDefaultCurrency();
return response.invoices.invoice.map(whmcsInvoice =>
Providers.Whmcs.transformWhmcsInvoice(whmcsInvoice, {
defaultCurrencyCode: defaultCurrency.code,
defaultCurrencySymbol: defaultCurrency.prefix || defaultCurrency.suffix
})
);
}
}
```
**Orders (Provisioning)**:
```typescript
@Injectable()
export class WhmcsOrderService {
async addOrder(params: WhmcsAddOrderParams): Promise<{ orderId: number }> {
return this.whmcs.call('AddOrder', {
clientid: params.clientId,
paymentmethod: params.paymentMethod,
pid: params.productIds,
billingcycle: params.billingCycles,
qty: params.quantities,
configoptions: params.configOptions,
notes: params.notes
});
}
async acceptOrder(orderId: number): Promise<WhmcsOrderResult> {
return this.whmcs.call('AcceptOrder', { orderid: orderId });
}
}
```
**Subscriptions**:
```typescript
@Injectable()
export class WhmcsSubscriptionService {
async getServices(clientId: number): Promise<Subscription[]> {
const response = await this.whmcs.call('GetClientsProducts', { clientid: clientId });
const defaultCurrency = this.currencyService.getDefaultCurrency();
return response.products.product.map(whmcsProduct =>
Providers.Whmcs.transformWhmcsSubscription(whmcsProduct, {
defaultCurrencyCode: defaultCurrency.code,
defaultCurrencySymbol: defaultCurrency.prefix || defaultCurrency.suffix
})
);
}
}
```
### WHMCS Webhooks (Inbound)
**Purpose**: Receive real-time updates from WHMCS (invoice paid, service activated, etc.)
```typescript
@Controller('webhooks/whmcs')
export class WhmcsWebhookController {
@Post()
async handleWebhook(
@Body() payload: any,
@Headers('x-whmcs-signature') signature: string
) {
// 1. Validate HMAC signature
if (!this.validateSignature(payload, signature)) {
throw new UnauthorizedException('Invalid webhook signature');
}
// 2. Process event
const eventType = payload.event;
switch (eventType) {
case 'InvoicePaid':
await this.handleInvoicePaid(payload);
break;
case 'ServiceActivated':
await this.handleServiceActivated(payload);
break;
// ... other events
}
return { received: true };
}
private validateSignature(payload: any, signature: string): boolean {
const secret = process.env.WHMCS_WEBHOOK_SECRET;
const computedSignature = createHmac('sha256', secret)
.update(JSON.stringify(payload))
.digest('hex');
return timingSafeEqual(
Buffer.from(signature),
Buffer.from(computedSignature)
);
}
}
```
---
## Freebit SIM Management
### Connection Architecture
```typescript
@Injectable()
export class FreebitConnection {
private readonly apiUrl = process.env.FREEBIT_API_URL;
private readonly apiKey = process.env.FREEBIT_API_KEY;
async request<T>(endpoint: string, params?: Record<string, any>): Promise<T> {
const response = await fetch(`${this.apiUrl}${endpoint}`, {
method: 'POST',
headers: {
'Authorization': `Bearer ${this.apiKey}`,
'Content-Type': 'application/json'
},
body: JSON.stringify(params)
});
return response.json();
}
}
```
### Freebit API Operations
**SIM Details**:
```typescript
@Injectable()
export class FreebitSimService {
async getSimDetails(iccid: string): Promise<SimDetails> {
const response = await this.freebit.request('/api/sim/details', { iccid });
// Transform with domain mapper
return FreebitProvider.transformFreebitAccountDetails(response);
}
async getSimUsage(iccid: string, startDate: Date, endDate: Date): Promise<SimUsage> {
const response = await this.freebit.request('/api/sim/usage', {
iccid,
start_date: startDate.toISOString(),
end_date: endDate.toISOString()
});
return FreebitProvider.transformFreebitUsageData(response);
}
}
```
---
## Data Transformation
### Transformation Flow
```
┌──────────────┐
│ External API │
│ Raw Response │
└──────┬───────┘
│ 1. Validate raw data
┌──────▼────────────────────┐
│ Domain Mapper │
│ • Parse raw schema │
│ • Transform to domain │
│ • Validate domain schema │
└──────┬────────────────────┘
│ 2. Return domain type
┌──────▼───────┐
│ Application │
│ Uses domain │
│ types only │
└──────────────┘
```
### Domain Mapper Pattern
```typescript
// packages/domain/billing/providers/whmcs/mapper.ts
export function transformWhmcsInvoice(
raw: unknown,
context: { defaultCurrencyCode: string; defaultCurrencySymbol: string }
): Invoice {
// 1. Validate raw data
const whmcs = whmcsInvoiceRawSchema.parse(raw);
// 2. Transform to domain model
const result: Invoice = {
id: parseInt(whmcs.invoiceid),
userId: parseInt(whmcs.userid),
status: mapWhmcsInvoiceStatus(whmcs.status),
amount: {
value: parseFloat(whmcs.total),
currency: whmcs.currencycode || context.defaultCurrencyCode
},
dueDate: new Date(whmcs.duedate),
invoiceNumber: whmcs.invoicenum,
createdAt: new Date(whmcs.date),
items: whmcs.items?.item?.map(transformWhmcsInvoiceItem) || []
};
// 3. Validate domain model
return invoiceSchema.parse(result);
}
```
### Context Injection Pattern
Some transformations need infrastructure context (like currency settings):
```typescript
// ✅ Correct: Inject context explicitly
const defaultCurrency = this.currencyService.getDefaultCurrency();
const invoice = Providers.Whmcs.transformWhmcsInvoice(rawInvoice, {
defaultCurrencyCode: defaultCurrency.code,
defaultCurrencySymbol: defaultCurrency.prefix || defaultCurrency.suffix
});
```
**Why this is clean:**
- Domain mapper is pure (deterministic for same inputs)
- Infrastructure concern (currency) is injected from BFF
- No service wrapper needed
---
## Error Handling
### Error Handling Strategy
1. **API Errors**: Log detailed error, return user-friendly message
2. **Validation Errors**: Return specific field errors
3. **Network Errors**: Retry with exponential backoff
4. **Business Errors**: Return domain-specific error codes
```typescript
@Injectable()
export class SalesforceOrderService {
async getOrderById(orderId: string): Promise<OrderDetails | null> {
try {
const result = await this.sf.query(soql);
return OrderProviders.Salesforce.transformSalesforceOrderDetails(result.records[0], []);
} catch (error) {
this.logger.error('Failed to fetch order from Salesforce', {
orderId,
error: error.message
});
// Don't expose internal details [[memory:6689308]]
throw new NotFoundException('Order not found');
}
}
}
```
### Retry Logic
```typescript
async callWithRetry<T>(
fn: () => Promise<T>,
maxRetries = 3,
delay = 1000
): Promise<T> {
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
if (attempt === maxRetries) throw error;
this.logger.warn(`Retry attempt ${attempt}/${maxRetries}`, { error: error.message });
await new Promise(resolve => setTimeout(resolve, delay * attempt));
}
}
}
```
---
## Caching Strategy
### Cache Configuration
```typescript
// Invoice cache
CACHE_TTL_INVOICES = 60-120s
CACHE_KEY_PATTERN = 'user:{userId}:invoices:page:{page}'
// Subscription cache
CACHE_TTL_SUBSCRIPTIONS = 120s
CACHE_KEY_PATTERN = 'user:{userId}:subscriptions'
// Catalog cache
CACHE_TTL_CATALOG = 300-900s (5-15 min)
CACHE_KEY_PATTERN = 'catalog:{type}'
```
### Cache Implementation
```typescript
@Injectable()
export class CacheService {
constructor(private readonly redis: Redis) {}
async get<T>(key: string): Promise<T | null> {
const cached = await this.redis.get(key);
return cached ? JSON.parse(cached) : null;
}
async set(key: string, value: any, ttl: number): Promise<void> {
await this.redis.setex(key, ttl, JSON.stringify(value));
}
async invalidate(pattern: string): Promise<void> {
const keys = await this.redis.keys(pattern);
if (keys.length > 0) {
await this.redis.del(...keys);
}
}
}
```
### Cache Invalidation
```typescript
// Invalidate invoice cache when webhook received
async handleInvoicePaid(payload: any) {
const userId = payload.userid;
// Process event
await this.processInvoicePayment(payload);
// Invalidate cache
await this.cacheService.invalidate(`user:${userId}:invoices:*`);
}
```
---
## Best Practices
### Integration Service Best Practices
**DO:**
1. ✅ Keep integration services thin - just fetch and transform
2. ✅ Use domain mappers directly - no wrappers
3. ✅ Build queries in utils - separate query construction
4. ✅ Handle errors gracefully - log and throw/return null
5. ✅ Cache when appropriate - reduce API calls
6. ✅ Inject context explicitly - currency, config, etc.
7. ✅ Return domain types - never return raw API responses
**DON'T:**
1. ❌ Create wrapper services - use domain mappers directly
2. ❌ Add business logic - belongs in domain or orchestrators
3. ❌ Transform twice - map once in domain
4. ❌ Expose raw types - always return domain types
5. ❌ Hard-code queries - use query builders
6. ❌ Skip error handling - always log failures
---
**Last Updated**: October 2025
**Status**: Active - Production System