AI Architecture Examples

A complete architecture for a production RAG system including document ingestion pipeline, embedding generation, vector store, retrieval service, LLM orchestration layer, caching, and monitoring. Designed for 99.9% uptime and sub-second latency.

// Production RAG architecture layers
// 1. Ingestion Pipeline: Sources -> Chunking -> Embedding -> Vector Store
// 2. Retrieval Service: Query -> Embedding -> Vector Search -> Re-ranking -> Access Control Filter
// 3. Generation Service: Context Assembly -> LLM Call -> Response Validation -> Caching
// 4. Evaluation Service: Continuous quality monitoring with automated metrics
// 5. Observability: Latency tracking, cost monitoring, quality dashboards

Architecture diagram (text):
┌─────────────┐    ┌──────────────┐    ┌─────────────┐
│   Sources   │───>│  Ingestion   │───>│ Vector Store│
│ (Docs, APIs)│    │  Pipeline    │    │  (Qdrant)   │
└─────────────┘    └──────────────┘    └──────┬──────┘
                                              │
┌─────────────┐    ┌──────────────┐    ┌──────▼──────┐
│    User     │───>│  Retrieval   │───>│  Re-ranker  │
│   Query     │    │   Service    │    │  + ACL      │
└─────────────┘    └──────────────┘    └──────┬──────┘
                                              │
                   ┌──────────────┐    ┌──────▼──────┐
                   │   Response   │<───│ Generation  │
                   │   + Cache    │    │   Service   │
                   └──────────────┘    └─────────────┘

An API gateway that routes AI requests to different models based on task type, cost constraints, and latency requirements. Includes fallback routing, rate limiting, cost tracking, and model performance comparison.

// AI Gateway routing configuration
const modelRouting = {
  "simple-classification": {
    primary: "claude-haiku",
    fallback: "gpt-4o-mini",
    maxLatency: 500, // ms
    maxCost: 0.001, // per request
  },
  "complex-reasoning": {
    primary: "claude-sonnet",
    fallback: "gpt-4o",
    maxLatency: 5000,
    maxCost: 0.05,
  },
  "code-generation": {
    primary: "claude-sonnet",
    fallback: "gpt-4o",
    maxLatency: 10000,
    maxCost: 0.10,
  },
};

// Gateway handles: routing, fallback, rate limiting,
// cost tracking, latency monitoring, and A/B testing

An architecture where AI processing is triggered by events (new document uploaded, email received, form submitted) and results are processed asynchronously. Uses message queues for decoupling and reliability.

// Event-driven AI processing
// Event sources -> Message Queue -> AI Workers -> Results Store -> Notifications

// Example: Document upload triggers AI processing
EventBus.on("document.uploaded", async (event) => {
  await queue.publish("ai.process.document", {
    documentId: event.documentId,
    tasks: ["extract", "classify", "summarise"],
    priority: event.priority,
    callback: "document.processed",
  });
});

// AI Worker processes asynchronously
Worker.consume("ai.process.document", async (job) => {
  const results = await processDocument(job.documentId, job.tasks);
  await resultsStore.save(job.documentId, results);
  await EventBus.emit(job.callback, { documentId: job.documentId, results });
});

An architecture pattern for adding AI capabilities to existing microservices without disrupting them. AI services run alongside existing services, sharing data through APIs and events, with feature flags for gradual rollout.

// AI as sidecar service pattern
// Existing service calls AI sidecar via internal API
// Feature flag controls whether AI result is used

async function processOrder(order) {
  // Existing logic
  const basePrice = calculatePrice(order);

  // AI augmentation (behind feature flag)
  if (featureFlags.isEnabled("ai-pricing-optimization", order.customerId)) {
    const aiPrice = await aiPricingService.optimize({
      basePrice,
      customer: order.customerId,
      context: order.items,
    });

    // Use AI price only if within acceptable range
    if (aiPrice.confidence > 0.8 && Math.abs(aiPrice.price - basePrice) / basePrice < 0.15) {
      return aiPrice.price;
    }
  }

  return basePrice;
}

An architecture that processes sensitive data with on-premises AI models while using cloud AI services for non-sensitive workloads. Includes data classification, routing logic, and unified monitoring across both environments.

// Hybrid cloud AI routing
async function routeAIRequest(request) {
  const classification = classifyDataSensitivity(request.data);

  if (classification.level === "restricted" || classification.level === "confidential") {
    // Process on-premises with local model
    return await onPremAI.process(request, {
      model: "local-llama-70b",
      gpu: "a100-cluster",
    });
  }

  // Process in cloud with commercial model
  return await cloudAI.process(request, {
    model: "claude-sonnet",
    region: classification.dataResidency,
  });
}

An architecture for serving AI predictions in real-time (sub-100ms) for applications like fraud detection, content recommendations, and dynamic pricing. Covers model serving, feature stores, and caching strategies.

// Real-time inference architecture
// Feature Store -> Model Server -> Response Cache

async function realTimeInference(request) {
  // Check response cache first (sub-1ms)
  const cached = await cache.get(request.cacheKey);
  if (cached) return cached;

  // Get pre-computed features from feature store (sub-10ms)
  const features = await featureStore.getOnlineFeatures(request.entityId);

  // Run inference on optimised model server (sub-50ms)
  const prediction = await modelServer.predict({
    model: "fraud-detection-v3",
    features: features,
  });

  // Cache result
  await cache.set(request.cacheKey, prediction, { ttl: 60 });

  return prediction;
}