The Architecture Decision AI Forced Us to Make

Six months ago, we added an AI feature: "Suggest similar products." Users loved it. It increased sales 15%.

But it broke our architecture assumptions.

Our system was designed for predictable, deterministic logic. "If user clicks Add to Cart, create cart item." Simple.

AI doesn't work like that. It's probabilistic. Sometimes it's slow. Sometimes it gives different results for the same input. Sometimes it just... fails randomly.

We had to rethink how we build systems around intelligence that isn't perfect.

What AI Changes About Architecture

1. Latency is Unpredictable

Old world: Database query takes 20ms. Every time. Predictable.

AI world: Model inference takes 50ms... or 500ms... or 5 seconds. Depends on input size, model load, cosmic rays.

Solution: We made AI calls async. User submits request, gets a "processing" response, result appears when ready.

2. Failures Are Common

Old world: If database fails, system is down. Fix it immediately.

AI world: Model API times out 5% of the time. That's just... how it is.

Solution: We added fallbacks. If AI fails, show cached recommendations. If those don't exist, show popular products. Always have a plan B.

3. Results Aren't Deterministic

Old world: Same input → same output. Always.

AI world: Same input → slightly different output each time (temperature, sampling, model updates).

Solution: We cache AI results for 24 hours. User sees consistent recommendations within a day, but they refresh daily.

Architectural Patterns for AI

Pattern 1: AI as a Separate Service

Don't embed AI in your core application. Make it a separate microservice.

Why? AI models need GPUs. Expensive. Scale them independently from your regular app servers.

Pattern 2: Circuit Breakers

If AI service is down, stop calling it. Fail fast. Use cached results.

if circuit_breaker.is_open("ai-service"):
    return cached_recommendations()
else:
    return ai_service.get_recommendations()

Pattern 3: Human-in-the-Loop

For critical decisions, AI suggests, human approves.

We use this for content moderation. AI flags suspicious content. Human reviews before taking action.

The Future I'm Seeing

In 2 years, every system will have AI components. Not because it's trendy, but because users expect it.

• Search that understands intent, not just keywords
• Customer support that answers questions instantly
• Code that reviews itself
• Bugs that get detected before deploy

The systems that win will be those that treat AI as a first-class component, not an afterthought.

Start designing for unpredictability. It's the new normal.