The LLM Integration Playbook: Building AI-Powered Products

After shipping three major AI features in production, here is what I have learned about integrating Large Language Models into real products.

The Hype vs. Reality Gap

LLMs are incredible. They are also unreliable, expensive, and slow. The gap between a demo and a production system is vast. Let me share the patterns that bridge it.

Pattern 1: The Validation Layer

Never trust LLM output directly. Always validate.

// Before
const response = await openai.chat.completions.create({ ... });
return response.choices[0].message.content;

// After
const response = await openai.chat.completions.create({ ... });
const parsed = JSON.parse(response.choices[0].message.content);
const validated = ResponseSchema.parse(parsed); // Zod validation
return validated;

Pattern 2: The Retry-with-Context Strategy

When validation fails, do not just retry blindly. Feed the error back to the model.

async function generateWithRetry(prompt: string, maxRetries = 3) {
  let lastError = null;
  
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const contextualPrompt = lastError 
      ? `${prompt}\n\nPrevious attempt failed with: ${lastError}. Please fix.`
      : prompt;
      
    const result = await generate(contextualPrompt);
    const validation = schema.safeParse(result);
    
    if (validation.success) return validation.data;
    lastError = validation.error.message;
  }
  
  throw new Error('Max retries exceeded');
}

Pattern 3: The Cost Control Matrix

The Bottom Line

LLMs are a tool, not magic. Treat them like any other unreliable external service: validate inputs, validate outputs, retry with grace, and always have a fallback.

The best AI products are the ones where users cannot tell which parts are AI and which are not.