The LLM Integration Playbook: Building AI-Powered Products
Val Chahul
The Operator
After shipping three major AI features in production, here is what I have learned about integrating Large Language Models into real products.
After shipping three major AI features in production, here is what I have learned about integrating Large Language Models into real products.
The Hype vs. Reality Gap
LLMs are incredible. They are also unreliable, expensive, and slow. The gap between a demo and a production system is vast. Let me share the patterns that bridge it.
Pattern 1: The Validation Layer
Never trust LLM output directly. Always validate.
// Before
const response = await openai.chat.completions.create({ ... });
return response.choices[0].message.content;
// After
const response = await openai.chat.completions.create({ ... });
const parsed = JSON.parse(response.choices[0].message.content);
const validated = ResponseSchema.parse(parsed); // Zod validation
return validated;Pattern 2: The Retry-with-Context Strategy
When validation fails, do not just retry blindly. Feed the error back to the model.
async function generateWithRetry(prompt: string, maxRetries = 3) {
let lastError = null;
for (let attempt = 0; attempt < maxRetries; attempt++) {
const contextualPrompt = lastError
? `${prompt}\n\nPrevious attempt failed with: ${lastError}. Please fix.`
: prompt;
const result = await generate(contextualPrompt);
const validation = schema.safeParse(result);
if (validation.success) return validation.data;
lastError = validation.error.message;
}
throw new Error('Max retries exceeded');
}Pattern 3: The Cost Control Matrix
Use Case | Model | Cost/1K tokens |
|---|---|---|
Classification | GPT-4o-mini | $0.00015 |
Summarization | Claude Haiku | $0.00025 |
Complex Reasoning | Claude Sonnet | $0.003 |
The Bottom Line
LLMs are a tool, not magic. Treat them like any other unreliable external service: validate inputs, validate outputs, retry with grace, and always have a fallback.
The best AI products are the ones where users cannot tell which parts are AI and which are not.
Discussion
Comments coming soon.