The Production AI Application Stack

Section 1

Executive Summary

A production AI application is more than:

A reliable system is a coordinated software platform containing:

1. 1

   User

2. 2

   Web / Mobile / API

3. 3

   Edge / CDN / WAF

4. 4

   Identity + Authorization

5. 5

   Application / Domain Logic

6. 6

   AI Orchestration

   • Model Routing
   • Prompt / Context Engineering
   • RAG / Retrieval
   • Memory / State
   • Tool Calling
   • Guardrails
   • Human Approval
7. 7

   Data + External Systems

8. 8

   Observability + Evaluation

9. 9

   Billing / Usage / Operations

10. 10

    Production Infrastructure

The core principle of this document is:

AI is a capability inside the application, not the application architecture itself.

Production systems must also engineer identity, authorization, data, payments, usage, security, observability, reliability, testing, deployment, and operational governance alongside AI.

Orchestration, tools, state/memory, guardrails, evaluation, observability, security, and cost control belong in the design from the start, not added later.

Section 2

Engineering Philosophy

2.1 Capability-first engineering

The system should first define what capability is required, then choose an implementation.

Bad:

Use Supabase because it is popular.

Better:

This prevents unnecessary vendor lock-in.

2.2 Production before polish

A beautiful interface that cannot reliably process payments, protect tenant data, recover from failures, or explain AI behavior is not production-ready.

The engineering order should generally be:

1. 1

   Requirements

2. 2

   Architecture

3. 3

   Data + Security Model

4. 4

   Core Application

5. 5

   AI Capabilities

6. 6

   Billing / Usage

7. 7

   Observability

8. 8

   Testing

9. 9

   Deployment

10. 10

    Production Readiness

2.3 Progressive complexity

Do not deploy enterprise infrastructure before the product requires it.

1. 1

   MVP

2. 2

   Growth

3. 3

   Scale

4. 4

   Enterprise

Every layer should have a reason to exist.

Section 3

Frontend

Responsibilities

• application interface
• responsive design
• accessibility
• state management
• forms
• optimistic UI
• loading / empty / error states
• AI streaming interfaces
• file uploads
• dashboards
• billing UI
• account management

Typical stack

• React
• Next.js
• TypeScript
• Tailwind CSS
• component system
• accessible UI primitives

Requirements

• server/client boundary must be deliberate
• avoid unnecessary client-side JavaScript
• validate all user input
• never expose secrets
• handle network failure gracefully
• design AI interfaces for partial/streaming responses
• provide cancellation for long-running AI tasks

Section 4

Application / Backend Layer

The application layer contains business rules.

Typical responsibilities:

• API endpoints
• server actions
• domain services
• business workflows
• authorization enforcement
• integrations
• AI orchestration
• webhook processing
• background job dispatch
• validation
• transactions

Recommended architecture:

1. 1

   UI

2. 2

   API / Server Action

3. 3

   Application Service

4. 4

   Domain Logic

5. 5

   Repository / Integration

6. 6

   Database / External Service

Avoid placing complex business logic directly inside UI components or API handlers.

Section 5

Database Architecture

5.1 Primary database

Default:

• PostgreSQL

Responsibilities:

• users
• organizations
• memberships
• application data
• billing state
• usage records
• audit logs
• AI task state
• workflow state

5.2 Data ownership

Every important record should have explicit ownership.

For multi-tenant applications:

1. 1

   organization

2. 2

   membership

3. 3

   user

4. 4

   organization-owned resources

Prefer:

on tenant-owned entities.

5.3 Database rules

• migrations must be version-controlled
• constraints should enforce invariants
• indexes should follow real query patterns
• sensitive fields should be minimized
• transactions should protect multi-step mutations
• soft deletion should only be used when justified
• auditability should be designed, not retrofitted

Section 6

Authentication and Authorization

Authentication answers:

Who are you?

Authorization answers:

What are you allowed to do?

They must remain separate concepts.

Required capabilities

• email/password or passwordless authentication
• OAuth where appropriate
• session management
• MFA for sensitive systems
• role-based access control
• organization membership
• resource-level authorization
• service identities
• API credentials
• AI-agent identities where applicable

Authorization model

Use layered controls:

1. 1

   Identity

2. 2

   Organization

3. 3

   Role

4. 4

   Permission

5. 5

   Resource ownership

6. 6

   Action-specific policy

Never rely exclusively on frontend authorization.

Section 7

Multi-Tenancy

For SaaS products, tenancy must be designed from the beginning.

Typical model:

Recommended controls:

• organization_id
• membership table
• role/permission model
• database-level row security where appropriate
• tenant-aware storage paths
• tenant-aware cache keys
• tenant-aware analytics
• tenant-aware AI retrieval
• tenant-aware audit logs

The application must never allow cross-tenant data leakage.

Section 8

File Storage and Processing

Use object storage for:

• images
• PDFs
• documents
• exports
• user attachments
• generated files
• AI source documents

Pipeline:

1. 1

   Upload

2. 2

   Authentication

3. 3

   Authorization

4. 4

   Virus / file validation

5. 5

   Object storage

6. 6

   Metadata record

7. 7

   Background processing

8. 8

   Extraction

9. 9

   Chunking

10. 10

    Embedding / indexing

11. 11

    Available for retrieval

Treat uploaded files as untrusted input.

Section 9

AI Model Layer

The application should not hard-code itself around a single model.

Create a model abstraction:

1. 1

   Application

2. 2

   AI Service

3. 3

   Model Router

   • Provider A
   • Provider B
   • Provider C

The router can select models based on:

• task complexity
• latency requirements
• cost
• context size
• structured output requirements
• availability
• privacy requirements
• modality

Model governance

Track:

• provider
• model
• version
• prompt version
• token usage
• latency
• cost
• failure
• fallback
• evaluation result

Section 10

AI Orchestration

A simple AI application:

1. 1

   Request

2. 2

   Prompt

3. 3

   Model

4. 4

   Response

An agentic application:

1. 1

   Request

2. 2

   Policy Check

3. 3

   Context Builder

4. 4

   Planner / Router

5. 5

   Agent Loop

   • Retrieve
   • Call Tool
   • Observe
   • Reason
   • Continue / Stop
6. 6

   Guardrails

7. 7

   Validation

8. 8

   Human Approval if required

9. 9

   Response / Action

Deterministic vs autonomous workflows

Prefer deterministic workflows when the process is known.

Use agentic behavior when the system genuinely benefits from:

• dynamic tool selection
• planning
• variable task decomposition
• adaptive retrieval
• multi-step reasoning
• uncertain workflows

Do not use agents merely because they are fashionable.

Section 11

Tool Calling

Tools allow AI systems to interact with real systems.

Examples:

• get_customer()
• create_invoice()
• search_documents()
• check_inventory()
• create_ticket()
• send_email()
• schedule_event()

Every tool requires:

• strict schema
• authentication
• authorization
• input validation
• output validation
• timeout
• retry policy
• rate limit
• audit logging
• idempotency where applicable

Tool risk classification

Low risk

• read-only search
• calculations
• document retrieval

Medium risk

• draft email
• create task
• update non-critical data

High risk

• financial transaction
• deleting data
• changing permissions
• sending external communication
• production infrastructure changes

High-risk actions should require stronger controls and, where appropriate, human approval.

Section 12

RAG and Knowledge Systems

RAG is more than:

A production retrieval system needs:

1. 1

   Source

2. 2

   Ingestion

3. 3

   Validation

4. 4

   Parsing

5. 5

   Chunking

6. 6

   Metadata

7. 7

   Embedding

8. 8

   Index

9. 9

   Retrieval

10. 10

    Filtering / reranking

11. 11

    Context construction

12. 12

    Model

13. 13

    Citation / provenance

Retrieval requirements

• tenant isolation
• document permissions
• metadata filtering
• source attribution
• chunk quality
• freshness
• deletion propagation
• retrieval evaluation
• poisoning protection

Retrieval strategies

Use the simplest strategy that works:

1. 01keyword search
2. 02vector search
3. 03hybrid search
4. 04reranking
5. 05graph / structured retrieval
6. 06agentic retrieval

Section 13

Memory and State

Memory must be intentionally designed.

Short-term state

Current task:

• conversation
• tool results
• intermediate state
• workflow state

Long-term memory

Potentially:

• preferences
• historical interactions
• business context
• learned facts

Long-term memory must have:

• ownership
• retention policy
• deletion controls
• privacy controls
• provenance
• relevance policy

Never store everything because storage is cheap.

Section 14

Guardrails and AI Security

AI security is an application-security discipline.

Threats

• prompt injection
• indirect prompt injection
• data exfiltration
• malicious documents
• tool abuse
• excessive agency
• unauthorized actions
• secret leakage
• cross-tenant retrieval
• insecure tool schemas
• model manipulation
• output-based attacks

Security pipeline

1. 1

   User Input

2. 2

   Input Validation

3. 3

   Policy / Security Check

4. 4

   Context Construction

5. 5

   Model

6. 6

   Tool Authorization

7. 7

   Tool Execution

8. 8

   Output Validation

9. 9

   Sensitive-data / policy checks

10. 10

    User

Principle

The model should never be the final authority for authorization.

The application must enforce permissions independently.

Section 15

Payments and Billing

Payments are a core application subsystem.

Capabilities:

• checkout
• subscriptions
• one-time purchases
• invoices
• payment status
• refunds
• failed payments
• trials
• plan changes
• cancellations
• webhook processing
• customer portal
• tax handling where applicable

Architecture:

1. 1

   User

2. 2

   Application

3. 3

   Payment Provider

4. 4

   Webhook

5. 5

   Webhook Verification

6. 6

   Billing Service

7. 7

   Database

8. 8

   Entitlements

Critical rules

• never trust client-side payment status
• verify webhook signatures
• make webhook handlers idempotent
• store provider event IDs
• reconcile provider state
• separate payment status from product entitlement
• audit billing changes

Section 16

Subscriptions and Entitlements

Subscription status is not the same thing as access.

Use:

1. 1

   Customer

2. 2

   Subscription

3. 3

   Plan

4. 4

   Entitlements

Example:

The application should evaluate entitlements centrally.

Section 17

Usage Metering

AI applications need usage accounting because AI has variable cost.

Track:

• requests
• tokens
• model
• provider
• duration
• tool calls
• storage
• retrieval operations
• background jobs
• paid features

Example:

1. 1

   AI Request

2. 2

   Usage Event

3. 3

   Cost Calculation

4. 4

   Usage Ledger

5. 5

   Quota / Billing

Usage records should be append-oriented and auditable.

Section 18

AI Cost Optimization

Track:

• Cost per request
• Cost per user
• Cost per organization
• Cost per feature
• Cost per successful task
• Cost per model

Optimization techniques:

• model routing
• caching
• prompt compression
• context filtering
• retrieval optimization
• smaller models for simple tasks
• asynchronous processing
• token budgets
• maximum tool steps
• user/tenant quotas

Never optimize cost by blindly reducing model quality.

Section 19

Background Jobs and Durable Workflows

Use background processing for:

• document processing
• embeddings
• email
• reports
• exports
• AI batch jobs
• scheduled tasks
• webhook retries
• long-running agents

Architecture:

1. 1

   Request

2. 2

   Create Job

3. 3

   Queue

4. 4

   Worker

5. 5

   Progress State

6. 6

   Completion / Retry / Failure

Required properties:

• idempotency
• retry policy
• timeout
• dead-letter handling
• observability
• progress state
• cancellation where appropriate

Section 20

Caching and Rate Limiting

Caching targets:

• expensive reads
• model responses where safe
• configuration
• permissions
• frequently accessed reference data

Do not cache sensitive information without a deliberate isolation strategy.

Rate limiting should exist at multiple levels:

1. 1

   IP

2. 2

   User

3. 3

   Organization

4. 4

   API Key

5. 5

   Endpoint

6. 6

   AI Feature

7. 7

   Tool

Section 21

Analytics

Product analytics answers:

What are users doing?

Track:

• activation
• retention
• feature usage
• conversion
• funnel completion
• errors
• AI feature adoption
• successful task completion

Do not use analytics as a substitute for application logs.

Section 22

Error Monitoring

Application monitoring should capture:

• exceptions
• failed requests
• stack traces
• release version
• user context
• organization context
• environment
• breadcrumbs

AI monitoring additionally needs:

• model failures
• tool failures
• malformed outputs
• retrieval failures
• safety events
• agent loop failures

Section 23

AI Observability

Traditional monitoring is insufficient for AI.

A useful AI trace should show:

Key metrics:

• task success rate
• model latency
• time to first token
• total latency
• token usage
• cost
• tool failure rate
• retry rate
• fallback rate
• escalation rate
• guardrail trigger rate
• evaluation score

Prompts should be treated as versioned artifacts.

Section 24

Evaluation

AI systems require evaluations in addition to traditional software tests.

Evaluation categories:

Deterministic

• schema validation
• expected tool call
• permission enforcement
• API response
• database state

Model quality

• factuality
• relevance
• instruction following
• tone
• classification accuracy

Agent quality

• task completion
• tool selection
• trajectory quality
• unnecessary steps
• recovery behavior

Safety

• prompt injection resistance
• unauthorized action prevention
• sensitive-data handling
• malicious tool requests

Evaluation should exist:

Section 25

Logging

Use structured logs.

Example conceptual event:

{
  "event": "ai.tool_call",
  "request_id": "...",
  "organization_id": "...",
  "user_id": "...",
  "tool": "create_invoice",
  "status": "success",
  "duration_ms": 842
}

Never log:

• passwords
• access tokens
• secret keys
• unnecessary personal data
• raw payment credentials

Section 26

Audit Logging

Audit logs answer:

Who did what, when, and through which mechanism?

Record important actions:

• login
• permission change
• billing change
• data export
• data deletion
• AI action
• tool execution
• administrative changes
• security events

For agentic systems:

must be distinguishable as the actor.

Section 27

Notifications

Support appropriate channels:

• in-app
• email
• push
• SMS where necessary

Use event-driven notification architecture:

1. 1

   Domain Event

2. 2

   Notification Service

3. 3

   Preference Check

4. 4

   Channel

Users should control non-critical notification preferences.

Section 28

Edge, CDN, WAF and Network Security

Use an edge layer for:

• CDN
• TLS
• WAF
• DDoS mitigation
• bot protection
• rate limiting
• routing

Application security must still exist behind the edge.

A WAF is not a replacement for secure application code.

Section 29

Infrastructure and Deployment

Default progression:

MVP

1. 1

   GitHub

2. 2

   Managed hosting

3. 3

   Managed database

4. 4

   Managed AI providers

Growth

Add:

• edge/WAF
• background workers
• Redis
• advanced observability
• dedicated queues
• more granular deployment environments

Enterprise

Potentially add:

• private networking
• SSO/SAML
• SIEM
• dedicated infrastructure
• regional deployment
• advanced secrets management
• disaster recovery architecture
• compliance controls

Section 30

CI/CD

Pipeline:

1. 1

   Commit

2. 2

   Lint

3. 3

   Type Check

4. 4

   Unit Tests

5. 5

   Integration Tests

6. 6

   Security Checks

7. 7

   Build

8. 8

   Preview

9. 9

   Approval

10. 10

    Production

11. 11

    Smoke Test

Database migrations must be part of controlled deployment.

Section 31

Testing Strategy

Use multiple layers.

1. 1

   Unit

2. 2

   Integration

3. 3

   API

4. 4

   End-to-End

5. 5

   Security

6. 6

   AI Evaluation

7. 7

   Load / Performance

AI-specific tests:

• prompt regression
• structured output validation
• tool authorization
• RAG retrieval quality
• agent task completion
• adversarial prompts
• cost budgets
• timeout behavior

Section 32

Backup and Disaster Recovery

Define:

• backup frequency
• retention
• restoration procedure
• RPO
• RTO
• database recovery
• object storage recovery
• configuration recovery
• secrets recovery
• third-party dependency failure strategy

A backup that has never been restored is not a proven backup strategy.

Section 33

Support and Operations

Production products need:

• customer support
• status page
• incident process
• admin console
• feature flags
• audit trail
• operational dashboards

Internal admin tools should be treated as production software.

Section 34

Production AI Engineering Skills System

The stack above becomes more useful as reusable AI-agent skills.

The skills system teaches an AI coding agent to engineer production systems, not only to generate code.

Section 35

Capability Skills vs Provider Skills

The skill system has two layers.

Layer A · Capability skills

These define engineering responsibilities.

Examples:

• payments-billing
• rag-engineering
• authentication-authorization
• ai-security
• observability
• database-engineering

Layer B · Provider skills

These define concrete implementations.

Examples:

• supabase
• vercel
• openai
• anthropic
• google-ai
• stripe
• paymongo
• xendit
• cloudflare
• posthog
• sentry

The agent should select:

1. 1

   Capability

2. 2

   Requirements

3. 3

   Provider

4. 4

   Implementation

rather than:

1. 1

   Vendor

2. 2

   Force application to fit vendor

Section 36

Skill Repository Structure

Section 37

Meta Skills

37.1 architecture-design

Before implementation, determine:

• product requirements
• users
• roles
• entities
• workflows
• integrations
• security boundaries
• data ownership
• AI requirements
• payment requirements
• infrastructure requirements
• observability requirements
• scalability requirements

Output:

• Architecture
• Data Model
• Security Model
• Integration Map
• Deployment Model
• Implementation Plan

37.2 ai-app-builder

This is the orchestrator.

It determines which skills are required for a given application.

Example:

1. 1

   User Request

2. 2

   architecture-design

3. 3

   Select required skills

4. 4

   Execute implementation

5. 5

   code-review

6. 6

   testing

7. 7

   production-readiness

37.3 code-review

Review:

• architecture
• correctness
• security
• database
• authorization
• AI behavior
• cost
• performance
• accessibility
• testing
• maintainability

37.4 production-readiness

Acts as the final gate.

It verifies:

• Authentication
• Authorization
• Tenant isolation
• Input validation
• Error handling
• Loading states
• Empty states
• Security
• Rate limiting
• Logging
• Monitoring
• Analytics
• AI observability
• Cost controls
• Payment correctness
• Webhook verification
• Tests
• Accessibility
• Performance
• Backups
• Documentation
• Deployment
• Incident response

Section 38

Standard SKILL.md Contract

Every skill should follow the same structure:

SKILL.md

1. 01Purpose
2. 02When to use
3. 03When not to use
4. 04Required inputs
5. 05Engineering principles
6. 06Architecture requirements
7. 07Implementation workflow
8. 08Security requirements
9. 09Performance requirements
10. 10Cost considerations
11. 11Testing requirements
12. 12Observability requirements
13. 13Failure handling
14. 14Anti-patterns
15. 15Verification checklist
16. 16Provider implementations
17. 17Example usage

Skills should be:

• actionable
• testable
• composable
• vendor-aware but vendor-independent
• explicit about failure modes
• explicit about security
• explicit about verification

Section 39

AI App Builder Orchestration

Example request:

Build a multi-tenant AI HR platform with subscriptions and an AI assistant.

The orchestrator should produce:

1. 1

   architecture-design

2. 2

   Application skills

   • frontend-engineering
   • backend-engineering
   • database-engineering
   • authentication-authorization
   • multi-tenancy
3. 3

   AI skills

   • ai-integration
   • ai-agents
   • ai-tool-engineering
   • rag-engineering
   • ai-security
   • ai-cost-optimization
4. 4

   SaaS skills

   • payments-billing
   • subscriptions
   • usage-metering
   • audit-logging
   • notifications-email
5. 5

   Production skills

   • background-jobs
   • rate-limiting
   • analytics
   • error-monitoring
   • ai-observability
6. 6

   Delivery skills

   • testing
   • ci-cd
   • deployment
7. 7

   code-review

8. 8

   production-readiness

The orchestrator must not blindly invoke every skill.

It should select skills based on actual requirements.

Section 40

Skill Execution Rules

The AI coding agent should:

1. 01Inspect the existing repository.
2. 02Identify the current architecture.
3. 03Determine which skills apply.
4. 04Read the relevant skill instructions.
5. 05Produce an implementation plan.
6. 06Identify dependencies.
7. 07Implement incrementally.
8. 08Verify each subsystem.
9. 09Run tests.
10. 10Perform security review.
11. 11Perform code review.
12. 12Run production-readiness checks.
13. 13Document changes.

The agent should avoid:

• rewriting working systems unnecessarily
• introducing dependencies without justification
• creating duplicate infrastructure
• bypassing authorization
• exposing secrets
• trusting client-side state
• blindly adding AI agents
• adding databases where simpler storage works
• adding queues where synchronous processing is sufficient

Section 41

Skill Dependency Graph

Skills should be composable.

Example:

Section 42

Engineering Governance

Every production application should have explicit decisions for:

Architecture

• application architecture
• data architecture
• AI architecture
• deployment architecture

Security

• authentication
• authorization
• secrets
• data isolation
• AI security
• tool permissions

Reliability

• retries
• timeouts
• queues
• fallbacks
• backups
• disaster recovery

AI

• model selection
• prompts
• RAG
• tools
• agent autonomy
• evaluations
• cost limits

Business

• plans
• entitlements
• payments
• usage
• refunds
• customer support

Section 43

Production Readiness Levels

Level 0 · Prototype

• Basic UI
• Basic backend
• Basic model integration

No production guarantees.

Level 1 · MVP

• Authentication
• Database
• Basic authorization
• AI integration
• Basic monitoring
• Basic error handling
• Deployment

Level 2 · Production

• Tenant isolation
• Billing
• Usage metering
• Rate limiting
• Observability
• Testing
• Backups
• Audit logs
• Security controls

Level 3 · Scale

• Queues
• Caching
• Advanced monitoring
• AI evaluation
• Cost optimization
• Performance engineering
• Disaster recovery

Level 4 · Enterprise

• SSO
• Advanced IAM
• Private networking
• SIEM
• Compliance
• Dedicated environments
• Advanced governance
• Regional controls
• Formal SLOs

Section 44

Reference MVP Stack

A strong default stack for many modern AI SaaS applications:

Frontend

React + Next.js + TypeScript

Styling

Tailwind CSS + accessible component system

Application

Next.js server-side application layer

Database

PostgreSQL

Authentication

Managed authentication provider

Storage

Managed object storage

AI

Multi-provider model layer

AI SDK

Provider abstraction + streaming + structured outputs + tools

RAG

PostgreSQL + vector extension where appropriate

Background Jobs

Managed durable workflow / job system

Payments

Stripe / PayMongo / Xendit depending on market

Analytics

Product analytics platform

Errors

Error monitoring platform

AI Observability

LLM tracing / evaluation platform

Email

Transactional email provider

Edge

CDN / WAF / rate limiting

Deployment

Managed application hosting

Source Control

GitHub

CI/CD

GitHub Actions + managed deployment pipeline

This is a default, not a mandatory stack.

Section 45

Example End-to-End AI Request

1. 1

   User

2. 2

   Browser

3. 3

   Edge

4. 4

   Authentication

5. 5

   API

6. 6

   Authorization

7. 7

   Rate Limit

8. 8

   AI Service

9. 9

   Context Builder

   • User Context
   • Tenant Context
   • Retrieved Knowledge
   • Conversation State
10. 10

    Model Router

11. 11

    LLM

12. 12

    Tool Decision

13. 13

    Tool Authorization

14. 14

    Tool Execution

15. 15

    Result Validation

16. 16

    Guardrails

17. 17

    Response

18. 18

    Usage Metering

19. 19

    AI Trace

20. 20

    Analytics

21. 21

    User

Section 46

Example Payment Lifecycle

1. 1

   User selects plan

2. 2

   Checkout

3. 3

   Payment Provider

4. 4

   Payment Result

5. 5

   Signed Webhook

6. 6

   Webhook Verification

7. 7

   Billing Service

8. 8

   Subscription State

9. 9

   Entitlement Update

10. 10

    Audit Event

11. 11

    Analytics

The browser must never be the authoritative source of subscription state.

Section 47

Example Document-to-RAG Lifecycle

1. 1

   Upload

2. 2

   Authorization

3. 3

   Object Storage

4. 4

   Metadata

5. 5

   Background Job

6. 6

   Parse

7. 7

   Validate

8. 8

   Chunk

9. 9

   Embed

10. 10

    Index

11. 11

    Retrieval

12. 12

    Permission Filter

13. 13

    Rerank

14. 14

    Context

15. 15

    LLM

16. 16

    Citation

Section 48

Example Agent Lifecycle

1. 1

   Task

2. 2

   Identity

3. 3

   Policy

4. 4

   Context

5. 5

   Planner

6. 6

   Tool Selection

7. 7

   Authorization

8. 8

   Tool

9. 9

   Observation

10. 10

    Evaluation

    • Continue
    • Retry
    • Escalate
    • Complete
11. 11

    Audit

12. 12

    Trace

13. 13

    Result

Section 49

What Should NOT Be Combined

Some responsibilities should remain independent.

• Authentication≠Authorization
• Payment status≠Entitlement
• Product analytics≠Security logs
• Application logs≠AI traces
• RAG≠Memory
• AI model≠Business logic
• Agent≠Workflow engine
• Prompt≠Authorization policy
• WAF≠Application security
• Database backup≠Disaster recovery

Keeping these boundaries clear makes systems easier to reason about and safer to operate.

Section 50

Core Engineering Principles

1. 01The model is not the architecture.
2. 02Authorization must be enforced outside the model.
3. 03Every tenant boundary must be explicit.
4. 04Payments must be server-authoritative.
5. 05AI usage must be measurable.
6. 06AI actions must be observable.
7. 07Prompts must be versioned.
8. 08Tools must have explicit permissions.
9. 09High-risk actions require stronger controls.
10. 10RAG requires provenance and access control.
11. 11Long-running work belongs in durable workflows.
12. 12Failures must be expected and recoverable.
13. 13Production AI requires evaluation, not just testing.
14. 14Use the simplest architecture that satisfies the requirement.
15. 15Skills should teach capabilities before vendors.
16. 16Every implementation should be verifiable.
17. 17AI agents should earn autonomy through reliability.
18. 18Do not add infrastructure without a measurable reason.
19. 19Operational visibility is part of the product.
20. 20Production readiness is a gate, not a feeling.

Section 51

Master Production Checklist

Product

• Requirements defined
• User journeys defined
• Roles defined
• Entitlements defined
• Failure states defined

Architecture

• Architecture documented
• Data model documented
• Integration boundaries documented
• Deployment model documented

Security

• Authentication
• Authorization
• Tenant isolation
• Input validation
• Secrets management
• Rate limiting
• AI security
• Tool permissions
• Audit logging

AI

• Model abstraction
• Prompt versioning
• Token budgets
• Cost tracking
• RAG permissions
• Tool validation
• Agent limits
• Guardrails
• Evaluation suite
• AI tracing

SaaS

• Billing
• Webhooks
• Subscription state
• Entitlements
• Usage metering
• Refund handling
• Customer support

Reliability

• Timeouts
• Retries
• Idempotency
• Background jobs
• Failure handling
• Backups
• Disaster recovery

Observability

• Structured logs
• Error monitoring
• Metrics
• Distributed tracing
• AI traces
• Cost monitoring
• Product analytics
• Alerts
• Status page

Delivery

• Automated tests
• AI evaluations
• CI/CD
• Preview environments
• Database migration strategy
• Rollback strategy
• Production smoke tests

UX

• Loading states
• Empty states
• Error states
• Accessibility
• Responsive design
• Mobile experience
• AI streaming UX
• Long-running task UX

Section 52

Final Reference Architecture

Section 53

The Core Mental Model

The modern AI application should be understood as:

Build applications where AI is one replaceable, observable, cost-controlled subsystem among identity, data, payments, security, and operations.

Section 54

Recommended Implementation Order

For a new AI SaaS product:

1. 01Product requirements
2. 02Architecture design
3. 03Database design
4. 04Authentication
5. 05Authorization
6. 06Multi-tenancy
7. 07Core application
8. 08File storage
9. 09AI integration
10. 10RAG / tools / agents as required
11. 11AI security
12. 12Usage metering
13. 13Payments
14. 14Background jobs
15. 15Observability
16. 16Analytics
17. 17Testing
18. 18CI/CD
19. 19Backup / recovery
20. 20Production readiness

Do not start with agents because the product is labeled "AI."

Start with the business workflow.

Then determine where AI earns its place.

Section 55

Putting it together

The Production AI Application Stack is a reference architecture and engineering operating model, not a fixed list of vendors.

Its purpose is to provide a repeatable way to build:

• AI SaaS products
• AI copilots
• internal enterprise applications
• RAG systems
• agentic applications
• workflow automation
• AI-powered business platforms
• AI-integrated traditional software

The architecture should evolve with the product.

The skills system ensures that AI coding agents can participate in this process without reducing software engineering to code generation.

The target state is:

1. 1

   Human Intent

2. 2

   Architecture

3. 3

   Capability Skills

4. 4

   Provider Selection

5. 5

   Implementation

6. 6

   Verification

7. 7

   Evaluation

8. 8

   Production

9. 9

   Continuous Improvement

Production AI engineering is more than prompt + model. It is software engineering, data engineering, security, AI engineering, payments, observability, operations, and continuous verification.