AKS in Production: Architecture Patterns for Reliable, Secure Kubernetes on Azure

Running a Kubernetes proof of concept is straightforward. Getting a cluster built with az aks create takes ten minutes, your application deploys, and everything works. Then you take it to production — and you discover that reliable, secure, production-grade Kubernetes is an entirely different engineering discipline.

Azure Kubernetes Service (AKS) abstracts the control plane, but the responsibility for workload reliability, security, networking, and operational posture is entirely yours. This article covers the patterns and decisions that separate production AKS deployments from ones that generate 3am pages.

Cluster Architecture Fundamentals

Node Pool Design

The most important structural decision in AKS is how you segment workloads across node pools. Mixing all workloads on a single node pool is a reliability, security, and cost problem simultaneously.

The recommended pattern:

A dedicated system node pool ensures Kubernetes infrastructure components are never evicted by resource-hungry application workloads. This is not optional for production; without it, CoreDNS eviction will make your cluster unreachable.

Availability Zones

Enable Availability Zone distribution across all node pools by specifying --zones 1 2 3 at creation time. This ensures that an AZ failure eliminates at most one-third of your nodes, rather than all of them.

Managed Identity and RBAC

Use System-Assigned Managed Identity for the cluster identity (or User-Assigned if you need predictable client IDs). Never use service principal credentials — they expire, require rotation, and introduce credential management overhead.

Enable Azure RBAC for Kubernetes (--enable-azure-rbac). This means Kubernetes RBAC is enforced through Azure AD group membership, giving you a single identity control plane for Azure resources and Kubernetes API access.

---

Networking: Choosing the Right CNI

The Container Network Interface (CNI) choice affects everything: IP address management, network policy capabilities, performance, and supportability.

Azure CNI vs. Kubenet vs. Azure CNI Overlay

Azure CNI assigns real Azure VNet IP addresses to every pod. Every pod is directly routable from the VNet. This provides maximum compatibility and performance but consumes IP addresses rapidly — a 100-node cluster with 30 pods per node needs 3,000+ IP addresses.

Kubenet uses NAT — pods get private IPs from a small cluster-internal range, and AKS NATs to the node IP. Lower IP consumption but higher latency and compatibility limitations.

Azure CNI Overlay (recommended for new deployments): pods get IPs from an overlay network, decoupled from the VNet address space. Solves the IP exhaustion problem of Azure CNI while maintaining performance and network policy support. This is the modern default for most workloads.

For high-performance workloads requiring direct pod-to-pod routing across nodes, Azure CNI with DPDK or eBPF-based Cilium provides additional performance headroom.

Network Policy

Enable network policy from cluster creation — this cannot be added later without recreating the cluster. Use Azure Network Policy Manager (built-in, integrates with Azure NSGs) or Cilium (more powerful, eBPF-based) depending on your requirements.

A default-deny policy in every namespace is a security baseline:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

With default-deny in place, every inter-service connection requires an explicit allow rule — making your network access model auditable and intentional.

---

Workload Security

Pod Security Standards

Enable Pod Security Admission at the cluster level. For production workloads, enforce the restricted standard: no root containers, read-only root filesystems, no privilege escalation, no host network/PID/IPC access.

For legacy workloads that can't yet meet restricted, apply baseline as a minimum, which blocks the most dangerous capabilities while allowing many real-world applications.

Workload Identity

Replace service account token files and hardcoded credentials with Azure AD Workload Identity. This allows pods to obtain short-lived Azure AD tokens using a federated trust with Kubernetes service accounts — no secrets required.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
  annotations:
    azure.workload.identity/client-id: "<managed-identity-client-id>"

The pod inherits Azure RBAC permissions from the managed identity, tokens rotate automatically, and there are no secrets to manage or leak.

Secrets Management

Do not store sensitive configuration in Kubernetes Secrets (base64 is not encryption). Use the Azure Key Vault Provider for Secrets Store CSI Driver to mount Key Vault secrets directly into pod filesystems as files or environment variables. Secrets are fetched at pod start time, Key Vault handles rotation, and pods automatically pick up new values on restart.

---

Reliability Patterns

Pod Disruption Budgets

Cluster upgrades, node pool scaling events, and Spot node evictions all involve draining pods. Without a PodDisruptionBudget (PDB), all instances of a deployment can be terminated simultaneously, causing an outage.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-application

Every production deployment should have a PDB that ensures at least one (preferably two) instances remain available during disruptions.

Resource Requests and Limits

Set both CPU and memory requests and limits on every container. Without requests, the Kubernetes scheduler can't make good placement decisions. Without limits, a single misbehaving container can exhaust node resources.

The pattern: set memory requests equal to the 95th percentile observed usage, memory limit to 1.5× the request. Set CPU request to observed average, CPU limit to 2× the request. These are starting points — tune with actual production metrics.

Horizontal Pod Autoscaler + KEDA

HPA scales based on CPU or memory utilisation. This is sufficient for many workloads but has a minimum response time of ~30 seconds.

KEDA (Kubernetes Event Driven Autoscaling) scales based on external metrics: Azure Service Bus queue depth, Azure Event Hubs lag, HTTP request rate. This enables scaling to zero for batch workloads and more responsive scaling for queue-driven services. KEDA is an AKS add-on; enable it with az aks enable-addons --addons keda.

---

Observability

Production AKS requires full observability across three layers: cluster, node, and workload.

Cluster layer: Enable Azure Monitor managed service for Prometheus and Azure Managed Grafana. This provides pre-built dashboards for cluster health, node utilisation, and Kubernetes control plane metrics with no configuration required.

Application layer: Instrument workloads with Application Insights SDK (or OpenTelemetry with Application Insights exporter) for distributed traces, custom metrics, and dependency tracking.

Log Analytics: Enable Container Insights add-on (--enable-addons monitoring) for structured log collection from all pods. Configure diagnostic settings on the AKS control plane to export API server logs to Log Analytics — this is essential for auditing and incident investigation.

Alert rules you need from day one:

• Node CPU/memory utilisation > 80% for 10 minutes
• Pod restart count > 5 in 1 hour (crashlooping)
• Deployment unavailable replicas > 0 for 5 minutes
• Control plane API server error rate > 1%
• Persistent volume claim space < 20%

---

Cluster Upgrades

AKS releases new Kubernetes versions regularly, and version N-2 reaches end of support. Kubernetes upgrades are not optional — they're a recurring operational obligation.

The upgrade strategy:

1. Enable automatic channel upgrades — Set --auto-upgrade-channel patch to automatically apply patch-level upgrades. This handles security patches without manual intervention.
2. Upgrade staging clusters first — Validate the new version against your workloads before upgrading production.
3. Use the cluster upgrade command, not node pool upgrades — az aks upgrade coordinates control plane and node pool upgrades correctly.
4. Ensure PDBs are in place before upgrading — the upgrade process drains nodes, and without PDBs you will have downtime.

Node OS updates: Enable Node OS Automatic Upgrades for security patches to the underlying VM OS between Kubernetes version upgrades.

---

Cost Optimisation for AKS

Cluster Autoscaler

Enable the Cluster Autoscaler on all node pools. Configure aggressive scale-in: scale-down-delay-after-add: 10m, scale-down-unneeded-time: 10m. Most teams leave the default 30-minute values, which means your cluster stays over-provisioned for 30 minutes after every scale-out event.

Spot Node Pools

Add a Spot node pool for fault-tolerant workloads: batch jobs, preprocessing pipelines, non-critical background workers. Spot pricing delivers 60–80% savings. Design workloads on Spot nodes to tolerate eviction (use Jobs with retry policies, not Deployments with state).

Namespace-Level Resource Quotas

Implement ResourceQuotas per namespace to prevent any single team from consuming disproportionate cluster resources:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi

---

Running production Kubernetes well is a significant operational investment — but it's also one of the most powerful platforms available for deploying complex, scalable workloads. The patterns above represent the baseline for a reliable, secure, and cost-efficient AKS deployment.

If you're planning an AKS migration or experiencing reliability or cost issues with an existing cluster, let's talk.