It usually starts the same way.An application goes down. Users report errors. A alert gets fired. Someone opens the Azure portal, someone else starts checking logs, and within minutes the team is juggling dashboards, alerts, traces, deployment history, configuration settings, and GitHub commits — all while pressure keeps rising.

At first glance, modern cloud-native systems look highly observable. We have Azure Monitor, Application Insights, Datadog, Prometheus, New Relic, distributed tracing, metrics, logs, alerts, and dashboards everywhere. But when a real incident happens, most teams still fall back to the same manual process: hunt for symptoms, jump across tools, form a hypothesis, test it, repeat.

That approach breaks down quickly in environments built on microservices, containers, App Services, multiple data sources, and 24/7 uptime expectations. The problem isn’t that we lack telemetry. The problem is that humans still have to correlate everything fast enough to reduce customer impact.

Not as a chatbot.
Not as “AI for the sake of AI.”
But as an operational teammate that can reason through incidents, use prior knowledge, investigate across telemetry and code, and help engineers move from symptoms to root cause much faster.

In this post, I’ll walk through two practical scenarios from my lab environment:

1. A .NET application outage returning HTTP 503 errors

2. Autonomous handling of a high response latency time alert

The interesting part isn’t that the agent can read logs. Plenty of tools can do that.
The interesting part is how the agent combines telemetry, knowledge files, runtime configuration, and engineering workflows into a single investigation loop.

Why SRE Agent?

Over the years, one pattern has shown up again and again: adding more dashboards and more alerts does not automatically improve reliability. In many organizations, it does the opposite. It creates more noise, more fatigue, and more context switching.

Most incidents are not caused by a single obvious failure. They’re caused by a chain of small things:

• a config drift no one noticed,

• a deployment mismatch,

• an auth change,

• a missing secret,

• a misleading health signal,

• known issue that exists somewhere in a runbook nobody has time to read during an outage.

This is where an SRE agent becomes interesting.Instead of only surfacing symptoms, it can:

• gather context from prior operational knowledge,

• inspect metrics, logs, and platform state,

• correlate signals across layers,

• check code and repository history,

• propose or apply remediation,

• and keep the human engineer in control for sensitive actions.

In other words, it doesn’t just help you observe a problem. It helps you investigate it.

This is Part 1 of a 2-part series

Part 1: What Azure SRE Agent can do — real incident investigation and resolution workflows

Part 2: How it works — architecture, configuration, tools, memory, permissions, and implementation details

Real world Incident Scenarios:

Scenario 1: Application Not Accesible with HTTP 503 error

The first scenario starts with a very common but frustrating production symptom: the application is down, but the platform looks fine.

I had a .NET application running on Azure App Service, backed by Azure SQL. When I opened the site, instead of the application UI, I was greeted with a generic application error page.

At this point, nothing about the error was especially helpful. It told me the app was broken, but not why.

So I did what most engineers do first: I checked the Azure portal.

The initial signal was misleading. App Service was running. The database was running. Basic resource health didn’t immediately scream “critical platform issue.” There were HTTP 5xx errors visible in monitoring, but CPU and memory were not showing the kind of dramatic spike you’d normally expect during a major failure.

This is exactly the kind of situation where manual incident response starts to drag:

• the platform is “up,”

• the app is effectively down,

• the dashboards are incomplete,

• and the root cause is hiding somewhere between runtime, config, and code.

So instead of manually hopping across five different tools, I asked the Azure SRE Agent to investigate:

“My dotnet application is not working, I see application error on webpage, can you investigate and find out why is that?”

What happened next is what made the experience feel different from traditional monitoring.

The agent did not begin with blind trial and error. It first pulled in context from its knowledge files , specifically files like debugging.md and deployment.md. That’s important, because real incident response is rarely just about live telemetry. It also depends on what the team already knows about the service.

In my setup, the agent’s operational memory is not stored as a messy chronological log. It is organized semantically by topic.

That structure matters. Instead of searching through one giant document, the agent can jump directly into focused knowledge areas such as:

• service overview,

• team and ownership,

• architecture,

• logs and queries,

• deployment details,

• auth behavior,

• debugging patterns,

• and known issue references.

That gives the agent something most dashboards do not: historical operational context.

In this case, the knowledge files already referenced a known recurring issue around SQL authentication. So the agent used that as a working hypothesis , but importantly, it didn’t stop there. It still moved on to validate the application’s real-time state through telemetry and platform inspection.

The agent checked:

• App Service health and metrics,

• container/runtime configuration,

• logs,

• startup behavior,

• and application exceptions.

It noticed that CPU and memory were not showing meaningful activity, which suggested something deeper than a normal runtime load problem. It also recognized that the container might not actually be starting correctly.

Because I had configured the agent in review mode, it did not make changes silently. When it needed to access or modify configuration, it explicitly asked for approval.

That review model is a big part of what makes this usable in real enterprise environments. The value is not “uncontrolled automation.” The value is fast investigation with human guardrails.

Once approved, the agent started walking through App Service configuration, logs, and runtime settings.

The first clear issue it found was not in the application code at all — it was in the hosting configuration.

The container image exposed port 8080, but the App Service configuration was missing the required WEBSITES_PORT setting. Even if the app code and database auth had been correct, App Service would still not have routed traffic properly to the container.

At the same time, the agent also confirmed the earlier knowledge-based suspicion: the application was using SQL authentication, while the SQL server was configured in a way that conflicted with that access pattern.

This is where the investigation started to become more powerful than a typical dashboard workflow.

A human engineer under pressure might fix the first visible issue and move on. But the agent continued reasoning through the dependency chain:

• one issue in container routing,

• another in database authentication,

• and likely more hidden misconfigurations behind them.

It applied the first round of fixes and re-validated the service.

But the application was still returning HTTP 503.

That’s the point where many incident bridges get stuck one fix has been applied, the expectation is recovery, and when recovery doesn’t happen, the whole investigation needs to widen again.

The agent did exactly that.It went back into the App Service configuration and noticed something subtle but critical: the DOCKER_REGISTRY_SERVER_PASSWORD setting was effectively empty. That meant the container could not pull its image from Azure Container Registry correctly.

Now the picture was clear. This outage was not caused by one dramatic failure. It was caused by multiple compounding misconfigurations:

1. Missing container port configuration (WEBSITES_PORT)

2. SQL authentication mismatch

3. Missing container registry credentials

4. And, as the final investigation summary confirmed, a connectivity issue related to SQL access configuration as well

After applying the necessary fixes, the agent validated the service again.

This time, the results were exactly what you want to see during an incident: the application recovered, the endpoints returned HTTP 200, and the UI became available again.

What I liked most about this scenario is that the agent did not behave like a scripted “runbook bot.” It behaved more like a junior-but-fast SRE partner:

• it started from known issues,

• validated live telemetry,

• checked config,

• correlated runtime behavior,

• asked for approval before making changes,

• and kept investigating even after the first fix did not fully resolve the incident.

The final recommendation was also the right long-term one:
instead of relying on fragile SQL credentials, move the application toward Managed Identity / Entra-based authentication to reduce future failures.

That is what a good SRE workflow should do: not only restore service, but also point toward a more durable operating model.

Scenario 2: Autonomous Handling of Incident

The first scenario showed guided investigation in review mode.
The second scenario shows what happens when the agent is connected to an incident platform and can begin working automatically from an alert.

This time, the trigger was not a user-facing outage page. It was a Sev2 Azure Monitor alert for high response time.

The alert indicated that requests were averaging over 5 seconds. That kind of alert is common — and often noisy. Sometimes it points to a real production issue. Sometimes it reflects a spike, a bad test path, or a monitoring rule that needs more context.

What matters is how quickly the investigation can distinguish between those possibilities.

As soon as the alert was detected, the SRE Agent acknowledged it and created an investigation plan.

This part is important.

Instead of immediately jumping to one likely cause, the agent laid out a structured plan:

1. gather knowledge from memory files,

2. search past incidents,

3. inspect the currently affected app,

4. correlate recent changes,

5. and then mitigate.

That sequencing mirrors how a strong human SRE would approach the issue — but the agent can do it in parallel and without fatigue.

During investigation, it pulled from prior knowledge, reviewed logs, and correlated the alert with application behavior. It discovered that the high response time was not simply a generic infrastructure slowdown. A key part of the latency was being driven by chaos/testing endpoints being hit and by application behavior that still needed code fixes.

The agent did something particularly valuable here: it connected the incident to the engineering workflow, not just to the monitoring workflow.

It created a GitHub issue summarizing the incident, identifying the root cause, and documenting the remediation path.

That issue became more than an alert record. It became an engineering artifact:

• what happened,

• what caused it,

• what had already been investigated,

• and what change was needed next.

The agent then moved toward code-level remediation:

• it identified the needed changes,

• prepared a fix branch / PR flow,

• and connected the operational incident to the software delivery pipeline.

The remediation path included both monitoring-side correction and application-side correction:

• refining the alert rule logic so chaos endpoints did not distort signal quality,

• and updating application behavior to reduce timeout-related failure patterns.

What makes this scenario compelling is that the agent did not stop at “I found the problem.” It helped push the process all the way toward:

• alert understanding,

• signal cleanup,

• code fix identification,

• deployment workflow,

• and verification.

That is a very different experience from the traditional pattern where one tool raises the alert, another tool shows logs, a third tool tracks the issue, and a fourth system eventually gets the fix.

Here, the incident workflow becomes much more continuous.

What Scenario 2 Demonstrates

If Scenario 1 was about guided troubleshooting, Scenario 2 is about something broader: operational orchestration.

The agent is not just a log reader. It becomes a bridge between:

• observability,

• incident handling,

• historical context,

• GitHub engineering workflows,

• and remediation execution.

That’s where the SRE model starts to evolve.

Instead of engineers spending most of their time gathering context, the agent does the context gathering first. The human engineer can then focus on:

• approving sensitive changes,

• validating business impact,

• reviewing code changes,

• and making the final operational judgment.

That shift matters because the biggest cost in many incidents is not only the outage itself — it is the time lost assembling the investigation.

Why THIS Matters

There’s a lot of hype right now around AI in operations. Some of it is useful. A lot of it is vague. That’s why I prefer to evaluate these systems through practical incident scenarios.

For me, the interesting question is not:

“Can AI read logs?”

The interesting question is:

“Can an agent meaningfully reduce investigation time, correlate across layers, and safely help engineers move toward resolution?”

From these scenarios, the answer looks increasingly like yes , with the right guardrails.

What stood out to me most was not raw automation. It was the combination of:

• memory (knowledge files and past learnings),

• reasoning (moving beyond one-signal diagnosis),

• workflow integration (Azure resources + GitHub),

• and human control (approval gates for risky actions).

That combination is what takes an SRE agent beyond a fancy assistant and makes it operationally relevant.

Conclusion

Traditional monitoring stacks are very good at telling us something is wrong.

What they usually do not do well is help us answer, quickly and reliably:

• What exactly failed?

• What changed?

• Is this a known issue?

• Is it platform, config, code, auth, or deployment?

• What is the safest next action?

• And how do we prevent this from happening again?

That is the gap Azure SRE Agent begins to fill.

In the first scenario, it helped trace a 503 outage across telemetry, memory, config, registry credentials, and SQL access issues.
In the second, it moved from alert detection into structured incident investigation and GitHub-driven remediation.

Recently, a customer asked me a question that really got me thinking:: "Can we build an AI agent that integrates with our datacenter — one that IT admins talk to in plain English, and it figures out which tools to run, in what order, to get the job done?"

Not a chatbot. Not a script wrapper. An agent that reasons about the problem, picks the right automation tools,runbooks & executes them, and explains what it found — all while respecting Security controls & ITSM policies

So I built one as MVP for Enterpise customer IT admins, Here are details of Solutions

Solution Overview

The solution is an AI-powered IT Operations agent that manages on-premises servers through conversation. You type "Why is my ABC server running slow?" and the agent SSHs into the server, runs diagnostics, analyzes the results, and tells you the root cause — in under few seconds depeneds on nature of questions

Following are high level components of this solution

Azure Arc : It Act as bridge between Azure and Datacenter servers, Your servers stay in your datacenter. Azure Arc installs a lightweight agent that creates a secure outbound private connection to Azure — no VPN, no public IPs, no firewall holes. The AI agent reaches your server securely through this relay using SSH

MCP Server: Building an MCP server for Azure Arc using fastMCP SDK , transform the API into two core AI-accessible components: Tools & Resources while Azure Arc API operations involve complex resource IDs and specific command syntax. An MCP server abstracts this so you can simply ask, "Which of my Arc servers in the datacenter have missing security patches"

AI Agent: Build using Foundry Agent SDK and hosted on Foundry Agent service, The agent receives your request, decides which tools to call and in what order, interprets the results, and follows up if needed. Ask it to investigate high CPU — it runs a full diagnosis first, spots a memory pressure error in the output, then pulls the event logs to confirm the root cause. That multi-step reasoning is what separates an agent from a converational bot

WebApp: Chat UI Interface which interacts with AI Agent running in Foundry Agent service runtime, it serves the chat UI, and it handles the agent conversation loop . When you type a message, the backend sends it to the Foundry Agent, waits for tool call requests, executes them by calling the MCP server's REST API, submits results back to the agent, and streams the final reply to your browser

Durable Functions: Alerts for Arc server configured in Azure monitor, when something goes wrong on Arc enable server — high CPU, a service crashes, disk filling up — Azure Monitor fires a webhook to a URL. That URL needs to be something that's always listening and ready to receive it.Functions gives us that always-on webhook endpoint without running a dedicated server 24/7. It sits idle costing nothing until an alert arrives, spins up in milliseconds, hands the alert to the agent, and goes back to sleep

Four Major componenst of Architecture:

Architecture Flow

Key Design Decisions

Azure Arc:
Azure Arc Agent enable onpremise servers to become Az ARM resource, which allow to managed Arc enable using Azure supported interface like UI, CLI and Rest APIs security without being exposed to internet

Foundry Agent Service :
Zero infrastructure to manage. No GPU provisioning, no container orchestration for the AI runtime.Built-in conversation state. Foundry manages threads natively — each conversation has persistent memory across messages.Native tool-calling orchestration. Enterprise identity and compliance, agent authenticates via Managed Identity and Azure RBAC — same identity layer as every other Azure resource

SSH over Run Command APIs:
Azure Arc gives you two ways to execute commands on your on-prem server.
Run Command (Azure REST API) — sends a PowerShell script to Azure, which queues it on the Arc agent, which executes it, which reports the result back through Azure. Round trip: 45-90 seconds. Every single time.

SSH via Arc — opens a real-time SSH tunnel through the Arc relay. The MCP server runs az ssh arc, connects to the server's SSH daemon, executes the PowerShell command, gets the output. Round trip: 5-7 seconds. That includes the relay setup, SSH handshake, and command execution.SSH user can be restricted to specific OS-level groups (Event Log Readers, Performance Monitor Users)

Security Layers

All Authentication and Authorization are faciliated via Entra ID using managed identity of Azure, no local creds or Hardcoded credentails

Layer 1 : Identity & Authentication

Managed Identity (All Cloud Components)

Layer 2: Authorization

Layer 3: Network & Transport

Layer 4: OS/App Controls

Pre-Requisites

• Azure Subscription

• Onpremises Windows/Linux Server onboarded to Azure via Arc Agent

• Az AI foundry Project with Reasoning LLM models (GPT models)

• Python 3.11

💡 Tip:** If you don't have an Arc server yet, you can use Azure Arc Jumpstart to set one up quickly with ArcBox.

Walkthrough of Solution Demo:

Part 1: Read-Only Operations (no ticket required)

First, we demonstrate read-only operations — health checks, diagnostics, event log queries, service status. These run freely without any change ticket because they don't modify the server.

Health-Check:
Question on health check of server triggers tools call in MCP for server health check which perform several steps on server and responsd with results with Server Perfromance, Last errors and its service review

Part 2: Change-Controlled Operation (ticket required)

Next, an admin requests a risky operation — restarting a critical service. The agent does not proceed. Instead, it informs the admin that this is a change-controlled operation and asks for a valid ServiceNow CHG or INC ticket number.

Part 3: Ticket Validation — Denied

The admin provides a ticket number. The agent validates it against ServiceNow and finds the ticket is pending approval — not yet authorized for execution. The agent politely denies the request and asks the admin to get the ticket approved first.

Part 4: Ticket Approved — Execution

The admin returns after the ticket is approved. The agent re-validates against ServiceNow, confirms the ticket is now approved, and proceeds to execute the service restart. It reports the result back to the admin with the ticket number referenced for audit

For more details on Internal and source code of this solution,check out github repo:
Osshaikh/IT-Ops-Agent

There are multiple options on Hosting Application leveraging Oracle as DB, Based on complexity i have categorised this into different solution categories.

Oracle on Native Azure VM

In Rehost or ‘Lift and Shift’ scenarios, on-premises workloads are migrated to run on Azure Native Virtual Machines (VMs). This approach involves moving your Oracle Database (DB) to standard Azure VMs, which you manage yourself, offering flexibility and cost savings compared to fully managed services.

Recommending native Azure VMs for Oracle workloads is appropriate in specific scenarios, balancing trade-offs between performance, management effort, and cost. Here are the key use cases:

• Small to Medium Workloads: Ideal for databases where high performance isn’t critical, such as smaller Online Transaction Processing (OLTP) or analytical workloads. Standard VM hardware often suffices, reducing costs compared to Exadata-based managed services like Oracle Database@Azure.

• Cost-Sensitive Projects: When budget constraints are a priority, native VMs avoid the premium pricing of managed services and specialized hardware. Using constrained core vCPUs can further lower Oracle licensing costs, making this option attractive for budget-conscious projects. (Architectures for Oracle Database on Azure Virtual Machines)

• Custom Configurations: If you require specific hardware or software setups unavailable with managed options—such as custom OS images or unique storage configurations—native VMs provide the flexibility to meet those needs.

• Non-Critical Applications: Suitable for development, testing, or less critical applications where downtime or performance hiccups are tolerable. This includes environments where experimentation is common, and managed services might be excessive.

• Full Control and Compliance: When regulatory or compliance requirements (e.g., data residency laws or security policies) demand complete control over the database environment, native VMs allow you to configure settings like Network Security Groups and Azure Firewall to align with those needs. (Overview of Oracle Applications and Solutions on Azure)

Oracle Licensing on Native Azure VMs

There are some constraints around Oracle Database licensing when running on native VMs in cloud environments like Azure or GCP. Microsoft Azure follows this licensing model:

• With Multi-Threading Enabled: Two vCPUs are counted as equivalent to one Oracle Processor license.

• With Multi-Threading Disabled: One vCPU equals one Oracle Processor license.

For this reason, we recommend using constrained vCPU VM series.These VMs offer the same underlying memory, throughput, and network capabilities as standard VMs but with a limited vCPU count. This helps fit within Oracle’s per-core licensing model, reducing costs.

How Constrained VMs Work:

• In a hyper-threaded VM, Azure allocates 2 threads per core, known as vCPUs.

• When hyper-threading is disabled, Azure presents single-threaded cores (physical cores), resulting in a 1:1 vCPU-to-CPU ratio.

Recommended Constrained VM SKUs for Oracle: Check the list of available sizes here: List of Available Sizes with Constrained vCPUs.

Pros

• Ideal Migration Scenarios: Perfect for cases where Oracle is currently running on Hyper-V or physical servers. Azure’s right-sizing tools, paired with constrained VM SKUs, make it a seamless fit for these environments.

• High Availability (HA): Oracle Databases in a Real Application Clusters (RAC) setup are used for HA. While RAC isn’t officially certified by Oracle on Azure, you can achieve HA using Azure’s native capabilities, such as Availability Zones and Oracle Data Guard replication across zones. This provides redundancy at both rack and datacenter levels.

• Cost Reduction: Using constrained VM SKUs reduces Oracle licensing costs without significantly impacting performance for many workloads.

• Licensing Optimization: Disabling multi-threading or hyper-threading on regular Azure VMs adjusts the CPU ratio to 1:1, further lowering license costs.

Cons

• Licensing Complexity: Oracle’s licensing rules can be challenging in cloud environments. You’ll often need to navigate Bring Your Own License (BYOL) scenarios and ensure VM configurations align with Oracle’s core-counting policies.

• RAC Limitations: Although high availability is achievable with Azure’s native features, RAC is not officially certified by Oracle on Azure, which might deter some users relying on official support.

Oracle on AVS (Azure VMware Solution)

In Scenario where customers are running their Oracle Databases on VMware environment, easiest migration path would be moving VMware on cloud with easier lfit & shift approach using VMware VMotion or Bulk migration method.

Although there are license contrains here as well, will though details one by one

Oracle does not recognize VMware as a hard partitioning technology, which has significant implications for licensing. Hard partitioning allows users to license only a subset of a server’s processors, thus offering more control and potentially lowering costs. However, since Oracle considers VMware a soft partitioning tool, customers must license all physical cores accessible by the Oracle software.

Careful management of VMware clusters is essential to minimize licensing costs. Cluster segregation can be an effective strategy to avoid having to license an entire vCenter. Organizations can limit the number of physical cores that need to be licensed by isolating clusters that run Oracle software. This requires rigorous VMware management practices to ensure that Oracle workloads are restricted to designated clusters and do not migrate freely across environments that are not fully licensed.

Key Points

Oracle makes a distinction between soft partitioning and hard partitioning in its licensing model:

• Soft Partitioning: Technologies that allow for dynamic allocation of resources but do not provide Oracle-recognized methods to limit the number of processors being used. VMware falls under this category, requiring licensing of all cores accessible.

• Hard Partitioning: Oracle-approved methods to limit resource usage to specific cores, such as Oracle VM Server, IBM LPAR, and certain physical hardware partitioning tools. These technologies allow sub-capacity licensing, making them more cost-effective in environments with fewer Oracle workloads.

For example, consider a 3-node AVS cluster using AV36P nodes, where each node has 36 physical cores:

• Total cores = 3 nodes × 36 cores/node = 108 cores.With a core factor of 0.5 (typical for Intel CPUs with hyper-threading), the number of processors to license = 108 × 0.5 = 54 processors.

• Oracle Database Enterprise Edition costs approximately $47,500 per processor (per Oracle’s Technology Global Price List). Thus, the licensing cost = 54 × $47,500 = $2,565,000.

Example: If a company has Oracle deployed on a VMware cluster with 3 Hosts, all of those hosts’ cores must be licensed, regardless of how many virtual CPUs are allocated to Oracle in the environment. Even if only a portion of the virtual CPUs is actively running Oracle, the company must account for all the physical hardware.

This makes AVS an expensive option for Oracle workloads unless mitigated by specific licensing agreements, such as an Unlimited License Agreement (ULA).

Strategies to Manage Licensing Costs

To minimize licensing costs:

• Cluster Segregation: Isolate Oracle workloads to a dedicated cluster (e.g., a 3-node AV36 cluster). This limits the number of cores requiring licenses but demands strict management to prevent Oracle VMs from migrating to unlicensed clusters via vMotion.

• Rigorous Management: Use VMware tools to enforce boundaries, ensuring Oracle software does not inadvertently span across the entire vCenter environment, which would require licensing all accessible cores.

Unlimited License Agreements(ULA):

• • ULAs allow unlimited use of specified Oracle software for a fixed term, avoiding per-core or per-cluster licensing.

    • For AVS, the ULA must include a license mobility clause, enabling on-premises licenses to be applied to the cloud. Oracle’s strategic partnership with Microsoft supports this mobility (per the Oracle and Microsoft Strategic Partnership FAQ).

    • This provides flexibility to scale Oracle deployments on AVS without additional licensing costs during the ULA period, making it a cost-effective option for large enterprises.

Pros

• Familiarity and Ease of Use: If your team uses VMware on-premises, AVS lets you stick with the same tools, reducing the learning curve and potential errors for managing Oracle databases.

• Enhanced Security: AVS offers a private cloud on dedicated hardware, providing better isolation for sensitive Oracle data, which is crucial for compliance.

• Similar Setup: AVS mirrors your on-premises VMware environment, so you can move your Oracle VM with minimal changes, reducing migration time and complexity.

• Familiar Tools: Using the same management tools means less learning and fewer errors, speeding up the process.

• Automated Migration: AVS provides tools designed for VMware migrations such vMotion,svMotion, automating much of the work, unlike native VM which may need more manual setup.

Cons:

• Lack of Scalability Savings: Businesses cannot license only the cores they need; instead, they must license all cores that could potentially run Oracle software, regardless of actual use.

• Higher Costs: Organizations often face exponentially higher costs because Oracle insists on licensing all the cores within a given cluster or data center rather than the specific hosts or VMs running Oracle.

• Compliance Challenges: The movement of virtual machines between physical hosts through VMware’s vMotion further complicates compliance, as the potential for Oracle workloads to migrate triggers the requirement to license more physical cores.

• Technically Oracle RAC supported on VMware environment, but as per Oracle policy is not officialy certified to run on non oracle public clouds

Oracle Database@Azure(Exadata)

Oracle Database@Azure is a service where Oracle databases run on Exadata X9M hardware within Microsoft Azure's data centers. This setup ensures low latency access to Azure resources, making it ideal for customers wanting Oracle’s database performance within Azure’s ecosystem.Enterprise-critical features like Oracle Real Application Clusters (Oracle RAC), Oracle Data Guard, Oracle GoldenGate, managed backups, self-managed Oracle Recovery Manager (RMAN) backups, Oracle Zero Downtime Migration (Oracle ZDM), on-premises connectivity, and seamless integration with other Azure services are supported.

Key features include:

• High Performance with Exadata X9M HardwareIt currently runs on Oracle Exadata X9M hardware, with configurations starting at a quarter-rack (minimum 2 database servers and 3 storage servers), scalable up to 32 database servers and 64 storage servers.

• Private Connectivity via Azure Virtual Network: customers can deploy Oracle Exadata within their Azure Virtual Network (VNet), using private IP addresses for secure, isolated access without public internet exposure, Helps meet data residency and security requirements, critical for industries like BFSI

• High Availability: Supports Oracle Real Application Clusters (RAC) and Oracle Data Guard for redundancy and disaster recovery, with built-in redundancy and failover capabilities and offer SLA of 99.9%

• Sub-1 Millisecond Network Latency:When applications and the database are in the same Azure region, network latency can be sub-1 millisecond due to co-location within Azure data centers.

• Integration with Azure Services: Integrates with Azure’s ecosystem, including Microsoft Entra ID for identity management and Azure AI/ML services,Combine Oracle’s database capabilities with Azure’s AI tools for advanced analytics or generative AI applications.

• Simplified Migration and Compatibility: Full compatibility with on-premises Oracle Database and Exadata deployments, supported by tools like Oracle Zero Downtime Migration,Lift and shift on-premises workloads to Azure without refactoring, minimizing migration effort and risk.

• Managed Service with Clear Responsibility Matrix: Oracle manages the Exadata hardware, Oracle Database software, and operating system (Oracle Linux 8.6+), while Microsoft manages the Azure infrastructure, and customers manage data, applications, and VNet.

• Cost Management and Licensing : Available through the Azure Marketplace with options to use existing Oracle licenses (BYOL) or Microsoft Azure Consumption Commitments

• Oracle Database software includes Enterprise Edition, with supported versions from 12c to 23c, customer-selectable based on compatibility and needs.

Responsiblity Matrix for Oracle Exadata

Oracle Software Stack in Exadata

• Oracle Database Software Versions: software includes Enterprise Edition, with supported versions from 12c to 23c, customer-selectable based on compatibility and needs.

• Additonal Components: Data Guard for high availability and disaster recovery & Golden Gate for data replication for DR

• Operating System: Oracle Linux 8.6 Later

• Licensing: Customers can bring their own Oracle Database licenses (BYOL) or purchase them through the Azure Marketplace,

Oracle Exadata X9M Overview

Oracle typically offers the X9M in several rack‑scale configurations:

• Quarter Rack:
  • Typically includes 2 database servers and 3 storage servers.
  • Often used for smaller-scale or departmental workloads.

• Half Rack:
  • Generally configured with 4 database servers and 6 storage servers.
  • Balances performance and capacity for medium‑sized enterprises.

• Full Rack:
  • Usually features 8 database servers and 12 storage servers.
  • Designed for very large, mission‑critical deployments requiring maximum throughput and IOPS.

Hardware Specifications for Oracle DB Exadata(X9M)

Exadata X9M consists of database servers and storage servers, with different configurations for high capacity (HC) and potentially extreme flash (EF)

Database Servers:

• The database servers in Exadata X9M, as used in Oracle Database@Azure, are equipped with advanced compute capabilities. Research suggests the following specifications, based on Exadata Cloud Services X9M documentation:

• Processor: 1 x AMD EPYC 7J13, offering 64 cores and 128 threads. The base clock speed is 2.6 GHz, with a turbo frequency up to 3.5 GHz, providing high performance for database operations (AMD EPYC 7J13 Processor)

• Cores: 64 cores, with 128 threads, supporting high compute workloads.

• Memory: 512 GB, expandable, supporting large in-memory database workloads, as seen in Exadata Cloud Services X9M configurations (Exadata Cloud Services X9M Specifications).

• Storage: 2 x 3.84 TB NVMe Flash SSD for local storage, with the option to expand to 4 x 3.84 TB, ensuring fast access to frequently used data (Oracle Exadata Database Machine Hardware Components).

Storage Server (High Capacity - HC):

• CPU Manufacturer and Family: Intel, specifically the Intel Xeon Scalable 4th generation (codenamed Sapphire Rapids), with the model being Intel® Xeon® Gold 6448Y

• Cores: 32 cores total, with 2 processors each having 16 cores, operating at 2.6 GHz, based on Exadata X9M HC storage server specs

• Memory: 256 GB DDR4 for general operations and 1.5 TB Persistent Memory, allocated for Exadata RDMA Memory (XRMEM)

• Storage: Includes 12 x 18 TB 7,200 RPM SAS HDDs for bulk storage (total raw capacity 216 TB) and 4 x 6.4 TB NVMe Flash SSDs for caching (total 25.6 TB)

Cons:

Cost: Oracle Exadata on Azure uses specialized Exadata hardware and is a managed service, likely resulting in higher costs compared to running Oracle on standard Azure VMs. Customers can choose from a variety of VM sizes in Azure VM, potentially finding more cost-effective options, especially with constrained vCPU VMs to reduce Oracle licensing costs

Scalability: Exadata on Azure offers scalability in predefined rack configurations (quarter-rack to full-rack), which might not be as flexible as scaling individual VM sizes in standard Azure VM. Customers can choose from various VM sizes in Azure VM, including constrained vCPUs for cost savings, offering more granular scaling.

Reference Artcile:
https://docs.oracle.com/en-us/iaas/Content/database-at-azure/oaa.htm
X9M: https://docs.oracle.com/en-us/iaas/exadatacloud/doc/exa-service-desc.html#GUID-EC1A62C6-DDA1-4F39-B28C-E5091A205DD3
License: Counting Licenses on Azure AVS - Palisade Compliance
Implementation: https://docs.oracle.com/en-us/iaas/Content/database-at-azure-exadata/odexa-exadata-services-azure.html

Azure Arc provides a versatile platform that allows you to manage servers running outside of Azure (whether on-premises or in other cloud environments) as if they were native Azure resources. When connecting these servers to Azure, you can choose from several connectivity options:

• Public (Default option)

• Private Link (Partially private)

• Proxy (Onprem)

• Proxy (Azure)

• Arc Gateway (Preview)

1. Public Connectivity

Overview: Public connectivity involves direct internet connections where servers communicate with Azure services over the public internet. This option is typically the easiest to set up, as it doesn’t require any changes to existing network infrastructure or private networking configurations.

Use Cases:

• Quick Deployment: Ideal for rapid deployments or testing environments where security is not the primary concern.

• Remote Locations: Suitable for remote sites where private networking options like VPNs or ExpressRoute are not feasible.

Security Considerations:

• Network Security: Servers must be protected by firewalls and other security measures to mitigate internet-based threats.

• Encryption: Implement TLS 1.2 or higher for data in transit to ensure secure communication.

Advantages:

• Cost-Effective: No additional costs associated with setting up private network connections.

Disadvantages:

• Security Risks: Direct internet connections are more vulnerable to security threats.

• Performance Variability: Potential for variable latency and bandwidth depending on internet conditions.

Configuration: No specific configuration is required other than whitelisting at least 15+ public endpoint URLs.

2. Private Link Connectivity

Overview: Private Link establishes a secure, private connection to Azure services through a private endpoint in your virtual network. This setup ensures that traffic between the server and Azure remains within the Microsoft backbone network, avoiding the public internet.

Use Cases:

• High Security Requirements: Suitable for environments that require stringent security and compliance, such as healthcare and government sectors.

• Regulatory Compliance: Helps meet regulatory mandates that require private network communication for sensitive data.

Security Considerations:

• Private Network: Minimizes exposure to internet-based threats by keeping traffic within a private network.

• Public Authentication Traffic: Note that traffic for authentication (Microsoft Entra ID) and Azure Resource Manager still routes through the public internet, which should be secured.

Advantages:

• Enhanced Security: Provides an additional layer of security by isolating traffic from the public internet.

• Reliable Performance: Offers more consistent and reliable performance by leveraging the Microsoft backbone network.

Disadvantages:

• Complexity: More complex to set up and manage, requiring network configuration changes and potentially additional infrastructure.

• Security: Although there will be still few Microsoft Entra & Azure Resource manager URLs go via public endpoint

• Cost: Involve additional expenses maintaining private endpoint connections.

Configuration: Leverage an existing circuit connection or IPsec tunnel to route Arc traffic privately. Set up a Private Link scope and configure a private endpoint in a subnet that has line-of-sight to the on-premises gateway VNet. For DNS resolution in a hybrid environment, deploy a DNS forwarder solution on Azure and configure conditional DNS forwarding on the on-premises DNS to forward Azure-related requests to the DNS forwarder VM.

• 

  Once private link is enabled, one could verify that name resolution for arc related endpoint (4 URL's) for telemetries are routed via private endpoint

• 

3. Proxy Connectivity

Overview: With proxy connectivity, servers connect to Azure services through a proxy server, which acts as an intermediary. This is particularly useful in environments where direct internet access is restricted or where compliance requires that all traffic passes through a proxy.

Use Cases:

• Controlled Environments: Ideal for environments that restrict direct internet access or require centralized management of outbound traffic.

• Compliance: Ensures that all traffic adheres to corporate security and compliance policies by routing it through a proxy.

Security Considerations:

• Traffic Monitoring: Proxies allow for inspection, logging, and monitoring of outbound traffic to detect and respond to suspicious activities.

• Controlled Access: Proxy servers can enforce policies on what traffic is permitted to reach Azure, enhancing overall security.

• SSL/TLS Inspection: FW/proxies can inspect SSL/TLS traffic to ensure secure communications and prevent data exfiltration.

Advantages:

• Enhanced Security: Provides additional security by isolating traffic from the public internet.

• Compliance: Facilitates compliance with industry regulations and standards.

Disadvantages:

• Complexity: More complex to set up and manage, requiring changes to network configurations and potentially additional infrastructure.

• Cost: May involve additional costs for setting up and maintaining the proxy server infrastructure.

Configuration: Azure Arc agent creates four services on your system, including an Arc proxy. Configure your on-premises firewall or proxy IP address in the Arc agent proxy settings, so that all traffic destined for Azure Arc passes through the proxy server, which then communicates with Azure endpoints on behalf of the Arc agent.

azcmagent config set proxy.url "http://ProxyServerFQDNorIP:port"
azcmagent config get proxy.url

Here is screenshot of on-premises server using proxy to facilitate traffic of Arc Agent
Since I have not whitelisted URL for SQL, traffic for Arc extension for SQL is failing

Sample Architecture for Arc server which could leverage onprem proxy solution:

4. Proxy (Private Connectivity)

Overview: In this scenario, the proxy server is hosted on Azure, and traffic from on-premises servers is routed privately to Azure via ExpressRoute or IPsec VPN. This approach ensures that all communication between on-premises servers and Azure remains private, leveraging the secure connection to the Azure-based proxy before reaching Azure services over the Microsoft backbone network.

Use Cases:

• High-Security Deployments: Suitable for environments that demand to keep traffic off internet or all traffic, ensuring that no data traverses the public internet

• Centralized Security Management: Ideal for organizations that want to centralize outbound traffic management while ensuring private connectivity.

Advantages:

• Enhanced Security: Offers a high level of security by ensuring that all traffic from on-premises servers remains off public internet until it connects Arc service,
  traffic can be routed across multiple layers of firewall in customer DMZ before it connects to Arc/Azure endpoint via Azure backbone network

• Compliance: Meets stringent security and compliance requirements by leveraging private connections.

Disadvantages:

• Complexity: More complex to set up due to the need for private networking configurations, including ExpressRoute or IPsec VPN.

• Cost: Higher costs associated with maintaining private connectivity and the Azure-based proxy infrastructure.

Configuration:

1. Enable Forward Proxy on Azure Firewall (Premium SKU): Configure the proxy service to expose its service over specific ports (e.g., HTTP/HTTPS).

   

2. Configure Proxy in Arc Agent: Use the azcmagent utility on your on-premises servers to configure the proxy settings using the IP and port details of the Azure-based proxy.

   

3. Verify Traffic: Monitor traffic on the Azure Firewall logs, ensuring that it passes through the proxy as configured. Based on the whitelisted rules, allow traffic to the necessary Arc endpoints.

   

4. DMZ Considerations: For added security, route traffic through a DMZ zone firewall before connecting to Arc endpoints via the Microsoft backbone network.
   Sample architecture for secure Arc traffic via proxy which traverse through multiple layers of firewall (ingress, egress) & connect Arc via backbone network

   

5. Arc Gateway (Preview)

Overview:

Managing Azure Arc often involves dealing with numerous endpoints—15+ for basic scenarios, and over 150 for full extension use. The Arc Gateway simplifies this by reducing the number of URLs that need to be whitelisted in enterprise proxies or firewalls.

Components:

• Arc Gateway: A centralized front end for all infrastructure traffic between Azure and on-premises servers.

• Azure Arc Proxy: A component within the Arc agent that routes traffic through the Arc Gateway, reducing the operational burden of managing multiple endpoints.

Use Cases:

• Simplified Management: Ideal for enterprises that want to reduce the operational complexity associated with managing numerous Arc-related endpoints.

• Secure, Centralized Traffic Routing: Ensures all Arc traffic is routed through a single, secure gateway.

Advantages:

• Reduced Complexity: Simplifies endpoint management by limiting the number of required URLs to 7 for all scenarios

• Improved Security: Centralizes traffic management, enhancing security and reducing the attack surface.

Disadvantages:

• Limited Availability: As a preview feature, it may not yet be suitable for production environments without thorough testing.

• Security: All traffic to Arc Gateway would traverse via public endpoint

Configuration:

1. Create Arc Gateway Resource: Set up the Arc Gateway within your Azure environment.

    az connectedmachine gateway create --name demo-gateway --resource-group demo-arc-gw --location eastus2 --gateway-type public --allowed-features '*' --subscription 'subsid'
   

   

2. Enable Association: Associate your Arc-enabled servers with the Arc Gateway.

   

    az connectedmachine setting update --resource-group demo-arc-gw --subscription subscription-name --base-provider Microsoft.HybridCompute --base-resource-type machines --base-resource-name arcbox-win2k19  --settings-resource-name default --gateway-resource-id '/subscriptions/subsid/resourceGroups/demo-arc-gw/providers/Microsoft.HybridCompute/gateways/demo-gateway'
   

3. Initiate Arc Proxy Service: Configure the Arc Proxy to leverage the Arc Gateway, reducing the number of required endpoints.

   

4. Verify Traffic Routing: Use the azcmagent utility to confirm that traffic is being routed through the Arc Gateway, reducing the list of required endpoints to just four.

   

5. Representation of traffic in visual architecture would look similar to this

   

In this blog post, I will demonstrate how to leverage the newly announced GenAI gateway features in API Management (APIM) to enhance the resiliency and capacity of your Azure OpenAI deployments using circuit breaker and load balancing patterns.

Background

Previously, I have shared various methods for using different routing algorithms (weightage/priority) to load balance traffic across multiple OpenAI endpoints to improve resilience and performance. However, these custom solutions often failed under extremely heavy traffic scenarios.

Since I have already discussed the challenges associated with OpenAI TPM and APIM patterns in a previous blog post, this post will focus directly on the solution.

Load Balancing Features in APIM for OpenAI

The solution comprises two main configuration parts: load balancing and the circuit breaker. These configurations can be managed via two methods: the APIM portal (UI) and Bicep (Infrastructure as Code).

Configuration of APIM via Portal (UI)

1. Support for Load Balancing Techniques: Recently announced at the last Microsoft Build event, APIM now supports weighted and priority-based load balancing techniques.

2. Creating Backend Resources: Begin by creating backend resources for each OpenAI endpoint in APIM. For instance, you can add two OpenAI endpoints from different regions.

• Import New API: Use the OpenAI service template to import a new API. Select an OpenAI endpoint and base URL suffix to form the APIM URI path, e.g., https://myapim.azure-api.net/(basesuffix).

• Add each of your Az OpenAI endpoint as backend resource in API management,

  

Configuration of LB & Circuit Breaker via Bicep

Due to the current lack of UI support for load balancer or circuit breaker configurations, these must be done via Bicep (Infrastructure as Code).

Load Balancer Configuration:

1. Define Load Balancer Method: You can configure the load balancer to use either weight or priority, or both. In this demo, we will use weight-based load balancing, where one endpoint will have a higher weight than another.

2. Code Snippet Adjustments: Modify the names of APIM and its backend pools, specifying the backend pools created in the previous step. This example uses two pools, but you can add more as needed.

    resource symbolicname 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' = {
      name: '(APIM-Name)/mybackendpool'
      properties: {
        description: 'Load balancer for multiple backends'
        type: 'Pool'
        pool: {
          services: [
            {
              id: '/subscriptions/(subscripitonid)/resourceGroups/DefaultResourceGroup-SUK/providers/Microsoft.ApiManagement/service/opai-apim/backends/backend1-openai-endpoint'
              priority: 1
              weight: 1
            }
            {
              id: '/subscriptions/(subscripitonid)/resourceGroups/DefaultResourceGroup-SUK/providers/Microsoft.ApiManagement/service/opai-apim/backends/backend2-openai-endpoint'
              priority: 1
              weight: 10
            }
          ]
        }
      }
    }
   

3. Deploy the Bicep Template: Deploy this Bicep template and look for changes in the backend pool to ensure it is deployed successfully.

4. Update API with New Backend Name: Once deployed, you will see the new backend name in APIM backends. Copy that name and replace it in the API you created in the initial steps.

   

     az deployment group create --resource-group Rresource-group-name --template-file apim.bicep
   

5. Circuit Breaker Configuration:

   1. Health Probe Setup: Deploy circuit breaker conditions for each backend pool resource, consisting of a 'failure condition' and a 'retry after' period.

   2. Failure Condition: Define the number of errors observed within a specific duration (e.g., 5 errors in 1 minute) and specify HTTPS error code ranges considered as errors.

   3. Retry After: Specify the duration after which APIM can retry the health probe on a failed backend pool endpoint. In this configuration, error code ranges from 400-599 and a duration of 1 minute are set for each backend pool.

    resource openaiprodbackend 'Microsoft.ApiManagement/service/backends@2023-09-01-preview' = {
      name: 'APIM-name/nameofbackendpool'
      properties: {
        url: 'https://openaiendpointname.openai.azure.com/openai'
        protocol: 'http'
        circuitBreaker: {
          rules: [
            {
              failureCondition: {
                count: 1
                errorReasons: [
                  'Server errors'
                ]
                interval: 'PT1M'
                statusCodeRanges: [
                  {
                    min: 400
                    max: 599
                  }
                ]
              }
              name: 'myBreakerRulePTU'
              tripDuration: 'PT1M'
              acceptRetryAfter: true
            }
          ]
        }
        description: 'OpenAI PTU'
        title: 'openai-PTU'
      }
    }

6. Verification via API

   Since there is no UI to verify the circuit breaker configuration, use the APIM backend REST API to confirm changes. Use Postman or refer to the APIM documentation for the validation tool. Specify the backend name in backendID and the APIM name in serviceName.

7. Bicep configuration is now completed, Next is to test LB & circuit breaker policy in action

   

   

Validation/Test

To test the circuit breaker with load balancing:

1. Induce Errors: Ensure the OpenAI endpoint throws errors within the 400-599 range. Perform a stress test on the OpenAI endpoint preferred in weight/priority, which should result in an HTTP 429 error ("Server is Busy").

2. Test APIM Response: Once the preferred endpoint is unresponsive, make a call via APIM to OpenAI and verify if the response is received from the second endpoint.

3. Example: Perform a stress test on the OpenAI endpoint in EastUS, resulting in a 429 error state. Validate with an ad-hoc call to the EastUS OpenAI endpoint to confirm it fails with 429. At the same time, a POST request to APIM should work fine, as it is handled by the secondary backend, which is the OpenAI endpoint in UKSouth.

4. Backend Endpoint 2: Backend endpoint 2 has a higher weight, located in EastUS. Perform a stress test on this endpoint using Postman.

5. Stress Test Results: Stress test on the OpenAI endpoint in EastUS should result in a 429 error state.

Ad-Hoc Call Validation: Validate with an ad-hoc call to the EastUS OpenAI endpoint to confirm it is failing with a 429 error.

Once the preferred endpoint is unresponsive, make a call via APIM to OpenAI and verify if the response is received from the second endpoint.

In this POST request to APIM should work fine, as it is handled by the secondary backend, which is the OpenAI endpoint in UKSouth.

Conclusion:

In conclusion, using the new GenAI gateway features in API Management can greatly improve the reliability and performance of Azure OpenAI services. By setting up smart traffic routing and safety measures, we ensure that your services stay responsive even under heavy load. This approach helps balance the workload and quickly redirects traffic if something goes wrong, making sure your users have a smooth and reliable experience.

Appendix:
Intelligent Load Balancing with APIM for OpenAI: Weight-Based Routing - Microsoft Community Hub

At last year's KubeCon North America, Microsoft announced the adoption of Karpenter in Azure Kubernetes Service (AKS) as an alternative to the Cluster Autoscaler (CA), referred to as Node Autoprovisioning (NAP). While Cluster Autoscaler has been the default node scaler in AKS/Kubernetes, there have been significant challenges that led to the adoption of Karpenter. This post delves into these challenges and explores how Karpenter addresses them.

Challenges with Cluster Autoscaler

Here is Node Autosclaing flow chart for Cluster-Autoscaler

• Limited to VMSS Groups: Cluster Autoscaler can only operate with Virtual Machine Scale Sets (VMSS) in AKS. Each VMSS consists of a specific group of VM instances with a specific VM SKU, hardware, and CPU:Memory ratio (e.g., Standard D4sv5 with 4 CPUs and 16 GB RAM).

  2. Node Latency: CA triggers the node pool API, which calls the VMSS instance API. This scaling process has latency, taking over a minute for a node to be ready in AKS.

  3. Node Pool Constraints: When deploying new pods, if the existing node capacity is exhausted, CA attempts to spin up a new node of the same VMSS SKU type. If that instance is unavailable, pods remain in a pending state.

  4. Scalability Limitations: CA can only scale up based on specific node pool SKU VMSS availability. It cannot leverage the capacity of other VM SKUs even if they have available resources.

Introducing Karpenter (Node Autoprovisioning)

Karpenter is an efficient node autoscaler for Kubernetes clusters, designed to optimize performance and cost. It can scale up and down worker nodes faster than Cluster Autoscaler and can launch appropriate individual nodes without creating traditional node groups in AKS.

• Key Features of Karpenter:

  • Efficiency: Faster scaling of Kubernetes nodes.

  • Flexibility: Launches nodes without needing VMSS.

  • Cost Optimization: Reduces overall costs and helps with patching of node images and Kubernetes versions.

  • Nodepool YAML based config which defined what types of nodes it can provision

Handling Disruptions

Disruption Controller responsible for Terminating/Replacing nodes in kubernetes cluster.
its uses one of 3 automated methods to finalise which nodes to handle via Disruption controller

• Expiration: Karpenter will mark nodes as expired and disrupt them after they have lived a set number of seconds. this parameters act as TTL for k8s nodes

spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 300s

• Consolidateafter: it used to configure disrupiton interval,amount of time it should wait before considering disruption cycle again

• Consolidation: It operates to actively reduce cluster cost by analyzing nodes
  Consolidation policy has two modes
  a)When Empty: Karpenter will only disrupt nodes with no workloads pods
  b)Whenunderutilized: It will attempt to reduce/replace nodes when underutilised

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: ondemand
spec:
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 60s

Enable NAP(Karpenter) on AKS

There are few pre requisites to enable NAP on AKS

• Install Az CLI with preview extension of version greater than 0.5.17

• Regsiter NAP provides called "NodeAutoProvisioningPreview"

• AKS with network configuration as Cilium + Overlay

Enable NAP on existing AKS cluster
Make sure existing AKS cluster has 'Azure' network plugin with Cilium as Network Policy. Key thing in this command is feature flag '--node-provisioning-mode Auto', which set NAP as default Node Autoscaler

az aks update --name aksclustername --resource-group rgname --node-provisioning-mode Auto

Deploy NAP with new AKS cluster

az aks create --name aksclustername--resource-group rgname--node-provisioning-mode Auto --network-plugin azure --network-plugin-mode overlay --network-dataplane cilium

Verify Karpenter Enablement:

kubectl api-resources | grep -e aksnodeclasses -e nodeclaims -e nodepools

aksnodeclasses                     aksnc,aksncs                        karpenter.azure.com/v1alpha2           false        AKSNodeClass
nodeclaims                                                             karpenter.sh/v1beta1                   false        NodeClaim
nodepools                                                              karpenter.sh/v1beta1                   false        NodePool

Disabling Cluster-Autoscaler

To switch from Cluster-Autoscaler to Karpenter, disable Cluster-Autoscaler on your AKS cluster:

az aks update --name aksclustername --resource-group aksrg --disable-cluster-autoscaler

Deploying a Sample Application

To see Node-Autoprovisioning in action, deploy a sample application:

osama [ ~ ]$ kubectl get nodes
NAME                                STATUS   ROLES   AGE     VERSION
aks-default-h2jxh                   Ready    agent   35m     v1.27.9
aks-nodepool1-41633911-vmss000000   Ready    agent   3d19h   v1.27.9

Scale replicas of Vote Application to trigger scale out events

osama [ ~ ]$ kubectl scale deployment azure-vote-front --replicas=12 -n karpenter-demo-ns
^[[Adeployment.apps/azure-vote-front scaled
osama [ ~ ]$ kubectl scale deployment azure-vote-back --replicas=12 -n karpenter-demo-ns
deployment.apps/azure-vote-back scaled

Verify auto scaling of nodes by reading via karpenter using below kubectl cmd

kubectl get events -A --field-selector source=karpenter --sort-by='.lastTimestamp' -n 10
NAMESPACE           LAST SEEN   TYPE     REASON                  OBJECT                                  MESSAGE
default             50m         Normal   Unconsolidatable        nodeclaim/default-95f54                 SpotToSpotConsolidation is disabled, can't replace a spot node with a spot node
default             50m         Normal   Unconsolidatable        node/aks-default-95f54                  SpotToSpotConsolidation is disabled, can't replace a spot node with a spot node
default             38m         Normal   DisruptionBlocked       nodepool/default                        No allowed disruptions due to blocking budget
default             5m33s       Normal   Unconsolidatable        nodeclaim/default-h2jxh                 Can't remove without creating 2 candidates
default             5m33s       Normal   Unconsolidatable        node/aks-default-h2jxh                  Can't remove without creating 2 candidates
default             2m12s       Normal   DisruptionBlocked       nodepool/system-surge                   No allowed disruptions due to blocking budget
karpenter-demo-ns   63s         Normal   Nominated               pod/azure-vote-front-6855444955-bnq7p   Pod should schedule on: nodeclaim/default-mrh7w
karpenter-demo-ns   63s         Normal   Nominated               pod/azure-vote-front-6855444955-gbwk6   Pod should schedule on: nodeclaim/default-mrh7w
karpenter-demo-ns   63s         Normal   Nominated               pod/azure-vote-front-6855444955-l2bgj   Pod should schedule on: nodeclaim/default-mrh7w
karpenter-demo-ns   63s         Normal   Nominated               pod/azure-vote-front-6855444955-nvc56   Pod should schedule on: nodeclaim/default-mrh7w
karpenter-demo-ns   63s         Normal   Nominated               pod/azure-vote-front-6855444955-22glj   Pod should schedule on: nodeclaim/default-mrh7w
karpenter-demo-ns   63s         Normal   Nominated               pod/azure-vote-front-6855444955-sxdl6   Pod should schedule on: nodeclaim/default-mrh7w
karpenter-demo-ns   63s         Normal   Nominated               pod/azure-vote-front-6855444955-t69w4   Pod should schedule on: nodeclaim/default-mrh7w

Customise Karpenter Config

Karpenter leverage new resource type in kubernetes Kind i.e. Nodepools

• Customise Nodepools: Specific specific VM series or VM family or even Specific CPU or Memory ratio.

• Select node based on features sets like GPU enable or Network Acceleration

• Defined Archiecture of CPU type either ARM or AMD based on capablity of specfic workload

• Architect your nodes for resiliency by configure zone topology

• Limit numbers of CPU & Memory could be utilised from nodes on nodelevel

Here is default Nodepool Yaml for karpenter(NAP), Which has confiuration on Node SKU types and Capacity, Also limit on nodes CPU:Memory along with Weight incase of Multiple nodepools

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 10s
  template:
    spec:
      nodeClassRef:
        name: default

      # Requirements that constrain the parameters of provisioned nodes.
      # These requirements are combined with pod.spec.affinity.nodeAffinity rules.
      # Operators { In, NotIn, Exists, DoesNotExist, Gt, and Lt } are supported.
      # https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#operators
      requirements:
      - key: kubernetes.io/arch
        operator: In
        values:
        - amd64
      - key: kubernetes.io/os
        operator: In
        values:
        - linux
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - ondemand
      - key: karpenter.azure.com/sku-family
        operator: In
        values:
        - E
        - D
      - key: karpenter.azure.com/sku-name
        operator: In
        values:
        - Standard_E2s_v5
        - Standard_D4s_v3
limits:
  cpu: "1000"
  memory: 1000Gi
weight: 100

Using Spot Node with Karpenter

• Add toleration in Sample AKS-Vote application i.e. "karpenter.sh/disruption:NoSchedule" which comes as default in spot node when provision with AKS Cluster

• Please refer my github repo for Application yaml and sample nodepool config

    spec:
          nodeSelector:
            "kubernetes.io/os": linux
          tolerations:
          - key: "kubernetes.azure.com/scalesetpriority"
            operator: "Equal"
            value: "spot"
            effect: "NoSchedule"
          containers:
          - name: azure-vote-front
            image: mcr.microsoft.com/azuredocs/azure-vote-front:v1
  

• Scale down your application replicas to allow Karpenter to evict existing on-demand nodes and replace them with Spot nodes:

    osama [ ~/karpenter ]$ kubectl get nodes
    NAME                                STATUS   ROLES   AGE     VERSION
    aks-nodepool1-41633911-vmss000000   Ready    agent   3d21h   v1.27.9
    aks-nodepool1-41633911-vmss00000b   Ready    agent   24m     v1.27.9
  
    osama [ ~/karpenter ]$ kubectl get pods -n karpenter-demo-ns -o wide
    No resources found in karpenter-demo-ns namespace.
  
    osama [ ~/karpenter ]$ kubectl scale deployment azure-vote-back --replicas=10 -n karpenter-demo-ns
    deployment.apps/azure-vote-back scaled
    osama [ ~/karpenter ]$ kubectl scale deployment azure-vote-front --replicas=10 -n karpenter-demo-ns
    deployment.apps/azure-vote-front scaled
    osama [ ~/karpenter ]$
  

• Deploy and scale vote application replicas so that karpenter spins up spot nodes based on nodepool configuration and schedule pods after toleration validation on spot

• Karpenter spins up new spot nodes and Nominate that node for sceduling sample vote-app

  
    osama [ ~/karpenter ]$ kubectl get events -A --field-selector source=karpenter --sort-by='.lastTimestamp'
    NAMESPACE           LAST SEEN   TYPE      REASON                       OBJECT                                    MESSAGE
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-pz8sp      Pod should schedule on: nodeclaim/default-52gbg
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-ckdcq      Pod should schedule on: nodeclaim/default-52gbg
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-v9nqj      Pod should schedule on: nodeclaim/default-52gbg
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-vswvs      Pod should schedule on: nodeclaim/default-52gbg
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-lnxmp      Pod should schedule on: nodeclaim/default-52gbg
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-jc2jz      Pod should schedule on: nodeclaim/default-52gbg
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-hwnbh      Pod should schedule on: nodeclaim/default-52gbg
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-r7msb      Pod should schedule on: nodeclaim/default-52gbg
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-96lm9      Pod should schedule on: nodeclaim/default-52gbg
    karpenter-demo-ns   104s        Normal    Nominated                    pod/azure-vote-back-687ddb67bd-5qcvk      Pod should schedule on: nodeclaim/default-52gbg
    default             1s          Normal    DisruptionLaunching          nodeclaim/default-bkz6c                   Launching NodeClaim: Expiration/Replace
    default             1s          Normal    DisruptionWaitingReadiness   nodeclaim/default-bkz6c                   Waiting on readiness to continue disruption
    default             1s          Normal    DisruptionBlocked            nodepool/system-surge                     No allowed disruptions due to blocking budget
    default             1s          Normal    DisruptionWaitingReadiness   nodeclaim/default-5vp7x                   Waiting on readiness to continue disruption
    default             1s          Normal    DisruptionLaunching          nodeclaim/default-5vp7x                   Launching NodeClaim: Expiration/Replace
  

  Configuring Multiple NodePools

• To configure separate NodePools for Spot and On-Demand capacity:

  Spot nodes configure with E series VM "Standard E2s_v5" and OnDemand with D series VM as "Standard_D4s_v5"

• In multi-nodepool scenario each nodepool needs to be configured with 'Weight' attribute, nodepool with highest weight would be priotized over another, here we have Spot node with weight:100 and ondemand with weight:60

osama [ ~ ]$ kubectl get nodepool default -o yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    budgets:
    - nodes: 100%
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h
  template:
    spec:
      nodeClassRef:
        name: default
      requirements:
      - key: kubernetes.io/arch
        operator: In
        values:
        - amd64
      - key: kubernetes.io/os
        operator: In
        values:
        - linux
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - spot
      - key: karpenter.azure.com/sku-family
        operator: In
        values:
        - B
      - key: karpenter.azure.com/sku-name
        operator: In
        values:
        - Standard_B2s_v2
  weight: 100

• If we do not specify an explicit SKU name, Karpenter will consider the entire VM series.

• To validate that the sample VoteApp is running on Spot nodes, use the following commands:

• The output should indicate that the nodes are of capacity type "spot":

    osama [ ~ ]$ kubectl get pods -n karpenter-demo-ns -o wide
    NAME                                READY   STATUS    RESTARTS   AGE   IP             NODE                NOMINATED NODE   READINESS GATES
    azure-vote-back-687ddb67bd-w7ghm    1/1     Running   0          63m   10.244.3.11    aks-default-5cr5f   <none>           <none>
    azure-vote-front-6855444955-64558   1/1     Running   0          63m   10.244.3.168   aks-default-5cr5f   <none>           <none>
    osama [ ~ ]$ kubectl describe node aks-default-5cr5f | grep karpenter.sh
                        karpenter.sh/capacity-type=spot
                        karpenter.sh/initialized=true
                        karpenter.sh/nodepool=default
                        karpenter.sh/registered=true
                        karpenter.sh/nodepool-hash: 12393960163388511505
                        karpenter.sh/nodepool-hash-version: v2
  

  Simulating Spot Node Eviction

  To test the spot eviction scenario, simulate a spot eviction using the Azure CLI:

    osama [ ~ ]$ az vm simulate-eviction --resource-group MC_aks-lab_aks-karpenter_eastus --name aks-default-5cr5f
    osama [ ~ ]$ date
    Tue May 21 06:20:02 PM IST 2024
  

• Monitor the availability of your VoteApp using a simple curl command:

    while true; do echo "$(date) $(curl -s -v -o /dev/null -w 'HTTP %{http_code}\n' http://voteapp.com 2>&1 | grep 'HTTP')"; sleep 2; done
  

• After running the spot simulation, the existing node will be marked for termination, and a new Spot node will be created to schedule the VoteApp pods. Within less than a minute, the VoteApp should start responding with HTTP 200 status codes.

•     root@MININT-8C81HDE:/home/osamaex while true; do echo "$(date) $(curl -s -v -o /dev/null -w 'HTTP %{http_code}\n' http://voteapp.com 2>&1 | grep 'HTTP')"; sleep 2; done
      Tue May 21 18:20:04 IST 2024 > GET / HTTP/1.1
      < HTTP/1.1 200 OK
      HTTP 200
      Tue May 21 18:20:07 IST 2024 > GET / HTTP/1.1
      < HTTP/1.1 200 OK
      HTTP 200
      Tue May 21 18:20:09 IST 2024 > GET / HTTP/1.1
      < HTTP/1.1 200 OK
      HTTP 200
      Tue May 21 18:20:12 IST 2024 HTTP 000  $Failure-Alert
      Tue May 21 18:21:14 IST 2024 > GET / HTTP/1.1
      < HTTP/1.1 200 OK                      $Successful-Response
      HTTP 200
      Tue May 21 18:22:58 IST 2024 > GET / HTTP/1.1
      < HTTP/1.1 200 OK
      HTTP 200
  

• Check the events logged by Karpenter:

•     kuctl get events -A --field-selector source=karpenter --sort-by='.lastTimestamp'
  

• Results of events logged by karpenter to replace spot noded with ondemand

•     osama [ ~ ]$ 
      NAMESPACE           LAST SEEN   TYPE      REASON           OBJECT                                  MESSAGE
      default             23s         Warning   FailedDraining   node/aks-default-5cr5f                  Failed to drain node, 10 pods are waiting to be evicted
      karpenter-demo-ns   22s         Normal    Evicted          pod/azure-vote-back-687ddb67bd-w7ghm    Evicted pod
      karpenter-demo-ns   22s         Normal    Evicted          pod/azure-vote-front-6855444955-64558   Evicted pod
      karpenter-demo-ns   21s         Normal    Nominated        pod/azure-vote-back-687ddb67bd-tb2pv    Pod should schedule on: nodeclaim/default-6zkkl
      karpenter-demo-ns   21s         Normal    Nominated        pod/azure-vote-front-6855444955-7wzss   Pod should schedule on: nodeclaim/default-6zkkl
  

• Verify that the pods are running on the new Spot node:

    kubectl get pods -n karpenter-demo-ns -o wide
    NAME                                READY   STATUS    RESTARTS   AGE   IP             NODE                NOMINATED NODE   READINESS GATES
    azure-vote-back-687ddb67bd-tb2pv    1/1     Running   0          18m   10.244.2.103   aks-default-6zkkl   <none>           <none>
    azure-vote-front-6855444955-7wzss   1/1     Running   0          18m   10.244.2.47    aks-default-6zkkl   <none>           <none>
  

Save Cost by utilizing Reserved Instance VM's

• NodePool configuration allows you to specify different VM series along with multiple VM SKUs. Create a separate NodePool with the highest weight value and specify all Reserved Instance VM SKU families or explicit SKU names using the karpenter.azure.com/sku-name or karpenter.azure.com/sku-familyparameter.

       spec:
          nodeClassRef:
            name: default
          requirements:
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
          - key: karpenter.sh/capacity-type
            operator: In
            values:
            - on-demand
          - key: karpenter.azure.com/sku-family
            operator: In
            values:
            - D
          - key: karpenter.azure.com/sku-name
            operator: In
            values:
            - [Standard_D2s_v3, Standard_D4s_v3, Standard_D8s_v3, Standard_D16s_v3, Standard_D32s_v3, Standard_D64s_v3, Standard_D96s_v3]
      weight: 90
  

Conclusion

The adoption of Karpenter in AKS signifies a major advancement in node scaling efficiency, flexibility, and cost optimization. By addressing the limitations of the Cluster Autoscaler and introducing dynamic, rapid provisioning of nodes, Karpenter provides a robust solution for managing Kubernetes clusters. Its flexibility in handling different VM types, faster scaling capabilities, and cost optimization make it a valuable addition to Kubernetes cluster management. By leveraging Karpenter, organizations can achieve more responsive and cost-effective Kubernetes deployments.

In that case my recommendation has been to deploy multiple OpenAI instance with S0 plan(Token-based-consumption) in any region where capacity is available & then used an load Balancer as fascade to distribute traffic across all your Az OPAI endpoints

Each OpenAI Models has its own limit called token limit which is measured based per minute i.e. TPM. In this case i am referring to GPT3.5 Model which comes with maximum TPM limit of 300K for an given region

Since there is surge in demand from customer all over world for LLM models like GPT, we have limited capacity of OpenAI instance in each region.

Many Customer specially D2C eventually get requirment of over 700-800K+ token for thier use case.
In that scenario customers tend to deploy OpenAI Instance with S0 plan across multiple location
Although TPM limit for all instance is not same because of capacity constrain overall

Following is real world example of customer which uses multiple OpenAI instance with different TPM limit for each region

OpenAI-EastUS :300k
OpenAI-WestUS: 240K
OpenAI-SouthUK: 150K
OpenAI-Southindia:100k
OpenAI-CentralIndia:50K

Why not use existing Load balancers & how it is different from other existing LB technique?

There are multiple load balancing option are now available using AppGW/FrontDoor or API management custom policies with algorithm based on roundrobin, random or even prirotiy based. Although there are some cons with each routing algorithm

AppGW/FrontDoor:Roundrobin pattern to Distribte traffic across OpeAI endpoints.
Can be used in scenario when all OpenAI instance has same TPM Limit.But it cannot do health probe of OpenAI endpoints if its TPM exhausted & received HTTP 429(Server is Busy) & it will continue to forward request to throttled openai endpoint

API management:
One differentiting capablities with APIM using its policies that it is aware about endpoint HTTP 429 status code and offers Retry mechanism which allows send traffic to another available endpoints from backend pool until current endpoints becomes active/healthy again.

APIM Policies for OpenAI:
Roundrobin: if you have multiple OpenAI endpoint across location in APIM backendpool, it simply select any of backend based on roundrobin provided its not being throtlled at that moment & it distribute traffic across sequentially doesnt honor OpenAI Endpoints TPM limits

Random: Its mostly similar with roundrobin method, primary differnence is that it selected backend endpoint randomly from backend pool. & its also doesnt honor openai TPM limit

Priority: In this case if you have endpoint across many location, you can assigned priotiy order sequence either based on TPM limit or latency from your base region,
But even then all traffic would always forwarded to endpoint which has lowest priority & rest of available openai endpoint would simply be waiting on standby unless lowest priority instance is throttled

Now in ths case customer requested for load balancing technique based on its TPM limit assigned against each OpenAI endpoint which should distribute traffic accordingly.

Weightage:

There is no direct feature capablities in APIM for weightage based routing.I have tried achieve same results using custom logic with APIM policies

Selection Process:
Backend logic used in this policy is based on weighted selection method to choose an endpoint route for retry.endpoint with higher weights are more likely to be chosen, but each endpoints route has at least some chance of being selected. This is because the selection is based on a random number that is compared against cumulative weights, which means the selection process inherently favors routes with higher weights due to the way cumulative weights are calculated and utilized. Let’s break down the process with a simple example to clarify how routes with higher weights are more likely to be selected.

*in this image, its deplects APIM configured based on Weight policy across OpenAI

There are three variable used in selelction process for backend
1)Totalweight
2)Cumulative weight
3)Random Number

Example: Lets understand this policy using following based on above image
Assume you have five openai endpoints as mentioned in image with following weight

Endpoint A: Weight = 50
Endpoint B: Weight = 100
Endpoint C: Weight = 150
Endpoint D: Weight = 300
Endpoint E: Weight = 600

Step 1:Calculate Total Weight
First, you calculate the total weight of all endpoint routes, which in this case would be 50+100+150+300+600=1200.

Step 2: Generate Random Weight

Next, a random number (let’s call it randomWeight) is generated between 1 and the total weight (inclusive). So, randomWeight is between 1 and 1200.

Step 3: Calculate Cumulative Weights
The cumulative weights are calculated to determine the ranges that correspond to each endpoint route. Here’s how they look based on the weights:

• Cumulative Weight after Endpoint A: 50 (just the weight of A,)

• Cumulative Weight after Endpoint B: 100 (the weight of A + B, 50 + 100)

• Cumulative Weight after Endpoint C: 150 (the weight of A + B + C, 50+100+150)

• Cumulative Weight after Endpoint D: 300 (the weight of A + B + C +D, 50+100+150+300)

• Cumulative Weight after Endpoint E: 600 (the weight of A + B + C + D +E, 150+100+150+300+500)

  Weight Distribution Percentage Calcuation:

  • Route A 50/1200×100%=4.17%

  • Route B 100/1200×100%=8.33%

  • Route C 150/1200×100%=12.50%

  • Route D 300/1200×100%=25.00%

  • Route E 600/1200×100%=50.00%

Step 4: Select the Endpoint Route Based on Random Weight

The randomWeight determines which Endpoint route is selected:

• If randomWeight is between 1 and 50, Endpoint A is selected (4.17% chance)

• If randomWeight is between 51 and 100, Endpoint B is selected(8.33% chance)

• If randomWeight is between 100 and 150, Endpoint C is selected(12.50% chance)

• If randomWeight is between 150 and 300, Endpoint D is selected(25 % chance)

• If randomWeight is between 300 and 600, Endpoint E is selected(50% chance)

OPAI TPM Exhausted Scenario:

• In this image, An first endpoint is throttled with HTTP response code 429 & APIM is routing request to other available backend based on weight

• When OpenAI endpoint starts getting traffic beyond its token capacity limit, it throws an HTTP Status code "429" which translate to that is 'server is busy'

• In that situation APIM is configured with health probe of endpoint based on '429' using Retry-After logic

• Once APIM gets HTTP 429 from its endpoint , it starts to priortize other endpoint in weightage order and forward most request next highest weight assigned endpoint

• Meanwhile it continue to retry health probe request to actuall thortlled endpoint after waiting for 60 second

OpenAI PTU Tier (Provision throughput Unit ):

To Ensure OpenAI endpoint with PTU plan should be always be preferred. you would need adjust PTU instance endpoint in such way that it out weight other endpoint significantly, effectively making it the most likely choice under normal circumstances.

1. Significantly Increase PTU endpoint Weight: Increase the weight of PTU endpoint so high compared to Endpoint E,D,C that the random selection virtually always lands on PTU based OpenAI endpoint.In Above example highest weight was 600 for Endpoint E, we could set PTU endpoint Weight something like 5000 or even more.This would make PTU endpoint overwhelmingly more likely to be selected.
2.Preferential Selection: Modify the selection logic to check for PTU endpoint availability first and choose it by default, only falling back to other endpoint A/B/C under certain conditions (e.g., Route C is down or throttling). This requires a bit of custom logic in your policy.
3.Use Priority Method: If your system allows, Clubbed a Prority based system where routes are tried in order of priority rather than by weight. PTU would be given the highest priority, with endpoint A,B,C & D as fallbacks.

Setup Policy in APIM:

• Create APIM instance with desired SKU if not exist already & enable managed identity on APIM

• Make sure to have same model(GPT35/4) & deployment name across all OpenAI endpoint

• Grant following Az RBAC "Cognitive OpenAI User" on All OpenAI resources via 'Access Control(IAM)' blade to APIM managed identity

• Download Az OpenAI API version Schema and inetrgrate with APIM using Import, you could read more on OpenAI Integration instruction here

• Once OpenAI API is imported, copy and paste this policy into APIM editor

• Modify or Replace from line no #15-43 with your existing All OpenAI instance endpoint details

• Assigned weight to each endpoint listed under routes as per its TPM Limits.

• Update your OpenAI model deployment name and routes index array sequence at line no #47,48 also at line no #129

• Perform Test run with APIM traces enabled to see policy logic in action

<policies>
    <inbound>
        <base />
        <!-- Getting OpenAI clusters configuration -->
        <authentication-managed-identity resource="https://cognitiveservices.azure.com" />
        <cache-lookup-value key="@("oaClusters" + context.Deployment.Region + context.Api.Revision)" variable-name="oaClusters" />
        <!-- If we can't find the configuration, it will be loaded -->
        <choose>
            <when condition="@(context.Variables.ContainsKey("oaClusters") == true)">
                <set-variable name="oaClusters" value="@{
                    JArray routes = new JArray();
                    JArray clusters = new JArray();
                    if(context.Deployment.Region == "West Europe" || true)
                    {
                        routes.Add(new JObject()
                        {
                            { "name", "openai 1" },
                            { "location", "eastus" },
                            { "url", "https://openaiendpoint-a.openai.azure.com/" },
                            { "isThrottling", false }, 
                            { "weight", "100"},
                            { "retryAfter", DateTime.MinValue } 
                        });

                        routes.Add(new JObject()
                        {
                            { "name", "openai 2" },
                            { "location", "UK SOuth" },
                            { "url", "https://openaiendpoint-b.openai.azure.com/" },
                            { "isThrottling", false },
                            { "weight", "150"},
                            { "retryAfter", DateTime.MinValue }
                        });

                        routes.Add(new JObject()
                        {
                            { "name", "openai 3" },
                            { "location", "Central india" },
                            { "url", "https://openendpointai-c.openai.azure.com/" },
                            { "isThrottling", false },
                            { "weight", "300"},
                            { "retryAfter", DateTime.MinValue }
                        });

                        clusters.Add(new JObject()
                        {
                            { "deploymentName", "gpt35turbo16k" },
                            { "routes", new JArray(routes[0], routes[1], routes[2]) }
                        });
                    }
                    else
                    {
                        //Error has no clusters for the region
                    }

                    return clusters;   
                }" />
                <!-- Add cluster configurations to cache -->
                <cache-store-value key="@("oaClusters" + context.Deployment.Region + context.Api.Revision)" value="@((JArray)context.Variables["oaClusters"])" duration="86400" />
            </when>
        </choose>
        <!-- Getting OpenAI routes configuration based on deployment name, region and api revision -->
        <cache-lookup-value key="@(context.Request.MatchedParameters["deployment-id"] + "Routes" + context.Deployment.Region + context.Api.Revision)" variable-name="routes" />
        <!-- If we can't find the configuration, it will be loaded -->
        <choose>
            <when condition="@(context.Variables.ContainsKey("routes") == true)">
                <set-variable name="routes" value="@{
                    string deploymentName = context.Request.MatchedParameters["deployment-id"];
                    JArray clusters = (JArray)context.Variables["oaClusters"];
                    JObject cluster = (JObject)clusters.FirstOrDefault(o => o["deploymentName"]?.Value<string>() == deploymentName);
                    if(cluster == null)
                    {
                        //Error has no cluster matched the deployment name
                    }
                    JArray routes = (JArray)cluster["routes"];
                    return routes;
                }" />
                <!-- Set total weights for selected routes based on model -->
                <set-variable name="totalWeight" value="@{
                int totalWeight = 0;
                JArray routes = (JArray)context.Variables["routes"];
                foreach (JObject route in routes)
                {
                    totalWeight += int.Parse(route["weight"].ToString());
                }
                return totalWeight;
                }" />
                <!-- Set cumulative weights for selected routes based on model-->
                <set-variable name="cumulativeWeights" value="@{
                JArray cumulativeWeights = new JArray();
                int totalWeight = 0;
                JArray routes = (JArray)context.Variables["routes"];
                foreach (JObject route in routes)
                {
                    totalWeight += int.Parse(route["weight"].ToString());
                    cumulativeWeights.Add(totalWeight);
                }
                return cumulativeWeights;
            }" />
                <!-- Add cluster configurations to cache -->
                <cache-store-value key="@(context.Request.MatchedParameters["deployment-id"] + "Routes" + context.Deployment.Region + context.Api.Revision)" value="@((JArray)context.Variables["routes"])" duration="86400" />
            </when>
        </choose>
        <set-variable name="routeIndex" value="-1" />
        <set-variable name="remainingRoutes" value="1" />
    </inbound>
    <backend>
        <retry condition="@(context.Response != null && (context.Response.StatusCode == 429 || context.Response.StatusCode >= 500) && ((Int32)context.Variables["remainingRoutes"]) > 0)" count="3" interval="0">
            <set-variable name="routeIndex" value="@{
            Random random = new Random();
            int totalWeight = (Int32)context.Variables["totalWeight"];
            JArray cumulativeWeights = (JArray)context.Variables["cumulativeWeights"];
            int randomWeight = random.Next(1, totalWeight + 1);
            int nextRouteIndex = 0;
            for (int i = 0; i < cumulativeWeights.Count; i++)
            {
                if (randomWeight <= cumulativeWeights[i].Value<int>())
                {
                    nextRouteIndex = i;
                    break;
                }
            }
            return nextRouteIndex;
        }" />
            <!-- This is the main logic to pick the route to be used -->
            <set-variable name="routeUrl" value="@(((JObject)((JArray)context.Variables["routes"])[(Int32)context.Variables["routeIndex"]]).Value<string>("url") + "/openai")" />
            <set-variable name="routeLocation" value="@(((JObject)((JArray)context.Variables["routes"])[(Int32)context.Variables["routeIndex"]]).Value<string>("location"))" />
            <set-variable name="routeName" value="@(((JObject)((JArray)context.Variables["routes"])[(Int32)context.Variables["routeIndex"]]).Value<string>("name"))" />
            <set-variable name="deploymentName" value="@("gpt35turbo16k")" />
            <set-backend-service base-url="@((string)context.Variables["routeUrl"])" />
            <forward-request buffer-request-body="true" />
        </retry>
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

Policy Internals:

This policy incorporates several key configuration items that enable it to efficiently manage and route API requests to OpenAI ChatGPT instances. so Highlighting few of configurations can provide valuable insights to blog readers

<authentication-managed-identity resource="https://cognitiveservices.azure.com" />
<cache-lookup-value key="@("oaClusters" + context.Deployment.Region + context.Api.Revision)" variable-name="oaClusters" />

• For authentication managed identity is used in this case instead of keys, so first line enable policy to leverage manage identity.

• In next block Attempts to fetch pre-configured OpenAI cluster information from the cache, significantly reducing the overhead of reconstructing the configuration for each API call.

 <choose>
            <when condition="@(context.Variables.ContainsKey("oaClusters") == true)">
                <set-variable name="oaClusters" value="@{
                    JArray routes = new JArray();
                    JArray clusters = new JArray();
                    if(context.Deployment.Region == "West Europe" || true)
                    {
                        routes.Add(new JObject()
                        {
                            { "name", "openai-a" },
                            { "location", "india" },
                            { "url", "https://openai-endpoint-a.openai.azure.com/" },
                            { "isThrottling", false }, 
                            { "weight", "600"},
                            { "retryAfter", DateTime.MinValue } 
                        });
                        clusters.Add(new JObject()
                        {
                            { "deploymentName", "gpt35turbo16k" },
                            { "routes", new JArray(routes[0], routes[1], routes[2]) }
                        });
                    }
                    else
                    {
                        //Error has no clusters for the region
                    }

                    return clusters;

• In choose block complex logic dynamically generates the configuration for OpenAI clusters if not found in the cache.

• Depending on the deployment region, the policy constructs clusters with specific attributes (e.g., name, location, URL, weight) reflecting each OpenAI instance's characteristics and intended traffic handling capacity.

• Each route is assigned a specific weight, dictating its selection probability for handling incoming requests, thereby facilitating a balanced and efficient distribution of traffic based on the defined capacities or priorities.

• Defines the OpenAI model (in this case, gpt35turbo16k) to which API requests should be routed. This specification allows the policy to support different OpenAI models,By clearly defining the deploymentName.

<choose>
            <when condition="@(context.Variables.ContainsKey("routes") == true)">
                <set-variable name="routes" value="@{
                    string deploymentName = context.Request.MatchedParameters["deployment-id"];
                    JArray clusters = (JArray)context.Variables["oaClusters"];
                    JObject cluster = (JObject)clusters.FirstOrDefault(o => o["deploymentName"]?.Value<string>() == deploymentName);
                    if(cluster == null)
                    {
                        //Error has no cluster matched the deployment name
                    }
                    JArray routes = (JArray)cluster["routes"];
                    return routes;
                }" />

• In this block it checks if the routes configuration for the current API request's deployment (e.g., a specific OpenAI model indicated by deployment-id) is already available within the context variables. If so, it proceeds to confirm and utilize these endpoint routes for further processing.

<!-- Set total weights for selected routes based on model -->
                <set-variable name="totalWeight" value="@{
                int totalWeight = 0;
                JArray routes = (JArray)context.Variables["routes"];
                foreach (JObject route in routes)
                {
                    totalWeight += int.Parse(route["weight"].ToString());
                }
                return totalWeight;
                }" />
                <!-- Set cumulative weights for selected routes based on model-->
                <set-variable name="cumulativeWeights" value="@{
                JArray cumulativeWeights = new JArray();
                int totalWeight = 0;
                JArray routes = (JArray)context.Variables["routes"];
                foreach (JObject route in routes)
                {
                    totalWeight += int.Parse(route["weight"].ToString());
                    cumulativeWeights.Add(totalWeight);
                }
                return cumulativeWeights;
            }" />

• Both of these variable are key elements in functioning of this policy

• Totalweight: by iterating over each route in the endpoint routes collection and summing up their weights, it represents the aggregate capacity of all endpoint routes. This total is crucial because it defines the range within which a random number can be generated to select a route proportionally based on its weight.

• Cumulative weight:As the policy iterates through endpoint routes, it progressively adds each route's weight to a running total, creating a series of increasing values. This cumulative approach allows for determining which route to select by generating a random number within the totalweight range and finding the segment (route) where this number falls.

 <retry condition="@(context.Response != null && (context.Response.StatusCode == 429 || context.Response.StatusCode >= 500) && ((Int32)context.Variables["remainingRoutes"]) > 0)" count="3" interval="30">
            <set-variable name="routeIndex" value="@{
            Random random = new Random();
            int totalWeight = (Int32)context.Variables["totalWeight"];
            JArray cumulativeWeights = (JArray)context.Variables["cumulativeWeights"];
            int randomWeight = random.Next(1, totalWeight + 1);
            int nextRouteIndex = 0;
            for (int i = 0; i < cumulativeWeights.Count; i++)
            {
                if (randomWeight <= cumulativeWeights[i].Value<int>())
                {
                    nextRouteIndex = i;
                    break;
                }
            }
            return nextRouteIndex;
        }" />

• Retry policy segment is mentioned within backend block is a mechanism designed to handle scenarios where an OpenAI ChatGPT instance—fails due to throttling (HTTP 429) or server errors (HTTP status codes >= 500). This segment also takes into account the availability of alternative endpoint routes for retrying the request.

• The maximum number of retry attempts is set to 3, meaning the policy will try up to three times to forward the request to a backend service before marking as unhealthy.Also retry attempts will be made on interval of 30 second one after another

• Random number is generated within the range of 1 to the totalWeight of all available endpoint routes. This random weight determines which route will be selected for the next retry attempt.it terates through the cumulative weights of all endpoint routes stored in cumulativeWeights variable

• While index of the selected route (nextRouteIndex) is determined and stored, guiding the API management platform on which route to use for the next retry attempt.

• By incorporating a retry mechanism with weighted random route selection, it ensures that requests are not simply retried on the same route that might be experiencing issues but are intelligently rerouted based on predefined weights reflecting each route's capacity

• system effectively balances the load across multiple routes, minimizing the impact of temporary issues on one or more endpoint

Appendix

For simple easy to deploy load balance options on AppGW/AFD refer my this repo

On overall reference Archiecture pattern of APIM & OpenAI can refer this doc

Historically, on Azure cloud customers needed to rely on Infrastructure-as-a-Service (IaaS) solutions, configuring virtual machines either as DNS forwarders or Network Virtual Appliances (NVA) to achieve correct name resolution across their on-premises environments and Azure. These setups were not only complex but also introduced potential single points of failure, diminishing the network's overall resilience and efficiency.

Recently I have encountered scenarios where customers expressed a desire to transition away from these IaaS-based DNS forwarders. Their goal was clear: to eliminate these single points of failure and significantly boost their network's resiliency, especially in disaster recovery (DR) scenarios. They envisioned an active-active DR solution where, even in the event of a DNS service outage in the primary region, their organization's name resolution would continue uninterrupted. Moreover, they expected name resolution to be handled by their regional Azure Private DNS (Az PDNS) service by default, only routing to another region if the primary was compromised.
Here is Architecture design for Prod/DR in Active Scenario

To fulfill these requirements, I have devised an architectural design tailored for production and DR in an active-active configuration. This design necessitates several critical adjustments:

1. DNS Resolver Service Configuration: Each DNS resolver service in every region must be capable of resolving the names (URLs) of all workloads.

2. Private DNS Zone Linking: As Platform-as-a-Service (PaaS) resources have separate instances running in both regions, their respective private DNS zones should be linked to the Virtual Network (VNet) of the PDNS resolver service.

3. Conditional Forwarder Zones: Production and DR Active Directory (AD) DNS servers must be set up with non-AD integrated conditional forwarder zones for Azure workloads (e.g., Blob storage, Az SQL.), enabling private access through the on-premises network.

4. Configuration of Conditional Forwarders: The conditional forwarder zones in Prod and DR AD-DNS servers must have a different sequence of primary and secondary endpoints. This ensures that DNS traffic from each environment or region is managed by its respective DNS resolver service, addressing the customer's need for regional traffic management and eliminating cross-region dependencies during DR scenarios.

Here is screenshot of conditional forwarders config & name resolution details.

Name Resolution from on-premises against Blob storage private endpoint.

Private DNS zone details for blob storage on azure

This architecture pattern not only signifies improvement in DNS management but also reflects in more resilient, efficient, and scalable overall azure infrastructure which complement each other in active-active DR scenarios.

Thanks for reading!

Do you have any other tips to improve my blog? Let me know in the comments, or just say hi 👋.

Until next time, Happy Learning