AWS Details Two-Node Ray LLM Inference on EKS
A packaged stack separates prompt processing from token generation across GPU nodes, while networking, cost and workload-level performance remain deployment-specific.
Edited by Tyronne Panaino
AWS published a multi-node Ray LLM deployment guide on October 12 that places one large language model across two GPU workers on Amazon Elastic Kubernetes Service. The reference design assigns prompt processing to one node and token generation to another, then presents both phases through one OpenAI-compatible endpoint.
The guide matters to infrastructure teams serving models that no longer fit comfortably on one GPU or whose long prompts interfere with ongoing generation. AWS packages the main software and networking dependencies into a Deep Learning Container, but the design still requires a Kubernetes cluster, specialized instances, a fast interconnect and workload-specific validation.
The container assembles a distributed serving stack
AWS's Ray LLM Deep Learning Container includes Ray Serve LLM as the serving layer and vLLM as the inference engine. It also includes NVIDIA's NIXL library for moving the key-value cache between nodes and the Elastic Fabric Adapter stack for high-bandwidth communication. A built-in application builder connects the two model phases behind an HTTP endpoint without requiring a custom handler or model archive.
The application is configured in YAML. Separate prefill and decode blocks identify the same model, set memory and context options and give each phase its own engine identity. NIXL coordinates the cache transfer, while the Libfabric backend directs that traffic over EFA. The example exposes the familiar model-listing and chat-completions routes used by OpenAI-compatible clients.
Packaging those components reduces one class of integration work: matching the CUDA runtime, serving framework, inference engine and network libraries. It does not make the deployment serverless or eliminate cluster operations. Teams still own capacity, network layout, identity, monitoring, model choice and the behavior of the application that calls the endpoint.
Prefill and decode have different bottlenecks
Large-model inference begins by processing the complete input and building a key-value cache. This prefill phase is compute-bound. Decode then generates output one token at a time while repeatedly reading and extending that cache, making memory bandwidth a central constraint. Running both phases on one GPU can cause a burst of prompt work to compete with generation already in progress.
The AWS pattern separates those phases. One GPU worker handles prefill and sends the resulting cache to a second worker responsible for decode. That makes it possible to choose and scale resources for each phase independently. The trade-off is the cache transfer itself: if the interconnect is too slow, communication can erase the benefit of separation.
AWS presents retrieval-augmented generation, document summarization and agent applications as workloads that may benefit because their prompts can be long or uneven. Those are intended use cases, not evidence that every workload will become faster or cheaper. A team needs request traces, prompt-length distributions, latency targets and utilization measurements from its own model before adopting the topology.
The EKS example has concrete infrastructure constraints
The reference architecture uses an EKS cluster in a virtual private cloud. A CPU-only `m7i.xlarge` node runs system services, the KubeRay operator and the Ray head. Two `g6.8xlarge` GPU nodes run the workers, with one assigned to prefill and the other to decode. AWS pins both GPU nodes to a single Availability Zone because EFA cannot cross zones.
That same-zone requirement narrows failure-domain and placement choices. The example also places nodes in private subnets with outbound access through a NAT Gateway. Before following it, AWS says an account needs quota for two `g6.8xlarge` instances, amounting to at least 64 virtual CPUs in the target Region, plus configured AWS credentials and the `kubectl`, `eksctl`, Helm and `envsubst` tools.
The worker pods request a GPU and an EFA interface. AWS runs them with privileged access because NIXL registers GPU memory with the network adapter. That operational detail deserves a security review rather than being copied into a production cluster without examining pod privileges, image provenance and namespace controls.
Evidence quality and what to test next
The evidence is a detailed first-party AWS tutorial for one model configuration and one cloud architecture. It is sufficient to explain the packaged components and reference topology, but this run did not fetch an independent cost study, resilience test or benchmark across varied models and request patterns. The article also does not establish that prefill/decode separation is the best design when prompts are short, traffic is light or network transfer dominates.
The next useful checkpoints are reproducible latency and throughput measurements against a single-node baseline, total hourly cost at realistic utilization, restart and scaling behavior, cache-transfer failures and cross-node observability. Teams should also test whether the OpenAI-compatible surface preserves every feature their clients expect instead of assuming protocol similarity means identical model behavior.
Status
Learning. AWS has documented and packaged a working multi-node serving pattern, while the performance and operational conclusions remain specific to its first-party example. Internal confidence is medium because no independent deployment evidence was fetched in this run.
Sources
Update note: Last reviewed 2026-10-13. We will revise this post if AWS changes the container stack or publishes broader cost, scaling or resilience evidence.
Sources
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.