GPU infrastructure interviews

A mix of NVIDIA A100s and some newer L40s units for our inference workloads.
We're also starting to test out a few H100s for our more demanding training jobs.

A100s 50%
L40s 30%
H100s 20%

We're using RoCE for networking between our GPU nodes and we're actively evaluating GPUDirect as a potential upgrade for our heavier data intensive training jobs.

It's a pretty even split though we're leaning slightly more toward inference for production workloads. Training is still a major focus especially as we bring more H100s online and our infrastructure has been flexible enough to handle both sides comfortably.

We're running Kubeflow for ML pipelines triton for inference and testing vLLM for large language models.

vLLM is still in testing, not yet in production. Beyond that we're pretty much standard with kubeflow and triton no major custom frameworks in the mix right now.

We have a good amount of visibility mostly at the per node level. We can see overall utilization and spot any underperforming nodes but getting down to a per workload breakdown is something we're still working on improving.

We use prometheus and Grafana for the core monitoring pulling in metrics from NVIDIA DCGM for the GPU specific data. It gives us a solid per node view and we've built a few custom dashboards to track the metrics that matter most to us.

Our average Gpu utilization sits roughly around 50-60 % most of the time. It does fluctuate a bit dipping during dev cycles and peaking during major training runs but that's the typical range we see.

Roughly $40k to $50k per month though we haven't broken it down to a strict per Gpu cost yet.

We're using bin packaging to improve pod density and have started a rightsizing initiative to avoid over provisioning. Spot instances are on our radar for non critical workloads but we haven't rolled them out yet.

We're in a decent spot with bin packing and rightsizing working well but the big gap is workload level visibility so we can fine tune resource allocation and better identify spot eligible workloads.

I think the biggest potential would be around smarter scheduling and rightsizing making sure each job gets exactly what it needs without waste. we'd also likely find more candidates for spot instances which could meaningfully reduce our monthly spend.

I'd prioritize cracking workload level visibility first. It feels like the foundational piece that would make spot eligibility and other optimizations much more effective down the road.

For the minimum viable picture I'd need per job resource consumption and cost attribution with key performance metrics like latency and throughput tied to each workload. That would let me tie usage directly to business impact and make smarter decisions about scheduling and rightsizing.

The big one is scaling effectively during peak demand over provisioning and better visibility into our long term usage patterns would help a ton with that. I'd also love to see more automation around scheduling and failover so we are not constantly manually babysitting our training runs.

I would say team capacity is the single biggest blocker right now. We have the budget and we're aware of the tooling out there but we are spread thin just keeping the lights on and handling day to day operations which leaves very little time for strategic improvements like this.

I think investing in a tool or service would feel more realistic for us right now. Hiring is a longer process and adds ongoing cost whereas a solid tool could give us a quicker win and start freeing up our exciting team almost immediate.

An ideal tool would give us real time workload level visibility and automated scheduling to optimizing resource usage so our team can stop babysitting and start focusing on higher impact work.

Mix of A100s and H100s for AI workloads.

We're about 70% H100s and 30% A100s across our fleet.Yes,er use InfinBand and GPUDirect for faster inter-node communication.

we're primarly using them for mix of both training and inference ,with about 60% training and 40% inference currently .The training work is mostly for our internal AI models ,while inference supports our customer-facing features.

Using vLLM,Triton,Ray and custom Kubeflow operators .

We have decent visibility through our monitoring tools ,average utilization is around 60%-70%.

We use NVIDIA DCGM with Prometheus and Grafana for monitoring.

Roughly $150k-200k monthly for GPU infrastructure and cloud services.

We use bin-packing , spot instances ,and schedule non-critical workloads off-peak.

It helps ,but we could improve with better workload scheduling.

Smarter bin-packing and prioritization of critical workloads over batch jobs

Utilization dips at night and weekends when batch jobs pause ,while daytime stays steady with inference.

Resource connection and manual scheduling of batch workloads.

Its weekly pain point , not daily but still requires significant manual effort .we often have to adjust batch job priorities and reschedule them manually to ensure inference workloads get priority during peak hours .

Smarter scheduler to auto -prioritize inference over batch jobs.

We deprioritize and accept slower batch completion times.

Delays of 2-4 hours peak inference hours .

Its mostly to avoid the operational overhead of preemption and restart ,as retraining from scratch wastes time and resources .keeping batch jobs running slower is simpler compromise than dealing with failed checkpoint and job restarts.

Eliminating it entirely would be the ultimate success ,but cutting it in half would be a solid improvement ideally , I want a scheduler that automatically handles priority decisions so I don't have to think about it at all.

I 'd want it to optimizie for inference SLA protection first ,since customer -facing workloads are our top priority .After that ,I'd want it to minimize the impact on batch job completion time while also considering overall cost efficiency.

Integration with Kubeflow and existing monitoring tool required.

We' d need a clear proof-of-concept timeline showing measurable ROI and internal skateholder buy-in from engineering and finance teams ,A successful pilot on a small cluster would help build confidence before scaling across the fleet.

50 % reduction in manual scheduling and inference SLA consistenly met .

Both equally ,within the next 3-6months.

Need better checking to reduce batch retry overhead.

They're separate but need an integrated solution together.

Yes ,that's exactly the solution we need.

Legacy non-Kubernetes workloads need coexitence with the solution.

Legacy workloads run on separate dedicated hardware ,not Kubernetes.

No , that covers everything we discussed today.

we run a mix of a100s for training and a few l40s for model inference and they work very well for our workloads

we are actually very close to a seventy thirty split with more a100s since training needs more resources and this works well for us

we use infiniband for the training nodes to get good speed and we also use gpudirect which has been very helpful for us

correct the l40s nodes just use regular ethernet and we use the a100s for fine tuning our models and the l40s to serve the live api requests

fine tuning is mostly done on demand when new data comes in and about seventy percent of our total gpu time goes to training and thirty percent to serving

we use ray to orchestrate our training jobs across the clusters and we use vllm for serving because it is very fast and works well for us

we can see general node level metrics using prometheus and grafana but trying to track real time utilization per training job or user is pretty fragmented and hard

our average utilization is around fifty percent on the a100s when we run active training runs and the l40s inference fleet averages about forty percent depending on user traffic

yes it is a daily thing especially at night for inference because we keep nodes warm to avoid cold starts and the training nodes sit idle between ray jobs

we spend around one hundred and fifty to two hundred thousand dollars a month on our gpu infrastructure which is a big part of our cloud bill

we do some basic bin packing manually and scheduling tweaks but we avoid spot instances because training runs get interrupted too easily

there is definitely a lot of headroom to make things better but we need automated ways to do it without hurting our developers or system stability

the biggest waste is the a100 training clusters staying active between our ray jobs and also keeping the l40s inference nodes warm overnight when traffic is very low

solving the a100 idle time would have a much bigger budget impact because those training nodes make up the bulk of our spend and cost a lot more to run

the ideal solution would automatically scale down the ray nodes immediately when the training jobs finish and also handle queuing so we can pack jobs better

immediate scale down would make a faster impact on our bill because stopping those expensive empty a100 nodes would cut the waste right away

we would need a dry run mode to verify the actions first and very clear checks that all training checkpoints are saved and no ray actors are running on the node

both saving money and having zero broken pipelines is success

we would look at existing tools first to see if they fit before trying to build anything in house

integration with our existing monitoring is the most important criteria for us

native prometheus metrics export is exactly what we need

NVIDIA H100

It's mix. Roughly 70% H100s and 30% A100s, with H100s handling most new AI workloads.

Yes. Our H100 cluster uses InfiniBand with GPUDirect RDMA for low-latency.

It's deployed on most of the H100 cluster

We use them for both training and inference.

Both

Training is clearly dominant. It is roughly 60% training and 40% inference

We primarily use PyTorch for model development, vLLM and NVIDIA Triton Inference Server for inference, Ray for distributed workloads.

we keep them separate. Training runs on a dedicated Kubernetes cluster optimized for multi-node GPU jobs, while vLLM and Triton are deployed on separate inference clusters to isolate production traffic, simplify scaling, and avoid resource contention.

We have good visibility, but it's not completely unified. We monitor GPU utilization, memory usage, temperatures, power, and job health across all clusters using Prometheus and Grafana, with NVIDIA DCGM Exporter providing GPU metrics. We have cluster-level dashboards, though cross-cluster reporting is still being improved.

On average, the training cluster runs at around 70–80% GPU utilization. During large distributed training jobs it often exceeds 90%, while between jobs or during scheduling gaps it can drop to around 50–60%.

The inference clusters average 40–50% GPU utilization. They typically run lower than training to maintain headroom for traffic spikes and meet latency targets, with utilization occasionally rising to 60–70% during peak demand.

We track costs monthly. Our GPU infrastructure costs are roughly $1–2 million per month, including compute, networking, power, and support contracts, though the exact figure varies with capacity expansion and utilization.

We have a dedicated FinOps team that owns cloud and infrastructure cost governance. Our platform engineering team works closely with them by providing utilization data, identifying optimization opportunities, and validating the impact of infrastructure changes.

We focus on a few key optimization strategies: GPU bin-packing to maximize utilization, idle GPU reclamation for abandoned or completed jobs, Kubernetes scheduling optimizations to reduce fragmentation, and rightsizing GPU allocations based on workload requirements. For fault-tolerant workloads, we also use spot instances where appropriate, but our production training and inference clusters primarily run on on-demand or reserved capacity for reliability.

It's still being refined, but we've seen around a 10–15% improvement in overall GPU utilization. That has translated into roughly 8–12% infrastructure cost savings, mainly through better scheduling, higher cluster efficiency, and reclaiming idle GPU capacity.

The biggest remaining opportunity is on the inference side. Training is already running at relatively high utilization, so the larger gains are likely to come from improving inference efficiency through dynamic autoscaling, better request batching, and more intelligent workload placement. We also see opportunities to reduce idle capacity during off-peak periods while still maintaining our latency targets.

No

It's a combination of factors. The biggest constraints are balancing latency SLAs with higher GPU utilization, limited cross-cluster scheduling capabilities, and the engineering effort required to validate optimization changes without disrupting production. Improving observability and automation is also a priority before making more aggressive optimization decisions.

I'd prioritize automation and tooling for scheduling and batching. We already have reasonable observability, but safer automation would let us implement optimization changes faster, reduce operational risk, and improve GPU utilization without impacting production performance.

I'd prioritize automation for scheduling and batching. Our observability is already good enough to identify optimization opportunities, but better automation would let us roll out changes safely, reduce manual effort, and improve GPU utilization without risking production stability.

The must-haves would be multi-cluster awareness, latency and SLA protection, safe rollback capabilities, and real-time cost and GPU utilization visibility. It should also integrate cleanly with Kubernetes, support policy-based automation, and provide clear audit logs so changes are transparent and easy to validate.

Nothing specific

I think I'd start with a centralized optimization and orchestration platform that provides unified visibility across all inference clusters and can safely automate scheduling, batching, and scaling decisions. That foundation would make it much easier to add cost optimization, policy enforcement, and performance tuning without relying on separate tools.

I think We're in the evaluation phase. We have a clear understanding of our requirements and are assessing a few platforms to see which best fits our Kubernetes environment, GPU infrastructure, and operational workflows before making any deployment decisions.

Yes

L40S

Mixed in with other models

35%

a100 and H100

30% A100 and 35% H100

No

Both

Rough split

Closer to 50/50

VLLM

No

We are 75% visibility

We track per team. The blind is 25% security monitoring

The 75% is tracking per team workload. We are using a homemade software to capture visibility.

All of the above

Just for alerting

We don’t have exact figures but somewhere between 5-10’million

It’s well optimized by finance

Finance is involved in all assets

Approval by finance

Getting C-suite approval

Budget and risk are weighed in as much as projected ROI

No measurable wins

Can’t really say

Better ROI math

Hardest to quantify savings perceived

Both

This is possible

Time-boxed test

A month is enough

All GPU nodes we use

Test on subsets

H100s

Just output

Rightsizing

Consolidation

Measure what’s possible

Both

Throughput

Measure continuously

No

No

A few weeks to a month

Sure 4-5 weeks is good

Across our entire infrastructure, we manage roughly 250 to 300 total nodes across three main clusters (Production, Staging, and Research). Our 50+ GPU nodes represent about 15% to 20% of our total footprint. The rest of the cluster consists of high-memory CPU nodes handling data ingestion, preprocessing pipelines, and general microservices. This ratio allows us to keep our heavy compute pipelines fully saturated without bottlenecking on data preparation.

Across the entire infrastructure, we manage about [Insert Number, e.g., 200] total nodes, so our GPU-to-total ratio sits around [Insert Ratio, e.g., 1:4]. This gives us plenty of CPU headroom for data ingestion, preprocessing, and orchestration without bottlenecking the training pipeline.On the GPU side, our 50+ nodes are primarily equipped with [Insert Model, e.g., NVIDIA H100 SXM5 80GB] GPUs, configured as [e.g., 8-GPU nodes]. We also maintain a smaller pool of [Insert Model, e.g., A100s or L40S] for lighter fine-tuning and inference workloads

Across all clusters—including dev, staging, data engineering, and production—we manage roughly [e.g., 200 to 250] total nodes. This puts our GPU-to-total ratio at about [e.g., 1:4 or 25%]. The remaining CPU-only nodes handle our heavy Apache Spark data preprocessing pipelines, ETL, and vector database indexing. To clarify the tiering, the 50+ nodes footprint refers strictly to our high-throughput H100 clusters dedicated to large-scale LLM pre-training and fine-tuning.Beyond those 50+ H100 nodes, we run an additional [e.g., 20 to 30] nodes equipped with A100s and L40Ss. Roughly speaking, our GPU split is about 70% H100s for heavy lifting and 30% A100/L40S for active inference, embedding generation, and smaller quantization tasks.

Yes, we use infiniband

To fulfill that scale of large model training, we deploy NDR InfiniBand (400 Gb/s per port) using a rail-optimized topology across all 50+ H100 nodes. This was a highly deliberate choice specifically engineered for multi-node training, rather than a legacy architectural constraint.

"We run a balanced mix. We pre-train large-scale models to build foundational knowledge, heavily fine-tune them for specific domains or reasoning tasks, and serve them for production inference.

around 40% 40% 20%

Lifecycle & Hardware Management: The NVIDIA GPU Operator serves as the robotic administrator to automate GPU driver installations, the NVIDIA Container Toolkit, and the Kubernetes Device Plugin. For shared or multi-tenant environments, Multi-Instance GPU (MIG) and Time Slicing are actively utilized.

We apply MIG (Multi-Instance GPU) extensively across both the H100 and A100 nodes for strict hardware isolation in multi-tenant or production inference scenarios. Time-slicing is primarily used on the L40S/A100 nodes for dev/test, and we even apply a hybrid approach on H100s where we time-slice within MIG slices.

Hybrid MIG + time-slicing is typically a deliberate, strategic decision to maximize hardware utilization and density, rather than an opportunistic reaction to headroom. It allows you to partition physical H100s into guaranteed hardware boundaries (MIG) for primary workloads, while layering time-slicing on top to maximize container density for secondary, bursty tasks.

Modern observability tools provide a deeply granular, real-time picture of GPU fleets. Teams are no longer limited to high-level summaries; they can track per-node metrics, per-workload breakdowns (like Kubernetes namespaces or pods), and per-process metrics simultaneously.

also are you going to give me my survey points

"We currently have robust per-node and per-workload visibility across our infrastructure. However, real-time, highly granular per-process tracking is an ongoing evolution. We are actively building out our tooling to achieve that level of deep observability as part of our target state."

Monthly spend easily ranges from $300,000 to $1,000,000+ for sustained, high-throughput inference and training workloads.

Utilizing the NVIDIA GPU Operator to timeshare or oversubscribe GPUs, allowing multiple developer or light-inference workloads to interleave on a single physical card. For heavier production use, Multi-Instance GPU (MIG) is leveraged to slice GPUs like the A100 into physically isolated segments.Dynamic Schedulers & Bin-Packing: Moving away from default cluster managers toward intelligent queue managers like Kueue or Volcano to pack workloads tightly. This ensures smaller jobs are consolidated onto fully loaded nodes rather than leaving GPUs in a fragmented, partially idle state.Spot Instances & Interruption Handlers: Shifting fault-tolerant batch workloads and iterative training routines to Spot Instances. Because Spot GPUs cost significantly less than on-demand instances, application-level checkpointing is integrated so jobs can quickly resume after preemptions.Rightsizing & Autoscaling: Utilizing tools like Karpenter to automatically provision and scale down infrastructure based on exact workload requirements. Continuous monitoring of actual SM (Streaming Multiprocessor) usage using DCGM Exporter helps rightsize workloads to avoid over-provisioning.Commitment Plans & Off-Peak Scheduling: Utilizing Savings Plans and Reserved Instances for steady-state workloads to capture baseline discounts, while scheduling heavy training jobs to execute during off-peak windows or lower-demand time zones.Combined, these tactics shift clusters from a model of dedicating one full physical GPU per workload to a fluid, elastic compute fabric that drastically reduces overall infrastructure costs.AWS EC2 Cost Optimization: How to Cut Your Compute Bill - ScaleOpsMay 6, 2026 — Key takeaways * Rightsizing is the highest-leverage EC2 cost optimization step because instance size affects cost, performance, an...ScaleOpsHow to make the most of your GPUs (part 1 - time-slicing) - Red HatJul 2, 2024 — This approach helps you get more throughput, keeping the overall system latency under specific service level agreements (SLAs), an...Red Hat5 Proven Strategies to Cut Your AI Infrastructure Cost by 60%Jun 23, 2026 — GPU Cost Optimization: 5 Proven Strategies to Cut Your AI Infrastructure Cost by 60% * Why GPU Costs Are the New Cloud Cost Proble...www.nops.ioGPU Cost Optimization: A Practical Guide for AI Teams - AmnicMar 25, 2026 — Start with a cost audit: * Tag every resource by team, project, and experiment ID from day one. * Export billing data to a BI tool...AmnicMIG vs Time-Slicing vs MPS: GPU Sharing on KubernetesApr 17, 2026 — If you do not need isolation (dev/test, single-team workloads, lightweight inference), you can choose between Time-Slicing and MPS...Sharon SahadevanFractional GPUs in Kubernetes: MIG, Time Slicing & Custom ...Jul 11, 2025 — 1. MIG: Multi-Instance GPU MIG (Multi-Instance GPU) is a hardware-level GPU partitioning feature introduced by NVIDIA for Ampere a...RafayKubernetes GPU Scheduler: Bin-Packing and Gang ...Feb 15, 2026 — Kubernetes GPU Scheduling is Inefficient Default Kubernetes GPU scheduler is simple: find a node with a free GPU, place pod there.www.rajathjohn.comTime-Slicing GPUs in Kubernetes — NVIDIA GPU OperatorGPU time-slicing enables workloads that are scheduled on oversubscribed GPUs to interleave with one another. This mechanism for en...NVIDIA DocsGPU Cost Optimization - InfracostSpot instance strategies represent one of the most effective methods for reducing GPU costs. Despite interruption risks, careful i...Infracost9:30Kubernetes GPU Scheduling: Efficient AI Workload ...YouTube·UplatzPractical Tips for Preventing GPU Fragmentation for Volcano ...Mar 31, 2025 — Our approach to resolving GPU fragmentation involved three key components: * Workload prioritization: Resources were ranked in des...NVIDIA DeveloperWhat are spot GPUs? Complete guide to cost-effective AI infrastructureAug 15, 2025 — Let's see what spot GPUs work great for: * AI model batch inference APIs: Most batch inference processes handle multiple requests ...Northflank

Today, bin-packing and time-slicing are live in our production environment for dev/test and light inference, driving immediate cost savings. Rightsizing and scheduling are actively in pilot, while full spot instance automation for heavy training remains on the roadmap pending checkpointing integration.

Deploying aggressive scale-to-zero autoscaling, bin-packing via Karpenter, and time-slicing via NVIDIA Kubernetes Device Plugin are core active tactics. Bin-packing reduces GPU node footprint by roughly 35-45%, and time-slicing cuts developer-specific GPU costs by up to 75%. The combination of Spot Instances and sharing yields 60-93% savings.

spot instances

i think bin packing is the most expensive but spot still has the biggest bang

Issues with staff

Oh i'm sorry. the fragmentation issues make it expensive

Memory fragmentation is most prevalent on the A100 and L40S architectures, primarily because these setups frequently rely on unoptimized PyTorch memory allocators and traditional, contiguous KV cache allocations.

NVIDIA TensorRT-LLM, new optimizations are rolling out to handle priority-based KV cache evictions directly. Offloading and compression techniques (like NVIDIA Dynamo's instant offload capabilities) are also actively being adopted to bypass VRAM limitations without stalling the GPU.

Offloading & Compression: These memory management techniques are largely still in the evaluation and pilot phase for many teams. While promising in research, they pose production engineering hurdles because older attention kernels do not always support partial token eviction.

is the survey almost done?

Continuous rightsizing will have the biggest infrastructure impact over the next 12 months. While Spot instances yield steeper discounts, rightsizing is the foundational pillar that eliminates baseline waste, stabilizes performance, and creates the baseline accuracy needed to actually implement automation, bin-packing, and memory optimizations without risking outages.

Kernel compatibility for KV cache eviction causes the most operational friction because it directly pits hardware math against user experience. Trying to evict tokens on the fly usually frees zero memory due to spatial fragmentation in paged attention. It ultimately stalls decode throughput if the underlying compiler cannot execute it perfectly.

NVIDIA A100 AND NVIDIA H100 Tensor Core GPUs

The fleet is split roughly 60% A100 and 40% H100. We use the H100 nodes for high-throughput LLM training and heavy AI workloads, while keeping the A100 nodes for standard data processing and inference tasks to optimize costs

Yes, we utilize InfiniBand and RoCE v2 for low-latency networking across our GPU clusters. We also rely heavily on GPUDirect RDMA to enable direct data exchange between GPU memories, which significantly boosts our distributed training performance

The workload is split, but we lean slightly more towards inference, which takes up about 60% of our capacity to support our production models. The remaining 40% is dedicated to continuous training and fine-tuning workloads on our H100 nodes

We are running PyTorch via the Kubeflow Training Operator along with DeepSpeed for our distributed training and fine-tuning workloads on the H100s. For the inference side on our A100s, we primarily deploy vLLM and NVIDIA Triton Inference Server managed by KServe to efficiently handle model serving and maximize GPU utilization

Yes, we use the NVIDIA GPU Operator to handle the baseline driver and time-slicing configuration. To manage the actual scheduling and avoid contention between the training and inference workloads, we rely on a combination of Kueue for batch training queue management and Karpenter for dynamic node provisioning, using taints and tolerations to keep the inference and training node pools strictly separated

We have pretty solid, real-time visibility at both the node and workload level. We use Prometheus to scrape metrics via the NVIDIA DCGM Exporter, which feeds into Grafana dashboards. This allows us to track real-time GPU utilization, memory allocation, and SM activity per pod and per node, so we rarely have blind spots

Our cloud infrastructure and monthly GPU spend runs around $60,000 to $80,000, bringing our annual cost to roughly $700,000 to $1,000,000 depending on how heavily we scale our training runs on the H100 nodes

We are actively leveraging Karpenter to automatically downscale or terminate idle nodes as soon as training jobs complete. Additionally, we enforce tight bin-packing strategies via Kueue to ensure high multi-tenant density and use fractional GPU configuration/time-slicing on our A100 nodes for smaller inference workloads to minimize wasted capacity

While we've captured the obvious low-hanging fruit with infrastructure-level scaling, we still see pockets of inefficiency in model-level waste. Specifically, during distributed training runs on our H100 nodes, we experience occasional memory fragmentation and sub-optimal data-loading pipelines that leave GPUs idle for brief windows, meaning there's definitely still room to optimize our code execution and cluster data throughput

The greatest potential for cost savings actually lies in fixing those data pipeline bottlenecks and memory fragmentation issues during training, because idle H100 time is incredibly expensive at our scale. If we can maximize data throughput and avoid those stall windows, we can significantly shorten our training cycles and directly reduce our cloud compute spend

We haven't fully itemized it down to the dollar, but based on our telemetry data showing roughly 15% to 20% aggregate idle time during training blocks, we estimate those stall windows cost us between $5,000 and $8,000 a month. Eliminating that waste would translate to a direct saving of up to $100,000 annually

If we could fix those issues instantly, success would look like a consistent 15% reduction in our overall training cycle time, while simultaneously pushing our average GPU Tensor Core utilization above 85% during active runs. Achieving that would give us the predictability we need to ship models faster without blowing past our budget

If I had to pick the single biggest lever, it would definitely be data pipeline optimization. Eliminating storage and data-loading bottlenecks to keep the H100 Tensor Cores constantly fed with batches will have a much more immediate, transformative impact on our overall training velocity than tackling fragmented memory allocations

Our biggest blocker right now is a combination of engineering bandwidth and deep visibility into the pipeline itself. While we have solid top-level metrics from DCGM, tracking exact I/O wait times, serialization overhead, and network throttling down to individual steps in the custom data-loader is incredibly time-consuming to set up, and our engineering team is currently stretched thin prioritizing model feature work

If we could get automated, deep visibility without the heavy instrumentation lift, it would completely change the math for us. Instead of our engineers spending weeks building custom tracing tools to find the bottlenecks, they could jump straight into actionable fixes during quick optimization sprints. It would turn a complex, multi-week engineering project into highly targeted fixes, allowing us to reclaim that $100k in annual waste without derailing our core model feature roadmap

The one thing it absolutely has to do is correlate low-level GPU stall metrics directly with our high-level data loader steps. It needs to point to the exact file, bucket, or preprocessing function causing the delay and tell us, "Your H100s are waiting right here for 500ms per batch because of this specific serialization block." If it can pinpoint the exact line or data source causing the stall out-of-the-box, it immediately becomes a high-value tool for us

What would drive us to actually adopt it is a truly frictionless, zero-code installation process combined with negligible performance overhead. If we can deploy it as a lightweight daemon or sidecar within our Kubernetes/Kubeflow clusters without having to modify our training scripts or rebuild containers, it eliminates the onboarding friction entirely. While integration with our existing Prometheus/Grafana stack for long-term alerting would be a nice bonus, the absolute dealbreaker for immediate adoption is that it requires near-zero engineering effort to get that first actionable insight

Absolutely. If a tool can deliver that level of specific, actionable insight on the very first training run—without making us jump through configuration hoops—it's a no-brainer for a proof-of-concept. Showing our team the exact root cause of an H100 stall right out of the gate is more than enough to justify dedicating a quick sprint to test it further and validate the potential $100k savings

We would definitely loop in both. The infrastructure team would handle the initial cluster-level deployment and make sure the sidecar or daemon runs smoothly without breaking anything. However, the machine learning engineers running those actual training jobs are the ones who ultimately need to look at the insights, validate them against the data-loader code, and implement the actual pipeline fixes to hit our optimization targets

A mix of NVIDIA H100s and A100s

It is roughly a half-and-half split between the two models.

Yes, we use InfiniBand and GPUDirect to accelerate pur network.

It is deployed across all of our GPU nodes.

We primarily use them for both training and inference.

It is roughly a half-and-half split between training and inference.

We run Ray, Kubeflow, and LLM on top of Kubernetes.

I am referring to the open source vLLM framework.

We have limited visibility and want better tools to track utilization.

We only use basic NVIDIA tooling like nvidia-smi and simple dashboards.

We manually SSH into nodes and pull logs because we lack an aggregation layer.

Our average GPU utilization is low, roughly around 30 to 40 percent.

We are experimenting with better job scheduling and batching to raise pur utilization.

We are still in the early days and have not seen major wins yet.

It is a mix of GPU sitting idle and jobs not fully saturating the hardware.

It is mostly an infrastructure issue due to memory constraints and scheduling.

We're facing contraints with both GPU memory and system memory on the nodes.

Memory constraints are the main blocker, but scheduling policies also play a role.

We lack clear visibility to separate idle time from underutilized running time.

We estimate about 40% is completely idle and 60% is running jobs.

Our monthly GPU infrastructure spend is roughly $150000

There is definitely frustration because we are spending a lot compared to what we actually use.

We see the greatest savings from fixing node scheduling and bad batching practices.

We would choose smarter scheduling tooling to automatically balance and optimize our workloads.

They're spread across multiple environments. Most of our GPU nodes are in production Kubernetes clusters, with smaller pools dedicated to development, testing, and machine learning workloads. We primarily use a managed cloud environment, with a small on-premises cluster for specialized internal workloads and validation.

Approximately 70% of our GPU nodes are in production, 15% are allocated to machine learning and AI development, 10% support development and testing, and the remaining 5% are in our on-premises environment for internal validation, proof-of-concept work, and specialized workloads.

We're primarily standardized on NVIDIA GPUs across the environment. Most production and AI workloads run on NVIDIA A100 and H100 GPUs, while some development and testing clusters still use NVIDIA L4 and T4 GPUs for lower-cost inference and validation workloads. We don't currently use AMD GPUs in production because our AI and Kubernetes tooling is optimized around the NVIDIA CUDA ecosystem.

Yes. Our production AI clusters use InfiniBand with GPUDirect RDMA to support distributed training and high-performance workloads, while the rest of our Kubernetes environments primarily use RoCE (RDMA over Converged Ethernet) for lower-latency networking. Our development and testing clusters generally rely on standard high-speed Ethernet without specialized network acceleration.

It's a mix of both, but the majority of our GPU capacity is dedicated to AI inference in production. Roughly 65% supports inference for customer-facing applications and internal AI services, while 35% is used for model training, fine-tuning, experimentation, and validation by our data science and machine learning teams.

For inference, we primarily use NVIDIA Triton Inference Server and vLLM to serve large language models efficiently. We also use Ray for distributed AI workloads and orchestration, with PyTorch as our primary machine learning framework and NVIDIA TensorRT to optimize inference performance in production.

They handle different workload types. Roughly 60% of our inference traffic runs through vLLM for large language model serving because of its efficient token throughput and memory management, while 40% runs through NVIDIA Triton for computer vision, recommendation models, and other optimized deep learning workloads that benefit from its multi-framework support.

Yes. We have centralized monitoring dashboards that provide near real-time visibility into GPU utilization across all Kubernetes clusters. On a typical day, 70–80% of our GPU fleet is actively utilized, and we can monitor metrics such as GPU utilization, memory usage, temperature, power consumption, and workload allocation at both the cluster and node level, with alerts for underutilized or oversubscribed resources.

We primarily use Prometheus and Grafana for monitoring and dashboards, with the NVIDIA DCGM Exporter to collect detailed GPU metrics. We also integrate with our cloud provider's native monitoring services for infrastructure health and alerting, giving us a single view of Kubernetes, GPU performance, and application workloads.

Yes. We track GPU infrastructure costs closely through our cloud cost management platform and internal FinOps reporting. Across cloud and on-premises environments, our GPU infrastructure costs are approximately $600,000–$800,000 per month, with annual spending in the range of $7–10 million, depending on AI training activity and production demand.

Based on our internal utilization reviews, I estimate that 15–20% of our GPU spend is tied to idle or underutilized capacity. If we could eliminate that waste through better scheduling, autoscaling, and workload placement, we could realistically save $1–2 million annually while maintaining our current service levels.

It's a combination of factors, but the biggest challenges are engineering bandwidth and workload complexity. Many AI workloads have unpredictable demand, and we intentionally keep spare GPU capacity available to meet performance and availability targets. We're continuously improving scheduling and autoscaling, but balancing cost optimization with reliability and fast response times is the primary constraint.

Realistically, I think we could recapture about 50–60% of those potential savings without affecting our availability or performance targets. That translates to roughly $600,000–$1.2 million annually, primarily through better workload scheduling, rightsizing GPU allocations, and improving autoscaling while still maintaining sufficient capacity for peak demand and failover scenarios.

The single biggest blocker is engineering bandwidth. We know where many of the optimization opportunities are, but our platform team is focused on supporting production AI workloads and delivering new capabilities, so GPU cost optimization projects often take a lower priority than reliability, security, and feature delivery.

If I could choose one capability, it would be an AI-driven GPU resource optimization platform that automatically analyzes workload patterns, predicts demand, and continuously optimizes scheduling, autoscaling, and workload placement across Kubernetes clusters. The biggest value would come from reducing idle GPU time and rightsizing allocations automatically, while maintaining our performance and availability targets without requiring constant engineering effort.

The biggest pain points today are GPU fragmentation and scheduling inefficiencies. We occasionally end up with available GPU capacity that can't be used efficiently because resources are spread across clusters or tied to workloads with different GPU and memory requirements. Better workload placement, bin-packing, and more intelligent scheduling across clusters would help improve utilization without affecting application performance or availability.

If I could solve just one tomorrow, it would be intelligent workload placement and scheduling. Improving that would have the biggest impact because it would reduce GPU fragmentation, increase overall utilization, improve bin-packing efficiency, and lower infrastructure costs without compromising performance or availability.

Running 100s primarily, with A100S GPUs in mixed cluster.

Mostly A100s, with some H100s and L40S GPUs mixed across nodes.

Yes, we use InfiniBand and RDMA with GPUDirect support across GPU nodes.

Primarily for distributing training workload, though InfiniBand and RDMA are standard across GPU nodes.

We run Kubeflow, NVIDIA NGC, PyTorch and TensorFlow on Kubernetes.

We use batch scheduling, running training jobs daily with frequent automated launches.

Most runs use 70–85% capacity, with batch jobs filling idle GPU resources.

We track per-node and per-workload metrics using Prometheus, Grafana, and NVIDIA DCGM.

We track historical averages too, typically seeing 75–80% GPU utilization cluster-wide.

We estimate around $500K monthly, including hardware, power, and infrastructure costs.

It’s reasonable, but we see opportunities to optimize scheduling and efficiency.

Mainly workload bin-packing and scheduling improvements; limited MIG or time-slicing currently.

Main blockers are tooling maturity, integration complexity, and limited engineering bandwidth.

Mostly ecosystem gaps around cost tracking, scheduling optimization, and bin-packing tools.

Better scheduling optimization would likely deliver the biggest efficiency gains overall.

Smarter bin-packing with dynamic scheduling and automated resource optimization.

A separate Kubernetes scheduling layer integrated with Kubeflow workflows.

Mostly a vision today; we haven't prototyped that layer yet.

noting else

We primarily use an equal mix of NIVIDIA H100 and A100 models

Yes, we utilize InfiniBand and RDMA/RoCE across our cluster

It was a foundational architecture decision to support future scaling

We use them for both large training and real time inference

Our Utilisation is currently split very close to 50/50

We primarily use PyTorch along with Kubeflow and TensorRT

run TensorRT through Triton inference Server for deployments

We handle scheduling via Kubernetes taints tolerations, and custom schedulers

It works well, but we occasionally encounter minor resource encounter minor resource contention

Training jobs face slight delays while interface latency remains stable

jobs typically pile up in the scheduling queue before running

Yes, we truck exact cluster utilisation via Grafana dashboards

We stay quite busy, averaging around eighty percent utilisation

We are actively planning to expand our overall cluster capacity

We are reconsidering the split to favor newer hardware

we are actively exploring options like the newer H200 chips

The goal is to eventually phase out the oldest models

we will keep them around for lower priority testing workloads

Our annual infrastructure spend is approximately two million dollars

We actively enforce rightsizing and aggressive bin-packaging across nodes

We maintain a tight ten percent buffer for unexpected spikes

It is good, but there is definitely room to improve

workload-specific optimization is our biggest gap

Tuning models for better GPU efficiency

The infrastructure team handle tuining

We use NAVIDIA's optimization frameworks

we focus specifically on TensorRT quantization

Only on our interence models

Yea, we use dynamic batching extensively

Multi-instance GPU utilization needs improvement

we are currently considering deploying it

increased scheduling complexity holds us back

managing taints alongside MIG slicing

Layering taints on top seems best

Automated workload rightsizing tools

We would need to build it

We covered all the main opportunities

The hardware refresh, for immediate gains

Our GPU equipped nodes are primarily running NVIDIA A100 Tensore Coe GPUs

Our nodes are uniform, we standardly run the 80GB PCIe variants of the A100 across the entire fleet to keep memory capacity consistent for our workloads

Yes, we run infiniBand fabric across the cluster and fully leverage GPUDirect RDMA.

It was heavily driven by specific workload requirements

I mean multi-GPU training Jobs

We are primarily running PyTorch Distributed for our multi-GPU training workloads

Yes, we are using Kubeflow

We've integrated an advanced secondary scheduler on top of Kubeflow, specifically using queue to handle multi-tenant queueing and strict scheduling

Yes it does

I'm seeing better job completion times

We've seen roughly a 20% to 25% improvement in job completion times

Yes it has freed up capacity to run concurrent training jobs

We see real-time utilization metric per node

We are running the standard Prometheus and Grafana stack

We are utilizing the NVIDIA DCGM exporter daemonset on our kubernetes nodes

Yes we do have visibility

We typically average around 70% to 75% GPU utilization across the entire fleet during active development cycles

It fluctuates based on human development cycles and job queue gaps

Yes, idle nodes

We typically track it on a per GPU hour metric to keep our workload efficiency calculations clean. For our A100 8GB instances, our blended rate sits right around $2 to $2.5 per hour per gpu

That rate represents our cloud infrastructure provider commitments but it is primarily compute only for the GPU instances themselves

Our primary optimization tactics focus on aggressive bin packing and gang scheduling via the secondary layer

Yes, we are hitting target

We still strive for more experience but we are currently satisfied

We're chasing optimization headroom around our model checkpointing and serialization processes.

It is never a blocking issue

More of the storage cost issue

We're focusing heavily on optimizing where and how they are stored via automatic data lifecycle policies.

Our current automation is primarily driven by age

It's mor about moving and deleting

Yes we are seeing meaningful cost savings

No, not at all

If we're taking cost out of the equation, our primary engineering friction comes down to multi tenancy queue dynamics.

We handle directly at the scheduling layer using kueue's multi tenant queueing mechanisms

Yes we to prevent queue starvazation

We use a hybrid set up consisting of a high performance parallel file systems for active training and standard object storage for long-term retention

If I had a clean slate, what's working really well and wouldn't change is the underlying hardware and storage foundation.

If I could re engineer the layers on top, I'd want to erlimate the sheer amount of configuration glue required to make kubernetes topology- aware

I lean heavily toward a purpose-built abstraction layer that handles it for us

not at all

We are using Hopper architecture class H200 and we are also using Blackwell architecture class B300

We are using mostly H200 and we some of B300 for specialized tasks

Yes we do use network accelerator like gpudirect and rdma

Across the entire fleet we have deployed

We are using it for training models and for inferencing

Most of it is for training models and some for inferencing

Ww are running pytorch and vLLM on top of kubernates

Yes we are using the same fleet for this

We have complete visibility and we are actively monitoring it in real time

We are using nvidia dcgm and datadog both for visibility

We checking datadog more as it gives complete visibility across our entire infrastructure

Yes i have some idea about that

Hundreds of thousands per month

We are active looking for ways to optimize this cost

We are looking for combination of all these ways as and when needed

We are satisfied with our current efficiency

Node types and the type of workload we are running both can be optimized for increasing efficiency and we are looking in to that

There is some scheduling complexity in infrastructure which creates some bottleneck but we are working to improve that

Design tools thats low cost and gives better visibility

Intelligent infrastructure that can better optimize and schedule processes

There is no such pain point currently

Faster job placement with cost savings

NVIDIA HOPPER H100, A100,L40S,BLACKWELL B200

IT IS EVENLY DISTRIBUTED BETWEEN B200,H100 AND A100

IT IS SUBSTANTIALLY SMALLER IN VOLUME

YES IT HAS NETWORK ACCELERATORS

NVIDIA BLUEFIELD 2, CONNECT X 7

NO THESE ARE DEPLOYED ACROSS ALL GPU NODES

THEIR MAIN PURPOSE IS TO ACCELERATE AI AND HIGH PERFORMANCE COMPUTING

YES THEY ACT AS A TRANSFORMERE ENGINE FOR NEXT GENERATION LLMS

IT ALSO ENHANCE SCALABLE AI WORKLOADS

NO IT IS MAINLY SERVING FOCUSED

tensorRT IS THE SOFTWARE WE USE MAINLY

WE ARE RUNNING IT DIRECTLY

WE ARE USING STANDARD KUBERNETES SCHEDULING

IT IS MORE OF A FREE FOR ALL SCHEDULER ASSISGNMENT

WE ARE RUNNING SINGLE GPU INFERENCE PODS

IT IS HANDLED BY STANDARD KUBERNETES SERVICE

WE ARE TRACKING IT PER NODE

IT IS JUST AGGERGATE PER NODE

NVIDIA DCGM

YES IT IS EXPORTED INTO A CENTRAL DASHBOARD

YES

I CAN'T TELL YOU THE EXACT FIGURES BUT I THINK IT IS IN MILLIONS

BIN PACKING

IT IS HANDLED BY KUBERNETS SCHEDULER LEVEL

IT IS IN THE EARLY ROLLOUT

WE HAVE NOT FACED ANY SUCH ISSUES YET

I THINK IT IS QUITE EFFECTIVE RIGHT NOW

BECAUSE WE ARE VERY PRODUCTIVE

I WAS MENTIONING ABOUT COST PER INFERENCE

YES TRACKING IT ACTIVELY

RIGHT NOW IT IS STABLE

IDLE TIME UTILIZATION

YES

YES THAT'S WHAT I MEAN

YES THAT'S IT

YES BY DYNAMIC SCALING IMPLEMENTATION

KEDA

REQUEST LATENCY

YES SCALING UP WHEN LATENCY CROSS THRESHOLD

3 TO 5 PENDING REQUESTS PER INSTANCE

3 TO 5 PENDING REQUEST PER INSTANCE

1000MS

IT DEPENDS ON CLOUD PROVIDER

AWS

NO IT DOES NOT HAVE A COOLDOEN PERIOD

YES IT HAS SOME EFFECT

MINOR POSITIVE

IT IS SOMEWHAT EFFECTIVE

CAN NOT SAY ABOUT THAT

NO THERE ARE NO SUCH THINGS

WE DID NOT CONSIDER ANYTHING ELSE

TOOLING COMPLEXITY

IT SHOULD BE A SIMPLER TOOL

TUNING THE THRESHOLD

MORE OF A SET IT AND FORGET IT

MATTER OF BETTER VISIBILITY

REAL TIME IDLE DETECTION

YES

BAKED INTO DCGM

NO PAIN POINTS

NOTHING MORE TO ADD UP

A100, H100, L40S

Heavier on A100/H100 for training, with a few L40S inference. Roughly 70/30 split.

Yes, we use InfiniBad and GPUDirect.

Deployed primarily on the A100/H100 training nodes, not on L40S inference nodes.

A mix of both, but heavier on training.

We use Kubeflow, Ray, vLLM, and Triton Inference Server.

vLLM handles most LLM inference, Triton supports broader model types. Training runs on Kubeflow and Ray.

Moderate visibility - we have dashboards but lack granular per-node GPU metrics.

We check logs, run kubectl describe nodes, and use nvidia-smi.

Mostly reactive monitoring, We have alerts for node failures but limited historical data.

Rough estimate, around $40k-$60k monthly across all GPU nodes.

That's the raw compute cost, excluding storage and networking.

We use spot instances for non-critical workloads and right-size node types.

Savings are noticeable, but churn and rescheduling add some overhead.

More of a gut feeling - we haven't formally quantified the overhead impact.

Decent but not optimal - there's clear headroom for better utilization and cost control.

Underutilized GPU nodes during off-peak hours and overprovisioned inference workloads.

We see 20-30% average utilization during off-peak but haven't formally tracked it.

Mostly workload patterns - batch training jobs scheduled duting daytime hours.

We've enabled some scheduling but haven't fully optimized for off-peak packing.

Better real-time visibility and historical utilization tracking.

Engineering time and existing tooling gaps - not enough bandwidth to implement.

Yes - if it's low-effort and cost-justified, I'd explore it.

NVIDIA A100 (40GB/80GB) — majority of the cluster, with some NVIDIA L40S nodes for inference workloads. A100 80GB: ~70% L40S: ~30%.

Yes — our GPU nodes use high-speed networking with InfiniBand/RDMA support, and we use GPU Direct where needed for distributed workloads and large-scale model training.

We use the GPUs for both training and inference. The workload is roughly split around 60% training and 40% inference, with training focused on model development and fine-tuning and inference supporting production applications.

We run a mix of GPU-accelerated frameworks on Kubernetes, including PyTorch and TensorFlow for training and fine-tuning workloads. For orchestration and serving, we use tools like NVIDIA GPU Operator, CUDA libraries, Kubernetes operators, and inference platforms such as NVIDIA Triton Inference Server for production deployments.

Yes, we use additional orchestration layers on top of Kubernetes for distributed training. We use Kubeflow and Ray for managing training workflows, scheduling distributed jobs, and scaling workloads across GPU resources. We also have some custom Kubernetes tooling for specific internal workflows.

We track GPU utilization at multiple levels, including per-node and per-workload visibility. We also monitor usage by team to help with capacity planning, resource allocation, and identifying underutilized GPU resources.

We use a combination of NVIDIA DCGM and nvidia-smi for GPU-level metrics, with Prometheus and Grafana for monitoring dashboards and alerting. We also integrate Kubernetes metrics to correlate GPU utilization with workloads and resource allocation.

Our average GPU utilization across the cluster is around 70–75%. It varies by workload, with training jobs typically driving higher utilization and inference workloads fluctuating based on demand.

Our GPU infrastructure costs are roughly in the range of several hundred thousand dollars per year. Including compute, storage, networking, and supporting infrastructure, the total spend is approximately $500K–$1M annually.

We have visibility into GPU infrastructure spend at both the infrastructure and team levels. Costs are tracked by cost center and workload ownership where possible, which helps with budgeting, capacity planning, and understanding resource usage across teams.

We are optimizing GPU usage through a combination of rightsizing workloads, improving scheduling efficiency, and better resource allocation. We use Kubernetes scheduling policies, workload monitoring, and some bin-packing approaches to reduce idle GPU capacity. For lower-priority workloads, we also consider time-scheduling and flexible capacity options where appropriate.

We are seeing measurable improvements, especially in reducing idle GPU capacity and improving overall utilization. The gains are ongoing as workloads change, but better scheduling, monitoring, and rightsizing have helped lower wasted capacity and improve cost efficiency.

The biggest cost-saving opportunities are in optimizing inference workloads, improving GPU scheduling efficiency, and reducing idle capacity during lower-demand periods.

I would start with improving scheduling efficiency and reducing idle GPU capacity.

For us, improving scheduling efficiency mainly means better bin-packing of workloads, more granular priority classes, and smarter resource allocation across teams.

The biggest blocker right now is having complete visibility into workload behavior and resource needs across teams.

It is a combination of both, but the larger challenge is a data and visibility problem. We have some monitoring in place, but we need more detailed workload-level insights and better reporting to understand usage patterns, resource requirements, and opportunities for optimization across teams. Improving processes around workload ownership and resource planning would also help.

We would like more granular visibility at the workload, namespace, and team levels. Per-pod GPU utilization, memory usage, job duration, queue time, and cost attribution by team would help us make better scheduling decisions.

The biggest immediate impact would come from per-workload and per-team GPU utilization data. Knowing which jobs are consistently underutilizing GPUs, how much capacity each team is consuming.

We have some visibility, but it is not complete enough to confidently identify all underutilizers. From what we can see, smaller development and experimentation workloads tend to have more idle capacity compared with larger scheduled training jobs.

Roughly 25–30% of our GPU capacity is used by development and experimentation workloads, while around 70–75% is allocated to scheduled training and production workloads.

Of the dev and experimentation capacity, I would estimate around 40–50% is idle or significantly underutilized on a typical day.

They are a mix of both, but the bigger issue is longer-running development jobs that keep GPU resources allocated while only using them intermittently.

We have some mechanisms in place, but they are not fully automated. We use Kubernetes scheduling policies and resource quotas, and we are evaluating options like time-slicing and preemption for lower-priority development workloads.

Time-slicing feels like the more feasible first step because it aligns better with our current tooling and workload patterns.

We have started testing time-slicing on a limited set of lower-priority development and experimentation workloads.

The early signal is positive — we are seeing better GPU sharing and some reduction in idle capacity. The main friction point is around workload expectations and user experience; development teams need clear guidance on performance trade-offs, scheduling behavior, and how shared GPU access affects their workflows.

We have started communicating the trade-offs to development teams, but it is still an ongoing effort.

The biggest concern is a combination of performance consistency and adoption. We want to make sure time-slicing does not negatively impact critical development workflows, especially for users who are sensitive to latency or training speed.

We have a basic classification approach today based on workload priority, team requirements, and business impact, but it is not fully automated.

The goal is to gradually automate the classification process rather than relying entirely on manual expansion.

We would use a combination of signals, including job duration, GPU utilization history, memory usage patterns, workload priority, team ownership, and whether the workload is interactive or batch-oriented.

We are leaning toward a hybrid approach. We would likely build some internal logic around our specific workload patterns and policies, while also evaluating existing Kubernetes and GPU management tools that can provide automation, scheduling insights, and resource optimization capabilities.

The must-have capabilities would be detailed GPU utilization visibility, workload-level cost attribution, automated scheduling optimization, and the ability to identify and reclaim idle resources.

I would prioritize detailed GPU utilization and workload-level visibility first. Without accurate data on how GPUs are being used, it is difficult to make effective scheduling or idle-reclamation decisions.

We would prefer something that integrates tightly with our existing Kubernetes and Kubeflow environment. A standalone platform could be considered if it provides significantly better GPU visibility, but seamless integration with our current workflows, metrics stack, and scheduling processes would be a major factor in adoption.

This is a near-term initiative rather than a long-term “when we have bandwidth” effort. We are looking at making progress over the next few months, starting with improving GPU visibility and metrics, then expanding into more automated scheduling and workload optimization as we validate the approach.

The biggest deal-breakers would be poor Kubernetes integration, lack of workload-level visibility, or limited support for GPU-specific metrics. We would also avoid tools that require major changes to our existing monitoring stack, add significant operational overhead, or do not provide actionable insights for scheduling and cost optimization.

Mix- mostly A100s and H100s, with a few L40S for lighter workloads.

About 50% H100s, 35% A100s, and 15% L40S.

Yes- we use InfiniBand for the H100 nodes, and RoCE for the A100 and L40S clusters. GPUDirect is enabled across the board.

Multi-GPU training for LLMs, plus some distributed inference. H100 for heavy lifting, A100s and L40S for training and serving.

About 70/30 training to inference, but it does shift during peak research cycles or product rollouts.

Kubeflow for training, Ray for distributed workloads, Triton for inference, plus vLLM experiments.

Separate clusters for training and serving- better resource management and security, but shared storage and networking.

Real-time per-node and per-workload metrics via Prometheus and Grafana, plus periodic reporting for capacity planning.

Training avg 60-70%, inference 40-50%- room to optimize.

$800K-$1M/month across both clusters

Full cost- hardware, storage, networking, and managed services. GPU hardware is the biggest piece.

Bin-packing, rightsizing, preemptible instances for light workloads, and experimenting with dynamic scaling.

10-15% gains so far, still early.

Bin-packing is the win; scheduling flexibility is the bottleneck.

Pod affinity and node selectors limit bin-packing; priority classes don't always match real-time demand.

Taints/tolerations in place for reserved capacity, but limited flexibility.

Manually adjust taints or spin up extra nodes.

10-15 minutes. Usually full nodes, even for smaller shifts.

Mostly over-provisioning for spikes. Packing could be tighter with better scheduling.

Spikes require 20-30% extra capacity, usually lasting 2-4 hours.

Most predictable- tied to product releases and research cycles.

we usually know 2-3 weeks out.

Consider but not prioritized.

Bandwidth- team is stretched thin on core infrastructure.

Integration gap.

Connecting forecast data to orchestration layer.

Logic- deciding when to pre-provision versus wait. Format is manageable.

Balancing cost vs response time. Policy could be simple, but trade-offs need thought.

5-10 minute wait is acceptable for most workloads.

Not quantified yet. Always-on reserve is expensive; provision-on-forecast seems cheaper but needs modeling.

Would need to build- no direct hook yet. Autoscaler is standard, not forecast-aware.

Yes, CornJob would cover most cases.

All three- lower latency, better utilization, and cost savings.

Multi-cluster orchestration and team structure are the next layers. Resource sharing across clusters is manual- potential for consolidation.

Workload placement. Storage/networking already shared. Placement decisions are manual.

GPU availability and latency requirements are the main drivers. Utilization checks are manual.

Automate if real-time visibility existed.

Utilization %, job priority, and affinity.

Custo controller- existing tools don't fully support our scheduling logic and constraints.

Sit above both- cross-cluster orchestrator.

Respect constraints first, then optimize. Affinity and latency rules override utilization.

Yes

Reallocate from current team. No headcount for new hires right now.

CornJob pre-provisioning- simpler, faster to implement, and addresses the biggest immediate pain point.

Yes

We run a mix of A100 and H100

About 60% A100 and 40% H100 currently

Yes, we use InffiniBand and GPUDirect extensivley

Strategic choice for AI model scaling needs

We run a heavy mix of both training and interface

About 60% training and 40% interface

We run Ray, Kubeflow, Triton, and vLLM

built from scratch, all run on same cluster

We use node pools and resource quotas effectivley

Quotas help, but we still see occasional interference

Inference spikes starve training jobs occasionally

Happens a few tiems weelkly

Manual for now - exploring automation solutions currently

Real time visibility with dashboard and alerts

Prometheus and Grafana; trust GPU utilization most

Training at 80%; inference around 60%

Numbers swing significantly during peak business hours

Inference hits 90%; training drops significanlt

Training jobs queue; recovery take about 10 minutes

Monthly spend is roughly in the six-figure range

Includes GPU instances and supporting infrastructure

Right-sizing instances and using spot pricing

Modest savings offset by growing workload demands

Efficiency is okay.

idle GPUs during low inference traffic times

they sit idle; not repurposed currenlty

Scheduling complexity and workload isolation challenges

Smarter scheduling and dynamic quota adjustments needed

Predictive scheduling based on traffic patterns

Exploring Karpenter and custom ML forecasting

Prefer off the shelf but open to build

Recapture 30% idle GPU time

Reduces queuing and boosts overall efficiency

Cross team buy in and dedicated engineering resources

Engineering capacity is the bigger hurdle

Building a strong business case with ROL

Cost savings plus faster model iteration

We have the data and can quantify both

Roughly 20% cost savings and 30% faster model iteration

Phased approach starting with simpler scheduling improvemnts

Start with time of day heuristics

Adaptive rules based on historical traffic patterns

We'lll pull historical data from Prometheus

We'll likely build it as a Kubernetes operator for simplicity

Exploratory now; timeline depends on engineering capacity allocation

Leading with both; but empahsing cost savings first

We'll present it as both the ask and the 6-month success vision

Phase one deployed with validated 15% recapture

Push toward full 30% target with ML forecasting

Six month total ; 15% by month 3, 30% by month 6

The biggest pushback will be engineering capacity allocation

Moderately confident; ROI cases visually get approved here

We'll present it in absolute dollar terms tied to monthly spend

Roughly $20000-$30000 in monthly savings

We'll pitch in the next few weeks after finalizing numbers

Confirming the exact engineering capacity ask and timeline

We'll ask for a small dedicated team for the full 6 month roadmap

I'm confident the ROI will get the team approved without major pushback

We'll kick off phase one immediately upon approval

We'll track it as GPUs freed up during low-traffic inference windows

We're good to go with everything locked in now

They are spread across multiple environments including separate production and development clusters for better management.

Ten prediction nodes, five development nodes, all in one region.

We run Nvidia A100s for production workloads and a mix of T4s for development.

They are all identical, each configured with 8 A100 GPUs per production node.

Yes, they are uniform. Each development node is required with 4 T4 GPUs.

Production nodes utilize infiniBand for high-throughput distributed training, while development nodes use standard ethernet.

We use a miz heavily distributed training on production and inference prototyping on development.

We rely primarily on PyTorch and ray for distributed training, and vLLM for inference.

We track basic cluster-level matrics via Prometheus and Grafana, but lack deep per GPU profilling.

We use nvidia-smi locally on nodes, but Prometheus is currently pulling generic node-level metrics.

Production utilization stays high during distributed training jobs, averaging around 75% to 80% overall.

We have minimal idle time between jobs, as our pipeline keeps them continuously loaded.

Dips are usually caused data loading bottlenecks or scheduling delays during multi-node initialization.

We don't share specific cost details publicly, so let's wrap things up here.

GPU nodes are distributed across multiple clusters, supporting production, development, testing, and AI workloads.

Production: 8 nodes, Development: 4 nodes, Testing: 3 nodes, AI workloads: 5 nodes.

We mainly run NVIDIA A100 and GPUs, with some newer H100 nodes for AI workloads.

A100: 15 nodes, mainly supporting production production and development. H100: 5 nodes, concentrated in the AI workload cluster.

Yes, we use RDMA?RoCE networking on GPU clusters, with GPUDirect enabled for high performance AI workloads.

It's mainly deployed on the 5 H100 AI workload nodes and selected A100 nodes used for high performance workloads.

We use the GPUs for both training and inference, with training focused on AI models and inference supporting production applications.

Roughly 60% training and 40% inference. Training including LLM fine tuning and batch model training; inference supportive real time AI service.

We use Kubeflow and Ray for training workflows, vLLM and Triton for inference serving, with NVIDIA CUDA and cuDNN for GPU acceleration.

We rely on Kubernetes native scheduling with NVIDIA device plugins, plus a queuing layer using Ray and Kubeflow for workload management.

We have real time visibility through NVIDIA DCGM and Kubernets monitoring tools, with per node and per workload GPU utilization tracking.

Average GPU utilization is around 70-75%, with higher utilization during training runs and moderate levels during inference workloads.

Our GPU infrastructure costs are roughly in the range of $80k to $120k per month, depending on workload demand and cloud usage.

We optimize GPU usage through workload scheduling, rightsizing, Kubernets bin packing autoscking and using spot instances for non critical workloads.

Kubbernetes bin packing delivers the biggest GPU utilization improvements currently.

Native Kubernetes scheduler bin packing with custom tuning.

We tune pod priority classes, node affinity rules , GPU resource requests, and scheduling constraints for better packing.

We set GPU requests based on actual workload needs, balancing accuracy with enough flexibility for efficient bin packing.

We use autoscaling for burst demand, with some pre provisioned GPU capacity for predictable high priority workloads.

We scale down idle workloads, reschedule jobs, and use spot capacity to avoid leaving GPUs unused.

We are satisfied overall, but there is still meaningful room to improve effciency through further automation and optimization.

The biggest gap is automate workload placement and capacity planning based on real time GPU demand patterns.

Dynamic workload movement in the bigger opportunity alongside smarter initial scheduling.

Mostly moving workloads off idle nodes and consolidating capacity with some real time rebalancing.

It's semi automated today Kubernetes tools handle scaling, while some consolidation decisions still need engineering oversight.

Typically it takes a few hours, with minimal disruption due to workload scheduling controls and SLA monitoring.

The biggest blocker is workload variability and the complexity of predicting GPU demand patterns accurately.

Mostly unpredictable job arrival patterns, with some uncertainty around GPU requirements for new workloads.

We run profiling and testing in development environments first, then tune GPU requirements before production deployment.

We monitor production usage, adjust GPU requests based on metrics

Freed capacity automatically flows back through Kubernets scheduling and is reused by other workloads.

The greatest saving potential is through smarter workload placement

Smarter scheduling means predictive placement and automated workload optimization.

WE currently deploy a dedicted cluster of fifty NVIDIA H100 enterprise GPU nodes.

Most nodes are in production, but a few are reserved for development testing.

We actually rely on standard high speed ethernet without any dedicated network accelerators.

Our primary focus in training large models, though we do handle some inference.

Training takes about seventy percent, leaving a meaningful thirty percent slice for inference.

We run Ray for distributed training and deploy vLLM for our inference.

We manage resource allocation across the cluster using Kueue on top of Kubernets.

We actually face frequent face frequent fragmentation issues, preventing large training jobs from scheduling properly.

It is mostly an occasional blocker, only happening during peak usage hours.

We usually just wait it out until peak hour workloads finish scaling down.

We have excellent real time dashboards showing utilization broken down by every workload.

We built a completely custom internal solution to mintor node and workload utlization.

WE needed tighter integration with our proprietary job scheduler and internal reporting infrastrucutre.

We pay roughly two dollars an fifty cents per hour for each GPU.

That rate include the base cloud intance cost plus our premium support package.

We see an average utilization of rougly fifty percent due to scheduling gaps.

It is currently accepted as a natural baseline for our unpredictable workload patterns.

The greatest saving potential definitely line sin fixing scheudling gagps.

We believe hitting an eighty percent utilization target is chievable with optiomized scheduling.

It require tuning Kueue bin packing logic alogside tricter priority and preemption rules.

We are actively looking for an external partner to help design this solution.

Granular cost visiblility across multi tenant workloads is another major hurdle for us.

We must charge back costs back costs directly to specific internal projects.

We successfully map Ray training jobs today, but struggle with vLLM request attribution. I want to finish.

It is a mix between A100s and L40S.

It is usually split 50-50 and what’s driving the mix is the transition between them.

It has RDMA/RoCE.

It is deployed on all 11-25 GPU nodes and what’s driving it is GPU training.

It is a meaningful mix of both.

It is closer to 50-50 and the actual workloads are batch training.

I am running Triton.

It is L40S and we have deployed 100 inference pods.

It is spread across my 11-25 nodes.

I am setting explicit GPU limits per pod.

I usually set it around .6

The are primarily for batch training.

A single A100 per job

I am doing it by choice

I am getting real-time metrics

Cloud-native dashboards

The are built into my managed Kubernetes service

60%

30,000 per GPU cost

Right sizing and bin-packing

I am actively doing both

It is doing well for me, I am seeing meaningful utilization gains.

I feel like it’s where I want it to be

I see the greatest potential for cost savings is the workloads and they are doing well.

I wished I had better scheduling.

I am thinking better in pod replacement

I am thinking drain and reschedule

It’s something that I have explored in the native tooling

I have gotten very far and it is working well and it deployed and working.

monitoring

I am looking to improve the granularity of what I am tracking per workload

I am able to attribute GPU usage back to my teams

Interference endpoints with Triton

I am tracking it today

It is more about standardizing how I report and act on the data

There’s are no gaps

Models like A100 and H100

The split is closer to 50/50

Yes we use accelerators from InfiniBand

We deployed across 25-50 nodes

It was something we added as we scaled

We do a split of training and inference

Closer to 50/50

We are running Kubeflow

No we use no other tools

Yes custom scheduling logic is in place

Yes we are time slicing across A100s and H100s

We are seeing good utilization gains

Yes the pain points of time slicing is workload interference

We are managing tradeoff by prioritizing one workload type over the other

This plays out operationally by having enough time to work on both

Yes it is baked into our custom scheduler

Yes we are looking at queuque depth

We try to keep some minimum guarantee for inference latency

Yes we always reserve 20% of GPU time

Yes it is enough to kepp inference SLAs happy

Yes we are seeing real time per workload breakdowns

Datadog because its good quality

yes we are using it to monitor GPU metrics

Yes the ballpark percentage is 70%

yes the GPU infrastructure is costing us 10 million

That is pure hardware costs

We havent taking any steps yet

nothing is holding us bacj

we see the greatest potential in small workloads

small batch inference requests

scheduling overhead are the biggest savings

No I have no sense of percentage

we know its there just not a priority

I think we are doing the best we can with our current level of GPU

I would not improve anything

H100's

A pure H100 fleet

GPUDirect

Across the entire fleet, because it simplifies the management of them, and it makes it easier to predictive managing of the entire fleet

I would say that we are using them for a mixture of both to get the most value from them

I would say that it is a 50/50 split on purpose

vLLM

vLLM exclusively

PyTorch

I would say that we have full visibility into GPU utilization

I would say that we use the tools built into the software I would say that we can see per node and per workload and historical and real time

I would say that to the best of my knowledge it's 70% utilization

I would answer that we are right sizing and bin packing and scheduling tuning

I would say that to me it feels like all 3 are doing equal amounts of work

I would say that we don't push for more than 70% because that is more than enough for my organization's needs

I would say that I don't currently have a sense of that because it's not a metric that falls under my responsibility

I would say that it's the FinOps teams responsibility to track that metric

I would say that in my experience all 3 felt like they took equal amounts of time to implement and therefore none of the 3 would be any more difficult to undo

I would respond that as referenced earlier we don't push further because as a organization we don't feel a need to

I would say that this is not something I have thought about, but a hypothetical number would be 80% just for the purpose of being able to handle future technology needs

Engineering leadership (Director / VP / CTO) responsible for platform or infrastructure

I would say that I can't really think of anything that would qualify for this, but the closest thing would probably be workload isolation on a very minor scale

I would say that it would be by Team because this simplifies things overall

It's that the isolation itself isn't creating much friction

I would definitely say that is capacity planning, because it feels like it increases very quickly

I would definitely say that it's definitely both that's driving the increase

I would definitely say that the current thinking within my organization is improving efficiency on what we have

I would say that currently I can't think of anything that qualifies as a acceptable answer because it's my understanding that everything has been taken care of

I would definitely say that I feel very certain that increased demand from teams would close the gap

I would say that there is no plan in place currently but we are working on developing one

I would definitely say that we would track a combination of all metrics to get as much visibility as possible

I would say that it's a split that we actively manage

I would say that it is based on observed demand patterns

I would definitely say that the split will stay ad it currently is because it works best for my organization

I would answer that to my organization it is handling the demand without expanding our existing fleet

I would say that summary is 100% accurate