Software framework for GPU optimization in AI: CUDA, ROCM, TRITON, TENSORRT – compiler path and performance implications

The throughput of deep learning depends on how the compiler stack mapped tensor program executes the GPU: thread/block planning, memory motion, and instruction selection (e.g., tensors, Core MMA Pipelines). In this article, we will focus on four main stacks: Cuda, Rocm, Triton, and Tensorrt, from a compiler perspective and explains which optimizations move the needle in practice.

What actually determines the performance of modern GPUs

Throughout the supplier, the same leverage recurs:

• Operator Schedule and Fusion: Reduce kernel launches and round trips to HBM; expose longer producers → consumption chains for registration/shared memory reuse. Tensorrt and Cudnn “Runtime Fusion Engine” illustrate this to get attention and communication.
• Tile and data layout: Match the tile shape with the tensor kernel/WGMMA/WMMA natural fragment size; avoid bank conflicts and partition camping in shared memory. Cutlass records twist-grade Gemm tiles for the core of the tensor and the CUDA core.
• Accuracy and quantization: FP16/BF16/FP8 for training/inference; INT8/INT4 for inference (calibration or QAT). Tensorrt automates calibration and kernel selection at these precisions.
• Graphic Capture and Runtime Specialization: Chart execution to amortize startup overhead; dynamic fusion of common subgraphs (e.g. note). The Cudnn 9 adds chart support to attention fusion engines.
• Automatic adjustment: Search for tile sizes, expansion factors and pipe depth for each Arch/Sku. Triton and Cutlass expose explicit automatic hooks; Tensorrt performs builder time policy selection.

With this lens, this is how each stack implements the above.

CUDA: NVCC/PTXA, Cudnn, Cutlass and Cuda pictures

Compiler path. CUDA code passes NVCC Then enter PTX PTXA Lower PTX to SASS (Arch-specific machine code). Control optimization requires feeding marks to the host and device stages; for the kernel, the key is -Xptxas. Developers often miss it -O3 Impact host code individually.

Kernel generation and library.

• fencing Provide parameter templates for GEMM/CONS, implement distortion-level tiling, tensor core MMA pipelines and misunderstandings designed for conflict-free access – organic references to writing peak kernels, including Hopper WGMMA Small path.
• Cudnn 9 Introduced runtime convergence engines (especially for attention blocks), native CUDA graph integration for these engines, and updates to new computing capabilities – aggregation reduces scheduling overhead and improves memory location in transformer workloads.

Performance impact.

• Transforming from unsequential Pytorch Ops to Cudnn Attention Convergence usually cuts kernel releases and global memory traffic; CUDA diagramit can reduce CPU bottlenecks in short sequences.
• On Hopper/Blackwell, it is decisive to keep the tile shape consistent with the natural size of the WGMMA/Tensor core. The Cutlass tutorial quantifies the throughput of the waste tensor throughput at the wrong size.

When Cuda is the right tool. You need to maximize control over instructional choices, occupancy, and SMEM orchestration; or you will extend the kernel beyond the library’s coverage while staying on the NVIDIA GPU.

ROCM: hip/jingle toolchain, Rocblas/Miopen and 6.x series

Compiler path. ROCM usage clang/llvm Compilation Fashionable (Similar to CUDA) Enter GCN/rDNA ISA. 6. The focus of the X series is Perf and Framework coverage. Release Notes track component-level optimization and HW/OS support.

Library and kernel.

• Rocblas and Miopen Implementation of GEMM/CORV originals with arched tiles and algorithmic selection, with a spirit similar to Cublas/Cudnn. The merged ChangElog highlights the iterative performance in these libraries.
• Recent ROCM Workstream includes better ones Triton Enablement of AMD GPUs enables Python-level core creation to reduce LLVM to AMD backends.

Performance impact.

• On AMD GPUs, matching LDS (Shared Memory) library width and vectorized global loads are as important as the shape of matrix tile shapes to align Smem Bank on NVIDIA. Compiler-assisted fusion in the framework (for example, note) plus auto-tuned libraries in Rocblas/Miopen usually truncate a large number of gaps to the handwritten kernel, including on the architecture/driver. Release documentation shows continuous tuner improvements for 6.0-6.4.x.

When ROCM is the right tool. You need native support and optimization for AMD accelerators, and hip portability with existing CUDA-style kernels and a clear LLVM toolchain.

Triton: DSL and compiler for custom kernels

Compiler path. Triton is a python-inserted DSL that can be passed LLVM;It handles vectorization, memory merging and registration allocation, while providing explicit control over block size and program ID. The build documentation shows LLVM dependencies and custom builds; NVIDIA’s developer material discusses Triton’s tweaks to new buildings that improve FP16/FP8 GEMM, such as Blackwell.

optimization.

• Automatic adjustment More than the size of the tile, num_warpsand pipeline stage; stationary masking For boundary conditions without scalar fallback; Shared memory Computing on staged and software pipelines overlaps with global load.
• Triton’s design is designed to automation While leaving the authors with block-level tiling selections, a part of CUDA-level optimization; the initial announcement outlines the separation of concerns.

Performance impact.

• The Triton glows when you need to have a unique kernel-external library coverage of a blended, shape (e.g., custom notice variants, standardized-activate-activate-motor chains). On modern NVIDIA parts, suppliers collaborate to report architecture-specific improvements to the Triton backend, thus reducing penalties for ordinary Gemms with Cutlass-style kernels.

When Triton is the right tool. You need to customize the near CUDA performance of Fused Ops without writing SASS/WMMA, and use automatic tuning to value Python-First Iteration.

Tensorrt (and Tensorrt-llm): Builder time graph optimization for inference

Compiler path. Tensorrt ingests onnx or framework diagrams and emits hardware-specific engine. During the build process, it executes Layer/Tensor Fusion,,,,, Accurate calibration (int8, fp8/fp16) and Kernel tactical choice;Best Practice Documentation describes these building stages. Tensorrt-LLM extends this with LLM-specific runtime optimization.

optimization.

• Graphic level: Constant folding, cons plate normalization, rotation bias activates fusion, pay attention to fusion.
• accurate: Post-training calibration (entropy/percentile/MSE) and per-tenster quantization, as well as smooth application/QAT workflow in tensile force-LLM.
• Runtime: Classification for multi-process/multi-GPU deployments (tensorrt-llm Docs) – KV cache, on-board batch processing and planning.

Performance impact.

• The biggest wins usually come from: end to end INT8 (or FP8 On supported Hopper/Blackwell), the frame was removed from overhead through a single engine and an active focus fusion was performed. Tensorrt’s builder is produced to every computer engine plan to avoid general-purpose kernels at runtime.

When tension is the right tool. Production inferences about NVIDIA GPUs where you can precompile optimized engines and benefit from quantization and large fusions.

Practical Guide: Selecting and Adjusting Stacks

1. Training and reasoning.
   • Training/experiment kernel → Cuda + Cutlass (NVIDIA) or ROCM + ROCBLAS/MIOPEN (AMD); Triton is used for custom Fused Ops.
   • NVIDIA’s production inference → tensorrt/tensorrt-llm, used for global graph-level gain.
2. Use building local instructions.
   • On Nvidia Hopper/Blackwell, make sure the tile map WGMMA/WMMA Dimensions; Cutlass material shows how to construct warp-level Gemm and Smem iterators.
   • On AMD, the use of LDS and vector width are opposite to the CU data diameter; shape-specific operations are performed using ROCM 6.x automatic regulator and triton-on-Rocm.
3. First fuse, then quantify.
   • Kernel/graphics fusion reduces memory traffic; quantization reduces bandwidth and increases mathematical density. Tensorrt’s Builder Time Fusions plus INT8/FP8 usually brings multiplication growth.
4. Use graph execution for short sequences.
   • The CUDA graph of the fusion of Cudnn attention fusion amortizes the emission overhead in self-rotation reasoning.
5. Think of the compiler flag as a first-class flag.
   • For CUDA, remember the device side flag: example, -Xptxas -O3,-v (and -Xptxas -O0 at diagnosis). Host only -O3 not enough.

refer to: