TensorRT Tuning: A Step-by-Step Latency Ladder for YOLOv8

A practical guide to TensorRT build and runtime parameters, with a worked trtexec tuning ladder for YOLOv8 and a reproducible Docker setup.

10-16 minutes(2156 words)complex

Quick Navigation

Difficulty: Intermediate
Estimated Time: 30-45 minutes
Prerequisites: NVIDIA GPU with recent drivers, Docker with NVIDIA Container Toolkit, A trained model exported to ONNX, Command line experience

What You'll Learn

This tutorial covers how to tune a TensorRT engine and verify the result:

  • How TensorRT works - Layer fusion, precision lowering, kernel auto-selection, and memory planning
  • Precision choices - FP16, BF16, INT8, and per-layer precision control
  • Build-time parameters - Workspace, optimization level, timing cache, and sparsity
  • Shapes and batching - How minShapes, optShapes, and maxShapes specialize an engine
  • Runtime parameters - Streams, CUDA Graph, data transfers, and profiling
  • A worked ladder - Six measured steps on YOLOv8s, with an accuracy check
  • Docker setup - A reproducible environment based on the NGC TensorRT image
  • Troubleshooting - Common failure modes and their causes

Prerequisites

  • NVIDIA GPU with recent drivers installed on the host
  • Docker with the NVIDIA Container Toolkit
  • A trained model exported to ONNX, such as yolov8s.onnx
  • Command line experience

Introduction

Teams moving a vision model from eager PyTorch to a TensorRT engine routinely report latency drops between 2x and 10x. That spread is the interesting part. The low end usually comes from running trtexec once with defaults. The high end comes from deliberate choices about precision, shapes, build effort, and runtime overhead.

TensorRT is NVIDIA's inference optimizer. It does not train anything. It takes a trained graph, fuses layers, selects kernels by timing them on your exact GPU, and emits a serialized engine bound to that GPU and that TensorRT version.

Every parameter you pass changes that search. Some widen it, some narrow it, and some trade accuracy for speed. This tutorial maps each knob to its effect when raised or lowered, walks through a full tuning ladder on YOLOv8s, and finishes with a Docker setup that keeps the whole pipeline reproducible.

Disclaimer: the latency and mAP figures in this tutorial are illustrative of the shape of the process. They are not benchmark results. Measure on your own hardware.

What TensorRT Does

TensorRT performs four jobs at build time, and each one maps to a tuning parameter later in this tutorial.

  • Layer fusion. Consecutive operations such as convolution, bias, and activation collapse into one kernel. This cuts memory round-trips, which dominate runtime on small and mid-size models.
  • Precision lowering. Weights and activations move from FP32 to FP16, INT8, or FP8 where the hardware supports it. Tensor cores execute these formats at much higher throughput.
  • Kernel auto-selection. For each layer, TensorRT benchmarks several CUDA implementations and keeps the fastest for your GPU. This is why engines are not portable across GPU architectures.
  • Memory planning. Tensors with non-overlapping lifetimes share buffers, which reduces allocation and copy overhead.

Key insight: An engine is a compiled artifact, not a model file. Treat it like a binary built for one target, and rebuild it whenever the GPU, driver stack, or TensorRT version changes.

Parameter Reference

The parameters below are grouped by what they control. Each table shows what happens when you raise or lower the value.

Precision

Precision selection is the first decision, because it changes both speed and accuracy.

ParameterWhen you raise itWhen you lower it
--fp16 or --bf16Enabling it roughly halves memory traffic and uses tensor cores. Accuracy is usually unchanged on detection and classification models. Start here.Staying at FP32 is the safest option but is typically 2x to 4x slower than the lowest precision.
--int8Enabling it gives the largest speedup. It needs a calibration set or a model already quantized with quantization-aware training.Not enabling it keeps accuracy stable and gives up the speedup.
Calibration set sizeA larger set (500 to 1000 representative images is a common range) gives more accurate activation ranges and a better quantized model, at the cost of a longer calibration pass.A smaller set calibrates faster, but outlier activations get clipped badly and accuracy can drop sharply.
--precisionConstraints=obey with --layerPrecisionsConstraining more layers (for example detection heads, softmax, or normalization) protects accuracy and costs speed.Constraining fewer layers is faster and riskier.
--bestLets TensorRT choose any precision per layer. It is a good exploratory flag.Constrain precision explicitly before shipping instead of relying on it.

Build-Time

These parameters control how hard the builder searches. They change build time and engine quality, not accuracy.

ParameterWhen you raise itWhen you lower it
--memPoolSize=workspace:4096 (MiB)A larger pool lets TensorRT consider kernels that need scratch memory, which are often the fastest.A pool that is too small excludes those kernels, so the engine gets slower or the build fails.
--builderOptimizationLevel (0 to 5, default 3)Higher levels time more tactics. Build time grows from minutes to much longer, and the engine is often a few percent faster.Lower levels build quickly and can leave 5% to 20% of performance on the table.
--timingCacheFile=timing.cacheWith a cache, later builds reuse earlier kernel measurements and finish far faster. Keep the file on a mounted volume.Without a cache, every build starts from zero.
--sparsity=enableOn Ampere and newer, a model pruned to the 2:4 pattern can run noticeably faster.On an unpruned model it changes nothing.

Shapes and Batching

Dynamic shapes are defined by three values: --minShapes, --optShapes, and --maxShapes. TensorRT optimizes primarily for optShapes and only guarantees correctness inside the min-to-max range.

ParameterWhen you raise itWhen you lower it
--optShapesTunes the engine for large inputs and makes it less efficient on small ones.Tunes for small inputs. Set it to the input size you actually serve most often.
--maxShapesAccepts larger inputs but reserves more memory and can limit optimizations.Saves memory and gives a tighter engine, and rejects anything bigger.
Width of the min-to-max rangeA wider range buys flexibility and costs specialization. If traffic has distinct modes, use several optimization profiles instead of one wide one.A narrower range improves tuning. A fixed shape (min, opt, and max equal) is the tightest engine you can build.
Batch sizeRaises throughput and GPU utilization but increases per-request latency and memory.Lowers latency but can leave the GPU underused.

Runtime

ParameterWhen you raise itWhen you lower it
--streams=NMore concurrent streams raise throughput if the GPU is not saturated, at the cost of per-request latency and memory.One stream gives stable, predictable latency.
--useCudaGraphEnabling it replays a captured kernel sequence and removes launch overhead. It matters for small, low-latency models and does almost nothing for large ones.Leaving it off keeps the default launch path.
--noDataTransfersEnabling it excludes host-device copies, so the number reflects pure GPU compute. Use it to isolate engine speed, then measure end-to-end separately.Leaving it off includes copies, which is closer to real serving.
--profilingVerbosity=detailed with --dumpProfileRicher layer names and per-layer timings help you find slow layers, with a small build overhead.Lower verbosity gives less detail to work with.

Change one parameter per run. Anything else makes the result impossible to attribute.

A Worked Tuning Ladder

The scenario is real-time detection with YOLOv8s on a single camera stream, on an RTX 4060. The goal is under 8 ms latency with no more than a 1 point mAP loss. As noted in the introduction, the figures below are illustrative, not benchmark results.

Step 0: FP32 baseline. This is the reference point. Expect roughly 9.5 ms in this scenario.

trtexec --onnx=yolov8s.onnx --saveEngine=fp32.engine

Step 1: FP16. The biggest single gain, with unchanged mAP. Latency falls to roughly 4.2 ms.

trtexec --onnx=yolov8s.onnx --fp16 --saveEngine=fp16.engine

Step 2: fixed shapes. One camera means batch 1, so collapse the range. Roughly 3.8 ms.

trtexec --onnx=yolov8s.onnx --fp16 \
  --minShapes=images:1x3x640x640 \
  --optShapes=images:1x3x640x640 \
  --maxShapes=images:1x3x640x640 \
  --saveEngine=fixed.engine

Step 3: search harder. Raise the workspace and optimization level, and persist the timing cache. Build time grows from about a minute to several, and latency drops to roughly 3.5 ms.

trtexec --onnx=yolov8s.onnx --fp16 \
  --minShapes=images:1x3x640x640 \
  --optShapes=images:1x3x640x640 \
  --maxShapes=images:1x3x640x640 \
  --memPoolSize=workspace:4096 \
  --builderOptimizationLevel=5 \
  --timingCacheFile=timing.cache \
  --saveEngine=tuned.engine

Step 4: cut launch overhead. The model is small, so CUDA Graph helps. Roughly 3.3 ms. These flags affect benchmarking, so check how your serving code behaves.

trtexec --loadEngine=tuned.engine --useCudaGraph --noDataTransfers

Step 5: try INT8, and check accuracy. If the target were still out of reach, INT8 with a calibration cache could bring latency to around 2.1 ms. In this scenario it also costs about 2.3 points of mAP, which breaks the budget.

Step 6: repair with mixed precision. Keep the detection head in FP16 and quantize the rest. Latency lands near 2.5 ms with a loss of about 0.6 points, inside the budget.

The pattern generalizes. Fix precision first, then shapes, then build effort, then runtime overhead, and leave INT8 for last because it is the only step that needs an accuracy re-check. Always measure latency and quality together, since a faster engine that misses objects is a regression.

Docker Setup

Docker keeps the driver, CUDA, and TensorRT versions pinned, which matters because engines are not portable between them.

Toolkit Install

On the host, confirm the driver works and install the NVIDIA Container Toolkit.

nvidia-smi
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

GPU Check

Verify that containers can see the GPU.

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

NGC Image

Pull the official TensorRT image from NGC. The tag 25.01-py3 is an example. Pick the newest tag from the NGC catalog that your host driver supports.

docker pull nvcr.io/nvidia/tensorrt:25.01-py3

Run it with your models mounted.

docker run --gpus all -it --rm -v "$(pwd)":/workspace/models --shm-size=2g nvcr.io/nvidia/tensorrt:25.01-py3

Dockerfile

For your own dependencies, build on top of the official image.

FROM nvcr.io/nvidia/tensorrt:25.01-py3
WORKDIR /app
RUN pip install --no-cache-dir onnx onnxruntime-gpu opencv-python-headless ultralytics
COPY . /app
CMD ["python3", "infer.py"]

Docker Compose

With Docker Compose, request the GPU through a device reservation.

services:
  trt:
    image: yolov8-trt
    volumes:
      - ./models:/app/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Three habits save the most time. Build the engine inside the same container image and on the same GPU model as production. Mount a volume for timing.cache and the .engine files so they survive restarts. On Jetson, use the l4t-tensorrt images with --runtime nvidia, and for LLMs use the TensorRT-LLM images instead.

Troubleshooting

  • could not select device driver "nvidia". The Container Toolkit is missing, or Docker was not restarted after configuring it.
  • CUDA driver version insufficient. The host driver is older than the image requires.
  • An engine that will not load. It was built on a different GPU architecture or TensorRT version.
  • Accuracy collapse after INT8. Poor or unrepresentative calibration data, or sensitive layers left quantized.
  • No speedup from tuning flags. The model is already bound by something else, usually pre-processing, post-processing, or data transfer. Profile before tuning further.

Conclusion

Most of the gain in TensorRT comes from three decisions: precision, shape specialization, and how much search effort the builder gets. The rest is incremental. A tuned engine is also a fragile one, because it is tied to a GPU, a driver, and a TensorRT release that you now have to track.

That shifts the real work from tuning to reproducibility. The teams that sustain their gains pin container images, persist timing caches, and rebuild engines in CI against the production GPU, with an accuracy gate before any INT8 engine ships.

The open question is how far this per-target compilation model can stretch as GPU generations arrive faster than model release cycles. If every hardware refresh invalidates your engines, the cost of your inference stack is no longer set by the model alone.


Tags: #TensorRT #YOLOv8 #NVIDIA #GPUInference #Docker #ONNX #INT8 #ComputerVision #MLOps #DeepLearning