6 LLM Deployment Formats in Production

Google and Anthropic agree on one thing about LLM inference:

The faster a model runs, the fewer machines can run it.

This sounds counterintuitive, but a model file gets faster by locking in hardware-specific choices into the file.

And those choices only fit the hardware they were made for.

That is why they build the same model more than once, one version per chip or runtime it has to run on.

The visual below covers the six formats used to run LLMs in production:

1-2) Pickle and safetensors hold raw weights. Both run anywhere Python runs, so neither is faster than the framework around it.

A .pt file is a list of instructions that torch.load executes and loading a checkpoint can also run a payload (possibly malicious).

Safetensors holds plain numbers instead, laid out the way memory needs them, so there is nothing to rebuild when the file opens.

Both still need Python, the model code, and a tokenizer alongside.

3) GGUF is llama.cpp’s format, built so a model can run on a machine with no ML framework installed.

To achieve this, weights, tokenizer, and chat template ship in one file, and llama.cpp or Ollama runs it directly.

4) ONNX is an interchange format, built so a model trained in one framework can run somewhere that framework was never installed.

It decides the operations and the order they run in, but leaves the hardware open, so the runtime picks CPU, CUDA, or an NPU when the file loads.

5) MLX is Apple’s framework for its own silicon, where the CPU and GPU share a single pool of memory.

It decides where the weights live, so they are never copied between the two, and that advantage does not help on any other machine.

6) TensorRT is NVIDIA’s compiler, which turns a model into machine code for one specific GPU.

Everything is settled at build time, down to instructions tested on the exact card it compiles for, which is why the file will not load on a different GPU architecture.

The safe default is to stop as high on the chart as the latency budget allows, because every row down trades speed with the types of hardware it can run on.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →
• Read Part 2 on understanding the core building blocks of LLMs →
• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →
• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →
• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →
• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →
• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →
• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →
• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →
• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →
• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →
• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →
• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →
• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

Good day!