A model parameter is a learned numerical value used by the network during inference. A “7B” model has roughly seven billion parameters. In a dense model, most model parameters participate in each token’s forward pass. In a sparse Mixture-of-Experts model, the checkpoint can contain far more total parameters than the number active per token. Parameter count helps estimate model-weight memory, but quality, latency and total runtime memory depend on much more than the headline number.
These are orientation bands, not hardware guarantees. Precision, architecture, context and runtime can change the real requirement substantially.
What a parameter actually is
Neural networks learn numerical values that transform one representation into another. These values are stored in tensors and are adjusted during training. The final learned values are the model parameters or weights.
The parameter count therefore describes the size of the learned function in a very literal storage sense. If a model has 8 billion parameters and every parameter is stored using 16 bits, the raw weight tensor is on the order of 16 GB before considering metadata, runtime overhead, caches and temporary allocations.
This arithmetic is useful, but it is only the starting point for deployment planning.
Dense models: the intuitive case
In a dense transformer, the same network layers are generally executed for every token. A 7B dense model therefore has approximately seven billion learned parameters in its checkpoint, and the inference process uses the dense network for each token.
Dense parameter count is useful for comparing approximate checkpoint scale, but two equally sized models can have very different capabilities because training data, architecture, tokenizer, post-training, context design and optimization quality differ.
MoE models: total parameters versus active parameters
Mixture-of-Experts models introduce sparse expert layers. A router selects a subset of experts for each token. This creates two important numbers:
- Total parameters: all model weights stored in the checkpoint.
- Active parameters per token: the approximate subset used for the computation of a token.
A 30B MoE with roughly 3B active parameters can have compute characteristics closer to a much smaller dense model for parts of the forward pass, but it still needs access to the expert weights. That is why “3B active” must not be used as if it were a 3B checkpoint size.
From parameter count to weight memory
| Storage precision | Approx. bytes/parameter | 8B model weights | 32B model weights |
|---|---|---|---|
| FP32 | 4 | ~32 GB | ~128 GB |
| FP16 / BF16 | 2 | ~16 GB | ~64 GB |
| 8-bit | ~1 + overhead | ~8 GB + overhead | ~32 GB + overhead |
| 4-bit | ~0.5 + overhead | ~4 GB + overhead | ~16 GB + overhead |
The table estimates weight storage only. Real inference memory also includes KV cache, runtime buffers, allocator overhead, multimodal encoders and sometimes multiple model components.
Why parameter count is a weak quality proxy
Parameter count became a convenient shorthand during the era when scaling larger dense transformers often correlated with capability improvements. But modern model design complicates that relationship. Distillation, synthetic-data post-training, sparse architectures, better tokenizers, reinforcement learning and specialized training can allow smaller models to outperform larger ones on particular tasks.
Use parameters to answer questions such as “How large is this checkpoint?” and “What hardware class should I investigate?” Use task-specific evaluation to answer “Is this model good enough for my workload?”
How to read model names without being misled
- Identify whether the model is dense or MoE.
- For MoE, record both total and active parameter counts.
- Identify the published precision or available quantizations.
- Check modality: vision encoders or audio components can add memory beyond the language backbone.
- Check context length and expected KV-cache requirements.
- Then map the checkpoint to a realistic runtime and hardware configuration.
Frequently asked questions
Does a 70B model always outperform a 7B model?
No. Model quality depends on architecture, training, post-training and the task. Parameter count is not a universal performance ranking.
How do I estimate raw weight memory?
As a rough first pass, multiply parameter count by bytes per parameter: FP32≈4, FP16/BF16≈2, 8-bit≈1 and 4-bit≈0.5, then add format and runtime overhead.
What does active parameters mean in a MoE model?
It is the subset of expert parameters selected for computation for a token. It does not mean the unused experts disappear from checkpoint storage.
Why can two 8B models require different memory?
Different architectures, vocabularies, attention designs, context lengths, multimodal encoders, quantization formats and runtimes can all change the real footprint.
Apply this knowledge
Move from the concept to a concrete deployment shortlist.
Primary sources and technical references
OpenWeightModels prefers publisher documentation, standards bodies, official repositories and original research papers. The source material remains authoritative where it changes.