FP8 LLMs for vLLM - a neuralmagic Collection

neuralmagic 's Collections

DeepSeek-R1-Distill Quantized

Granite 3.1 Quantization

Sparse-Llama-3.1-2of4

Vision Language Models Quantization

FP8 LLMs for vLLM

Llama-3.2 Quantization

Llama-3.1 Quantization

INT8 LLMs for vLLM

INT4 LLMs for vLLM

Sparse Foundational Llama 2 Models

Compression Papers

DeepSparse Sparse LLMs

Sparse Finetuning MPT

Compressed LLMs from the Community

FP8 LLMs for vLLM

updated Oct 17, 2024

Accurate FP8 quantized models by Neural Magic, ready for use with vLLM!

RedHatAI/Meta-Llama-3.1-405B-Instruct-FP8

Text Generation • 406B • Updated Oct 9, 2024 • 311 • 31
RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8

Text Generation • 8B • Updated Oct 9, 2024 • 128k • 43
RedHatAI/Meta-Llama-3.1-70B-Instruct-FP8

Text Generation • 71B • Updated Mar 25 • 7.18k • 50
RedHatAI/Phi-3-medium-128k-instruct-FP8

Text Generation • 14B • Updated Oct 9, 2024 • 41 • 5
RedHatAI/Mistral-Nemo-Instruct-2407-FP8

Text Generation • 12B • Updated Jul 19, 2024 • 3.48k • 18
RedHatAI/Meta-Llama-3-8B-Instruct-FP8

Text Generation • 8B • Updated Jul 18, 2024 • 2.69k • • 24
RedHatAI/Meta-Llama-3.1-405B-Instruct-FP8-dynamic

Text Generation • 406B • Updated Oct 19, 2024 • 4.16k • 15
RedHatAI/Meta-Llama-3-70B-Instruct-FP8

Text Generation • 71B • Updated Jul 18, 2024 • 853 • • 13
RedHatAI/Mixtral-8x7B-Instruct-v0.1-AutoFP8

Text Generation • 47B • Updated Jul 18, 2024 • 58 • 3
RedHatAI/Meta-Llama-3-8B-Instruct-FP8-KV

Text Generation • 8B • Updated Sep 15 • 20.6k • • 8
RedHatAI/Meta-Llama-3-70B-Instruct-FP8-KV

Text Generation • 71B • Updated Jun 26, 2024 • 65 • 3
RedHatAI/Mixtral-8x22B-Instruct-v0.1-AutoFP8

Text Generation • 141B • Updated Aug 12, 2024 • 40 • 3
RedHatAI/Qwen2-72B-Instruct-FP8

Text Generation • 73B • Updated Jul 18, 2024 • 729 • 15
RedHatAI/Qwen2-7B-Instruct-FP8

Text Generation • 8B • Updated Jul 18, 2024 • 13k • • 2
RedHatAI/Qwen2-1.5B-Instruct-FP8

Text Generation • 2B • Updated Jul 18, 2024 • 8.5k
RedHatAI/Qwen2-0.5B-Instruct-FP8

Text Generation • 0.5B • Updated Jul 18, 2024 • 553 • 3
RedHatAI/Mistral-7B-Instruct-v0.3-FP8

Text Generation • 7B • Updated Jul 18, 2024 • 2.01k • 3
RedHatAI/Llama-2-7b-chat-hf-FP8

Text Generation • 7B • Updated Jul 18, 2024 • 361
RedHatAI/Phi-3-mini-128k-instruct-FP8

Text Generation • 4B • Updated Oct 9, 2024 • 43
RedHatAI/gemma-2-9b-it-FP8

Text Generation • 9B • Updated Sep 22 • 932 • 5
RedHatAI/Qwen2-57B-A14B-Instruct-FP8

Text Generation • 57B • Updated Jul 18, 2024 • 523 • 1
RedHatAI/DeepSeek-Coder-V2-Lite-Instruct-FP8

Text Generation • 16B • Updated Jul 18, 2024 • 24.6k • 8
RedHatAI/DeepSeek-Coder-V2-Lite-Base-FP8

Text Generation • 16B • Updated Jul 18, 2024 • 22
RedHatAI/DeepSeek-Coder-V2-Base-FP8

Text Generation • 236B • Updated Jul 22, 2024 • 24
RedHatAI/DeepSeek-Coder-V2-Instruct-FP8

Text Generation • 236B • Updated Jul 22, 2024 • 31 • 7
RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Text Generation • 8B • Updated Sep 22 • 37.5k • 8
RedHatAI/Meta-Llama-3.1-70B-Instruct-FP8-dynamic

Text Generation • 71B • Updated Oct 19, 2024 • 1.72k • 7
RedHatAI/Mistral-Large-Instruct-2407-FP8

Text Generation • 123B • Updated Oct 9, 2024 • 26
RedHatAI/Meta-Llama-3.1-8B-FP8

Text Generation • 8B • Updated Mar 25 • 6.94k • 10
RedHatAI/Meta-Llama-3.1-70B-FP8

Text Generation • 71B • Updated Oct 9, 2024 • 735 • 2
RedHatAI/starcoder2-15b-FP8

Text Generation • 16B • Updated Oct 9, 2024 • 22
RedHatAI/starcoder2-3b-FP8

Text Generation • 3B • Updated Oct 9, 2024 • 16
RedHatAI/starcoder2-7b-FP8

Text Generation • 7B • Updated Oct 9, 2024 • 14
RedHatAI/Meta-Llama-3.1-405B-FP8

Text Generation • 410B • Updated Oct 9, 2024 • 27
RedHatAI/gemma-2-2b-it-FP8

3B • Updated Oct 9, 2024 • 276 • 1
RedHatAI/Llama-3.2-1B-Instruct-FP8-dynamic

Text Generation • 1B • Updated Oct 9, 2024 • 195k • 3
RedHatAI/Llama-3.2-3B-Instruct-FP8-dynamic

Text Generation • 4B • Updated Oct 9, 2024 • 100 • 3
RedHatAI/Llama-3.2-3B-Instruct-FP8

Text Generation • 4B • Updated Oct 16, 2024 • 1.85k • 6
RedHatAI/Llama-3.2-1B-Instruct-FP8

Text Generation • 1B • Updated Oct 16, 2024 • 116k • 3
RedHatAI/Llama-3.2-1B-FP8

1B • Updated Oct 14 • 10.7k
RedHatAI/Phi-3.5-mini-instruct-FP8-KV

Text Generation • 4B • Updated Oct 1, 2024 • 72 • 2
RedHatAI/Llama-3.1-Nemotron-70B-Instruct-HF-FP8-dynamic

Text Generation • 71B • Updated Oct 23 • 8.51k • 14