--- license: mit base_model: microsoft/NextCoder-14B tags: - fp8 - quantized - code - nextcoder - microsoft - llmcompressor - vllm - premium-quality - 2048-calibration - code-optimized library_name: transformers pipeline_tag: text-generation --- # NextCoder-14B-2048-Calibration-FP8 **Premium FP8 quantization with 2,048 code-optimized calibration samples** This is a **premium FP8 quantized version** of [microsoft/NextCoder-14B](https://huggingface.co/microsoft/NextCoder-14B) featuring rigorous code-optimized multi-dataset calibration for production-grade reliability. Quantized by [TevunahAi](https://huggingface.co/TevunahAi) on enterprise-grade hardware. ## 🎯 Recommended Usage: vLLM For optimal performance with **full FP8 benefits** and code-optimized quality, use **vLLM** or **TensorRT-LLM**: ### Quick Start with vLLM ```bash pip install vllm ``` **Python API:** ```python from vllm import LLM, SamplingParams from transformers import AutoTokenizer # vLLM auto-detects FP8 from model config llm = LLM(model="TevunahAi/NextCoder-14B-2048-Calibration-FP8", dtype="auto") # Prepare prompt with chat template tokenizer = AutoTokenizer.from_pretrained("TevunahAi/NextCoder-14B-2048-Calibration-FP8") messages = [{"role": "user", "content": "Write a Python function to calculate fibonacci numbers"}] prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) # Generate sampling_params = SamplingParams(temperature=0.7, max_tokens=512) outputs = llm.generate([prompt], sampling_params) for output in outputs: print(output.outputs[0].text) ``` **OpenAI-Compatible API Server:** ```bash vllm serve TevunahAi/NextCoder-14B-2048-Calibration-FP8 \ --dtype auto \ --max-model-len 4096 ``` Then use with OpenAI client: ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="token-abc123", # dummy key ) response = client.chat.completions.create( model="TevunahAi/NextCoder-14B-2048-Calibration-FP8", messages=[ {"role": "user", "content": "Write a Python function to calculate fibonacci numbers"} ], temperature=0.7, max_tokens=512, ) print(response.choices[0].message.content) ``` ### vLLM Benefits - ✅ **Weights, activations, and KV cache in FP8** - ✅ **~14GB VRAM** (50% reduction vs BF16) - ✅ **Native FP8 tensor core acceleration** on Ada/Hopper GPUs - ✅ **Single GPU deployment** on RTX 4090, RTX 5000 Ada, or H100 - ✅ **Premium 2048-sample code-optimized calibration** - ✅ **Production-grade code quality** ## ⚙️ Alternative: Transformers (Not Recommended) This model can be loaded with `transformers`, but **will decompress FP8 → BF16 during inference**, requiring ~28GB+ VRAM. For 14B models, **vLLM is strongly recommended**.
Transformers Example (Click to expand) ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch # Loads FP8 weights but decompresses to BF16 during compute model = AutoModelForCausalLM.from_pretrained( "TevunahAi/NextCoder-14B-2048-Calibration-FP8", device_map="auto", torch_dtype="auto", low_cpu_mem_usage=True, ) tokenizer = AutoTokenizer.from_pretrained("TevunahAi/NextCoder-14B-2048-Calibration-FP8") # Generate messages = [{"role": "user", "content": "Write a Python function to calculate fibonacci numbers"}] text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = tokenizer([text], return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=512) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` **Requirements:** ```bash pip install torch>=2.1.0 transformers>=4.40.0 accelerate compressed-tensors ``` **System Requirements:** - **~28GB+ VRAM** (decompressed to BF16) - Multi-GPU setup or high-end single GPU - CUDA 11.8 or newer **⚠️ Warning:** Most consumer GPUs will struggle with transformers inference at this size. Use vLLM for practical deployment.
## 📊 Model Details | Property | Value | |----------|-------| | **Base Model** | [microsoft/NextCoder-14B](https://huggingface.co/microsoft/NextCoder-14B) | | **Architecture** | Dense (14B parameters) | | **Quantization Method** | FP8 E4M3 weight-only | | **Framework** | llm-compressor + compressed_tensors | | **Calibration Samples** | **2,048** (4-8x industry standard) | | **Calibration Type** | Code-optimized (4 datasets) | | **Storage Size** | ~14GB | | **VRAM (vLLM)** | ~14GB | | **VRAM (Transformers)** | ~28GB+ (decompressed to BF16) | | **Target Hardware** | NVIDIA RTX 4090, RTX 5000 Ada, H100 | | **Quantization Date** | November 27, 2025 | | **Quantization Time** | 91.3 minutes (~1.5 hours) | ## 🏆 Premium Code-Optimized Calibration This model was quantized using TevunahAi's **premium code-focused calibration process**: ### Calibration Details - **Total Samples:** 2,048 (4-8x industry standard) - **Datasets Used:** 4 code-focused sources - **Coverage:** Comprehensive across coding tasks | Dataset | Samples | Purpose | |---------|---------|---------| | **HuggingFaceH4/CodeAlpaca_20K** | 512 | Code instruction pairs | | **garage-bAInd/Open-Platypus** | 512 | STEM/reasoning (includes code) | | **teknium/OpenHermes-2.5** | 512 | Diverse instructions | | **theblackcat102/evol-codealpaca-v1** | 512 | Evolved code examples | ### Why Code-Optimized Calibration? Most FP8 quantizations use generic chat data for calibration. TevunahAi uses **2,048 samples from 4 code-focused datasets**, ensuring: - ✅ **Superior code generation quality** - ✅ **Better handling of programming syntax** - ✅ **Optimized for multiple languages** - ✅ **Accurate completion of complex code** - ✅ **Production-grade reliability for coding tasks** **For code models, generic calibration isn't enough. TevunahAi uses code-specific data.** ## 🔧 Why FP8 for Code Models? ### With vLLM/TensorRT-LLM: - ✅ **50% memory reduction** vs BF16 (weights + activations + KV cache) - ✅ **Single GPU deployment** on RTX 4090 (24GB) or RTX 5000 Ada (32GB) - ✅ **Faster inference** via native FP8 tensor cores - ✅ **Better throughput** with optimized kernels - ✅ **Code-optimized calibration** maintains quality ### With Transformers: - ✅ **Smaller download size** (~14GB vs ~28GB BF16) - ✅ **Compatible** with standard transformers workflow - ⚠️ **Decompresses to BF16** during inference (no runtime memory benefit) - ❌ **Requires 28GB+ VRAM** - impractical for most setups **For 14B code models, vLLM is essential for practical deployment.** ## 💾 Model Files This model is stored as sharded safetensors files (all required for inference). The compressed format enables efficient storage and faster downloads. ## 🚀 NextCoder Model Family Microsoft's NextCoder family offers state-of-the-art code generation. The 14B version provides the sweet spot: | Model | Parameters | VRAM (vLLM) | Quality | Use Case | |-------|------------|-------------|---------|----------| | **7B** | 7B | ~7GB | Good | Fast iteration, prototyping | | **14B** | 14B | ~14GB | Better | Complex tasks, better reasoning | | **32B** | 32B | ~32GB | Best | Flagship performance, production | **14B Benefits:** - ✅ **2x capacity** vs 7B model - ✅ **Superior reasoning** for complex algorithms - ✅ **Better context handling** for larger codebases - ✅ **Single GPU deployment** on RTX 4090/5000 Ada - ✅ **Excellent quality-per-cost** ratio ## ⚖️ Comparison: Standard vs Premium Calibration TevunahAi offers two quantization tiers for this model: | Version | Calibration | Samples | Datasets | Quant Time | Use Case | |---------|-------------|---------|----------|------------|----------| | Standard FP8 | Basic | 512 | 1 generic | ~35 min | Quick deployment | | **Premium FP8** (this) | Code-optimized | **2,048** | **4 code-focused** | **91 min** | Production-grade | ### When to Choose Premium: - ✅ Production deployments - ✅ Quality-critical applications - ✅ API services at scale - ✅ Benchmarking and evaluation - ✅ Enterprise code generation ### When Standard is Fine: - ✅ Quick testing - ✅ Development/prototyping - ✅ Resource-constrained environments - ✅ Non-critical applications ## 🔬 Quantization Infrastructure **Professional hardware for premium calibration:** - **CPUs:** Dual Intel Xeon Max 9480 (224 threads, 128GB HBM2e @ 2000 GB/s) - **Memory:** 256GB DDR5-4800 (16 DIMMs, 8-channel per socket, ~614 GB/s) - **Total Memory Bandwidth:** ~2,614 GB/s aggregate - **Peak Memory Usage:** ~170GB during quantization (model + calibration datasets) - **GPU:** NVIDIA RTX 5000 Ada Generation (32GB VRAM, native FP8 support) - **Software:** Ubuntu 25.10 | Python 3.12 | PyTorch 2.8 | CUDA 13.0 | llm-compressor **Why This Matters:** - **91 minutes** of rigorous quantization and validation - **Code-specific calibration** requires specialized datasets - Professional infrastructure enables quality impossible on consumer setups ## 📚 Original Model This quantization is based on [microsoft/NextCoder-14B](https://huggingface.co/microsoft/NextCoder-14B) by Microsoft. NextCoder-14B features: - **State-of-the-art code generation** capabilities - **Strong performance** across multiple programming languages - **Excellent instruction following** for coding tasks - **Larger capacity** than 7B for complex coding tasks - **MIT license** for commercial use For comprehensive information, please refer to the [original model card](https://huggingface.co/microsoft/NextCoder-14B). ## 🔧 Hardware Requirements ### Minimum (vLLM): - **GPU:** NVIDIA RTX 4090 (24GB) or RTX 5000 Ada (32GB) - **VRAM:** 14GB minimum, 16GB+ recommended - **CUDA:** 11.8 or newer ### Recommended (vLLM): - **GPU:** NVIDIA RTX 4090 / RTX 5000 Ada / H100 - **VRAM:** 24GB+ - **CUDA:** 12.0+ ### Transformers: - **GPU:** Multi-GPU setup or high-end single GPU (32GB+) - **VRAM:** 28GB+ (single GPU) or distributed - **Not recommended** for practical deployment ## 📖 Additional Resources - **vLLM Documentation:** [docs.vllm.ai](https://docs.vllm.ai/) - **TensorRT-LLM:** [github.com/NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) - **TevunahAi Models:** [huggingface.co/TevunahAi](https://huggingface.co/TevunahAi) - **llm-compressor:** [github.com/vllm-project/llm-compressor](https://github.com/vllm-project/llm-compressor) ## 📄 License This model inherits the **MIT License** from the original NextCoder-14B model. ## 🙏 Acknowledgments - **Original Model:** Microsoft NextCoder team - **Quantization Framework:** Neural Magic's llm-compressor - **Quantized by:** [TevunahAi](https://huggingface.co/TevunahAi) ## 📝 Citation If you use this model, please cite the original NextCoder work: ```bibtex @misc{nextcoder2024, title={NextCoder: Next-Generation Code LLM}, author={Microsoft}, year={2024}, url={https://huggingface.co/microsoft/NextCoder-14B} } ``` --- ## 🌟 Why TevunahAi Premium Calibration FP8? ### Task-Optimized Calibration TevunahAi doesn't use one-size-fits-all calibration: | Model Type | Calibration Focus | Example Datasets | |------------|-------------------|-----------------| | **Code Models** | Code-specific | CodeAlpaca, evol-codealpaca | | **General Models** | Diverse instructions | UltraChat, SlimOrca | | **MoE Models** | Balanced distribution | Multi-task datasets | **The right calibration for the right model.** ### The Difference is in the Details | Aspect | Standard FP8 | TevunahAi Premium FP8 | |--------|--------------|----------------------| | **Calibration Samples** | 128-512 | **2,048** | | **Datasets** | Single generic | **4 code-focused** | | **Calibration Time** | ~35 min | **91 min** | | **Edge Case Handling** | Adequate | **Superior** | | **Code Quality** | Good | **Excellent** | | **Production Ready** | Maybe | **Absolutely** | | **Infrastructure** | Consumer/Prosumer | **Enterprise-grade** | ### Professional Infrastructure - **2.6 TB/s** aggregate memory bandwidth - **2,048 samples** across 4 code-focused datasets - **Quality-first** approach over speed - **Enterprise-ready** results for production code generation **When deploying code models in production, accept no compromises.** ---
**Professional AI Model Quantization by TevunahAi** *Code-optimized premium calibration on enterprise-grade infrastructure* [View all models](https://huggingface.co/TevunahAi) | [Contact for custom quantization](https://huggingface.co/TevunahAi)