7 key steps for large model private deployment: Model selection (recommend Qwen2.5-72B, Apache 2.0 license) → Computing resource evaluation (72B model INT4 quantization requires 2×A100 40G) → Inference engine selection (vLLM preferred for production) → Model quantization (AWQ-INT8 accuracy loss <1%, memory savings 50%) → Containerized deployment → Performance optimization (Continuous Batching increases throughput 2-3x) → Monitoring and operations. According to CAICT data, the annual growth rate of enterprise-level large model private deployment demand in 2025 exceeded 60%.
Step 1: How to Choose a Model?
Mainstream Open-Source Model Comparison
| Model | Parameters | Chinese Capability | Inference Speed | Open Source License | Recommended Scenario |
|---|---|---|---|---|---|
| Qwen2.5-72B | 72B | ★★★★★ | Medium | Apache 2.0 | First choice for general scenarios |
| Qwen2.5-7B | 7B | ★★★★ | Fast | Apache 2.0 | Lightweight scenarios |
| DeepSeek-V3 | 671B MoE | ★★★★★ | Fast | MIT | Sufficient budget |
| ChatGLM4-9B | 9B | ★★★★ | Fast | Apache 2.0 | Conversational scenarios |
| Llama3.1-70B | 70B | ★★★ | Medium | Llama3 | Primarily English |
| Yi-1.5-34B | 34B | ★★★★ | Faster | Apache 2.0 | Cost-effective choice |
Selection Recommendations
Step 2: How to Evaluate Computing Resource Requirements?
GPU Requirement Reference
| Model | FP16 | INT8 | INT4 |
|---|---|---|---|
| 7B | 1×A100 40G | 1×A10 24G | 1×RTX4090 24G |
| 34B | 2×A100 80G | 1×A100 80G | 1×A100 40G |
| 72B | 4×A100 80G | 2×A100 80G | 2×A100 40G |
Cost Estimation
| Configuration | Purchase Cost | Monthly Rental Cost | Applicable Scenario |
|---|---|---|---|
| 1×RTX4090 | ¥15,000 | ¥3,000/month | 7B model testing |
| 1×A100 40G | ¥80,000 | ¥15,000/month | 7B-34B model |
| 2×A100 80G | ¥250,000 | ¥40,000/month | 34B-72B model |
| 4×A100 80G | ¥500,000 | ¥80,000/month | 72B+ model |
Step 3: How to Choose an Inference Engine?
| Engine | Throughput | Latency | Ease of Use | Recommended Scenario |
|---|---|---|---|---|
| vLLM | ★★★★★ | ★★★★ | ★★★★ | Preferred for production |
| TGI | ★★★★ | ★★★★ | ★★★★ | Compatibility priority |
| TensorRT-LLM | ★★★★ | ★★★★★ | ★★★ | Latency-sensitive scenarios |
| Ollama | ★★★ | ★★★ | ★★★★★ | Local dev & testing |
Our recommendation: Use vLLM for production (highest throughput, active community), and Ollama for development/testing (one-click deployment).
Step 4: How to Do Model Quantization?
Comparison of Quantization Methods
| Method | Accuracy Loss | Speed Improvement | Model Size Reduction | Applicable |
|---|---|---|---|---|
| FP16→INT8 (AWQ) | <1% | 2x | 2x | General recommendation |
| FP16→INT4 (GPTQ) | 1%-3% | 3x | 4x | Resource constrained |
| FP16→INT4 (GGUF) | 2%-5% | 3x | 4x | CPU inference |
Quantization Effect Reference
Quantization effects of Qwen2.5-72B on Chinese benchmarks:
| Quantization | C-Eval | Inference Speed (tokens/s) | Memory Usage |
|---|---|---|---|
| FP16 | 83.5 | 25 | 144GB |
| AWQ-INT8 | 82.8 | 48 | 72GB |
| GPTQ-INT4 | 81.2 | 72 | 40GB |
Step 5: How to Configure Containerized Deployment?
```yaml
docker-compose.yml example
services:
vllm:
image: vllm/vllm-openai:latest
deploy:
resources:
reservations:
devices:
count: 2
command: >
--model Qwen/Qwen2.5-72B-Instruct-AWQ
--quantization awq
--tensor-parallel-size 2
--max-model-len 8192
--gpu-memory-utilization 0.9
ports:
```
Step 6: How to Optimize Performance?
| Optimization Item | Method | Effect |
|---|---|---|
| Continuous Batching | Dynamic batching | Throughput improvement 2-3x |
| PagedAttention | Memory paging management | Memory utilization increase 40% |
| Prefix Caching | System prompt caching | Latency reduction for same prefix requests by 50% |
| Speculative Decoding | Small model speculates, large model verifies | Inference speed improvement 2-3x |
Step 7: How to Do Monitoring and Operations?
Key Monitoring Metrics
| Metric | Alert Threshold |
|---|---|
| GPU Utilization | >95% for 5 minutes |
| Inference Latency P99 | >5 seconds |
| Request Failure Rate | >1% |
| Memory Usage | >90% |
| Model Service Availability | <99.9% |
Operations Strategy
FAQ
How much investment is needed for large model private deployment?
For 7B model private deployment: hardware ¥15,000 (1×RTX4090) + deployment ¥30,000-50,000, total investment ¥50,000-70,000. For 72B model: hardware ¥250,000 (2×A100 80G) + deployment ¥80,000-120,000, total investment ¥330,000-370,000. According to IDC data, the average initial investment for enterprise large model private deployment is ¥250,000-500,000, with annual operating costs of ¥50,000-100,000.
Which is more cost-effective, private deployment or API calls?
When monthly call volume is below 5 million tokens, API calls are more cost-effective (monthly cost approx. under ¥10,000). When monthly call volume exceeds 5 million tokens, private deployment is more economical (fixed costs are controllable). The breakeven point for 72B model private deployment is approximately 8 million tokens per month. According to NVIDIA calculations, from a 3-year TCO perspective, private deployment in high-usage scenarios saves 40%-60% compared to API calls.
Is there a performance gap between privately deployed models and API versions?
There is a slight difference. Taking Qwen2.5-72B as an example: the API version (Tongyi Qianwen Max) uses FP16 precision and the latest optimizations, while the private AWQ-INT8 quantized version has an accuracy loss of about 0.7%. For the vast majority of enterprise scenarios, this gap can be ignored. However, for scenarios with extremely high accuracy requirements (such as medical diagnosis or legal compliance), it is recommended to deploy the FP16 version privately or use a larger parameter model.
Interested in large model private deployment solutions? Book a free computing resource evaluation