Define models & targets
Record model size, precision, context, concurrency and latency. Separate inference, fine-tuning and full training; their memory estimates differ.
Local AI workstation design
Plan models, GPU memory, power and software as one system.
Architecture outline
Data / model SSD → System RAM & CPU → GPU inference → Local application
These planning and selection methods support discussion. Adapt the final configuration to workloads, site conditions and validation results.
Design steps & decisions
GPU memory budget
For an 8B model, weights alone need about 16GB at FP16 or 4GB at 4-bit. Add KV cache, quantisation metadata, runtime overhead and headroom. These figures are not total runnable VRAM requirements.
System balance
Check GPU dimensions, power, PCIe lanes and cooling. System RAM must support data processing and loading; plan NVMe capacity for OS, models and datasets.
Repeatable testing
Pin drivers, framework, model and prompts. Record time to first token, generation rate, peak VRAM and sustained temperature. Verify workload distribution before selecting multiple GPUs.
Prepare for a design discussion
Provide current equipment and versions, key applications and workloads, users or concurrency, data volume and growth, site power and networking, budget, timing and critical service windows.
Deliverables to agree
Requirements brief, configuration and BOM, compatibility records, test conditions and acceptance checklist. Agree the actual scope of installation, migration, maintenance and on-site services during project confirmation.
Continue selection & planning
LET’S PLAN YOUR CONFIGURATION
Discuss your requirements ↗