Ripper Aussie / Fine-tuning

Runpod RTX A6000 48GB for QLoRA Fine-Tuning: My Real Experiment

My actual QLoRA workflow using an RTX A6000 48GB on Runpod, including local baseline results, training configuration, storage decisions and the metrics I want to beat.

This is the experiment that pushed me from local inference into rented GPU training.

My local setup is a Ryzen 7 255 mini PC with Radeon 780M graphics and 32GB of system memory. The baseline model is Qwen3.5-4B in Q6_K GGUF format, around 3.46GB.

The local baseline

Before paying for cloud training, I measured what the local setup could already do.

• 30/30 validation cases passed with thinking enabled.
• Average latency with thinking enabled was 78.38 seconds per item.
• With thinking disabled, the same 30/30 cases passed.
• Average latency dropped to 3.04 seconds per item.
• Observed runs were roughly 2.68–3.98 seconds per item.
• That made the non-thinking path about 25.8× faster.

I then ran a harder benchmark. Increasing GPU offload on the integrated Radeon only moved classification accuracy from roughly 82% to 84%, review accuracy stayed around 72%, and fully-correct results moved from 58% to 60%. Latency changed only marginally, from about 3.60 to 3.55 seconds.

Why the A6000 48GB

I selected an RTX A6000 with 48GB VRAM on Runpod at about US$0.53/hour.

The A6000 is not the newest or fastest GPU in every benchmark. I chose it because 48GB gives plenty of headroom for a 4B-class model, QLoRA training state, sequence-length experiments and batch-size changes without immediately turning the session into a VRAM optimisation exercise.

The planned QLoRA configuration

The first training run is intentionally conservative:

• base model: Qwen3.5-4B;
• 4-bit training path;
• LoRA rank: 16;
• LoRA alpha: 32;
• dropout: 0.05;
• learning rate: 2e-4;
• 2 epochs initially;
• sequence length around 1024 for the first pass;
• Unsloth as the preferred training stack.

The objective is not to burn GPU time. It is to determine whether the adapter improves hard boundary cases without making easy cases worse.

Validation stays separate

I created a dedicated validation set before moving to the cloud. It contains similar but distinct boundary cases rather than copies of the held-out test set.

Line counts and JSON validation are done locally before paid compute starts.

Storage discipline

I assume the container disk is disposable. Datasets, checkpoints, adapters and useful output go to persistent storage or are copied out before the Pod is terminated.

This is not just housekeeping. Losing a checkpoint after an otherwise successful training run can make the cheapest GPU session the most expensive one.

What I will measure

I am not treating lower training loss as success. The adapted model has to beat the hard local baseline.

• classification accuracy;
• review-decision accuracy;
• fully-correct end-to-end result rate;
• latency per item;
• confidence calibration;
• regressions on obvious cases.

The current hard baseline to beat is approximately 84% classification accuracy, 72% review accuracy and 60% fully-correct results.

Why renting makes sense for this experiment

At about US$0.53/hour, ten hours of A6000 compute is roughly US$5.30 before storage and other charges. That is fundamentally different from buying a dedicated NVIDIA GPU for occasional training.

The local machine handles the cheap, repeatable work. The cloud handles the short period where CUDA and VRAM materially change the experiment.

What I have not measured yet

I have deliberately not published a claimed “after” result because the fine-tuning run has not yet produced a final benchmark I trust. When it does, I will add the actual training duration, peak VRAM, total GPU cost and before/after accuracy here.