Ripper Aussie / Local + Cloud

How I Run a 4B LLM Locally and Move Training to Runpod

My hybrid AI workflow: Qwen3.5-4B locally for fast iteration, then Runpod for CUDA and QLoRA training when the local hardware stops being the right tool.

I do not see local AI and cloud GPUs as competing approaches. In my setup they solve different parts of the same problem.

The local model

I run Qwen3.5-4B in Q6_K GGUF format locally. It is small enough to be practical on a compact development machine and fast enough, with the right settings, to support repeated classification experiments.

With thinking disabled, my validation workload averaged about 3.04 seconds per item while still passing all 30 validation cases in that particular set.

Why local iteration matters

During development I may change prompts, schemas, labels, validation logic and test cases repeatedly. Doing that locally means:

• no startup delay;
• no hourly billing;
• no network dependency;
• fast access to logs and files;
• no need to upload every small change.

Where local hardware stopped helping

I tested additional GPU offload on the integrated Radeon. The improvement on the harder benchmark was modest: classification accuracy moved from roughly 82% to 84%, fully-correct results from 58% to 60%, and latency barely changed.

That suggested the next gain was more likely to come from model adaptation than from pushing more layers onto the integrated GPU.

That is where Runpod enters the workflow

The cloud phase is narrowly defined. I need NVIDIA CUDA, more VRAM and enough headroom to run QLoRA experiments cleanly.

I do not move the entire development environment to the cloud. I move only the expensive training phase.

The handoff from local to cloud

1. prepare and clean training data locally;
2. create a separate validation set;
3. validate file structure locally;
4. freeze the first training configuration;
5. start the cloud GPU;
6. run a smoke test;
7. fine-tune;
8. save the adapter and checkpoints;
9. bring the resulting artefacts back to the local machine;
10. benchmark them against the original local baseline.

Why this architecture is cost-efficient

The local machine is paid for and useful regardless of whether I am training a model. The cloud GPU is only running during the small window where it provides a real capability advantage.

That keeps cloud cost variable and makes the local machine the stable development environment.

What stays private

Local execution also gives me the option to keep sensitive inference workloads entirely on my own machine. Moving training to the cloud does not mean every production inference request must move there as well.

The outcome I want

The ideal result is a locally deployable model or adapter that is better because it was trained on stronger remote hardware, but does not require that remote hardware for day-to-day use.