cheapest gpu overall for a gemma 3 27b lora?

[mikakotakenomiya]

has anyone compared the total rental cost for training a qlora on gemma 3 27b? im trying to figure out whether something cheaper like an l40s actually saves money or if an a100/h100 finishes fast enough to cost less overall. mostly care about the final cost for one completed run, not the hourly rate.

[kratos0718]

Worth separating two things here: whether the card fits the job, and what one
completed run costs. For QLoRA at 27B they're almost unrelated.

Memory first, since that's what usually drives card choice — and at 4-bit it
shouldn't:

27B weights @ NF4 ≈ 12.6 GiB
LoRA r=16 on q,v ≈ 18M trainable params
adapter (bf16) ≈ 0.03 GiB
+ Adam m,v (fp32) ≈ 0.14 GiB

So the frozen base plus the trainable state is roughly 13 GiB. Everything on
your list clears that with room to spare:

A100-40 ~26 GB left for activations/KV
L40S-48 ~34 GB left
A100/H100-80 ~66 GB left

The point being: none of these are memory-constrained for a 27B QLoRA run. What
fills the remaining space is activations, which scale with batch size ×
sequence length — and with gradient checkpointing on (unsloth enables it by
default) that drops a lot too. So the extra 32 GB on an 80 GB card only buys you
something if you actually push batch size or context length up to use it.

Which means your question reduces to throughput per unit cost:

cost_per_run = hourly_rate ÷ (steps_per_hour)

and not to the hourly rate on its own, which is what you already suspected.

The part I'd push back on is assuming the H100 finishes proportionally faster.
QLoRA spends a lot of its time on the dequantisation path, which tends to be
memory-bandwidth bound rather than FLOP bound, so an H100's compute advantage is
under-used relative to its price premium. Ada-class cards like the L40S
frequently come out ahead on cost-per-run for 4-bit LoRA for exactly that
reason. That's a tendency, not a guarantee — it flips once you're big enough on
batch/sequence to become compute bound.

Rather than trust anyone's numbers (including mine), this is cheap to measure
directly: run ~50 steps on each candidate with your real dataset and sequence
length, take steps/sec, and compute

hourly_rate ÷ (steps_per_sec × 3600) × total_steps

That costs a few minutes of rental per card and gives you the actual answer for
your config, which is the only one that matters.

Two things that will move the cost more than the card choice:

• sequence length (activations scale with it, and so does time per step)
• packing / avoiding padding — if you're padding to max length you may be paying
  for a lot of nothing

For transparency: I've done QLoRA fine-tuning with unsloth on a much smaller
model (Qwen2-VL-2B), not on 27B, so treat the throughput reasoning above as
reasoning rather than a benchmark. The memory figures are just arithmetic and
you can check them.