Hiding a transformer from its own GPU

The attack

A transformer running on a GPU leaves a trace. Kernels fire in a particular order, with particular durations, touching particular memory addresses — and that pattern is characteristic of the architecture producing it. Decepticon (Rafi et al., IISWC 2023) turns that trace into an extraction attack.

The insight it exploits is that fine-tuned models inherit their parent's execution fingerprint. A model fine-tuned from BERT still runs like BERT. So an adversary who can observe kernel timings on shared hardware can identify which pre-trained model sits underneath a proprietary system, then clone it by fine-tuning the same public parent — recovering weights close enough that the original paper reports a 94% match in predictions against a BERT-based victim.

The fingerprint is the vulnerability. Everything below is an attempt to destroy it.

Two defenses

We recreated the Decepticon attack as described, then built two countermeasures that manipulate GPU kernel execution to corrupt the signal the attack depends on. They take opposite approaches.

Constant-load padding

Maintain a continuous stream of short, busy workloads alongside the real inference. The genuine kernel executions are still there, but they're buried in manufactured activity, and every model's execution profile is forced toward the same uniform shape. If all fingerprints look alike, none of them identifies anything.

Jitter scheduling

Perturb the kernel schedule directly. Introduce bursts and delays so that the timing graph smears across runs — the same model executed twice produces two different traces. Rather than hiding the fingerprint, this makes it unstable.

What it cost

We measured both against the recreated attack on a BERT-base victim, recording median inference latency and how often the attack correctly identified the parent model.

Configuration	Median latency	Overhead	Attack top-1	Attack top-3
Undefended	305.79 ms	—	95.0%	100%
Constant-load	2,026.72 ms	6.6×	9.6%	40.3%
Jitter	406.71 ms	1.33×	61.1%	75.0%

Attack top-1 and top-3 are the rates at which Decepticon placed the correct parent model first, or within its first three guesses.

Reading the result

Neither defense is a solution, and that's the finding worth stating plainly.

Constant-load padding does what it was built to do — it takes the attack from near-certain identification down to below ten percent, which is worse than guessing among a handful of candidates. But a 6.6× slowdown is not a cost any production inference service would absorb. It's a demonstration that the fingerprint can be destroyed, not a deployable mitigation.

Jitter is the practical one. A 33% overhead is within the range a security-conscious deployment might accept, and it cuts identification by a third. But 61% top-1 still means the attacker is right more often than not, and 75% top-3 means they can usually narrow the field to three candidates. That's meaningful leakage.

It's also worth being precise about the threat model. Both defenses assume an adversary observing kernel-level timing on shared hardware. They do nothing about extraction attacks that operate purely on model inputs and outputs, and they don't address the broader question of whether architectural obscurity is worth paying for at all when the parent models are public and few in number.

Context

This work grew out of a literature survey of transformer security that I first-authored during the same apprenticeship, which mapped adversarial attacks onto machine-learning pipeline phases and adversary knowledge levels. Writing it is what surfaced the gap: hardware and side-channel defenses were consistently underexplored relative to input-level ones, and almost nobody was quantifying what those defenses cost.

Presented at the CV PATH speed research presentations (Bakersfield College, CSUB, and UC Merced) in September 2025, and as an invited seminar talk at CSUB. The poster is here.