# Playing around with FrameFT

Experiments · Oct 15, 2026

I recently came across [FrameFT](https://arxiv.org/abs/2608.26430), a fine-tuning method from an ICML 2026 paper by Harshavardhan Adepu, Li Zhang, Sanjiv Kumar and Vikas Singh, at UW–Madison and Google DeepMind. What caught my eye was the staggeringly low number of parameters it trains; depending on the config it can easily be more than 20× smaller than LoRA without a performance impact!

The method is based on a frozen ‘tight fusion frame’ and a trainable sparse matrix. The fusion frame is just a fixed matrix, similar to [Fourier features](https://arxiv.org/abs/2006.10739) in simple MLPs. I wanted to test this out myself to see how impactful fusion frames are. I forked [the code](https://github.com/ArjunSahlot/FrameFT), swapped the frame for two simpler matrices, and ran 63 fine-tuning runs on my local compute.

I found that a random (but orthogonal) matrix did about as well as the fusion frame, but an identity matrix did clearly worse. This makes me think the secret sauce isn't necessarily in the formulation of the frame, but rather in its orthogonality and the way it mixes the weights.

## Fine-tuning with very few numbers

A language model mostly just consists of large matrices called weights. RoBERTa-base, the model used in the paper and this experiment, has about 124M of them. Fine-tuning just nudges those weights in a clever way to optimize for a certain task.

Nudging all 124M is very inefficient and as costly as training. The resulting checkpoints will be as big as the original model. So the usual trick of PEFT (parameter-efficient fine-tuning) is to freeze the model and learn a smaller weight update, ΔW. [LoRA](https://arxiv.org/abs/2106.09685) does this by splitting an M×N ΔW into an M×r and an r×N matrix, which multiply out to the right shape. FrameFT does this with a preset frame matrix B, which stays constant, and a sparse matrix S that learns, computing out to ΔW = BᵀSB. The sparsity of S sets the parameter count: only its nonzero entries, its coefficients, get trained.

**What each method trains in RoBERTa-base.**

| Method | Trained numbers | What trains |
| --- | --: | --- |
| Full fine-tuning | 124,055,040 | every weight in the model |
| LoRA | 294,912 | two thin matrices beside each query and value matrix |
| FrameFT | 24,000 | 1,000 coefficients in S for each query and value matrix |

One query matrix up close (layer 7, 768 × 768):

- **Full fine-tuning:** W itself trains. Written as an update, ΔW is a full 768 × 768 matrix with all 589,824 entries trained.
- **LoRA:** W stays frozen and gains a side path, Down (768 × 8) then Up (8 × 768), both trained.
- **FrameFT:** W stays frozen and gains a side path, Bᵀ then S then B. The frame B is fixed; only 1,000 entries of S train.

*RoBERTa-base. LoRA is rank 8 and FrameFT 1,000 coefficients, both on the query and value matrices of all 12 layers. The dots in S are the real positions FrameFT trains on MRPC. Every method also trains a new classifier on top (592,130 numbers), left out of these counts, as in both papers.*

FrameFT trains 1,000 coefficients at randomly picked spots in S, on the query and value matrices of each of the model's 12 layers. That's 24 matrices with 1,000 coefficients each, 24,000 in all.

## What's special about the frame?

The frame is where the paper's math lives. It's built with an algorithm called Spectral Tetris, and it depends on a ‘subspace size’ parameter, ρ. At ρ = 2 the frame is almost completely filled in, and every doubling of ρ zeroes out about half of what's left. Understandably, ρ = 2 scores the best, if only slightly, and it's the setting behind most of the paper's numbers.

Interestingly, at ρ = 2 the matrix turns out to be a [Fourier matrix](https://en.wikipedia.org/wiki/DFT_matrix), the same math that splits a sound into its frequencies. [Fourier features](https://arxiv.org/abs/2006.10739) help a simple network learn fine detail much faster, so I wondered if the Fourier matrix was doing something similar here.

But how much of a difference does the frame actually make? To find out, I compared it against a couple of simpler matrices.

## The experiment

I compared three types of frames:

- **Frame**: the paper's own, with the authors' settings for each task (ρ = 2 on RTE and MRPC, ρ = 4 on STS-B).
- **Random**: start off with a random matrix, then apply QR decomposition to make it orthogonal.
- **Identity**: multiplying by it changes nothing, so this is basically ‘no frame’. Each coefficient trains one weight directly.

**Before and after QR.** The random matrix my runs used, measured three ways.

| | Random | After QR |
| --- | --- | --- |
| Typical size of an entry | 0.036 | 0.036 |
| Length of a row | 0.94 to 1.08 | exactly 1 |
| Angle between two rows | 81° to 99° | 90°, every pair |
| How much it stretches a direction | 0.0007× to 2× | 1×, every direction |

*The random matrix my runs used: 768 × 768 numbers from a bell curve (seed 0), divided by √768 so its rows are about length 1, then QR-decomposed. Angles are over all 294,528 pairs of rows; stretches are the matrix’s 768 singular values. Each strip is scaled to its tallest bar. Before QR, 48 of the 768 directions shrink to under a tenth of their length. After QR the numbers look just as random, but every row is exactly length 1 and exactly perpendicular to the others, so the matrix never stretches anything.*

Everything else – model, data, hyperparameters – stayed the same. The change was just [one small function](https://github.com/ArjunSahlot/FrameFT/blob/8fe400b0faa895189cfadcd345b603c1dde55032/peft/src/peft/tuners/frame/layer.py#L114):

```python
def build_basis(basis, n, l, basis_seed):
    if basis == "frame":     # the paper's tight fusion frame
        tff, _ = construct_real_tff(n // l, l // 2, n // 2)
        return tff.view(-1, n)
    if basis == "random":    # random, then made orthogonal with QR
        g = torch.Generator().manual_seed(basis_seed)
        x = torch.randn(n, n, generator=g, dtype=torch.float64)
        q, r = torch.linalg.qr(x)
        return (q * torch.sign(torch.diagonal(r))).float()
    if basis == "identity":  # each coefficient is one weight
        return torch.eye(n)
```

Due to compute constraints, I could only run three of GLUE's tasks, with 7 seeds each:

- **RTE**: does the first sentence imply the second?
- **MRPC**: do two sentences say the same thing?
- **STS-B**: how similar are two sentences, from 0 to 5?

## Results

Like the paper, I score each run by its best validation score at any point in training. The switch on the figure shows the score at the very end instead.

**FrameFT on RoBERTa-base by basis, best epoch.** Mean over seeds 42–48; frame vs random wins out of 7 seeds.

| Task | Frame | Random | Identity | Frame beats random | Paper (frame) |
| --- | --: | --: | --: | --: | --: |
| RTE | 79.7 | 79.9 | 72.0 | 3 of 7 | 79.8 |
| MRPC | 90.2 | 89.3 | 88.1 | 6 of 7 | 92.3 |
| STS-B | 90.6 | 90.5 | 89.2 | 4 of 7 | 90.9 |

**FrameFT on RoBERTa-base by basis, final epoch.** Mean over seeds 42–48; frame vs random wins out of 7 seeds.

| Task | Frame | Random | Identity | Frame beats random | Paper (frame) |
| --- | --: | --: | --: | --: | --: |
| RTE | 76.2 | 76.0 | 70.2 | 4 of 7 | 79.8 |
| MRPC | 88.8 | 87.1 | 86.2 | 7 of 7 | 92.3 |
| STS-B | 90.5 | 90.4 | 89.1 | 3 of 7 | 90.9 |

*Validation scores, one dot per seed. A line joins three runs that differ only in the basis. The authors’ own hyperparameters, unchanged; STS-B is correlation ×100. Seeds 42 to 48; a seed fixes the data order and every starting value. The paper’s line is FrameFT’s score in its Table 1.*

Clearly, identity isn't a good option: it scored below both of the others on every task and every seed. But random comes close, only trailing by a small amount on MRPC (about a point). As a sanity check, my frame runs reproduce the paper on RTE (79.7 against its 79.8) and come close on STS-B (90.6 against 90.9). MRPC comes in 2 points low (90.2 against 92.3). Maybe there's a config issue with MRPC which I haven't been able to figure out yet.

## Interpreting the results

Why does identity perform so poorly? The issue isn't that it can't learn; in fact, the training losses go down about the same for all of them. The gap happens on the validation data, meaning weaker generalization. But why? I think it comes down to how the weights get mixed. Without a frame, ΔW is just S: each coefficient changes exactly one weight, wiring one input neuron to one output neuron, and the other 588,824 weights never move. With a frame, every coefficient moves nearly every weight a little, so each one hears a blend of all the inputs. Interestingly, RoBERTa has a few neurons, particularly [neurons 77 and 588](https://arxiv.org/pdf/2105.06990#page=5), that happen to be significantly more important than others. This skewed importance likely exaggerates the effect: without a frame, the few coefficients that land on those neurons get an outsized say, while a frame blends them into every coefficient.

How come random gets so close? The fancy linear algebra makes for a satisfying way to build a frame, but a random matrix matches its performance. I believe this is because the secret sauce is in the mixing and the orthogonality, and a random orthogonal matrix has both. The frame still has strengths on paper: it's structured, and at bigger ρ it gets sparse, which a random matrix never does. In practice, though, at least here, random mixing is just as good, and those strengths don't show up in the scores.

All of this comes with a (really, really) big asterisk: one small model, three small tasks, seven seeds each, and a single random matrix. This is mainly just an exploration experiment to understand FrameFT, rather than to come up with new findings.

## How it ran

Everything ran on my laptop's GPU, an RTX 5070 Ti with 12 GB of memory, one run at a time. The paper's batches of 32 didn't fit in memory, so I used gradient accumulation: each batch runs as two halves of 16, with their gradients added up before the update, which works out to the same math. I set up a queue so runs could go back to back overnight, and had a Luna agent check on them every ten minutes or so, under strict rules never to touch anything a run measures. If there were minor issues, it could fix them and still have results waiting by morning. Over six nights that came to 63 runs and 38 GPU-hours.

**When the runs happened, in UTC.** 63 runs over 6 nights, 38.2 GPU-hours in total.

| Night | Runs | First started (UTC) | Last finished (UTC) | Hours |
| --- | --: | --- | --- | --: |
| Sep 28 | 14 | 06:22 | 16:51 | 10.5 |
| Sep 30 | 10 | 08:45 | 16:07 | 7.3 |
| Oct 1 | 12 | 10:11 | 14:26 | 4.2 |
| Oct 2 | 9 | 09:35 | 14:58 | 5.4 |
| Oct 3 | 9 | 10:01 | 15:22 | 5.3 |
| Oct 4 | 9 | 10:12 | 15:40 | 5.4 |

*Each block is one run, placed at the time it started (UTC) and as wide as it took. One run at a time, on one laptop GPU. An RTX 5070 Ti laptop GPU (12 GB) under WSL2, with an agent checking on the queue through the night. UTC is five hours ahead of Madison that week, so the nights read as mornings.*

## Unanswered questions

- How would a random matrix that isn't orthogonal compare?
- What if, instead of 1,000 randomly placed coefficients, the positions were picked by pruning over a training run?
- Does any of this hold up on bigger models and more benchmarks?
- What's going on with the MRPC numbers?

[All writing](https://arjunsahlot.com/writing)