Filling the Idle Tiles: Tile-Grouped Convolutions for Keyword Spotting on Array Accelerators

The smallest accurate keyword spotters are the always-on wake-word detectors that listen for "Hey Siri" or "OK Google" on a battery. They owe their size to a single trick. Each ordinary convolution is split into two parts: a depthwise step that filters each feature channel over time on its own, and a pointwise step that mixes channels at each time step. This split cuts parameters and multiply-accumulate operations sharply. On a matrix accelerator, which is built from a fixed square grid of multipliers, the depthwise step does almost nothing useful. One channel's filter has only a few taps, so it fills a sliver of the grid while every other multiplier waits. The field reports efficiency as parameters and operation counts, so this waste never appears in its tables.

This note proposes to put the waste to work. The depthwise step would be replaced by a grouped convolution whose group size equals the accelerator's tile width. Each tile then holds a small dense filter bank rather than one lonely filter. Our first-order model predicts that the grouped layer takes about the same number of tile-cycles as the depthwise layer, possibly fewer, while doing roughly tile-width times more arithmetic per channel. That extra capacity should buy accuracy at no measured cost in latency or energy. The claim is testable: on a cycle-exact systolic array, a tile-grouped keyword spotter should match the latency of a depthwise model while beating its accuracy. It should also beat widening or densifying the model at matched latency.

Core Claim

Operation count is the wrong cost for compact speech models on tiled hardware. The right cost is how many tiles a layer occupies and for how many cycles. Under that cost, the capacity a layer can hold for free is set by the tile. Depthwise convolution leaves most of it unused. Take a one-dimensional convolution over $C$ channels and $T$ time frames, with kernel length $k$ and group size $g$, lowered to a matrix product and run weight-stationary on an $S \times S$ array. To first order it occupies

$$ N_{\text{tiles}}(g) = \frac{C}{g}\left\lceil \frac{g k}{S} \right\rceil \left\lceil \frac{g}{S} \right\rceil $$

weight tiles. Each tile streams all $T$ frames, so tile-cycles scale as $N_{\text{tiles}} T$. The expression counts groups, times the tiles needed to hold each group's $gk$-by-$g$ weight matrix. Depthwise ($g=1$) gives $C$ tiles, each only $k/S^2$ full. Tile-grouped ($g=S$) gives $Ck/S$ tiles that are completely full. So whenever $k \le S$, the grouped layer uses no more tile-cycles than the depthwise one, yet has $S$ times the weights per channel.

This model ignores data movement and the host-side reshaping overhead. Gemmini showed that overhead dominates on lightweight networks, so it is exactly what the experiments must measure rather than assume.

The contribution is threefold. It supplies the hardware cost accounting that compact keyword-spotting work omits. It replaces a zero-padding penalty with learned capacity. And it tests whether this yields a better accuracy–latency frontier than the obvious alternatives.

Relation to Prior Work

Three lines of prior work motivate the gap without closing it.

The first is the compact keyword-spotting architectures, which report only parameters and operations. MatchboxNet builds residual stacks of one-dimensional time-channel separable convolutions. MarbleNet reuses the design for voice activity detection. ST-Conv adds dilated separable temporal filters, a recurrent layer and shared-weight attention. None of them profiles the models on array hardware.

The second is architecture search. EdgeSpeechNets synthesizes non-uniform layer widths, and DARTS-based search for keyword spotting searches cells with no hardware term in its objective. Neither constrains shapes to the tile.

The third is the hardware papers, which document the failure but in other domains. Tetris-SDK finds that its adaptive in-memory mapping gains nothing on MobileNet's depthwise layers. Tellingly, it reaches for grouped convolutions to relieve channel-depth limits in vision networks, and that finding is the closest precedent to ours. Gemmini names depthwise layers an unresolved systolic-array bottleneck.

We differ from Tetris-SDK in three ways: our domain is one-dimensional streaming speech, the group size is tied to the tile width, and the extra capacity is treated as an accuracy lever rather than a mapping aid.

Two papers constrain the design. The Case for Co-Designing Model Architectures with Hardware shows that reshaping dimensions to tile multiples buys throughput in large models, for example 20% faster GPT-3 2.7B training from a change in head count. There the padding overhead is amortized. Deep Tensor Convolution on Multicores warns that padding hurts small networks disproportionately. Our design resolves this tension by filling the padding with trained weights instead of zeros.

Structured tile pruning supplies a complementary method. It removes whole systolic-tile blocks of feed-forward weights in speech translation Transformers. We reuse its tile-level view, but we add capacity where tiles are empty rather than remove it where they are full.

Research Program

We will first build the cost accounting. MatchboxNet, MarbleNet and ST-Conv will be lowered to 8-bit integer kernels on Gemmini with 8×8, 16×16 and 32×32 arrays, in both weight-stationary and output-stationary dataflows. We will record per-layer cycles, PE utilization and host time under cycle-exact simulation. PE utilization means useful multiply-accumulates divided by the number of PEs times cycles, and host time is time spent reshaping tensors on the RISC-V core. This produces the first hardware profile of these models. It also tells us whether the depthwise layers really dominate wasted tile-cycles, or whether the host overhead dominates instead.

Next we build the tile-grouped variants. Each depthwise layer becomes a grouped convolution with $g = S$, channel counts are rounded to multiples of $S$, and the pointwise layers are kept. For the single-channel input layer, which works on 40 feature bins, we fold adjacent time frames into the channel dimension so the first tile is filled rather than padded. This targets the early-layer channel deficit noted by Gholami et al..

We then sweep $g$ from 1 to $2S$. The first-order model predicts a knee in tile-cycles at $g=S$ and a monotone rise in capacity, so accuracy against measured latency should bend sharply at the tile width. Training follows each source paper's recipe, and quantization to 8 bits follows the analytic post-training approach of Banner et al. as a fixed step, identical across all models.

Finally, streaming. Always-on deployment slides a window over continuous audio. We therefore also measure cost per 10 ms hop with cached convolution state, since the tile argument could change when $T$ is small.

Baselines

The naive reference point is each published model compiled as is, with the toolchain zero-padding dimensions to the tile. It shows how much waste exists before anyone intervenes.

The strongest existing accuracy baseline is MatchboxNet-3x1x64, which reports 97.21% accuracy on Google Speech Commands v1 with 77k parameters (MatchboxNet). It is the strongest competitor because it is the leading depthwise-separable design, exactly the structure we claim wastes the array. For the smallest regime we use ST-Conv, which reports 95.7% with 32.2K parameters and 3.09M multiplications (ST-Conv). No reported result was found on Lacuna for the latency, energy or PE utilization of either model on any array accelerator, so we produce those numbers ourselves.

For fairness, every model is retrained by us with the same data, augmentation, epochs, hyperparameter search budget and 8-bit quantization, on the same Gemmini configuration. Our retrained numbers, not the published ones, are the comparison points. Beating the baseline means higher accuracy at equal or lower measured cycles and energy per inference, within the parameter memory the baseline uses plus a stated margin. It also counts if we match accuracy at clearly fewer cycles.

A dense baseline replaces the depthwise layers with full convolutions, bounding what capacity alone can do. KW-MLP is the dense matrix-multiply alternative (97.56% on v2-35 at 0.424M parameters). A widening ablation scales MatchboxNet's channel count to the same measured latency as our model. It isolates whether the gain comes from filling tiles or simply from more parameters. The $g$ sweep itself is the ablation for the key ingredient: a gain that appears at $g=S$ but not at mismatched group sizes of similar parameter count attributes the result to tile alignment.

Why not just…?

Why not just let the compiler pad or pick a better kernel for depthwise layers? Padding is the naive baseline above, and our cost model says padding and depthwise waste are the same tiles. Deep Tensor Convolution on Multicores shows padding costs small networks the most. A smarter compiler could, however, route depthwise layers to the host's vector unit instead of the array. That is a real threat on chips with strong vector units. We will run this routing as an additional baseline. Gemmini suggests the host becomes the bottleneck on lightweight models, but if host routing matches our latency, the claim shrinks to array-only chips, and we will say so.

Why not just use dense convolutions or a gated MLP that maps cleanly onto matrix tiles? Dense layers fill tiles but multiply parameter memory. KW-MLP needs 0.213M–0.424M parameters (KW-MLP), against 77k for MatchboxNet. Grouping at the tile width is the smallest change that fills the tile. The dense baseline, compared at matched latency and memory, settles whether the middle point is actually better.

Why not just run hardware-aware architecture search with a latency table? This is the strongest alternative. A search whose space includes group size and whose objective uses measured Gemmini latency might find tile-grouping by itself. We will run exactly this, extending the DARTS setup of NAS for KWS with a latency term. Two outcomes are possible. If the search converges to $g \approx S$ and does no better than our one-line rule, the contribution is the cost model and a rule that removes the search. If the search does better, the rule is insufficient and we report that.

Why not just prune tiles, as in speech translation? Tile pruning (Rouas et al.) removes full tiles from overparameterized feed-forward layers. It cannot fix depthwise layers, whose tiles are nearly empty rather than full. We test it as a composable step on the pointwise layers only.

Proposed Experiments and Decisive Checks

The evaluation uses Google Speech Commands v1-12 and v2-35 for accuracy, following the protocols used by MatchboxNet, ST-Conv and KW-MLP. We also report AVA-speech true positive rate for a MarbleNet-style voice activity variant, against MarbleNet's reported 0.858 (MarbleNet). Hardware metrics are cycles and energy per one-second clip, cycles per streaming hop, PE utilization and host fraction, all from cycle-exact Gemmini simulation across three array sizes and two dataflows.

The first decisive check is whether measured tile-cycles for $g=S$ fall at or below those for $g=1$, as the cost model predicts. If data movement or host reshaping erases this, the idea fails at its root, whatever the accuracy.

The second is whether, at matched measured latency, the tile-grouped model beats MatchboxNet, the widened MatchboxNet and the dense variant in accuracy, with the advantage peaking at $g=S$ and shifting when $S$ changes. A shift that tracks the array size would be the strongest evidence that the tile, not the parameter count, drives the result.

The third is whether the advantage survives streaming with short hops and host-side vector routing. It tells us whether this is a design rule for always-on keyword spotting or only for offline array execution.