ALGOTECHNIC
All articles
8 min read

Why Transformers Became the Default Architecture for Foundation Models

How attention, parallel training and pretrain–transfer made Transformers the leading foundation-model architecture, without eliminating CNNs or inference bottlenecks.

  • transformers
  • foundation models
  • transfer learning
Text tokens and image patches enter attention blocks, then branch into reusable representations for several downstream tasks.

A foundation model is trained on broad data and then adapted to many downstream tasks. That recipe needs more than a large network. It needs an architecture that can absorb broad datasets, use accelerator hardware efficiently and expose representations that transfer without a redesign for every task. By April 2022, the Transformer had become the leading answer to those requirements, first in language and increasingly in vision.

This does not make it the default for all machine learning. Its rise rests on a specific combination: self-attention mixes information across a sequence, matrix operations expose parallel work during training, and pretraining turns the resulting network into a reusable base. The same design also imposes quadratic attention costs and leaves autoregressive generation sequential. Those constraints belong in any production decision.

Attention makes context a matrix operation

A recurrent network processes a sequence through a chain of hidden states. Computing position t requires the state from t−1, so the positions within one example cannot be evaluated in parallel. A Transformer instead projects each input embedding into queries, keys and values. For one attention head, it computes softmax(QKᵀ/√dₖ)V: each query is compared with every key, normalised into weights and used to combine the values.

Every output position therefore has a direct route to every permitted input position. A word can draw on a distant word without passing information through each intervening hidden state. Multiple heads learn separate projections, allowing different relations to be represented in parallel. Position embeddings restore order information that the attention operation itself does not supply, while feed-forward layers, residual connections and normalisation complete each block.

The crucial systems advantage is the shape of the computation. Queries, keys and values for all positions can be packed into matrices, and the attention scores can be evaluated together. During training, the targets are already available. Masked-token objectives can predict selected positions from their surroundings, while a causal mask prevents each next-token prediction from seeing later positions. Both provide many prediction targets in one pass. Parallelism comes from removing the recurrent dependency across input positions, not from making the total computation free.

This maps well to accelerators designed for large matrix multiplications. Bigger batches and models can keep more arithmetic units occupied, although memory capacity, communication between devices and kernel efficiency still determine realised throughput. The architecture removes one bottleneck; it does not suspend hardware limits.

Pretraining turns a backbone into shared infrastructure

Self-attention explains how a Transformer mixes context, but it does not explain why teams adopt the same backbone across tasks. Pretrain–transfer learning supplies that reason. A model first learns from a broad corpus using labels derived from the data itself. It is then adapted to a narrower task through fine-tuning, a learned output head or conditioning at inference time.

BERT made the arrangement unusually clear. Its bidirectional encoder learns to predict masked tokens from both left and right context. For downstream classification or span prediction, a small task-specific output layer can be attached and the network fine-tuned end to end. The input conventions and internal representation remain largely intact. Autoregressive Transformers use a different objective, predicting the next token from previous ones, but preserve the same division between broad pretraining and later adaptation.

This changes where engineering effort goes. Pretraining is expensive and can be amortised across many applications; adaptation reuses learned features instead of starting from random weights. A common block design also creates a stable interface for tokenisers, checkpoints, optimisation code and deployment tooling. Teams can improve a shared base or replace an output head without inventing a new architecture for each task.

Reuse also concentrates risk. Pretraining data, model defects and evaluation blind spots can travel into every adapted system. A production team should treat a checkpoint as shared infrastructure: record its data and provenance, test each intended use, and avoid assuming that broad pretraining guarantees behaviour in a narrow domain. Transfer reduces task-specific training; it does not remove task-specific validation.

Training computes all token positions in parallel, dense attention forms an n-by-n score matrix, and autoregressive generation produces tokens in dependent steps.
Known targets allow training to compute every position together, but dense attention still forms an n-by-n score matrix. Autoregressive inference must produce tokens in order, so output length remains a serial latency cost.

Vision adopted the recipe, not just the layers

The Vision Transformer (ViT) tested how little of the architecture needed to change beyond NLP. It divides an image into fixed-size patches, projects each patch into an embedding and treats the patches as tokens. A class token and positional embeddings join the sequence, which then passes through a standard Transformer encoder. For transfer to a new classification task, the prediction head can be replaced while the backbone is retained.

The important result is not that pixels naturally resemble words. It is that a common sequence-processing backbone can learn useful visual representations once the input has been expressed as tokens. The pretrain–transfer recipe survives the change of modality, along with much of the implementation and scaling knowledge developed for language.

ViT also exposes the cost of removing domain structure. A convolutional network builds locality and translation equivariance into its layers. A plain ViT must learn most two-dimensional relationships from data; its patch extraction and positional handling supply only limited vision-specific bias. The original work found that large-scale pretraining made this approach competitive, while training on a smaller dataset favoured models with stronger visual priors.

That finding is a conditional result, not a law that vision Transformers always require enormous private datasets. Data-efficient image Transformers subsequently used stronger augmentation, regularisation and optional distillation from a convolutional teacher to train effectively on ImageNet alone. The training recipe can compensate for some missing inductive bias. Nor does success in image classification establish universal replacement: convolutional and hybrid models remain credible choices, particularly where data, latency or mature task-specific architectures matter.

The costs move rather than disappear

Dense self-attention compares each of n queries with n keys. The resulting score matrix has n² entries, so attention computation and memory grow quadratically with sequence length. Feed-forward and projection operations also account for substantial computation and scale differently. An asymptotic comparison alone cannot predict serving speed, but doubling a context can quadruple the attention component.

This constraint connects otherwise separate design choices. Longer documents increase token count. Higher-resolution images produce more patches when patch size stays fixed; smaller patches have the same effect. A larger context window is therefore not a free product feature. It changes memory planning, batch size, accelerator utilisation and response time.

Local windows, sparse patterns, low-rank approximations and sequence compression address this cost by restricting or restructuring attention. Each introduces a trade-off: some discard direct global interactions, some approximate the full operation, and some need specialised kernels to turn lower theoretical complexity into higher throughput. The right choice depends on the workload and hardware, not on complexity notation alone.

Autoregressive inference presents a second, distinct limit. Training can score all known target positions in parallel under a causal mask. Generation cannot: after producing a token, the model must feed it back before computing the next one. Caching past keys and values avoids recomputing the whole prefix, but it does not remove the dependency between output steps. Long answers incur a serial chain of model evaluations, while each step still attends over a growing prefix.

A deployment budget must consequently separate input cost from output latency. Long prompts stress attention memory and prefill computation; long responses add sequential decoding time. Batching can improve device utilisation across requests, but cannot make one response’s tokens independent of one another.

Choose a backbone by its transfer and serving economics

Transformers became central to foundation models because their computational structure and the pretrain–transfer recipe reinforce one another. Attention offers direct contextual mixing and parallel training; self-supervised objectives turn unlabelled data into reusable representations; small adaptation layers let many applications share a backbone. BERT demonstrated that arrangement in language understanding, and ViT showed that it could cross modalities.

The useful engineering question is not whether Transformers have replaced every alternative. It is where a shared pretrained representation repays its training and inference costs. Encoder, decoder and encoder–decoder variants suit different outputs. Data volume, context length, patch resolution and required response length alter the budget. Domain-specific architectures remain rational when their inductive biases or serving profiles fit better.

Default status, within foundation models, therefore means a reusable starting point rather than a universal optimum. Adopt the backbone for its transfer properties, but design the system around the costs that attention and sequential generation retain.

Sources and further reading