Traditional GPT vs. looping Transformers: Key differences

For almost a decade, the story of GPT has been remarkably simple: make the Transformer bigger, train it on more data, and give it more computation. Layer after layer, the same basic architecture has powered an extraordinary progression from early GPT models to today’s frontier systems.

But what happens when adding more layers is no longer the only way to make a Transformer compute more?

A different idea is emerging: what if a Transformer could reuse its own layers, effectively looping through the same computation multiple times? Instead of building a deeper model by adding a new set of parameters for every layer, a model could increase its effective depth by running some of those layers again. The result is a subtle architectural change with potentially important consequences for how models reason, how much computation they use, and how their capabilities scale.

In this article, we’ll describe the Transformer block that sits at the heart of GPT-style models: self-attention, feed-forward networks, normalization, and residual connections. We’ll then compare the familiar traditional Transformer stack with the newer idea of a looped or recurrent-depth Transformer, where the same computational blocks can be reused.

How does a traditional Transformer architecture work?

A conventional GPT-style Transformer processes tokens through a sequence of Transformer layers, with each layer having its own learned parameters. Although the layers contain broadly the same components- self-attention, feed-forward networks, normalization, and residual connections- each has a different set of weights. Adding layers therefore increases both the model’s computational depth and its parameter count, with a corresponding increase in memory requirements. (see figure below)

What is a looping Transformer, and how does it work?

A Looping Transformer takes a different approach: instead of adding a new set of parameters for every additional layer, it uses layer reuse through parameter sharing. A smaller group of Transformer layers is executed repeatedly, and each pass uses the same learned parameters (see figure above)

This decouples computational depth from parameter count. A model can perform substantially more computation by repeatedly applying a relatively small set of layers, without storing a separate set of weights for every pass. In other words, looping trades additional computation for a smaller parameter footprint.

This is particularly interesting for local and edge AI. A looped model can potentially retain the expressive power of a deeper network while requiring significantly less memory, making it possible to run models on smaller and cheaper GPUs and, for sufficiently small models, even on CPUs.

The trade-off is straightforward: looping does not make computation free. The shared layers still run repeatedly, so inference can take longer. The architectural question, therefore, is whether reusing fewer parameters for more computation can match the capability of a larger model with many independent layers. Recent work on looped Transformers explores this parameter-efficiency trade-off precisely.

What does depth mean in a Transformer model?

In a traditional Transformer, depth is determined by the number of layers in the model. Each layer adds another transformation to the computation and has its own learned parameters, so a deeper model is also a larger and more complex model.

With a Looping Transformer, depth has a different meaning. Instead of making the model deeper by adding more layers, the same layers are executed repeatedly in a loop. The model can therefore achieve greater computational depth without increasing its parameter count proportionally. In this sense, depth comes from repeated computation rather than model complexity: a relatively small model can perform many sequential transformations simply by looping over its layers.

How do looping Transformers differ from reasoning models?

Looping happens within the model’s high-dimensional hidden space, repeatedly transforming internal representations that are largely opaque and difficult to inspect. Reasoning models such as OpenAI’s o-series and DeepSeek-R1, in contrast, extend computation through explicit reasoning tokens in a much lower-dimensional, human-readable space, making their intermediate steps more observable and easier to inspect or control. The trade-off is therefore not only computational: looping provides hidden computational depth, while explicit reasoning provides a degree of interpretability and observability. In the figure below, you can see the explicit reasoning, in red, that involves additional runs through the LLM: the reasoning loop is triggered and happens in the same chat with the user.

How do looping Transformers perform on reasoning tasks?

As usual, our approach is to provide code to show and test the ideas, rather than just describing them. For this reason, I have prepared a repository with everything you need to compare a model with a traditional transformer architecture with a model using the looping architecture. Rather than comparing large frontier models on a conventional benchmark, we wanted to see whether the idea holds up in a much more constrained environment: a language model running locally on a CPU. This is one reason we deliberately chose small language models (SLMs) for the experiment. A multi-billion-parameter model is still substantial, but small enough to run on an ordinary laptop without a dedicated GPU, making the experiment reproducible rather than requiring a specialized machine.

Which language models did we compare?

We tested two similarly sized models with different architectural approaches: Nanbeige4.2-3B, which uses a Looped Transformer, and Qwen3.5-4B, which provides a compact non-looped comparison model. The goal was not to establish a rigorous ranking between the two models; their training data and procedures differ, but to get an intuitive sense of whether the extra computation from layer reuse can compensate for fewer unique parameters.

What reasoning tasks did we test?

Task name	Task description
Ordering	Given a chain such as Alice is taller than Bob, Bob is taller than Charlie, Charlie is taller than David, identify who is shortest.
Logical implication	Given a chain such as A implies B, B implies C, C implies D, D implies E, determine what follows from A.
Spatial reasoning	Given a sequence such as red is north of blue, blue is north of green, green is north of yellow, determine the position of red relative to yellow.
Irrelevant information	Given several relationships mixed with unrelated facts, identify the answer that requires following the relevant chain while ignoring the distractors.

The test itself is intentionally tiny. We give both models a handful of short reasoning problems involving chains of relationships: ordering objects, following logical implications, and traversing simple spatial relationships (see table above). These tasks require the model to combine several pieces of information before producing an answer, rather than simply retrieving a fact from its training data. For example, instead of asking whether A implies B, we can ask whether A implies E when the model is given a chain of implications from A through B, C, and D to E. We use the same prompts and evaluation criteria for both models and run them deterministically.

Running the test is an exercise in patience, especially on an older laptop, but in the end it will print a summary with the execution results for both models:

========================================================================
SUMMARY
========================================================================
Model                             Accuracy     Avg latency
------------------------------------------------------------------------
Nanbeige4.2-3B (looped)       4/4            32.92s
Qwen3.5-4B                    4/4            26.27s

How did the looped and traditional models compare?

The result is surprisingly simple: the smaller looped model achieves results comparable to the larger non-looped model on these reasoning tasks. Qwen3.5-4B has more parameters, while Nanbeige4.2-3B relies on reusing its Transformer layers to perform additional computation. Yet, on this small experiment, that difference in parameter count does not translate into an obvious difference in the ability to solve the problems.

This is not evidence that looped Transformers are universally better but, perhaps, more interesting for developers: even on a CPU, a relatively small model using parameter sharing can perform similarly to a somewhat larger model on tasks that require multiple reasoning steps. That makes looping an interesting way of thinking about the trade-off between how many parameters a model has and how many times those parameters can be used.

Conclusion

For developers, looping architectures could make capable AI more accessible on local and edge hardware. Instead of simply choosing a larger model, developers may be able to trade parameters for computation, reducing memory requirements while maintaining capability. The practical takeaway is simple: when evaluating an SLM (but even an LLM), don’t look only at parameter count; also consider how its architecture uses depth and computation.

The post Traditional GPT vs. looping Transformers: Key differences appeared first on LogRocket Blog.