IBM researchers break up with traditional transformers in new gen AI model architecture

IBM researchers have pitched a new architecture for generative AI models that fundamentally changes transformer models, the backbone of LLMs, to reduce parameter size.

The scientists presented the paper at the International Conference on Machine Learning (ICML) that took place in Seoul, South Korea, this week. 

The paper presents a new function class for generative models and an architecture family that puts it into practice called Continued Fraction Generative Networks, or CoFrGeNets.

In a blog published on July 9, IBM said the architecture was a “practical and conceptual shift, pointing to lighter-weight generative AI models that perform competitively, and in many cases even better” than the larger models they tested against. 

IBM argues that novel model architectures represent an alternative to hill-climbing on current transformer models – which historically has meant adding more parameters with more energy and compute needed to facilitate longer training times. 

Amit Dhurandhar, a principal research scientist at IBM Research, who worked on the paper said the impetus was to try to discover new tools for building more efficient models that are compact and scalable, without sacrificing performance, per the IBM blog.