The Transformer architecture introduced in 2017 became the T in GPT, combining neural attention mechanisms with generative pre-training.

Format Web Page

Location PART 3 - THE THIRD AXIS CHAPTER 14: AGI & THE SYNTHESIZER Language, in Contrast to Cyc

"The Transformer architecture introduced in 2017 became the T in GPT, combining neural attention mechanisms with generative pre-training."

OpenAI's early GPT research explicitly combined Transformers with unsupervised pre-training to develop a task-agnostic language model. This establishes the architectural connection between Google's Transformer research and the later GPT family.

View Source

Source title Improving language understanding with unsupervised learning

Source authors Alec Radford

Source year 2018

Confidence high

Importance high