Dataconomy
  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
Subscribe
No Result
View All Result
  • AI
  • Tech
  • Cybersecurity
  • Finance
  • DeFi & Blockchain
  • Startups
  • Gaming
Dataconomy
  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
Subscribe
No Result
View All Result
Dataconomy
No Result
View All Result

NVIDIA’s nGPT model cuts AI training time by 20x

byKerem Gülen
October 21, 2024
in Artificial Intelligence

NVIDIA has unveiled a major advancement in AI model training with the launch of the Normalized Transformer (nGPT). This new architecture, designed to enhance the training process for large language models (LLMs), has the potential to speed up training times by 4 to 20 times, all while maintaining model stability and accuracy. The nGPT model streamlines the training process, using fewer resources and offering a more efficient solution to AI development.

What makes nGPT different: Hyperspherical learning

At the core of nGPT’s efficiency is a concept called hyperspherical representation learning. In traditional transformer models, data is often processed without a consistent geometric framework. NVIDIA’s nGPT changes this by mapping all key components—such as embeddings, attention matrices, and hidden states—onto the surface of a hypersphere. This geometric setup helps ensure that all layers of the model remain balanced during training, creating a more stable and efficient learning process.

This approach reduces the number of training steps significantly. Rather than applying weight decay directly to model weights like previous models, nGPT relies on learned scaling parameters, which optimize how the model adjusts during training. Importantly, this method eliminates the need for other normalization techniques like LayerNorm or RMSNorm, making the process both simpler and faster.

Stay Ahead of the Curve!

Don't miss out on the latest insights, trends, and analysis in the world of data, technology, and startups. Subscribe to our newsletter and get exclusive content delivered straight to your inbox.

NVIDIA’s nGPT model cuts AI training time by 20x
NVIDIA’s nGPT model cuts AI training time by 20x (Image credit)

Faster training with fewer resources

The results of nGPT’s architecture are clear. In tests conducted using the OpenWebText dataset, NVIDIA’s nGPT consistently outperformed traditional GPT models in terms of both speed and efficiency. With text inputs as long as 4,000 tokens, nGPT required far fewer training rounds to achieve similar validation loss, drastically cutting down the time it takes to train these complex models.

Additionally, nGPT’s hyperspherical structure provides better embedding separability. This means the model can more easily distinguish between different inputs, leading to improved accuracy during standard AI tests. The improved generalization of the model also enables it to perform better on tasks beyond its initial training, speeding up convergence while maintaining high levels of precision.

NVIDIA’s nGPT model cuts AI training time by 20x
NVIDIA’s nGPT model cuts AI training time by 20x (Image credit)

Why this matters for AI training

A key advantage of nGPT is its ability to combine both normalization and representation learning into one unified framework. This design simplifies the model’s architecture, making it easier to scale and adapt for more complex hybrid systems. This could potentially lead to the development of even more powerful AI systems in the future, as nGPT’s approach could be integrated into other types of models and architectures.


Featured image credit: Kerem Gülen/Ideogram

Tags: AIFeaturedNvidia

Related Posts

OpenAI Sora 2 requires opt-in for Nintendo, Pokémon content

OpenAI Sora 2 requires opt-in for Nintendo, Pokémon content

October 6, 2025
OpenAI Agent Builder debuts with drag-and-drop workflows

OpenAI Agent Builder debuts with drag-and-drop workflows

October 6, 2025
OpenAI introduces GPT-5 Codex for advanced developer workflows

OpenAI introduces GPT-5 Codex for advanced developer workflows

October 6, 2025
Eufy offered customers  per video to train its theft detection AI

Eufy offered customers $2 per video to train its theft detection AI

October 6, 2025
ChatGPT reportedly reduces reliance on Reddit as a data source

ChatGPT reportedly reduces reliance on Reddit as a data source

October 3, 2025
Light-powered chip makes AI computation 100 times more efficient

Light-powered chip makes AI computation 100 times more efficient

October 3, 2025

LATEST NEWS

Leaked real-world photos show Huawei Mate 80 series design

Snapchat Memories limits free storage to 5 GB

OpenAI Sora 2 requires opt-in for Nintendo, Pokémon content

OpenAI and Jony Ive developing new AI device, report details challenges

Microsoft warns of critical vulnerability as Windows 10 support deadline nears

OpenAI Agent Builder debuts with drag-and-drop workflows

Dataconomy

COPYRIGHT © DATACONOMY MEDIA GMBH, ALL RIGHTS RESERVED.

  • About
  • Imprint
  • Contact
  • Legal & Privacy

Follow Us

  • News
    • Artificial Intelligence
    • Cybersecurity
    • DeFi & Blockchain
    • Finance
    • Gaming
    • Startups
    • Tech
  • Industry
  • Research
  • Resources
    • Articles
    • Guides
    • Case Studies
    • Glossary
    • Whitepapers
  • Newsletter
  • + More
    • Conversations
    • Events
    • About
      • About
      • Contact
      • Imprint
      • Legal & Privacy
      • Partner With Us
No Result
View All Result
Subscribe

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy Policy.