Data

Tools

Indexes

Applications

Experiments

Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub topics: stable-llm-pretraining

bluorion-com/ZClip

Official implementation of the paper: "ZClip: Adaptive Spike Mitigation for LLM Pre-Training".

Language: Python - Size: 500 KB - Last synced at: 14 days ago - Pushed at: 14 days ago - Stars: 125 - Forks: 8

Related Keywords
adaptive-clipping 1 adaptive-gradient-clipping 1 enable-high-learning-rate 1 gradient-clipping 1 gradient-norm-clipping 1 llm-stable-training 1 loss 1 loss-spike 1 pre-training-stability 1 stable-llm-pretraining 1 stable-training 1 traning-stability 1 zclip 1