Training transformers where every layer W = V·Uᵀ from initialization reveals a corpus-determined optimal rank - looking for arXiv endorser (cs.LG) [D]

The Native Factorized Weights (NFW) experiment introduces a new approach to training transformers. By initializing every linear layer with factorized weights, represented as W = V·Uᵀ, the model can learn to optimize its rank directly from the data. This method, also referred to as 'Sliver layers', has the potential to simplify the training process and improve model efficiency.

The experiment's findings suggest that this approach can lead to a corpus-determined optimal rank, which could have significant implications for the development of more efficient and effective transformer models.

The NFW method differs from traditional approaches, which often involve training a standard transformer and then compressing it using techniques like SVD or LoRA adapters. By incorporating factorization from the outset, the model can learn to adapt to the specific requirements of the task at hand.

Further research and experimentation are needed to fully explore the potential of the NFW method and its applications in various areas of machine learning.

Source: Training transformers where every layer W = V·Uᵀ from initialization reveals a corpus-determined optimal rank - looking for arXiv endorser (cs.LG) [D]. Read the full piece at the source.