Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models Paper • 2501.12370 • Published 21 days ago • 10
Theory, Analysis, and Best Practices for Sigmoid Self-Attention Paper • 2409.04431 • Published Sep 6, 2024 • 1