둔비의 공부공간

https://arxiv.org/abs/2102.02887 Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training In this paper, we introduce a new perspective on training deep neural networks capable of state-of-the-art performance without the need for the expensive over-parameterization by proposing the concept of In-Time Over-Parameterization (ITOP) in sparse train arxiv.org ..

https://arxiv.org/abs/1911.11134 Rigging the Lottery: Making All Tickets Winners Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to arxiv.org 2020년 ICLR paper (Utku Evci et al.) Google Brain, ..

보호되어 있는 글입니다.

ICLR 2022 https://arxiv.org/abs/2106.14568 Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity The success of deep ensembles on improving predictive performance, uncertainty estimation, and out-of-distribution robustness has been extensively studied in the machine learning literature. Albeit the promising results, naively training multiple..

DSD 논문을 인용한 후속논문들에 대한 간단한 요약 Monarch: Expressive Structured Matrices for Efficient and Accurate Training 기존 compute/memory를 줄이는 방법들은 여러 문제가 있었다. (방법들) replace dense weight matrices with structured ones, such as sparse & low-rank matrices and the Fourier transform. 비효율적인 efficiency quality trade-off dense-to-sparse fine-tuning 할때, dense weight matrix의 approximate를 다루기 쉬운 알고리즘의 부족 Monarch라는 hardwa..

Neurips 2021 https://arxiv.org/abs/2106.12379 AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks The increasing computational requirements of deep neural networks (DNNs) have led to significant interest in obtaining DNN models that are sparse, yet accurate. Recent work has investigated the even harder case of sparse training, where the DNN weights are arxiv.org Network의 ..