둔비의 공부공간

https://arxiv.org/abs/2305.15975 Triplet Knowledge DistillationIn Knowledge Distillation, the teacher is generally much larger than the student, making the solution of the teacher likely to be difficult for the student to learn. To ease the mimicking difficulty, we introduce a triplet knowledge distillation mechanismarxiv.orgXijun Wang et al.   KD에서, teacher가 student보다 크기 때문에, teacher의 solution을..

보호되어 있는 글입니다.

https://arxiv.org/abs/1906.03728 The Generalization-Stability Tradeoff In Neural Network PruningPruning neural network parameters is often viewed as a means to compress models, but pruning has also been motivated by the desire to prevent overfitting. This motivation is particularly relevant given the perhaps surprising observation that a wide varietyarxiv.org(2020 neurips) 일반화 분야에서 flatness는 주로 ..

ACL 2022, Microsoft https://arxiv.org/abs/2204.06625 CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing Model ensemble is a popular approach to produce a low-variance and well-generalized model. However, it induces large memory and inference costs, which are often not affordable for real-world deployment. Existing work has resorted to sharing weights among ..

Neurips 2021, Tsinghua University https://arxiv.org/abs/2110.14430 Adversarial Neuron Pruning Purifies Backdoored Deep Models As deep neural networks (DNNs) are growing larger, their requirements for computational resources become huge, which makes outsourcing training more popular. Training in a third-party platform, however, may introduce potential risks that a malicious traine arxiv.org https..

https://arxiv.org/abs/2005.06870 Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly find the optimal network parameters and sparse network structure in a unified optimization process with trainable pruning thresholds. These thresholds can have fin arxiv.org (..