둔비의 공부공간

ICLR 2025 Accpeted https://openreview.net/forum?id=DmEHmZ89iB Single Teacher, Multiple Perspectives: Teacher Knowledge...Do diverse perspectives help students learn better? Multi-teacher knowledge distillation, which is a more effective technique than traditional single-teacher methods, supervises the student from...openreview.net   기존 KD와 다르게, Multi-teacher를 사용해서 더욱 다양한 관점을 student에게 전달하는 논문들이 ..

https://arxiv.org/abs/2308.10524 Dataset QuantizationState-of-the-art deep neural networks are trained with large amounts (millions or even billions) of data. The expensive computation and memory costs make it difficult to train them on limited hardware resources, especially for recent popular large languagearxiv.org ICCV 2023 Accepted.  일단, 우리가 생각하는 model quantization을 기대하고 논문을 접하면 안된다.그냥 data를..

ViT Quantization을 빠르게 보려고 하는데, 공개되어있는 github가 있길래 읽어봤다. https://github.com/HuangOwen/VVTQ GitHub - HuangOwen/VVTQ: Official PyTorch implementation of paper "Variation-aware Vision Transformer Quantization"Official PyTorch implementation of paper "Variation-aware Vision Transformer Quantization" - HuangOwen/VVTQgithub.com Accpeted는 안된 것 같다.     논문의 저자들이 하고자하는 이야기는 다음과 같다. "ViT Quantization은 불안정하며..

블로그에 자세히 정리한다고 생각하고 글을 쓰다 보니, 논문 리뷰가 아니라 번역에 가깝게 옮겨놓게 되면서[작성중] 카테고리에 비공개로 쓰다가 만 논문들이 너무 많아졌다.그냥 한번 읽고, 기억나는 내용을 중점으로 정리하는 방법으로 바꿔야겠다.   https://arxiv.org/abs/1902.08153 Learned Step Size QuantizationDeep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as..

https://openaccess.thecvf.com/content/CVPR2023/html/Wang_DaFKD_Domain-Aware_Federated_Knowledge_Distillation_CVPR_2023_paper.html CVPR 2023 Open Access RepositoryDaFKD: Domain-Aware Federated Knowledge Distillation Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, Zhigang Zeng; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 20412-2..

Federated + Pruninghttps://arxiv.org/pdf/2303.06237 AAAI 2023 Accepted *communication cost : Federated Learning에서 server와 client간의 통신에서 발생하는 overhead를 말한다. AbstractFederated Learning은 privacy-preserving distributed deep learning paradigm이다.하지만, Federated Learning은 computation과 communucation cost가 필요하기 때문에 특히 mobile같은 환경의 장비에겐 부담된다. 기존 Pruning 방법들은 client server간의 낮은 양방향 communication cost, clie..