Federated learning algorithm based on knowledge distillation

Jiang, Donglin, Shan, Chen and Zhang, Zhihui (2021) Federated learning algorithm based on knowledge distillation. In: 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE).

PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Available under Licence Creative Commons Attribution.
Download (1MB) | Preview


Federated learning is a new scheme of distributed machine learning, which enables a large number of edge computing devices to jointly learn a shared model without private data sharing. Federated learning allows nodes to synchronize only the locally trained models instead of their own private data, which provides a guarantee for privacy and security. However, due to the challenges of heterogeneity in federated learning, which are: (1) heterogeneous model architecture among devices; (2) statistical heterogeneity in real federated dataset, which do not obey independent-identical-distribution, resulting in poor performance of traditional federated learning algorithms. To solve the problems above, this paper proposes FedDistill, a new distributed training method based on knowledge distillation. By introducing personalized model on each device, the personalized model aims to improve the local performance even in a situation that global model fails to adapt to the local dataset, thereby improving the ability and robustness of the global model. The improvement of the performance of local device benefits from the effect of knowledge distillation, which can guide the improvement of global model by knowledge transfer between heterogeneous networks. Experiments show that FedDistill can significantly improve the accuracy of classification tasks and meet the needs of heterogeneous users.

Item Type: Conference or Workshop Item (Paper)
Keywords: Federated learning; Knowledge distillation; Non-independent-identical-distribution; Heterogeneous network
Schools/Departments: University of Nottingham Ningbo China > Faculty of Science and Engineering > School of Computer Science
University of Nottingham Ningbo China > Faculty of Science and Engineering > School of Mathematical Sciences
Identification Number: https://doi.org/10.1109/ICAICE51518.2020.00038
Depositing User: Yu, Tiffany
Date Deposited: 07 May 2021 08:51
Last Modified: 07 May 2021 08:51
URI: https://eprints.nottingham.ac.uk/id/eprint/65181

Actions (Archive Staff Only)

Edit View Edit View