Hours:
16 hours (4 credits)

Room:

Aula Riunioni del Dipartimento di Ingegneria dell’Informazione, Via G. Caruso 16, Pisa - Ground Floor

To register to the course, click here

Short Abstract:

Knowledge transfer learning is an advanced learning paradigm that focuses on transferring the knowledge acquired by a complex model to a simpler or more specialized one. The central idea is to enable a lightweight model to mimic the behavior and decision boundaries of a more complex model, often referred to as the teacher model. This is achieved by training the student model on the soft labels or probabilistic outputs generated by the teacher, rather than on the original hard labels of the dataset. The student model not only replicates the predictive capability of the teacher but also inherits its generalization capacity, often leading to faster convergence and improved performance on related but distinct tasks. Such techniques have proven effective in scenarios involving heterogeneous information sources, semi-supervised and unsupervised learning, and distributed or federated environments where direct access to raw data is limited.

In addition to traditional transfer learning, the course introduces the emerging and complementary concept of knowledge unlearning as part of the broader field of knowledge management in artificial intelligence. Unlearning refers to the adjustment of previously acquired knowledge from a model without requiring complete retraining. It is particularly relevant in contexts where data privacy, fairness, or regulatory compliance necessitate the forgetting of specific information or biases. When integrated with transfer learning, unlearning can be viewed as a technique for selective knowledge transfer, enabling systems to retain only the most relevant and reliable information from the source model while discarding outdated or undesirable patterns.

Despite the wide adoption of transfer learning in many domains such as computer networking, decision support systems, and natural language processing, the underlying theoretical foundations of the method remain underdeveloped. Current approaches often rely heavily on empirical performance without a rigorous understanding of why knowledge transfer works effectively across different domains and architectures. This gap underscores the need for formal design techniques and mathematical frameworks capable of explaining and predicting the behavior of transfer learning systems. For these reasons, in this course, we introduce an exploration of the mathematical tools suitable for the theoretical analysis of transfer learning, with particular emphasis on spectral analysis methods. Spectral techniques enable a detailed examination of how information propagates through different layers and representations, revealing insights into the mechanisms of knowledge compression, regularization, and generalization.

Course Contents in brief:

1. Introduzione alle NN e al regime di NTK [1] [2]
2. Analisi shallow NN in regime di NTK [3] [4]
3. Analisi Deep NN in regime di NTK [5] [6] [7]
4. Metodi per l’analisi spettrale delle NN in regime NTK (Teoremi di Gershgorin, Fisher-Courant)  [8] [12]
5. Introduzione al Transfer Learning: Knowledge distillation [9] [10] [11]
6. Analisi Knowledge distillation in regime NTK [9] [10]
7. Analisi Unlearning in regime NTK [13] [14] [15]

Schedule:

From 16 April 2026

1. Day1 – [16:18]
2. Day2 – [16:18]
3. Day3 – [16:18]
4. Day4 – [16:18]
5. Day5 – [16:18]
6. Day6 – [16:18]
7. Day7 – [16:18]
8. Day8 – [16:18]