Learning to teach in multiagent reinforcement learning with teams of N > 2 Agents

Abstract

Sharing unique insights while learning how to collectively perform a task is a central function of human intelligence in groups. While navigating a complex task independently can be feasible, communicating with and studying from other agents can yield faster and better learning performance when one's peers possess new and relevant teachings (i.e. a shortcut to learning). Numerous works have demonstrated the value of incorporating these mechanisms of communication and teaching in distributed intelligence systems. However they often make assumptions about apriori knowledge of which peers are experts and their frameworks are limited to peers who when paired together have a low likelihood of producing negative transfer as a result of teaching and studying from each other. This is problematic, for it is much more often the case in the real world that agents possess a variety of knowledge, not all of which is compatible with that of other agents.

To this end, we introduce Selective Knowledge Sharing and Dropout (SKSD). Our algorithm addresses the problem of expert advising level knowledge identification in learning-to-teach using a novel knowledge assessment mechanism, which identifies expertise among peer teachers via online Bayesian estimation. The algorithm uses this assessment to inform a novel knowledge sharing mechanism which facilitates the transfer of advising level knowledge among peers using actor and critic distillation resulting in accelerated student learning and prevention of negative transfer among teachers. With aims to utilize a variety of teacher knowledge, we extend previous learning-to-teach systems to more than two peers and address the challenges associated with sharing knowledge when a minority of agents are marginalized out of the group sharing process due to the method of advice aggregation.

Our empirical results demonstrate that SKSD significantly outperforms previous methods in learning-to-teach in terms of student learning performance, even when a majority of teachers in an advising group are non-experts.