Show simple item record

dc.contributor.authorWu, Tailin
dc.contributor.authorFischer, Ian
dc.contributor.authorChuang, Isaac L.
dc.contributor.authorTegmark, Max Erik
dc.date.accessioned2020-07-02T16:14:43Z
dc.date.available2020-07-02T16:14:43Z
dc.date.issued2019-09
dc.date.submitted2019-08
dc.identifier.issn1099-4300
dc.identifier.urihttps://hdl.handle.net/1721.1/126053
dc.description.abstractThe Information Bottleneck (IB) method provides an insightful and principled approach for balancing compression and prediction for representation learning. The IB objective I ( X ; Z ) - β I ( Y ; Z ) employs a Lagrange multiplier β to tune this trade-off. However, in practice, not only is β chosen empirically without theoretical guidance, there is also a lack of theoretical understanding between β , learnability, the intrinsic nature of the dataset and model capacity. In this paper, we show that if β is improperly chosen, learning cannot happen—the trivial representation P ( Z | X ) = P ( Z ) becomes the global minimum of the IB objective. We show how this can be avoided, by identifying a sharp phase transition between the unlearnable and the learnable which arises as β is varied. This phase transition defines the concept of IB-Learnability. We prove several sufficient conditions for IB-Learnability, which provides theoretical guidance for choosing a good β . We further show that IB-learnability is determined by the largest confident, typical and imbalanced subset of the examples (the conspicuous subset), and discuss its relation with model capacity. We give practical algorithms to estimate the minimum β for a given dataset. We also empirically demonstrate our theoretical conditions with analyses of synthetic datasets, MNIST and CIFAR10.en_US
dc.language.isoen
dc.publisherMDPI AGen_US
dc.relation.isversionof10.3390/e21100924en_US
dc.rightsCreative Commons Attribution 4.0 International licenseen_US
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/en_US
dc.sourceMDPIen_US
dc.titleLearnability for the Information Bottlenecken_US
dc.typeArticleen_US
dc.identifier.citationWu, Tailin; Fischer, Ian; Chuang, Isaac L.; Tegmark, Max. "Learnability for the Information Bottleneck." Entropy 21,10 (2019): 924.en_US
dc.contributor.departmentMassachusetts Institute of Technology. Department of Physicsen_US
dc.relation.journalEntropyen_US
dc.eprint.versionFinal published versionen_US
dc.type.urihttp://purl.org/eprint/type/JournalArticleen_US
eprint.statushttp://purl.org/eprint/status/PeerRevieweden_US
dc.date.updated2019-11-20T15:53:55Z
dspace.date.submission2019-11-20T15:54:02Z
mit.journal.volume21en_US
mit.journal.issue10en_US
mit.metadata.statusComplete


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record