IST Lunch Bunch

Any set of probabilistic functions of some input variables is a "representation" of the inputs. We introduce a principled and practical approach to unsupervised representation learning based on a novel hierarchical decomposition of information. Intuitively, we optimize representations and stack them to capture increasingly long-range multivariate dependencies. Each layer of the hierarchy provides a progressively tighter bound on the information in the data. These bounds allow us to quantify the information value of each layer and hidden unit in the representation. We demonstrate the usefulness of these representations on diverse data from human behavior, language, gene expression, and finance.