Learning efficient visual embedding models under data constraints

Abstract

Deep learning models require large-scale datasets to learn rich sets of low and mid-level patterns and high-level semantics. Therefore, given a high-capacity neural network, one way to improve the performance of a model is increasing the size of the dataset which the model is trained over on. Considering that it is easy to get the amount of computational power required to train a network, data becomes a serious bottleneck in scaling up the existing machine learning pipelines. In this thesis, we look into two main data bottlenecks that rise in computer vision applications: I. the difficulty of finding training data for diverse sets of object categories, II. the complication of utilizing data containing sensitive user information for the purpose of training neural network models. To address these issues, we study zero-shot learning and decentralized learning schemes, respectively. Zero-shot learning (ZSL) is one of the most promising problems where substantial progress can potentially be achieved through unsupervised learning, due to distributional differences between supervised and zero-shot classes. For this reason, several works investigate the incorporation of discriminative domain adaptation techniques into ZSL, which, however, lead to modest improvements in ZSL accuracy. In contrast, we propose a generative model that can naturally learn from unsupervised examples, and synthesize training examples for unseen classes purely based on their class embeddings, and therefore, reduce the zero-shot learning problem into a supervised classification task. The proposed approach consists of two important components: I. a conditional Generative Adversarial Network that learns to produce samples that mimic the characteristics of unsupervised data examples, and II. the Gradient Matching (GM) loss that measures the quality of the gradient signal obtained from the synthesized examples. Using our GM loss formulation, we enforce the generator to produce examples from which accurate classifiers can be trained. Experimental results on several ZSL benchmark datasets show that our approach leads to significant improvements over the state of the art in generalized zero-shot classification. Collaborative learning techniques provide a privacy-preserving solution, by enabling training over a number of private datasets that are not shared by their owners. However, recently, it has been shown that the existing collaborative learning frameworks are vulnerable to an active adversary that runs a generative adversarial network (GAN) attack. In this work, we propose a novel classification model that is resilient against such attacks by design. More specifically, we introduce a key-based classification model and a principled training scheme that protects class scores by using class-specific private keys, which effectively hides the information necessary for a GAN attack. We additionally show how to utilize high dimensional keys to improve the robustness against attacks without increasing the model complexity. Our detailed experiments demonstrate the effectiveness of the proposed technique.