• About
  • Policies
  • What is open access
  • Library
  • Contact
Advanced search
      View Item 
      •   BUIR Home
      • Scholarly Publications
      • Faculty of Engineering
      • Department of Computer Engineering
      • View Item
      •   BUIR Home
      • Scholarly Publications
      • Faculty of Engineering
      • Department of Computer Engineering
      • View Item
      JavaScript is disabled for your browser. Some features of this site may not work without it.

      Understanding how orthogonality of parameters improves quantization of neural networks

      Thumbnail
      View / Download
      1.8 Mb
      Author(s)
      Eryılmaz, Şükrü Burç
      Dündar, Ayşegül
      Date
      2022-05-10
      Source Title
      IEEE Transactions on Neural Networks and Learning Systems
      Print ISSN
      2162-237X
      Electronic ISSN
      2162-2388
      Publisher
      IEEE
      Pages
      1 - 10
      Language
      English
      Type
      Article
      Item Usage Stats
      8
      views
      8
      downloads
      Abstract
      We analyze why the orthogonality penalty improves quantization in deep neural networks. Using results from perturbation theory as well as through extensive experiments with Resnet50, Resnet101, and VGG19 models, we mathematically and experimentally show that improved quantization accuracy resulting from orthogonality constraint stems primarily from reduced condition numbers, which is the ratio of largest to smallest singular values of weight matrices, more so than reduced spectral norms, in contrast to the explanations in previous literature. We also show that the orthogonality penalty improves quantization even in the presence of a state-of-the-art quantized retraining method. Our results show that, when the orthogonality penalty is used with quantized retraining, ImageNet Top5 accuracy loss from 4- to 8-bit quantization is reduced by up to 7% for Resnet50, and up to 10% for Resnet101, compared to quantized retraining with no orthogonality penalty.
      Keywords
      Deep neural networks
      Orthogonality regularization
      Perturbation theory
      Quantization
      Permalink
      http://hdl.handle.net/11693/111435
      Published Version (Please cite this version)
      https://www.doi.org/10.1109/TNNLS.2022.3171297
      Collections
      • Department of Computer Engineering 1561
      Show full item record

      Browse

      All of BUIRCommunities & CollectionsTitlesAuthorsAdvisorsBy Issue DateKeywordsTypeDepartmentsCoursesThis CollectionTitlesAuthorsAdvisorsBy Issue DateKeywordsTypeDepartmentsCourses

      My Account

      Login

      Statistics

      View Usage StatisticsView Google Analytics Statistics

      Bilkent University

      If you have trouble accessing this page and need to request an alternate format, contact the site administrator. Phone: (312) 290 2976
      © Bilkent University - Library IT

      Contact Us | Send Feedback | Off-Campus Access | Admin | Privacy