Accelerated learning for Restricted Boltzmann Machine with momentum term

Szymon Zareba*, Adam Gonczarek, Jakub M. Tomczak, Jerzy Światek

*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Restricted Boltzmann Machines are generative models which can be used as standalone feature extractors, or as a parameter initialization for deeper models. Typically, these models are trained using Contrastive Divergence algorithm, an approximation of the stochastic gradient descent method. In this paper, we aim at speeding up the convergence of the learning procedure by applying the momentum method and the Nesterov's accelerated gradient technique. We evaluate these two techniques empirically using the image dataset MNIST.

Original languageEnglish
Title of host publicationProgress in Systems Engineering - Proceedings of the 23rd International Conference on Systems Engineering
PublisherSpringer Verlag
Pages187-192
Number of pages6
ISBN (Print)9783319084213
DOIs
Publication statusPublished - 1 Jan 2015
Externally publishedYes
Event23rd International Conference on Systems Engineering, ICSEng 2014 - Las Vegas, NV, United States
Duration: 19 Aug 201421 Aug 2014

Publication series

NameAdvances in Intelligent Systems and Computing
Volume1089
ISSN (Print)2194-5357

Conference

Conference23rd International Conference on Systems Engineering, ICSEng 2014
Country/TerritoryUnited States
CityLas Vegas, NV
Period19/08/1421/08/14

Keywords

  • Contrastive Divergence
  • Deep learning
  • Nesterov's momentum
  • stochastic gradient descent

Fingerprint

Dive into the research topics of 'Accelerated learning for Restricted Boltzmann Machine with momentum term'. Together they form a unique fingerprint.

Cite this