The Unreasonable Effectiveness of the Forget Gate with Jos van der Westhuizen

Today we're joined by Jos Van Der Westhuizen, PhD student in Engineering at Cambridge University. Jos' research focuses on applying LSTMs, or Long Short-Term Memory neural networks, to biological data for various tasks. In our conversation, we discuss his paper "The unreasonable effectiveness of the forget gate," in which he explores the various "gates" that make up an LSTM module and the general impact of getting rid of gates on the computational intensity of training the networks. Jos eventually determines that leaving only the forget-gate results in an unreasonably effective network, and we discuss why. Jos also gives us some great LSTM related resources, including references to Jurgen Schmidhuber, whose research group invented the LSTM, and who I spoke to back in Talk #44.