Softmax Recurrent Unit: A new type of RNN cell

L. (Lucas) Vos, T.M. van Laarhoven

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

Abstract

Recurrent Neural Networks (RNNs) have been very successful in many state-of-the-art solutions for natural language tasks like machine translation. However, LSTM, the most common RNN cell, is complex and utilizes a lot of components. We present the Softmax Recurrent Unit (SMRU), a novel and elegant design of a new type of RNN cell. The SMRU has a simple structure, which is solely based around the softmax function. We present four different variants of the SMRU and compare them to both the LSTM and GRU on various tasks and datasets. These experiments show that the SMRU achieves competitive performance, surpassing either the LSTM or the GRU on any the given task, while having a much simpler design.
Original languageEnglish
Title of host publicationESANN 2020 proceedings
Subtitle of host publication28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning
Place of PublicationBruges, Belgium
PublisherEuropean Symposium on Artificial Neural Networks
Pages309-314
Number of pages6
ISBN (Print)978-2-87587-074-2
Publication statusPublished - Oct 2020
EventThe 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning - Online, Bruges, Belgium
Duration: 2 Oct 20204 Oct 2020
https://www.esann.org/

Symposium

SymposiumThe 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning
Abbreviated titleESANN 2020
Country/TerritoryBelgium
CityBruges
Period2/10/204/10/20
Internet address

Fingerprint

Dive into the research topics of 'Softmax Recurrent Unit: A new type of RNN cell'. Together they form a unique fingerprint.

Cite this