Softmax Recurrent Unit: A new type of RNN cell

L. (Lucas) Vos; T.M. van Laarhoven

Softmax Recurrent Unit: A new type of RNN cell

L. (Lucas) Vos
, T.M. van Laarhoven

Research output: Chapter in Book/Report/Conference proceeding › Conference Article in proceeding › Academic › peer-review

Abstract

Recurrent Neural Networks (RNNs) have been very successful in many state-of-the-art solutions for natural language tasks like machine translation. However, LSTM, the most common RNN cell, is complex and utilizes a lot of components. We present the Softmax Recurrent Unit (SMRU), a novel and elegant design of a new type of RNN cell. The SMRU has a simple structure, which is solely based around the softmax function. We present four different variants of the SMRU and compare them to both the LSTM and GRU on various tasks and datasets. These experiments show that the SMRU achieves competitive performance, surpassing either the LSTM or the GRU on any the given task, while having a much simpler design.

Original language	English
Title of host publication	ESANN 2020 proceedings
Subtitle of host publication	28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning
Place of Publication	Bruges, Belgium
Publisher	European Symposium on Artificial Neural Networks
Pages	309-314
Number of pages	6
ISBN (Print)	978-2-87587-074-2
Publication status	Published - Oct 2020
Event	The 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning - Online, Bruges, Belgium Duration: 2 Oct 2020 → 4 Oct 2020 https://www.esann.org/

Symposium

Symposium	The 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning
Abbreviated title	ESANN 2020
Country/Territory	Belgium
City	Bruges
Period	2/10/20 → 4/10/20
Internet address	https://www.esann.org/

Access to Document

https://www.esann.org/sites/default/files/proceedings/2020/ES2020-187.pdf

Cite this

@inproceedings{061a679a757a41338e75256fb87081e1,

title = "Softmax Recurrent Unit: A new type of RNN cell",

abstract = "Recurrent Neural Networks (RNNs) have been very successful in many state-of-the-art solutions for natural language tasks like machine translation. However, LSTM, the most common RNN cell, is complex and utilizes a lot of components. We present the Softmax Recurrent Unit (SMRU), a novel and elegant design of a new type of RNN cell. The SMRU has a simple structure, which is solely based around the softmax function. We present four different variants of the SMRU and compare them to both the LSTM and GRU on various tasks and datasets. These experiments show that the SMRU achieves competitive performance, surpassing either the LSTM or the GRU on any the given task, while having a much simpler design.",

author = "Vos, \{L. (Lucas)\} and \{van Laarhoven\}, T.M.",

year = "2020",

month = oct,

language = "English",

isbn = "978-2-87587-074-2",

pages = "309--314",

booktitle = "ESANN 2020 proceedings",

publisher = "European Symposium on Artificial Neural Networks",

address = "Belgium",

note = "The 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2020 ; Conference date: 02-10-2020 Through 04-10-2020",

url = "https://www.esann.org/",

}

Vos, L & van Laarhoven, TM 2020, Softmax Recurrent Unit: A new type of RNN cell. in ESANN 2020 proceedings: 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. European Symposium on Artificial Neural Networks, Bruges, Belgium, pp. 309-314, The 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium, 2/10/20. <https://www.esann.org/sites/default/files/proceedings/2020/ES2020-187.pdf>

Softmax Recurrent Unit: A new type of RNN cell. / Vos, L. (Lucas); van Laarhoven, T.M.
ESANN 2020 proceedings: 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Bruges, Belgium: European Symposium on Artificial Neural Networks, 2020. p. 309-314.

Research output: Chapter in Book/Report/Conference proceeding › Conference Article in proceeding › Academic › peer-review

TY - GEN

T1 - Softmax Recurrent Unit

T2 - The 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning

AU - Vos, L. (Lucas)

AU - van Laarhoven, T.M.

PY - 2020/10

Y1 - 2020/10

N2 - Recurrent Neural Networks (RNNs) have been very successful in many state-of-the-art solutions for natural language tasks like machine translation. However, LSTM, the most common RNN cell, is complex and utilizes a lot of components. We present the Softmax Recurrent Unit (SMRU), a novel and elegant design of a new type of RNN cell. The SMRU has a simple structure, which is solely based around the softmax function. We present four different variants of the SMRU and compare them to both the LSTM and GRU on various tasks and datasets. These experiments show that the SMRU achieves competitive performance, surpassing either the LSTM or the GRU on any the given task, while having a much simpler design.

AB - Recurrent Neural Networks (RNNs) have been very successful in many state-of-the-art solutions for natural language tasks like machine translation. However, LSTM, the most common RNN cell, is complex and utilizes a lot of components. We present the Softmax Recurrent Unit (SMRU), a novel and elegant design of a new type of RNN cell. The SMRU has a simple structure, which is solely based around the softmax function. We present four different variants of the SMRU and compare them to both the LSTM and GRU on various tasks and datasets. These experiments show that the SMRU achieves competitive performance, surpassing either the LSTM or the GRU on any the given task, while having a much simpler design.

UR - https://www.i6doc.com/en/book/?gcoi=28001100609140

M3 - Conference Article in proceeding

SN - 978-2-87587-074-2

SP - 309

EP - 314

BT - ESANN 2020 proceedings

PB - European Symposium on Artificial Neural Networks

CY - Bruges, Belgium

Y2 - 2 October 2020 through 4 October 2020

ER -

Softmax Recurrent Unit: A new type of RNN cell

Abstract

Symposium

Access to Document

Other files and links

Fingerprint

Cite this