Abstract
Recurrent Neural Networks (RNNs) have been very successful in many state-of-the-art solutions for natural language tasks like machine translation. However, LSTM, the most common RNN cell, is complex and utilizes a lot of components. We present the Softmax Recurrent Unit (SMRU), a novel and elegant design of a new type of RNN cell. The SMRU has a simple structure, which is solely based around the softmax function. We present four different variants of the SMRU and compare them to both the LSTM and GRU on various tasks and datasets. These experiments show that the SMRU achieves competitive performance, surpassing either the LSTM or the GRU on any the given task, while having a much simpler design.
| Original language | English |
|---|---|
| Title of host publication | ESANN 2020 proceedings |
| Subtitle of host publication | 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning |
| Place of Publication | Bruges, Belgium |
| Publisher | European Symposium on Artificial Neural Networks |
| Pages | 309-314 |
| Number of pages | 6 |
| ISBN (Print) | 978-2-87587-074-2 |
| Publication status | Published - Oct 2020 |
| Event | The 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning - Online, Bruges, Belgium Duration: 2 Oct 2020 → 4 Oct 2020 https://www.esann.org/ |
Symposium
| Symposium | The 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning |
|---|---|
| Abbreviated title | ESANN 2020 |
| Country/Territory | Belgium |
| City | Bruges |
| Period | 2/10/20 → 4/10/20 |
| Internet address |