Gradient-Descent for Randomized Controllers Under Partial Observability

Linus Heck, Jip Spel, Sebastian Junges, Joshua Moerman, Joost-Pieter Katoen

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

Abstract

Randomization is a powerful technique to create robust controllers, in particular in partially observable settings. The degrees of randomization have a significant impact on the system performance, yet they are intricate to get right. The use of synthesis algorithms for parametric Markov chains (pMCs) is a promising direction to support the design process of such controllers. This paper shows how to define and evaluate gradients of pMCs. Furthermore, it investigates varieties of gradient descent techniques from the machine learning community to synthesize the probabilities in a pMC. The resulting method scales to significantly larger pMCs than before and empirically outperforms the state-of-the-art, often by at least one order of magnitude.
Original languageEnglish
Title of host publicationVerification, model Checking, and Abstract Interpretation
Subtitle of host publication23rd International Conference, VMCAI 2022 Philadelphia, PA, USA, January 16–18, 2022 Proceedings
EditorsBernd Finkbeiner, Thomas Wies
PublisherSpringer
Pages127-150
Number of pages24
Edition1
ISBN (Electronic)978-3-030-94583-1
ISBN (Print) 978-3-030-94582-4, 978-3-030-94584-8
DOIs
Publication statusPublished - 2022
EventThe 23rd international conference Verification, Model Checking, and Abstract Interpretation - Philadelphia, United States
Duration: 16 Jan 202218 Jan 2022
Conference number: 23
https://popl22.sigplan.org/home/VMCAI-2022
https://link.springer.com/book/10.1007/978-3-030-94583-1

Publication series

SeriesLecture Notes in Computer Science
Volume13182
ISSN0302-9743
SeriesTheoretical Computer Science and General Issues (LNCS subseries)
Volume13182

Conference

ConferenceThe 23rd international conference Verification, Model Checking, and Abstract Interpretation
Abbreviated titleVMCAI 2022
Country/TerritoryUnited States
CityPhiladelphia
Period16/01/2218/01/22
Internet address

Fingerprint

Dive into the research topics of 'Gradient-Descent for Randomized Controllers Under Partial Observability'. Together they form a unique fingerprint.

Cite this