Only practical knowledge or knowing the algorithm? Notions and necessities of explainable artificial intelligence in long-term care

The number of older adults living independently at home is expanding, which is often said to bring the need for more technological assistance. Dutch policy aims to allow older adults to remain living at home as long as possible. In such policies, the use of technologies supports older adults to perform daily practices. Artificial intelligence (AI), as part of these technologies, has the potential to improve personalized care and ageing in place, both at home and in residential care settings. However, the internal machineries of AI systems often remain hidden as a black box for the users, which can include caregivers or older adults. Interest in explainable AI (XAI) originates from this ‘black boxing’, as a technique to assist users in understanding the underlying logic of the decision-making process, and in identifying mistakes, transforming the opaque black box into an interpretable twin ‘white box’. Current research is mostly located in the technical domains, and it remains unknown how various stakeholders see XAI. To fill this gap, we conducted 21 scenario-based interviews with professionals to investigated XAI in three long-term care contexts: company, care management and care provision. We draw on the theory of the co-constitution of ageing and AI to reach our aim of understanding ‘what is XAI’ in the different contexts, and the enactments of XAI in their practices. Participants express different notions and necessities of XAI, varying from knowing algorithms towards practical understanding. The needed level of explainability is divers in the different contexts of care. As a follow-up, we recommend to include older adults and perform research into the enactment of XAI in practice, and the form or degree of XAI needed and for whom.