Fingerprint Surface-Based Detection of Web Bot Detectors

Hugo Jonker*, Benjamin Krumnow, Gabry Vlot

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingChapterAcademicpeer-review

Abstract

Web bots are used to automate client interactions with websites, which facilitates large-scale web measurements. However, websites may employ web bot detection. When they do, their response to a bot may differ from responses to regular browsers. The discrimination can result in deviating content, restriction of resources or even the exclusion of a bot from a website. This places strict restrictions upon studies: the more bot detection takes place, the more results must be manually verified to confirm the bot’s findings.
To investigate the extent to which bot detection occurs, we reverse-analysed commercial bot detection. We found that in part, bot detection relies on the values of browser properties and the presence of certain objects in the browser’s DOM model. This part strongly resembles browser fingerprinting. We leveraged this for a generic approach to detect web bot detection: we identify what part of the browser fingerprint of a web bot uniquely identifies it as a web bot by contrasting its fingerprint with those of regular browsers. This leads to the fingerprint surface of a web bot. Any website accessing the fingerprint surface is then accessing a part unique to bots, and thus engaging in bot detection.
We provide a characterisation of the fingerprint surface of 14 web bots. We show that the vast majority of these frameworks are uniquely identifiable through well-known fingerprinting techniques. We design a scanner to detect web bot detection based on the reverse analysis, augmented with the found fingerprint surfaces. In a scan of the Alexa Top 1 Million, we find that 12.8% of websites show indications of web bot detection.
Original languageEnglish
Title of host publicationComputer Security - ESORICS 2019
Subtitle of host publication24th European Symposium on Research in Computer Security, Luxembourg, September 23-27, 2019, Proceedings, Part II
EditorsKazue Sako, Steve Schneider, Peter Y.A. Ryan
Place of PublicationCham
PublisherSpringer
Chapter28
Pages586-605
Number of pages20
ISBN (Electronic)9783030299620
ISBN (Print)9783030299613
DOIs
Publication statusPublished - 15 Sep 2019

Publication series

SeriesLecture Notes in Computer Science (LNCS)
Volume11736
ISSN0302-9743

    Fingerprint

Cite this

Jonker, H., Krumnow, B., & Vlot, G. (2019). Fingerprint Surface-Based Detection of Web Bot Detectors. In K. Sako, S. Schneider, & P. Y. A. Ryan (Eds.), Computer Security - ESORICS 2019: 24th European Symposium on Research in Computer Security, Luxembourg, September 23-27, 2019, Proceedings, Part II (pp. 586-605). Springer. Lecture Notes in Computer Science (LNCS), Vol.. 11736 https://doi.org/10.1007/978-3-030-29962-0_28