Fingerprint Surface-Based Detection of Web Bot Detectors

Hugo Jonker*, Benjamin Krumnow, Gabry Vlot

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingChapterAcademicpeer-review

Abstract

Web bots are used to automate client interactions with websites, which facilitates large-scale web measurements. However, websites may employ web bot detection. When they do, their response to a bot may differ from responses to regular browsers. The discrimination can result in deviating content, restriction of resources or even the exclusion of a bot from a website. This places strict restrictions upon studies: the more bot detection takes place, the more results must be manually verified to confirm the bot’s findings.
To investigate the extent to which bot detection occurs, we reverse-analysed commercial bot detection. We found that in part, bot detection relies on the values of browser properties and the presence of certain objects in the browser’s DOM model. This part strongly resembles browser fingerprinting. We leveraged this for a generic approach to detect web bot detection: we identify what part of the browser fingerprint of a web bot uniquely identifies it as a web bot by contrasting its fingerprint with those of regular browsers. This leads to the fingerprint surface of a web bot. Any website accessing the fingerprint surface is then accessing a part unique to bots, and thus engaging in bot detection.
We provide a characterisation of the fingerprint surface of 14 web bots. We show that the vast majority of these frameworks are uniquely identifiable through well-known fingerprinting techniques. We design a scanner to detect web bot detection based on the reverse analysis, augmented with the found fingerprint surfaces. In a scan of the Alexa Top 1 Million, we find that 12.8% of websites show indications of web bot detection.
Original languageEnglish
Title of host publicationComputer Security - ESORICS 2019
Subtitle of host publication24th European Symposium on Research in Computer Security, Luxembourg, September 23-27, 2019, Proceedings, Part II
EditorsKazue Sako, Steve Schneider, Peter Y.A. Ryan
Place of PublicationCham
PublisherSpringer
Chapter28
Pages586-605
Number of pages20
ISBN (Electronic)9783030299620
ISBN (Print)9783030299613
DOIs
Publication statusPublished - 15 Sep 2019

Publication series

SeriesLecture Notes in Computer Science (LNCS)
Volume11736
ISSN0302-9743

Fingerprint

Websites
World Wide Web
Detectors

Cite this

Jonker, H., Krumnow, B., & Vlot, G. (2019). Fingerprint Surface-Based Detection of Web Bot Detectors. In K. Sako, S. Schneider, & P. Y. A. Ryan (Eds.), Computer Security - ESORICS 2019: 24th European Symposium on Research in Computer Security, Luxembourg, September 23-27, 2019, Proceedings, Part II (pp. 586-605). Cham: Springer. Lecture Notes in Computer Science (LNCS), Vol.. 11736 https://doi.org/10.1007/978-3-030-29962-0_28
Jonker, Hugo ; Krumnow, Benjamin ; Vlot, Gabry. / Fingerprint Surface-Based Detection of Web Bot Detectors. Computer Security - ESORICS 2019: 24th European Symposium on Research in Computer Security, Luxembourg, September 23-27, 2019, Proceedings, Part II. editor / Kazue Sako ; Steve Schneider ; Peter Y.A. Ryan. Cham : Springer, 2019. pp. 586-605 (Lecture Notes in Computer Science (LNCS), Vol. 11736).
@inbook{7e71fdd136c34d36a10f8ff5729974b3,
title = "Fingerprint Surface-Based Detection of Web Bot Detectors",
abstract = "Web bots are used to automate client interactions with websites, which facilitates large-scale web measurements. However, websites may employ web bot detection. When they do, their response to a bot may differ from responses to regular browsers. The discrimination can result in deviating content, restriction of resources or even the exclusion of a bot from a website. This places strict restrictions upon studies: the more bot detection takes place, the more results must be manually verified to confirm the bot’s findings.To investigate the extent to which bot detection occurs, we reverse-analysed commercial bot detection. We found that in part, bot detection relies on the values of browser properties and the presence of certain objects in the browser’s DOM model. This part strongly resembles browser fingerprinting. We leveraged this for a generic approach to detect web bot detection: we identify what part of the browser fingerprint of a web bot uniquely identifies it as a web bot by contrasting its fingerprint with those of regular browsers. This leads to the fingerprint surface of a web bot. Any website accessing the fingerprint surface is then accessing a part unique to bots, and thus engaging in bot detection.We provide a characterisation of the fingerprint surface of 14 web bots. We show that the vast majority of these frameworks are uniquely identifiable through well-known fingerprinting techniques. We design a scanner to detect web bot detection based on the reverse analysis, augmented with the found fingerprint surfaces. In a scan of the Alexa Top 1 Million, we find that 12.8{\%} of websites show indications of web bot detection.",
author = "Hugo Jonker and Benjamin Krumnow and Gabry Vlot",
year = "2019",
month = "9",
day = "15",
doi = "10.1007/978-3-030-29962-0_28",
language = "English",
isbn = "9783030299613",
series = "Lecture Notes in Computer Science (LNCS)",
publisher = "Springer",
pages = "586--605",
editor = "Kazue Sako and Schneider, {Steve } and Ryan, {Peter Y.A.}",
booktitle = "Computer Security - ESORICS 2019",

}

Jonker, H, Krumnow, B & Vlot, G 2019, Fingerprint Surface-Based Detection of Web Bot Detectors. in K Sako, S Schneider & PYA Ryan (eds), Computer Security - ESORICS 2019: 24th European Symposium on Research in Computer Security, Luxembourg, September 23-27, 2019, Proceedings, Part II. Springer, Cham, Lecture Notes in Computer Science (LNCS), vol. 11736, pp. 586-605. https://doi.org/10.1007/978-3-030-29962-0_28

Fingerprint Surface-Based Detection of Web Bot Detectors. / Jonker, Hugo; Krumnow, Benjamin; Vlot, Gabry.

Computer Security - ESORICS 2019: 24th European Symposium on Research in Computer Security, Luxembourg, September 23-27, 2019, Proceedings, Part II. ed. / Kazue Sako; Steve Schneider; Peter Y.A. Ryan. Cham : Springer, 2019. p. 586-605 (Lecture Notes in Computer Science (LNCS), Vol. 11736).

Research output: Chapter in Book/Report/Conference proceedingChapterAcademicpeer-review

TY - CHAP

T1 - Fingerprint Surface-Based Detection of Web Bot Detectors

AU - Jonker, Hugo

AU - Krumnow, Benjamin

AU - Vlot, Gabry

PY - 2019/9/15

Y1 - 2019/9/15

N2 - Web bots are used to automate client interactions with websites, which facilitates large-scale web measurements. However, websites may employ web bot detection. When they do, their response to a bot may differ from responses to regular browsers. The discrimination can result in deviating content, restriction of resources or even the exclusion of a bot from a website. This places strict restrictions upon studies: the more bot detection takes place, the more results must be manually verified to confirm the bot’s findings.To investigate the extent to which bot detection occurs, we reverse-analysed commercial bot detection. We found that in part, bot detection relies on the values of browser properties and the presence of certain objects in the browser’s DOM model. This part strongly resembles browser fingerprinting. We leveraged this for a generic approach to detect web bot detection: we identify what part of the browser fingerprint of a web bot uniquely identifies it as a web bot by contrasting its fingerprint with those of regular browsers. This leads to the fingerprint surface of a web bot. Any website accessing the fingerprint surface is then accessing a part unique to bots, and thus engaging in bot detection.We provide a characterisation of the fingerprint surface of 14 web bots. We show that the vast majority of these frameworks are uniquely identifiable through well-known fingerprinting techniques. We design a scanner to detect web bot detection based on the reverse analysis, augmented with the found fingerprint surfaces. In a scan of the Alexa Top 1 Million, we find that 12.8% of websites show indications of web bot detection.

AB - Web bots are used to automate client interactions with websites, which facilitates large-scale web measurements. However, websites may employ web bot detection. When they do, their response to a bot may differ from responses to regular browsers. The discrimination can result in deviating content, restriction of resources or even the exclusion of a bot from a website. This places strict restrictions upon studies: the more bot detection takes place, the more results must be manually verified to confirm the bot’s findings.To investigate the extent to which bot detection occurs, we reverse-analysed commercial bot detection. We found that in part, bot detection relies on the values of browser properties and the presence of certain objects in the browser’s DOM model. This part strongly resembles browser fingerprinting. We leveraged this for a generic approach to detect web bot detection: we identify what part of the browser fingerprint of a web bot uniquely identifies it as a web bot by contrasting its fingerprint with those of regular browsers. This leads to the fingerprint surface of a web bot. Any website accessing the fingerprint surface is then accessing a part unique to bots, and thus engaging in bot detection.We provide a characterisation of the fingerprint surface of 14 web bots. We show that the vast majority of these frameworks are uniquely identifiable through well-known fingerprinting techniques. We design a scanner to detect web bot detection based on the reverse analysis, augmented with the found fingerprint surfaces. In a scan of the Alexa Top 1 Million, we find that 12.8% of websites show indications of web bot detection.

U2 - 10.1007/978-3-030-29962-0_28

DO - 10.1007/978-3-030-29962-0_28

M3 - Chapter

SN - 9783030299613

T3 - Lecture Notes in Computer Science (LNCS)

SP - 586

EP - 605

BT - Computer Security - ESORICS 2019

A2 - Sako, Kazue

A2 - Schneider, Steve

A2 - Ryan, Peter Y.A.

PB - Springer

CY - Cham

ER -

Jonker H, Krumnow B, Vlot G. Fingerprint Surface-Based Detection of Web Bot Detectors. In Sako K, Schneider S, Ryan PYA, editors, Computer Security - ESORICS 2019: 24th European Symposium on Research in Computer Security, Luxembourg, September 23-27, 2019, Proceedings, Part II. Cham: Springer. 2019. p. 586-605. (Lecture Notes in Computer Science (LNCS), Vol. 11736). https://doi.org/10.1007/978-3-030-29962-0_28