How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability

Benjamin Krumnow; H.L. Jonker; Stefan Karsch

doi:10.1145/3555050.3569131

How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability

Benjamin Krumnow, H.L. Jonker, Stefan Karsch

Research output: Chapter in Book/Report/Conference proceeding › Conference Article in proceeding › Academic › peer-review

Abstract

Automated browsers are widely used to study the web at scale. Their premise is that they measure what regular browsers would encounter on the web. In practice, deviations due to detection of automation have been found. To what extent automated browsers can be improved to reduce such deviations has so far not been investigated in detail. In this paper, we investigate this for a specific web automation framework: OpenWPM, a popular research framework specifically designed to study web privacy. We analyse (1) detectability of OpenWPM, (2) resilience of OpenWPM's data recording, and (3) prevalence of OpenWPM detection. Our analysis (1) reveals OpenWPM is easily detectable. Our investigation of OpenWPM's data recording integrity (2) identifies novel evasion techniques and previously unknown attacks against OpenWPM's instrumentation. We investigate and develop mitigations to address the identified issues. Finally, in a scan of 100,000 sites (3), we observe that OpenWPM is commonly detected (∼14% of front pages). Moreover, we discover integrated routines in scripts specifically to detect OpenWPM clients. In conclusion, our case study shows that even the most popular web measurement framework, OpenWPM, is more gullible than expected, and this gullibility is rarely accounted for in studies.

Original language	English
Title of host publication	CoNEXT '22: Proceedings of the 18th International Conference on emerging Networking EXperiments and Technologies
Editors	Giuseppe Bianchi, Alessandro Mei
Place of Publication	New York
Publisher	Association for Computing Machinery (ACM)
Pages	171-186
Number of pages	16
ISBN (Electronic)	9781450395083
DOIs	https://doi.org/10.1145/3555050.3569131
Publication status	Published - 30 Nov 2022
Event	18th International Conference on emerging Networking EXperiments and Technologies - Roma, Italy Duration: 6 Dec 2022 → 9 Dec 2022 Conference number: 18 https://conferences2.sigcomm.org/co-next/2022/#!/home

Conference

Conference	18th International Conference on emerging Networking EXperiments and Technologies
Abbreviated title	CoNEXT '22
Country/Territory	Italy
City	Roma
Period	6/12/22 → 9/12/22
Internet address	https://conferences2.sigcomm.org/co-next/2022/#!/home

Keywords

bot detection
privacy
reliability
security
web bots
web measurements

Access to Document

10.1145/3555050.3569131

WPM vs. WPM_hide
Krumnow, B. (Creator), Harvard Dataverse, 27 Oct 2022
DOI: 10.7910/dvn/remwxs, https://dataverse.harvard.edu/citation?persistentId=doi:10.7910/DVN/REMWXS
Dataset

Cite this

Krumnow, B., Jonker, H. L., & Karsch, S. (2022). How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability. In Giuseppe Bianchi, & Alessandro Mei (Eds.), CoNEXT '22: Proceedings of the 18th International Conference on emerging Networking EXperiments and Technologies (pp. 171-186). Association for Computing Machinery (ACM). https://doi.org/10.1145/3555050.3569131

Krumnow, Benjamin ; Jonker, H.L. ; Karsch, Stefan. / How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability. CoNEXT '22: Proceedings of the 18th International Conference on emerging Networking EXperiments and Technologies. editor / Giuseppe Bianchi ; Alessandro Mei. New York : Association for Computing Machinery (ACM), 2022. pp. 171-186

@inproceedings{66a55260e6d642a8b17c4c4a69a2b333,

title = "How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability",

abstract = "Automated browsers are widely used to study the web at scale. Their premise is that they measure what regular browsers would encounter on the web. In practice, deviations due to detection of automation have been found. To what extent automated browsers can be improved to reduce such deviations has so far not been investigated in detail. In this paper, we investigate this for a specific web automation framework: OpenWPM, a popular research framework specifically designed to study web privacy. We analyse (1) detectability of OpenWPM, (2) resilience of OpenWPM's data recording, and (3) prevalence of OpenWPM detection. Our analysis (1) reveals OpenWPM is easily detectable. Our investigation of OpenWPM's data recording integrity (2) identifies novel evasion techniques and previously unknown attacks against OpenWPM's instrumentation. We investigate and develop mitigations to address the identified issues. Finally, in a scan of 100,000 sites (3), we observe that OpenWPM is commonly detected (∼14% of front pages). Moreover, we discover integrated routines in scripts specifically to detect OpenWPM clients. In conclusion, our case study shows that even the most popular web measurement framework, OpenWPM, is more gullible than expected, and this gullibility is rarely accounted for in studies.",

keywords = "bot detection, privacy, reliability, security, web bots, web measurements",

author = "Benjamin Krumnow and H.L. Jonker and Stefan Karsch",

year = "2022",

month = nov,

day = "30",

doi = "10.1145/3555050.3569131",

language = "English",

pages = "171--186",

editor = "{Giuseppe Bianchi} and {Alessandro Mei}",

booktitle = "CoNEXT '22: Proceedings of the 18th International Conference on emerging Networking EXperiments and Technologies",

publisher = "Association for Computing Machinery (ACM)",

address = "United States",

note = "18th International Conference on emerging Networking EXperiments and Technologies, CoNEXT '22 ; Conference date: 06-12-2022 Through 09-12-2022",

url = "https://conferences2.sigcomm.org/co-next/2022/#!/home",

}

Krumnow, B , Jonker, HL & Karsch, S 2022, How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability. in Giuseppe Bianchi & Alessandro Mei (eds), CoNEXT '22: Proceedings of the 18th International Conference on emerging Networking EXperiments and Technologies. Association for Computing Machinery (ACM), New York, pp. 171-186, 18th International Conference on emerging Networking EXperiments and Technologies, Roma, Italy, 6/12/22. https://doi.org/10.1145/3555050.3569131

How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability. / Krumnow, Benjamin ; Jonker, H.L.; Karsch, Stefan.
CoNEXT '22: Proceedings of the 18th International Conference on emerging Networking EXperiments and Technologies. ed. / Giuseppe Bianchi; Alessandro Mei. New York: Association for Computing Machinery (ACM), 2022. p. 171-186.

Research output: Chapter in Book/Report/Conference proceeding › Conference Article in proceeding › Academic › peer-review

TY - GEN

T1 - How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability

AU - Krumnow, Benjamin

AU - Jonker, H.L.

AU - Karsch, Stefan

N1 - Conference code: 18

PY - 2022/11/30

Y1 - 2022/11/30

N2 - Automated browsers are widely used to study the web at scale. Their premise is that they measure what regular browsers would encounter on the web. In practice, deviations due to detection of automation have been found. To what extent automated browsers can be improved to reduce such deviations has so far not been investigated in detail. In this paper, we investigate this for a specific web automation framework: OpenWPM, a popular research framework specifically designed to study web privacy. We analyse (1) detectability of OpenWPM, (2) resilience of OpenWPM's data recording, and (3) prevalence of OpenWPM detection. Our analysis (1) reveals OpenWPM is easily detectable. Our investigation of OpenWPM's data recording integrity (2) identifies novel evasion techniques and previously unknown attacks against OpenWPM's instrumentation. We investigate and develop mitigations to address the identified issues. Finally, in a scan of 100,000 sites (3), we observe that OpenWPM is commonly detected (∼14% of front pages). Moreover, we discover integrated routines in scripts specifically to detect OpenWPM clients. In conclusion, our case study shows that even the most popular web measurement framework, OpenWPM, is more gullible than expected, and this gullibility is rarely accounted for in studies.

AB - Automated browsers are widely used to study the web at scale. Their premise is that they measure what regular browsers would encounter on the web. In practice, deviations due to detection of automation have been found. To what extent automated browsers can be improved to reduce such deviations has so far not been investigated in detail. In this paper, we investigate this for a specific web automation framework: OpenWPM, a popular research framework specifically designed to study web privacy. We analyse (1) detectability of OpenWPM, (2) resilience of OpenWPM's data recording, and (3) prevalence of OpenWPM detection. Our analysis (1) reveals OpenWPM is easily detectable. Our investigation of OpenWPM's data recording integrity (2) identifies novel evasion techniques and previously unknown attacks against OpenWPM's instrumentation. We investigate and develop mitigations to address the identified issues. Finally, in a scan of 100,000 sites (3), we observe that OpenWPM is commonly detected (∼14% of front pages). Moreover, we discover integrated routines in scripts specifically to detect OpenWPM clients. In conclusion, our case study shows that even the most popular web measurement framework, OpenWPM, is more gullible than expected, and this gullibility is rarely accounted for in studies.

KW - bot detection

KW - privacy

KW - reliability

KW - security

KW - web bots

KW - web measurements

U2 - 10.1145/3555050.3569131

DO - 10.1145/3555050.3569131

M3 - Conference Article in proceeding

SP - 171

EP - 186

BT - CoNEXT '22: Proceedings of the 18th International Conference on emerging Networking EXperiments and Technologies

A2 - Giuseppe Bianchi,

A2 - Alessandro Mei,

PB - Association for Computing Machinery (ACM)

CY - New York

T2 - 18th International Conference on emerging Networking EXperiments and Technologies

Y2 - 6 December 2022 through 9 December 2022

ER -

Krumnow B , Jonker HL, Karsch S. How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability. In Giuseppe Bianchi, Alessandro Mei, editors, CoNEXT '22: Proceedings of the 18th International Conference on emerging Networking EXperiments and Technologies. New York: Association for Computing Machinery (ACM). 2022. p. 171-186 doi: 10.1145/3555050.3569131

How gullible are web measurement tools? a case study analysing and strengthening OpenWPM's reliability

Abstract

Conference

Keywords

Access to Document

Fingerprint

Datasets

WPM vs. WPM_hide

Cite this