Jul 19, 2021
It recently came to our attention that our MADweb 2021 paper “First, Do No Harm: Studying the manipulation of security headers in browser extensions” has two critical errors which causes our results to be incorrect.
To understand if an extension manipulates a request, we use the Chrome Devtools Protocol (CDP) to monitor network requests and responses. The idea is to make two requests, once without any extension (ground-truth data) and once with the extension under test. Subsequently, we extract the headers for both attempts and determine if they differed between ground truth and extension. To avoid temporal drift as best as possible, we collect the ground-truth headers for a given site and subsequently test this site with the extension under test. However, there are still two major problems with this approach.
Something we did not think of at the time is the fact that servers may omit/add certain headers for some, but not all, requests. This can be caused by load balancers or through A/B testing. As an example,
https://www.linkedin.com/litms/api/metadata/user includes a
report-uri in 4/10 requests as of July 19, 2021. While we normalized parts that are random before comparison (such as nonces or the actual URL in
report-uri), we did not account for different headers coming back from the same URL within a few seconds.
Hence, if any extension was “unlucky” in receiving the version without
report-uri for the above-mentioned URL, we would compare the ground truth (with
report-uri) against the one observed during test and incorrectly flag the extension as having manipulated the CSP. In the paper, we noted that 1,938 domains had used report-uri in their CSP and it was seemingly dropped on 25 domains, yet by 676 extensions. While our data does not allow us to pinpoint the exact number of incorrect detections (since the sites have changed since then), we strongly believe this to a major contribution to incorrectly flagging many extensions as having manipulated a CSP.
There is also the opposite case, in which the ground truth did not collect a header, but the test with an extension did. This would lead to extensions being flagged as having injected a header where in fact this was pure chance in the second response. We also found other examples, such as HSTS and XFO, which cause similar problems. As an example, we receive different HSTS max-age values for
pinterest.com (either 63072000 or 31536000, seemingly depending on the time of day).
Through an extensive analysis, we (unfortunately only recently) discovered that there is a race condition between Chrome extensions and the Chrome Devtools Protocol (CDP). While we cannot fully validate our suspicion, our tests showed the following problem: If an extension registers an event handler to react to receiving/sending headers, the corresponding event is not necessarily also sent to the CDP session. This means that we might not be able to collect any data for a certain extension. If, however, the site currently visited in the test had some header in the ground-truth crawl and even sends it on the second try, we might still not collect any headers for the given URL. Hence, this leads to a situation where we would incorrectly flag an extension as having dropped a security header.
We do not know the exact impact this had on our findings, since the work was conducted in late 2020. However, we analyzed the data and found that we flagged many more extensions with the
all_urls privilege as being potential dangerous than we did for those with specific host permissions. Given our (belated) insights, this makes perfect sense: as these extensions operate on all URLs, we tested them against the top 100 sites (which include, among others, Pinterest and LinkedIn), which could trigger problem #1. Similarly, the chance of loosing the race seemingly is the same across all requests. Hence, the more URLs we test, the higher the chances problem #2 may occur.
Given these insights, we are unable to stand by the results. Moreover, we cannot reproduce the results with the outlined methodology, as this would inevitably lead to the same problems again. While potentially double-checking might work to alleviate Problem #1, we will be unable to address Problem #2 with the outlined approach.
As a result of this, we have asked the chairs to withdraw our paper from the workshop.