Off a tip from Joan Donovan and her team at the Shorenstein Center I’ve been experimenting a bit lately to examine new domains related to the COVID-19 pandemic. I’m writing this blog post to document a way of using web archiving tools like the WARC format and the warcio Python module, in the context of Jupyter Notebooks as a way to study the web. I think we often consider WARC as useful for saving what a page looks like so that it can be “played back” later, using something akin to the Internet Archive’s Wayback Machine. But (as I hope you’ll see) WARC is also a useful and convenient method for storing web content along with critical information about how it was retrieved, so you can study it later.
Specifically, I’ve been looking at domains that have been flagged by the componay DomainTools who help other companies with brand protection, domain monitoring, domain valuation, and cybercrime investigations. In late March DomainTools released a public dataset of COVID-19 related domains that they have flagged as being likely threats. Threats can include phishing scams, and attacks, as well as attempts to capitalize on the pandemic.
The DomainTools dataset is updated every day, and since April 5 I’ve been downloading it and creating a snapshot at the Internet Archive. This dataset is not only a unique picture of the use of web platforms to capitalize on the Coronavirus crisis, but also provides a rare (but narrow) view into the operations of a cybersecurity firm.
Given the news of how Shopify is being used by various actors to make money off of the crisis I thought it could be interesting to look at the DomainTools data, and see how prevalent Shopify sites were.
The first step is to collect the homepages for these domains. In this Jupyter notebook I used the warcio library from the Webrecorder project to walk through the domains, and fetch the homepage, and write the data to a WARC file. The notebook contains some logic for probing where the homepage for a domain might be, but you can see the basics of how easy it is to write to a WARC file as you crawl the web in the warcio documentation.
Using warcio here is very useful because it means I don’t have to manage writing the HTTP responses to disk myself. In addition the WARC data contains the full HTTP transaction history (including HTTP headers) to the file as well. It took a good week to collect the over 100,000 domains that DomainTools had marked with a high threat score (99). This is because I did it in the simplest possible way, one domain at a time, and a fair number (26,335) no longer had responsive web servers running.
With the WARC data in hand I then worked in this notebook to scan the HTTP responses looking for evidence of Shopify. Just reading the file line by line showed that indeed there were quite a few sites that set a Shopify cookie, that is used in their analytics tracking:
Set-Cookie: _shopify_y=53b92b06-c72e-40a3-a64e-d6b0f81b6c47; path=/; expires=Thu, 14 Apr 2022 01:41:58 GMT
Given that little hint it’s then possible to read the WARC data using warcio, and identify domains that appear to be served up using Shopify using some code like this (see the notebook for the details):
This found 372 domains. Joining this list back up with the DomainTools data lets us see how these shops have been created over time, with the caveat that the created time could be when the domain was registered or when DomainTools first noticed the domain.
So it looks like they peaked on March 15. But if we compare that graph to the graph of all the domains being tracked by DomainTools we can see it coincides roughly with the general trend in domains that DomainTools is tracking:
It’s important to stress that what we are seeing here is as much (perhaps more) a picture of how DomainTools collects, analyzes and classifies DNS, as it is a picuture of the Coronavirus web. You can find the chart generation for these graphs in their notebooks.
Given that Shopify claims to have started blocking shops that sold testing equipment I thought it could be interesting to see how many of the Shopify websites mention “test kit”. This is pretty easy to do with warcio with some similar code, but this time looking at the HTML response content:
This found 388 domains, but only one appeared to be using Shopify: rapidmedicalsystems.com. We can join this list as we did before back with the original DomainTools data to see test-kit sites created over time, which largely fits the same pattern we saw before.
If you are interested in inspecting these domains for yourself I’ve made a snapshot of the DomainTools data with additional columns for Shopify and Test Kits, to indicate if they tested true for either of those. A few caveats to note is that these domains were determined using the 2020-04-11 snapshot of the DomainTools dataset, and only domains that have were flagged with the highest risk score of 99 (78% of the domains) were included.