Novaya Gazeta

On March 4 the Russian newspaper Novaya Gazeta announced that it would be discontinuing its coverage of the war in Ukraine after the Kremlin criminalized reporting on the topic:

Russia’s Novaya Gazeta newspaper, whose editor Dmitry Muratov was a co-winner of last year’s Nobel Peace Prize, said on Friday it would remove material on Russia’s military actions in Ukraine from its website because of censorship. The newspaper said it would continue to report on the consequences that Russia is facing, including a deepening economic crisis and the persecution of dissidents. – Reuters

Novaya Gazeta had to choose between completely shutting down or starting to censor their own news coverage. Doing otherwise would mean their reporters would be risking imprisonment or worse (seven of their journalists have been murdered since 2000).

So how has this new law shaped Navaya Gazeta’s reporting?

The Internet Archive’s Wayback Machine has good archival coverage of Novaya Gazeta novayagazeta.ru. The Internet Archive’s CDX API is a web service that allows you to programmatically query the archive to see what article URLs existed prior to the crackdown. So it should be a simple matter of testing each URL in the list to see if it’s available on the web right?

It turns out it’s a little bit trickier than that, but basically yes. Keep reading if you are interested in learning how to use the Wayback Machine to find removed content, and what 95 of the removed Novaya Gazeta articles are.

Article URLs

First of all there are many types of resources from the novayagazeta.ru website stored in the Internet Archive. The CDX API lets you limit by a URL prefix, date and also the content type, to hone in on pages instead of images, video, JavaScript or CSS.

The Novaya Gazeta has been around for almost 30 years, so there is a lot of content in the Internet Archive that is no longer on the novayagazeta.ru website because of site redesigns, CMS changes, etc. So it’s important to look at recently archived content, for example from 2020 onwwards.

import wayback
import datetime

wb = wayback.WaybackClient()

# search Wayback for archived HTML pages at novayagazeta.ru
results = wb.search(
    url='https://novayagazeta.ru/articles/',
    matchType='prefix',
    from_date=datetime.date(2020, 1, 1),
    filter_field='mimetype:text/html'
)

# collect the unique URLs that are found
urls = set()
for record in results:
    urls.add(record.url)

When I ran this today it yielded 64,146 URLs. Each article URL looks something like:

https://novayagazeta.ru/articles/2022/03/19/minoborony-rf-vpervye-zaiavilo-o-primenenii-giperzvukovogo-oruzhiia-v-khode-boevykh-deistvii-v-ukraine-news

It’s possible to narrow this list down further by removing URLs that have tracking parameters in a query string. It’s also possible to limit to articles that were published in 2022. This reduces the list that needs to be checked to 2,875 URLs. You an see this Jupyter notebook for the details.

Missing Articles

Checking if the URL is available or not on the live novayagazeta.ru website is not as simple as it could be. Usually a webserver will respond with a 404 Not Found when it is asked for a page that it doesn’t know about. But it appears that novayagazeta.ru responds with with a 200 OK message even when the page is not found. This is actually not that unusual on the web, and is one of the reasons why research into linkrot is so tricky.

To make matters worse the HTML that the server responds with doesn’t contain a not found message. The page is dynamically generated–it contains JavaScript code that needs to be executed which then in turn creates the not found message you see in your browser:

So in order to check these URLs it’s necessary to use a browser that will execute the JavaScript and then look for the “page missing” text in the rendered page. Since I was working in a Jupyter notebook I opted to use the pyppeteer library to drive a headless Chromium browser to visit each URL, and then look to see if the page had the missing message in it.

import pyppeteer

async def check(urls):
    missing = []
    browser = await pyppeteer.launch()
    page = await browser.newPage()
    for url in urls:
        await page.goto(url, {'waitUntil': 'networkidle2'})
        content = await page.evaluate('document.body.textContent', force_expr=True)
        is_missing = 'Ошибка. Похоже, вы перешли по неправильной ссылке. Попробуйте найти материал через поиск на сайте или сообщите нам через ctrl+enter, если что-то сломалось.' in content
        if is_missing:
            missing.append(url)
    await browser.close()
    return missing

results = await check(urls)

Running this found 196 articles (6.8%) which were no longer available on the novayagazeta.ru website. To further understand these it is possible to use a similar technique with the browser to look up these 196 URLs to find the Wayback Machine URL to view the archived page, and to extract the title from the page. This yielded these results:

Missing Archives

One thing that pops out when you scroll the table over and look at the title column is that “Новая Газета - novayagazeta.ru” repeats for 63 of them. For example:

https://web.archive.org/web/20220111140927/https://novayagazeta.ru/articles/2022/01/11/espch-pri

These are incomplete archival snapshots where only part of the page was archived. This highlights why quality assurance is such a difficult and labor intensive part of web archiving practice.

In addition there are 31 rows where the wayback_url is empty. For example:

https://web.archive.org/web/20220306115321/https://novayagazeta.ru/articles/2022/02/24/my-budem-stremitsia-k-demilitarizatsi

These are cases where the only snapshot in the Internet Archive appeared to be the missing pages. The snapshot wasn’t created in time before it was removed. Or, perhaps in some cases the URL never existed on the Novaya Gazeta website. It’s hard to say which without looking in other web archives.

The List

So, that was a bit more work than expected, but here is a final list of 95 articles from the Novaya Gazeta website published in 2022 that appear to have been removed. The big caveat here is that this is not a complete list, because not all the articles may have been archived by the Internet Archive, and my analysis only looked at articles published in 2022 (perhaps older articles have been removed as well). Novaya Gazeta are the only ones who know the real list of removed articles.

I cannot read Russian, so I spent a little time spot checking some of these in Google Translate. It wasn’t entirely clear to me that all of them were about the Ukraine, but several of them did appear to be. If you do understand Russian and have some time to analyze them please let me know and I will link out to you.