top hosts referenced in wikipedia (part 2)

Jodi Schneider pointed out to me in an email that my previous post about the top 100 hosts referenced in wikipedia may have been slightly off balance since it counted *all* pages on wikipedia (talk pages, files, etc), and was not limited to only links in articles. The indicator for her was the high ranking of, which seemed odd to her in the article space.

So I downloaded the enwiki-latest-page.sql.gz, loaded it in, and then joined on it in my query to come up with a new list. Jodi was right, it’s a lot more interesting:

This removed a lot of the interwiki links between the English wikipedia and other language wikipedias (which would be interesting to look at in their own right). It also removed administrative links to things like Also interesting is that it removed from the list, which probably were linked to from user profile pages? The neat thing is it introduced new sites into the top 100 like the following:

We can see a lot more pop culture media present: newspapers, magazines, sporting information. Also we can see research oriented websites like,, make it into the top 100.

I work for the US federal government so I was interested to look at what .gov domains were in the top 100:

hostname links 419816 62134 57423 48530 33018 25962 25941 20178 20063 18880 15115 12196

Which points to the importance of federal biomedical, geospatial, scientific, demographic and biographical information to wikipedians. It would be interesting to take a look at higher education institutions at some point. Doing these one off reports is giving me some ideas about what linkypedia could turn into. Thanks Jodi.

7 thoughts on “top hosts referenced in wikipedia (part 2)

  1. Thanks for posting these results.

    I think you’ve posted the original list again rather than the updated one because is the top entry. It also appears to be served with the wrong MIME type (or perhaps is twice compressed?)

    This is just the tip of the iceberg and it would be interesting to see what kinds of additional insights could be teased out. Some of the things that come to mind include identifying patterns used by bots and excluding/segregating them so that you can see more organic results, aggregating equivalent domains (e.g. Google Books in different TLDs), reversing domains and aggregating subtotals (gov.nih.*, gov.nps.*), aggregating by type/subject (e.g. &, etc, etc. Days of fun for someone who wants to slice & dice the data!

  2. Actually was in the top of the results when limiting to articles as well. I just removed it from my barchart again.

    As far as I can see with curl, the content-type header for the list is “application/x-gzip” which is right?

    ed@curry:~/Projects/openpub/examples$ curl -I
    HTTP/1.1 200 OK
    Date: Thu, 26 Aug 2010 02:03:31 GMT
    Server: Apache/2.2.14 (Ubuntu)
    Last-Modified: Wed, 25 Aug 2010 11:30:30 GMT
    ETag: "f8013-b44b97-48ea4357ce180"
    Accept-Ranges: bytes
    Content-Length: 11815831
    Vary: Accept-Encoding
    Connection: close
    Content-Type: application/x-gzip

    Those are some good ideas for further analysis, thanks!

  3. Ed, when you say “referenced in Wikipedia” do you mean actually used as a “References” or “Notes” link to substantiate a citation, or do you simply mean “external link” of any variety, anywhere on an article page?

    If the latter, might I suggest you use some different terminology in this blog post?

    Great work, by the way. It’s the sort of stuff that the Wikimedia Foundation itself should be doing, but their too busy hiring new staff to not do much.

  4. @thekosher, well by external links I mean links included in enwiki-latest-externallinks.sql.gz made available by It’s my understanding that these are referenced citations, and not just any link on an article page. The stats in this post also used the enwiki-latest-page.sql.gz dump to limit to only external links in articles with the following SQL:

    SELECT * FROM externallinks, page 
    WHERE externallinks.el_from = page.page_id
    AND page.page_namespace = 0

    I hope that clarifies things somewhat.

Leave a Reply