One of the nice things that we did with twarc2 is to design it so you can add plugins relatively easily. These plugins extend twarc’s basic functionality to do different things with collected Twitter data. This is just a quick post about a new plugin twarc-hashtags.
twarc-hashtags was born of necessity. As I mentioned in the last post I’ve been doing some work with Alejandra Josiowicz to examine tweets about Brazilian activist Marielle Franco. Thanks to the Academic Research product track we were able to collect all the tweets matching the phrase Marielle Franco.
twarc2 search --archive '"marielle franco"' tweets.jsonl
But we wanted to discover what hashtags were used in this initial dataset in order to broaden the search using relevant hashtags and then run it again.
pip install twarc-hashtags you get a new
hashtags which you can use to generate a CSV
dataset that represents the hashtags present in your data:
twarc2 hashtags tweets.jsonl hashtags.csv
The generated CSV is pretty simple. It has two columns:
tweets. While your data is being
read a little SQLite database is populated which has three columns:
This allows for easy counting (using SQL) but also for a bit more
For example, if you would like to see the hashtags grouped by month you can:
twarc2 hashtags tweets.jsonl hashtags.csv --group month
The generated dataset will have an additional column for
time containing a year-month value, e.g. 2020-03. You can
do the same for day, week, and year if you want to group
In addition you can limit the number of hashtags to display. So if you just wanted to see the top 20 hashtags per month you could:
twarc2 hashtags tweets.jsonl hashtags.csv --group month --limit 20
And finally, since loading the SQLite database can take some time
(for example if you are looking through 3.8 million tweets like I was)
you can load it the first time and then use
afterwards to skip the import step and use the existing SQLite database.
This will allow you to try grouping by something different, or using a
new limit, without needing to parse all that data again.
twarc2 hashtags tweets.jsonl hashtags.csv --group week --limit 10 --no-import
Maybe some fancier output than CSV will get added over time (ideas are welcome). Having the output in CSV means you can pretty easily drop it into another tool, like D3, GoogleSheets, Tableau, DataWrapper, etc.
I’ve been meaning to try RawGraphs after seeing it come up in a thread that Anne Helmond kicked off about some of her data visualization work some time ago. Here’s what the top 25 hashtags look like over time as a BumpGraph. If you want you can use the rawgraphs file (it’s JSON) to upload it yourself and tweak it.
If you click on the image it might be a bit easier to read (you can hover on the streams to see what they are)–but clearly it needs a bit of work still. One thing it shows pretty clearly though is the emergence of #QuemMandouMatarMarielle (Who Ordered Marielle to be Killed), the yellow band, that started the year after her murder, and has continued since.
If you get a chance to try the twarc-hashtags plugin please let me know!