Dependency Hell
XKCD 2347 was published this week, which got some attention on the maintainers discussion list. One thing I learned there is that GitHub has added the ability to get the dependencies for a GitHub project using the their GraphQL API. So once you’ve got set up with an API token you can issue a GraphQL query like this:
{"%s", name: "%s") {
repository(owner:
description50) {
dependencyGraphManifests(first:
nodes {
blobPath
dependencies {
nodes {
packageName
repository {
name
owner {
login
}
primaryLanguage {
name
}
}
requirements
hasDependencies
}
}
}
}
} }
and get back some data like this:
{
"data": {
"repository": {
"description": "A command line tool (and Python library) for archiving Twitter JSON",
"dependencyGraphManifests": {
"nodes": [
{
"blobPath": "/DocNow/twarc/blob/main/setup.py",
"dependencies": {
"nodes": [
{
"packageName": "pytest",
"repository": {
"name": "pytest",
"owner": {
"login": "pytest-dev"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": true
},
{
"packageName": "python-dotenv",
"repository": {
"name": "python-dotenv",
"owner": {
"login": "theskumar"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": true
}
]
}
},
{
"blobPath": "/DocNow/twarc/blob/main/requirements/python2.txt",
"dependencies": {
"nodes": [
{
"packageName": "mock",
"repository": {
"name": "python-mock",
"owner": {
"login": "calvinchengx"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": true
},
{
"packageName": "pytest",
"repository": {
"name": "pytest",
"owner": {
"login": "pytest-dev"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": true
},
{
"packageName": "python-dateutil",
"repository": {
"name": "python-dateutil",
"owner": {
"login": "paxan"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": true
},
{
"packageName": "requests-oauthlib",
"repository": {
"name": "requests-oauthlib",
"owner": {
"login": "requests"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": true
},
{
"packageName": "unicodecsv",
"repository": {
"name": "python-unicodecsv",
"owner": {
"login": "jdunck"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": false
}
]
}
},
{
"blobPath": "/DocNow/twarc/blob/main/requirements/python3.txt",
"dependencies": {
"nodes": [
{
"packageName": "pytest",
"repository": {
"name": "pytest",
"owner": {
"login": "pytest-dev"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": true
},
{
"packageName": "python-dateutil",
"repository": {
"name": "python-dateutil",
"owner": {
"login": "paxan"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": true
},
{
"packageName": "requests-oauthlib",
"repository": {
"name": "requests-oauthlib",
"owner": {
"login": "requests"
},
"primaryLanguage": {
"name": "Python"
}
},
"requirements": "",
"hasDependencies": true
}
]
}
}
]
}
}
}
}
The cool thing here is that GitHub is doing the work of figuring out
how dependencies are managed in the project
(requirements.txt
, pom.xml
,
Cargo.toml
, Gemfile
, etc). Sometimes it looks
like the results return a loading
error message, which
later return ok. So maybe GitHub are lazily calculating some of this
information?
To test the API I created a small command line utility called xkcd2347 which you give a GitHub repository and it will print out the dependencies:
$ xkcd2347.py docnow/twarc
pytest: https://github.com/pytest-dev/pytest
python-dotenv: https://github.com/theskumar/python-dotenv
mock: https://github.com/calvinchengx/python-mock
python-dateutil: https://github.com/paxan/python-dateutil
requests-oauthlib: https://github.com/requests/requests-oauthlib
unicodecsv: https://github.com/jdunck/python-unicodecsv
If you like you can look at the dependencies of the dependencies
using the --depth
option:
$ xkcd2347.py --depth 2 docnow/twarc
pytest: https://github.com/pytest-dev/pytest
pallets-sphinx-themes: https://github.com/pallets/pallets-sphinx-themes
pygments-pytest: https://github.com/asottile/pygments-pytest
sphinx: https://github.com/sphinx-doc/sphinx
sphinx-removed-in: https://github.com/MrSenko/sphinx-removed-in
sphinxcontrib-trio: https://github.com/python-trio/sphinxcontrib-trio
python-dotenv: https://github.com/theskumar/python-dotenv
bumpversion: https://github.com/peritus/bumpversion
click: https://github.com/pallets/click
flake8: https://github.com/PyCQA/flake8
ipython: https://github.com/ipython/ipython
mock: https://github.com/calvinchengx/python-mock
portray:
pytest: https://github.com/pytest-dev/pytest
pytest-cov: https://github.com/pytest-dev/pytest-cov
sh: https://github.com/amoffat/sh
tox: https://github.com/tox-dev/tox
twine: https://github.com/pypa/twine
typing: https://github.com/python/typing
wheel: https://github.com/IsCoolEntertainment/debpkg_python-wheel
mock: https://github.com/calvinchengx/python-mock
python-dateutil: https://github.com/paxan/python-dateutil
requests-oauthlib: https://github.com/requests/requests-oauthlib
oauthlib: https://github.com/oauthlib/oauthlib
requests: https://github.com/psf/requests
mock: https://github.com/calvinchengx/python-mock
requests-mock:
unicodecsv: https://github.com/jdunck/python-unicodecsv
The hierarchy could probably be better displayed: currently each level of dependency is presented as a space.
If you like you can use it as a Python library too:
import xkcd2347
gh = xkcd2347.GitHub(key="yourkeyhere")
for dep in gh.get_dependencies(repo_owner='docnow', repo_name='twarc'):
print(dep['packageName'])
If you run it with --level 0
it will walk as far into
the dependency hierarchy as it can. But beware…these go very deep. The
utility is smart enough to cache GraphQL query results: so it won’t keep
hammering on the GitHub API for the same information over and over. It
also takes care not to wander down circular dependencies, for situations
where a package has a dependency somewhere in the tree of dependencies
that includes itself.