List Nomisweb Datasets with JSON¶
date: 2015-06-29
You can use the Nomisweb API to discover (list) datasets and to obtain them once you have identified what you need. The API supports a variety of formats but in this tutorial I use JSON because it’s less verbose than other formats so is a smaller file.
jq
and curl
¶
To list all the available datasets I used a programme called jq
(‘Json Query’ I assume). You can install jq
on an Ubuntu machine
with sudo apt install jq
or sudo snap install jq
. You should
install a version >= 1.5. You will also need curl
(sudo apt install curl
).
The following script uses curl and jq to obtain and filter the results,
retaining ID, name, and description. It then saves the output to
nomisweb-datasets.txt
:
Line 1 specifies which shell to use.
Line 2 specifies the relevant json file listing all the datasets, specified in the API documentation.
Line 3 is a series of filters piped together. It essentially obtains everything; then filters out everything but
keyfamilies
at that level; then filters everything butkeyfamily
(it just so happens there’s nothing to filter at that level); then obtains all datasets (‘.[]
’); before finally extracting the id, name, and description from each index.Line 4 saves this to the file rather than print it to the terminal window as is the default.
nomisr
¶
If you’re an R user, querying and obtaining data from the Nomis API is
now even easier with the `nomisr
package <https://docs.evanodell.com/nomisr/>`__.