Skip to content

Downloading and finding data from WIS2

Introduction

In this session you will learn how to discover data from the WIS2 Global Discovery Catalogue (GDC) and download data from a WIS2 Global Broker (GB).

Preparation

Note

Before starting please login to your student VM.

Downloading data with pywis-pubsub

The first practical session used MQTT Explorer to connect to the Météo-France Global Broker.

Let's use the pywis-pubsub to subscribe using a command line tool.

pywis-pubsub subscribe --help

Update the sample configuration (see the sections marked TBD) to connect to the Météo-France Global Broker:

vi ~/exercise-materials/pywis-pubsub-exercises/config.yml

Open MQTT Explorer and connect to the Météo-France Global Broker.

Update the following values in the configuration:

  • broker
    • username: everyone
    • password: everyone
    • host: globalbroker.meteo.fr
    • port: 8883
  • subscribe_topics: fill in one to many topics origin/# and cache/# (on separate lines)
  • storage.option.path: add a directory to where data should be downloaded

Run the pywis-pubsub command:

pywis-pubsub subscribe -c ~/exercise-materials/pywis-pubsub-exercises/config.yml

Question

What is the format of the data notifications that are displayed on the screen?

Question

Is there data being downloaded? How can we run the pywis-pubsub command to be able to download the data (hint: review the options when running the pywis-pubsub subscribe --help command)?

Stop the pywis-pubsub command (CTRL-C) and update the configuration to be able to download the data to /tmp/wis2-data.

Finding data with pywiscat

Let's use pywiscat to query the GDC

pywiscat wis2 search

Question

How many records are returned from the search?

Let's try querying the GDC with a keyword:

pywiscat wis2 search -q radar

Question

What is the data policy of the results?

Try additional queries with -q

Tip

The -q flag allows for the following syntax:

  • -q sea: find all records with the word "sea"
  • -q "NOT sea": find all records that do not contain the word "sea"
  • -q "sea AND ice": find all records with both "sea" and "ice"
  • -q "sea OR ice": find all records with both "sea" and "ice"
  • -q "sea NOT ice": find all records with "sea" but not "ice"
  • -q "sea~": fuzzy search

If searching for terms with spaces, enclose in double quotes.

Let's get more details on a specific search result that we are interested in:

pywiscat wis2 get <id>

Tip

Use the id value from the previous search.

Conclusion

Congratulations!

In this practical session, you learned how to:

  • use pywis-pubsub to subscribe to a Global Broker and download data to your local system
  • use pywiscat to discover datasets from the Global Discovery Catalogue