Skip to content

Consus Monitoring

What is Consus Monitoring?#

Consus Monitoring is a microservice that aggregates metadata about the last harvesting run of a catalogue, additionally it displays the next scheduled harvesting run for a catalogue.

What information does Consus Monitoring display?#

Currently, Consus Monitoring aggregates the following information about a catalogue:

  • the number of harvested datasets during the last harvesting run
  • the number of skipped datasets during the last harvesting run
  • the number of updated datasets during the last harvesting run
  • the number of added datasets during the last harvesting run
  • the number of deleted datasets during the last harvesting run
  • the number of errors that occured during the last harvesting run
  • the timestamp of the last harvesting run
  • the timestamp of the next harvesting run

Where does Consus Monitoring get its data from?#

Consus Monitoring gets its data from two entities: Consus Scheduling and the Elasticsearch search engine. The Scheduling component is queried for the next scheduled harvesting run timestamp while Elasticsearch provides the results for all other information mentioned above.

Consus Monitoring Classes#

Consus Monitoring consists of five classes written in Java:

  • The MainVerticle is the application entry point that handles requests coming from the API and starts the other Verticles.
  • The SchedulerConnectionVerticle retrieves data from Consus Scheduling. It can either return an array with all catalogues that have associated triggers in the Scheduling compontent or the timestamp of the next harvesting run for a certain catalogue.
  • The ElasticsearchConnectionVerticle sends queries to Elasticsearch and collects the responses in a JSON object.
  • The ElasticsearchQueryBuilder uses the Elasticsearch Query DSL to create queries in the form of JSON objects.
  • The ApplicationConfig contains the configuration for the service.

Note

Since Consus Monitoring is developed on the basis of the Vert.x Framework , classes that communicate in an asynchronous manner with each other or send requests to external APIs are denoted with the ending "Verticle".

How is the Consus Monitoring API used?#

Consus Monitoring provides its services via an API. Currently, there are three endpoints:

  • GET / returns the API documentation
  • GET /catalogues returns a list of catalogue IDs that have an associated trigger in Consus Scheduling
  • GET /catalogues/{catalogue id} returns harvesting metadata for a certain catalogue