Skip to content

Quick Start#

Launching a basic version of Piveau is very easy and straightforward. This guide shows how to launch a minimal version of the Hub and a simple harvesting job with Consus to get some data into it. piveau uses docker as its default deployment environment and runs on most operating systems (Linux, macOS and Windows).

Run pre-build docker images#

Prerequisites#

  • Docker & Docker Compose

Run hub and consus#

You need two Docker Compose files, which you can download here. The images will be pulled from different docker hubs.

docker compose -f docker-compose-hub.yml up -d 
docker compose -f docker-compose-consus.yml up -d

You can check if all services are running, e.g. piveau-hup-repo at http://localhost:8082

Harvest data#

Before harvesting the data, you need to register a catalog in the piveau hub. Download the GovData Catalogue RDF and execute the following command:

curl -i -X PUT -H "X-API-Key: yourRepoApiKey" -H "Content-Type: text/turtle" --data @govdata.ttl localhost:8082/catalogues/govdata

The command should yield status code 201 Created.

You can start the harvesting process by choosing one of the following two options:

  1. Get the govdata pipe from here. Next, use the CLI interface of the scheduler either via telnet localhost 5000 or http://localhost:8095/shell.html and launch the govdata pipe there: launch govdata
  2. Create an immediate trigger using the scheduler API:
    curl -i -X PUT -H 'Content-Type: application/json' --data '{"status": "enabled", "id": "immediateTrigger"}' 'localhost:8090/pipes/govdata/triggers/immediateTrigger'

Important

GovData consists of several thousand datasets. The harvesting process can take quite some time.

That's it!#

You can now start using your new basic piveau instance. Start browsing datasets here: http://localhost:8080

Screenshot

Build & run from source#

You need the following piveau services for this guide:

  • piveau-hub-repo
  • piveau-hub-search
  • piveau-hub-ui
  • piveau-consus-scheduling
  • piveau-consus-exporting-hub
  • piveau-consus-transforming-js
  • piveau-consus-importing-ckan
  • piveau-consus-importing-rdf

You find the sources for the Hub here and for Consus here. You also need two Docker Compose files, which you can download here. In addition, some images from the public Docker Hub will be pulled, like Elasticsearch.

Prerequisites#

  • Java JDK >= 17
  • Maven >= 3.6
  • Node >= 10 <= v14
  • npm >= 5.6.0 <= 6.14.16
  • Docker & Docker Compose
  • Git
  • curl

Directory structure#

Place the sources and Docker Compose files in the following directory structure:

.
├─ piveau-hub-repo
├─ piveau-hub-search
├─ piveau-hub-ui
├─ piveau-consus-scheduling
├─ piveau-consus-exporting-hub
├─ piveau-consus-transforming-js
├─ piveau-consus-importing-ckan
├─ piveau-consus-importing-rdf
├─ docker-compose-consus-build.yml
└─ docker-compose-hub-build.yml
For example:

git clone https://gitlab.com/piveau/hub/piveau-hub-repo

Important

On Windows: Make sure that you are using LF line separators!

Piveau Hub - The Data Storage#

Compile#

Open a shell and execute the following commands:

mvn -f piveau-hub-repo/pom.xml clean package
mvn -f piveau-hub-search/pom.xml clean package
cd piveau-hub-ui && npm install
cp config/user-config.sample.js config/user-config.js && npm run build
cd ..

Build and run images#

Build and start everything with Docker Compose:

docker compose -f docker-compose-hub-build.yml up --build

This may take a couple of minutes. Upon completion, you can check if all services are running, e.g piveau-hup-repo at http://localhost:8081

Piveau Consus - Get some data with a harvesting run#

Compile#

Download the pipe specification for the German Open Data portal GovData. The scheduling component requires it. Open a shell and execute the following commands:

mvn -f piveau-consus-importing-rdf/pom.xml clean package -DskipTests
mvn -f piveau-consus-importing-ckan/pom.xml clean package -DskipTests
mvn -f piveau-consus-transforming-js/pom.xml clean package -DskipTests
mvn -f piveau-consus-exporting-hub/pom.xml clean package -DskipTests
cp pipe-govdata.yaml piveau-consus-scheduling/src/main/resources/pipes
mvn -f piveau-consus-scheduling/pom.xml clean package -DskipTests

Build and run images#

Build and start everything with Docker Compose:

docker compose -f docker-compose-consus-build.yml up --build

This may take a couple of minutes. Upon completion, you can check if all services are running, e.g piveau-importing-rdf at http://localhost:8090

Using Helm Charts#

It is also possible to use helm charts to deploy piveau components or modules to a kubernetes cluster.

Paca Repository#

All piveau helm charts are available through a publicly available repository. To add it to you repo list:

$ helm repo add paca https://paca.fokus.fraunhofer.de/repository/helm-charts

List all currently available helm charts in the paca repository:

$ helm search repo paca

piveau Modules and Components#

Beside the three main modules consus, hub, and metrics, each single component has a helm chart.

When you are already properly connected to your cluster (including namespace), installing piveau consus just requires the following command: 

$ helm upgrade --install -f my-values.yaml paca/piveau-consus

See README for more detailed information on how to use your own value file to individualize your deployment.