piveau Profile#

Experimental Feature

The described features, configuration and APIs are work in progress.

A piveau Profile is central configuration that allows you to customize the metadata model of piveau instance. Hence, it enables you to modify, extend and configure the DCAT-AP-based core model. Furthermore it allows you add additional metadata types beyond catalogues, datasets and distributions. Currently, piveau Profile is supported by hub-repo and hub-search and if used it needs to be enabled in both services.

Concept#

A piveau Profile is a directory that contains a manifest file package.json and one or more SHACL files. The SHACL files defines the metadata model and are using a domain-specific properties.

.
└─ my-profile
    ├─ piveau.json
    ├─ shacl-file-1.ttl
    ├─ shacl-file-2.ttl
    ├─ my-vocabulary.ttl
    └─ samples

The profile will take effect on the following aspects:

The mapping in Elasticsearch
The indexing process from RDF to the mapping
The OpenAPI description of the hub-search API
The availability of additional entities in hub-repo and hub-search

Enabling the Feature#

The following config applied for both hub-repo and hub-search will enable and configure a piveau Profile.

{
    "PIVEAU_FEATURE_FLAGS": {
        "piveau_profile": true
    },
    "PIVEAU_PROFILE": {
        "type": "directory",
        "path": "/path/to/my/profile/my-profile"
    }
}

piveau.json#

Here is an example of a piveau.json:

{
  "version": "1",
  "id": "my-profile",
  "core": [
    {
      "id": "dataset",
      "description": "My model for a dataset",
      "path": "my-profile.ttl",
      "name": "dataset",
      "shapeUri": "http://data.europa.eu/r5r/Dataset_Shape"
    }
  ],
  "resources": [
    {
      "id": "my-resource",
      "description": "My Resource",
      "path": "my-resource.ttl",
      "name": "my-resource",
      "typeUri": "https://piveau.eu/ns/voc#CustomResource",
      "shapeUri": "https://piveau.eu/ns/voc#CustomResourceShape",
      "inCatalogue": false, //(1)!
      "searchSettings": { //(2)!
        "facets": [],
        "searchParams": []
      }
    }
  ]
}

inCatalogue: Set to true if the resource should be required to be part of a catalogue. Like a DCAT Dataset. Default is false.
searchSettings: Configures the piveau-hub-search service for custom resources. facets corresponds to PIVEAU_HUB_SEARCH_ES_CONFIG.index.*.facets; searchParams corresponds to PIVEAU_HUB_SEARCH_ES_CONFIG.index.*.searchParams

Note: "core" and "resources" are optional. Use "core" to customize core resources like catalogues or datasets, and "resources" to define customized resources.

Danger

The id field does not support special characters or uppercase letters. Supported are Numbers, lowercase letters (a-z) and a minus sign (-).

Custom Vocabularies#

You can bundle a custom vocabulary with a piveau Profile. Store the vocabulary file in the profile directory and add it to the optional vocabularies array in piveau.json:

{
  "id": "my-profile",
  "vocabularies": [
    {
      "id": "my-publishers",
      "uriRef": "https://example.org/vocabularies/my-publishers",
      "file": "my-vocabulary.ttl",
      "remote": {
        "address": "https://example.org/vocabularies/my-publishers.ttl"
      },
      "tags": [
        "core",
        "skos"
      ]
    }
  ]
}

file is resolved relative to the profile directory. Use the id to identify the vocabulary in piveau configuration, and set uriRef to the vocabulary's concept-scheme URI. The vocabulary file should contain the SKOS concepts that the profile's metadata model references. See Create and integrate a custom vocabulary for an example SKOS vocabulary and its use in metadata.

To retrieve the latest vocabulary from a remote source, add the optional remote object. Its address is the download URL. Keep file to provide the local vocabulary resource when the remote source is not used.

After loading the profile, install the vocabulary with the installVocabularies hub-repo command. The example uses the core tag, so it is included when the command installs the default set of core vocabularies.

Resource Materialization#

hub-search can materialize selected fields from related source resources into a destination resource. It stores the selected values in a generated array field on the destination. Both the source and destination must be custom resource types defined in the profile's resources array.

Configure a materialization in the destination resource's searchSettings.materializations array in piveau.json:

{
  "id": "destination-resource",
  "searchSettings": {
    "materializations": [
      {
        "targetField": "materialized-sources",
        "sourceResourceType": "source-resource",
        "sourceReferenceField": "destination-id",
        "projection": ["id", "title", "description"]
      }
    ]
  }
}

Key	Description
`targetField`	Name of the generated array field on the destination resource that contains the materialized source values.
`sourceResourceType`	Source resource type whose documents supply the projected values. It must match a resource definition `id` in the profile.
`sourceReferenceField`	Direct source field containing the canonical URI of the destination resource.
`projection`	Non-empty list of unique, direct source fields copied into each generated array item.

The source reference field must be a direct property in the source resource's SHACL shape. It must use StandardResourceKeyword and declare sh:maxCount 1:

@prefix ex: <https://example.org/> .
@prefix pv: <https://piveau.eu/ns/voc#> .
@prefix sh: <http://www.w3.org/ns/shacl#> .

ex:SourceResourceShape
    a sh:NodeShape ;
    sh:property [
        sh:path ex:destination ;
        sh:nodeKind sh:IRI ;
        sh:maxCount 1 ;
        pv:mappingClass "StandardResourceKeyword" ;
        pv:mappingName "destination-id"
    ] .

If the reference property declares sh:minCount 1, hub-search rejects source writes that omit the reference field. Without sh:minCount 1, source resources without a destination reference are allowed. A source type can participate in multiple materializations, while each reference field can contain at most one destination URI.

The projected fields must be direct properties in the source resource's SHACL shape. The generated targetField must not already be defined by the destination SHACL shape or another materialization.

When a source or destination resource changes, hub-search rebuilds the affected destination field from all matching sources. Each array item contains only the configured projected fields, and the items are ordered by source resource ID. Chained materializations are not supported: a resource type cannot be both a materialization source and a materialization destination.

Models with SHACL#

The foundation of a piveau Profile are SHACL files that use domain-specific properties.

@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix dcatap: <http://data.europa.eu/r5r/> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix pv: <https://piveau.eu/ns/voc#> .

pv:My_Profile
    a pv:PiveauProfile ;
    pv:profileVersion "1" ;
    dct:title "My Profile"@en ;
    dct:description "Based and inspired by DCAT-AP 2.1.1"@en ;
    owl:versionInfo "0.0.1" .


dcatap:Catalog_Shape
    a sh:NodeShape ;    
    sh:name "Catalog"@en ;
    sh:property [
        pv:mappingClass "SimpleMultiLangTitle" ;
        pv:mappingName "title" ;
        sh:minCount 1 ;
        sh:nodeKind sh:Literal ;
        sh:path dct:title ;
        sh:severity sh:Violation ;
    ] ;
    sh:targetClass dcat:Catalog .

Danger

Installing or updating a piveau Profile requires a restart of the services. You cannot change the mapping of an existing field (e.g., change text to keyword, or change an analyzer). This would invalidate already indexed data. If you need to do this, you typically have to reindex your data into a new index with the correct mapping. You can add new, entirely new top-level fields to an existing index's mapping. For example, if your index initially only had product_id and name, you could add description or price without reindexing.

To resolve the strict_dynamic_mapping_exception error while indexing, see the Troubleshoot section.

You can also add new fields (sub-fields) to an existing text field if those sub-fields are based on analysis that can be derived from the already tokenized text field, or if they are keyword type (which only needs the raw string, which is typically also stored). For example, if you have a name field of type text, you can later add a name.keyword sub-field, because the _source contains the original string, from which the keyword version can be derived upon reindexing.