Configuration

General Concepts

Each Dump Things Service instance maintains a persistent configuration. The configuration is stored in a private area of the storage root and will be loaded when a Dump Things Service instance is started.

Providing an initial configuration

If the persistent configuration is empty, for example, because the service is started on an empty <storage root>, an initial configuration can be provided by configuration files:

  1. via a configuration file in <storage root>/.dumpthings.yaml, or

  2. a configuration file can be passed to the command line with the -c/--config option.

Performing dynamic configuration changes

Every Dump Things Service instance (since version 6) supports dynamic reconfiguration via administration API endpoints. That means, a server can be started with an empty configuration and the configuration can be modified at runtime.

If a Dump Things Service instance is started with an empty configuration, the command line option --admin-token-hash has to be provided. This option will establish a “bootstrap” administrator token in the Dump Things Server instance that can be used to access the administration endpoints.

Note: the bootstrap administrator token does not become part of the persisted configuration. That means, it will not be represented in a configuration dump, i.e., in the result of GET /admin_tokens.

Elements of the configuration

Dump Things Service instances have three main configuration elements:

  • Collections

  • Tokens

  • Administrator Tokens

A Dump Things Service instance has administration endpoints for all three configuration elements, which allow to create, modify, and delete these elements. Any change to the configuration is persisted immediately and will be effective for all subsequent requests to the service.

All configuration changes are audited (see audit for details). The audit log is stored in the storage root at the path <storage root>/__dumpthings__/config_audit.

Configuration modification model

A Dump Things Service instance maintains an internal representation of the current configuration. The persistent configuration is kept up-to-date with the internal configuration.

If, for example, a collection is added via the administration interface, the service will add the collection to its internal configuration representation. It will then persist the new configuration and updated its internal state to “activate” the new configuration.

In other words if the administrator interface is used to add a collection, a token, or an administrator-token, the element will be added to the already existing configuration elements. If an element is removed via the administration interface, this element will be removed from the internal configuration, while all other configuration elements remain unchanged.

To determine the current configuration of the service, the administrator API provides endpoints to read collection configurations, token configurations, and admin token configurations.

To modify an existing configuration element, the administrator can fetch its current state, modify it, and PUT it back to the service.

The tool dump-things-download-config (see Command-Line Tools) can be used to fetch the current configuration of a running Dump Things Service instance and write it to a configuration file (see config_file:). This file can be used, for example, to create a Dump Things Service with an identical configuration at another location.

For a detailed description of the configuration endpoints see HTTP Endpoints.

Configuration File Structure

A configuration file (version 2) has the following top-level structure:

type: collections   # must be "collections"
version: 2          # must be 2

collections:
  <collection-name>:
    ...

tokens:
  <token-name>:
    ...

 admin_tokens:
   <admin-token-name>:
     ...

Collections

Each entry under <collection-name> defines a collection. The following keys are supported:

schema (required)

URL of the LinkML schema for this collection.

default_token (required)

Name of the token to use when no token is provided by a client. All permissions in the default token are OR-ed with any client-provided token.

Note: a token of the given name must be defined in the tokens section of the configuration file. At least the default token must be created before a collection can be created. Although tokens reference collections, tokens can always be created because their creation does check whether referenced collections exist yet.

curated (required)

Path to the curated area of the collection. This should be a relative path, which will be interpreted relative to the storage root.

incoming (required if tokens with write access exist for the collection)

Path to the incoming area. Required if the collection has any tokens with write access associated to it. This should be a relative path, which will be interpreted relative to the storage root.

backend (optional)

Storage backend configuration. If omitted, record_dir+stl is used. See Storage Backends for details.

use_classes (optional)

List of LinkML class names that should receive store/validate endpoints (the classes must be subclasses of Thing). All other classes defined in the schema are ignored.

ignore_classes (optional)

List of LinkML class names to exclude from store/validate endpoints (the classes must be subclasses of Thing).

auth_sources (optional)

List of authentication source configurations. See Authentication & Authorization.

submission_tags (optional)

Tags for an optional automated annotation of submitted records (see `Submission Annotation Tags`_ below).

audit_backends (optional)

List of audit-backend configurations. See `Audit Backends`_ below.

collections:
  personal_records:
    default_token: no_access
    schema: https://concepts.datalad.org/s/demo-research-information/unreleased.yaml
    curated: curated
    incoming: incoming
    use_classes:
      - Organization
      - Person
      - Project

Authentication configuration is described in Authentication & Authorization. If no authentication sources are defined for a collection, a config source is used by default, that means, the service will use the tokens that are configured in the configuration file.

The service might annotate uploaded records with a submitter ID and a submission time (version below 6 will always annotate submitted records, version 6 and higher will only annotate submitted records, if the query parameter add_submission_tag is set to true.

By default, the following tags are used for submission annotation:

  • Submitter ID: http://purl.obolibrary.org/obo/NCIT_C54269

  • Submission time: http://semanticscience.org/resource/SIO_001083

Override these defaults per collection with submission_tags:

collections:
  collection_1:
    ...
    submission_tags:
      submitter_id_tag: schema:user_id
      submission_time_tag: schema:time

The service supports audit logs of changes made via the curation interface. Configure them per collection with audit_backends:

collections:
  collection_1:
    ...
    audit_backends:
      - type: gitaudit
        path: /path/to/audit-log-dir
        auto_flush_timeout: 60   # seconds (default: 60, minimum: 1)

Currently the only supported audit backend type is gitaudit. A gitaudit backend stores changes in a bare Git repository at path. After auto_flush_timeout seconds without new entries the current changeset is committed.

Use dump-things-gitaudit-report to display the audit log and dump-things-gitaudit-rebuild-index to rebuild its index (see Command-Line Tools).

In configuration files, collections reference tokens and tokens reference collections. This can create a chicken-and-egg problem when creating new collection and new tokens dynamically. This is resolved by allowing tokens to refer to collections that are not created. In other words:

Create tokens first, then create collections.

The following collection names are reserved and must not be used:

  • api

  • collections

  • tokens

  • admin_tokens

  • __dump_things__

Tokens

Each entry under tokens defines a token. The following keys are supported:

representation (required)

The plain-text value the client must supply in the X-DumpThings-Token HTTP header. If hashed is True this must be the SHA-256 hex digest of the plain-text token.

hashed (optional, default: False)

If True, the representation is treated as a SHA-256 hash. Use the command dump-things-hash-token (see :doc:commands) to compute the correct hash.

user_id (required)

An identifier that is added as an annotation to each uploaded record.

collections (required)

Mapping from collection name to per-collection rights.

Each collection entry inside a token has:

mode

Access mode. Available modes (in the following list “own incoming area” refers to the incoming area that is associated with the token and the collection):

  • NOTHING – no access.

  • READ_CURATED – read the curated area only.

  • READ_COLLECTION – read curated data and own incoming area.

  • WRITE_COLLECTION – read curated data, read own incoming area, write to own incoming area.

  • READ_SUBMISSIONS - read own incoming area.

  • WRITE_SUBMISSIONS – read and write own incoming area.

  • SUBMIT - read curated area only, write to own incoming area.

  • SUBMIT_ONLY - write to own incoming area.

  • CURATOR - curator-level access: read & write to curated area, read & write to all incoming areas.

incoming_label

Label that defines the subdirectory (aka zone) in the incoming area for this token and collection. Multiple tokens can share the same label, which means they share a zone.

tokens:
  basic_access:
    representation: anonymous
    hashed: False
    user_id: anonymous_user
    collections:
      rooms_and_buildings:
        mode: READ_CURATED
        incoming_label: ''

  contributor_bob:
    representation: bob
    user_id: Bob
    collections:
      rooms_and_buildings:
        mode: WRITE_COLLECTION
        incoming_label: new_rooms_and_buildings

Tips & Tricks

You can configure the service so that incoming records are immediately visible in the curated area by setting the final incoming path to equal the curated path:

collections:
  datamgt:
    default_token: anon_read
    curated: datamgt/curated
    incoming: datamgt

tokens:
  trusted-submitter-token:
    user_id: trusted_submitter
    collections:
      datamgt:
        mode: WRITE_COLLECTION
        incoming_label: curated   # datamgt/curated == curated path

Start the version 6 server on the same storage root as the version 5 server. The version 6 server will automatically migrate the configuration from <storage root>/.dumpthings.yaml from the version 5 format to the new format and persist it. Use the -c/--config option on first start to initialise the persisted configuration from a different file than <storage root>/.dumpthings.yaml.

Once started, the configuration is stored in the internal database and the file <storage_root>/.dumpthings.yaml or the -c option can be omitted on subsequent starts:

dump-things-service /path/to/storage -c /path/to/old-config.yaml