Storage Backends

The service supports multiple storage backends. A backend can be configured per collection in the configuration file. If no backend is defined for a collection, record_dir+stl is used by default.

Available Backends

record_dir

Stores records as YAML files in a directory structure defined by Dump Things Storage v0. Reads the backend configuration from a record collection configuration file as described in the same specification.

record_dir+stl (default)

Same as record_dir, but adds a schema-type layer (STL):

schema_type attributes are stripped from the top-level mapping of a record before it is stored.
When a record is read back, a schema_type attribute is re-inserted using the schema to determine the correct class URI.

This is useful when endpoints return records of multiple classes, because clients can determine the class of each record from the schema_type attribute.

sqlite

Stores records in a SQLite database. One database file named __sqlite-records.db is created per curated/incoming area.

sqlite-backends have the advantage that they support search patterns to limit the size of a result on the server.

sqlite+stl

Same as sqlite, but with the same schema-type layer as record_dir+stl.

Note

record_dir and sqlite backends can co-exist independently in the same directory. There are no file-name collisions between backends, as long as no class name starts with . or __.

Configuring a Backend

Specify the backend in the collection configuration with the backend key:

collections:
  my_collection:
    schema: https://example.org/schema.yaml
    default_token: anon_read
    curated: my_collection/curated

    backend:
      type: record_dir+stl   # explicit default

collections:
  another_collection:
    schema: https://example.org/schema.yaml
    default_token: anon_read
    curated: another_collection/curated

    backend:
      type: sqlite