Skip to content

DDR-2401: Refactor Ddr::IngestFolder, remove configuration file and per-location ingest permissions

David Chandek-Stark requested to merge DDR-2401 into main

This was a substantial refactoring of "DPC Folder Ingest", which is some of the oldest code in the repo. While the principal initial goal was to remove the requirement to grant additional permissions on a per-ingest-location basis, additional changes made it possible to remove the configuration file completely. New environment variables have been added to avoid hard-coding implementation-specific details:

  • INGEST_BASE_PATHS - A string of colon-separated paths from which ingests are permitted (by any authz'd user)

    INGEST_BASE_PATHS="/nas/dpc_archive:/nas/rubenstein_archive"

  • INGEST_FOLDER_CHECKSUM_FILE_DIR - The path to the default checksum location.

    INGEST_FOLDER_CHECKSUM_FILE_DIR="/nas/dpc_staging/checksums"

  • INGEST_FOLDER_EXCLUDE_FILES - A string of colon-separated fnmatch-style patterns to exclude from ingest. If not provided the patterns **/.DS_Store, **/Thumbs.db, and **/*-sha1.txt are used as a default.

    INGEST_FOLDER_EXCLUDE_FILES="**/*.tmp:**/*.log"

Edited by David Chandek-Stark

Merge request reports