Using the tool in an instrument repository

To reproduce the CI tests locally before committing, run the following from the repository root, in the conda environment created from its conda.yml (conda env create -f conda.yml):

pip install git+https://github.com/mccode-dev/mcstas-ess-instr-citool.git
mcstas-ess-instr-citool -a runci .

While developing, -a runcifast (or -a checkbuild) gives faster feedback, since it runs each example with only 100 neutrons (or only compiles the instruments), and stops at the first failure. mcstas-ess-instr-citool -a check . quickly validates the layout, and mcstas-ess-instr-citool . shows a summary of the project. If the working copy contains temporary files which the layout does not allow (e.g. output from running the instruments), the checks fail on them; --lenient ignores them instead (with a warning), but CI will still fail if they are committed.

A GitLab CI configuration (.gitlab-ci.yml) for ESS instrument repositories at git.esss.dk, doing the same:

runci:
  tags:
    - python311  # as used by other ESS instrument repositories
  variables:
    MAMBA_ROOT_PREFIX: "$CI_PROJECT_DIR/.micromamba"
  script:
    # The package cache is shared by all jobs on the runner (outside the
    # project directory), so packages are only downloaded and stored once,
    # and the environment mostly consists of hard links into the cache:
    - export CONDA_PKGS_DIRS="$HOME/.cache/mcstas-ess-instr-citool/conda-pkgs"
    # Uncompressed micromamba executable (the runner has no bzip2), in a hidden
    # directory since other files are not allowed at the top level:
    - curl -Ls -o .micromamba/bin/micromamba --create-dirs https://github.com/mamba-org/micromamba-releases/releases/latest/download/micromamba-linux-64
    - chmod +x .micromamba/bin/micromamba
    - .micromamba/bin/micromamba create -y -q -n ci -f conda.yml
    - .micromamba/bin/micromamba run -n ci pip install -q git+https://github.com/mccode-dev/mcstas-ess-instr-citool.git
    - .micromamba/bin/micromamba run -n ci mcstas-ess-instr-citool -a runci .
  after_script:
    # Remove the environment (also after failures), so it does not stay on the
    # runner's disk:
    - rm -rf .micromamba

The micromamba files are kept in the hidden .micromamba/ directory, since other files are not allowed at the top level of the repository. The package cache is shared by all jobs on the runner (CONDA_PKGS_DIRS, outside the project directory), so packages are only downloaded and stored once, and the environment mostly consists of hard links into the cache. The environment is removed at the end of each job (after_script), so it does not stay on the runner. Without this, each job put about 5 GB and 180,000 files into its project directory, which filled the disk of a runner (see issue #2).

For faster feedback, in particular for instruments with long tests, the pipeline can be split into two stages: first runcifast (examples with only 100 neutrons, stopping at the first failure), and only if that passes the full runci (see issue #3). The common setup is then shared by both jobs:

stages:
  - quick
  - full

.citool:
  tags:
    - python311  # as used by other ESS instrument repositories
  variables:
    MAMBA_ROOT_PREFIX: "$CI_PROJECT_DIR/.micromamba"
  before_script:
    - export CONDA_PKGS_DIRS="$HOME/.cache/mcstas-ess-instr-citool/conda-pkgs"
    - curl -Ls -o .micromamba/bin/micromamba --create-dirs https://github.com/mamba-org/micromamba-releases/releases/latest/download/micromamba-linux-64
    - chmod +x .micromamba/bin/micromamba
    - .micromamba/bin/micromamba create -y -q -n ci -f conda.yml
    - .micromamba/bin/micromamba run -n ci pip install -q git+https://github.com/mccode-dev/mcstas-ess-instr-citool.git
  after_script:
    - rm -rf .micromamba

runcifast:
  extends: .citool
  stage: quick
  script:
    - .micromamba/bin/micromamba run -n ci mcstas-ess-instr-citool -a runcifast .

runci:
  extends: .citool
  stage: full
  script:
    - .micromamba/bin/micromamba run -n ci mcstas-ess-instr-citool -a runci .

The jobs of the second stage only run when those of the first one pass. Each job creates its own environment, which is quick since the packages come from the shared cache.