LephareInformer fails during `prepare` with missing LSST filter files (`total_u.pb`) despite default cache setup

[anat-dsht]

Description

I am running the LephareInformer stage from rail.estimation.algos.lephare in a JupyterLab environment on a remote cluster (CCIN2P3). The pipeline fails during the prepare step due to missing filter files, even though the default cache directories are automatically set and detected.

The error suggests that required LSST filter files (e.g. total_u.pb) are not present in the expected LEPHAREDIR directory.

---

Full notebook (reproducible example)

from rail.utils import catalog_utils
from rail.estimation.algos import lephare 
from rail.core.data import TableHandle

# Load catalog config
catalog_utils.load_yaml("catalogs.yaml")
catalog_utils.apply("cardinal_roman_rubin")

# Create informer
informer = lephare.LephareInformer.make_stage(
    name="inform",
    output_mode="return",
    hdf5_groupname="",
    do_prepare=True,
)

# Remove env variables to rely on default cache behavior
import os 
os.environ.pop("LEPHAREDIR")
os.environ.pop("LEPHAREWORK")

# Load data
train_data = TableHandle(
    "train", path="public/pz_challenge_taskset_1_cardinal_training_10yr.hdf5"
)
test_data = TableHandle(
    "test", path="public/pz_challenge_taskset_1_cardinal_test_10yr.hdf5"
)

# Run informer
model = informer.inform(train_data)

---

Console output

LEPHAREDIR is being set to the default cache directory:
/pbs/home/a/adesouhant/.cache/lephare/data
More than 1Gb may be written there.

LEPHAREWORK is being set to the default cache directory:
/pbs/home/a/adesouhant/.cache/lephare/work

Default work cache is already linked. 
This is linked to the run directory:
/pbs/home/a/adesouhant/.cache/lephare/runs/20260415T192106

Then later:

User defined LEPHAREDIR is set. Code runs depend on all required
auxiliary data being present at /pbs/home/a/adesouhant/.cache/lephare/data.
User defined LEPHAREWORK is set. All intermediate files will be written to:
/pbs/home/a/adesouhant/.cache/lephare/runs/inform

---

Error

ValueError: Can't open filter file 
/pbs/home/a/adesouhant/.cache/lephare/data/filt/lsst/total_u.pb

Traceback points to:

• lephare.prepare
• FilterSvc.from_keymap

---

Observations

• The cache directory is correctly created and detected

• However, the directory:

  /pbs/home/a/adesouhant/.cache/lephare/data/filt/lsst/
  

  does not contain the required .pb files

• do_prepare=True does not seem to generate these filter files automatically

• Removing LEPHAREDIR and LEPHAREWORK does not fix the issue (they are reset internally)

---

Expected behavior

Either:

1. Required auxiliary data (filters, templates, etc.) should be automatically downloaded/initialized
   or
2. A clear error message should indicate that manual setup is required

---

Actual behavior

The pipeline fails with a low-level error:

ValueError: Can't open filter file ...

without guidance on how to obtain or generate the missing files.

---

Questions

1. Is there a recommended way to install/download LePhare auxiliary data when using RAIL?
2. Should LSST filter files (total_u.pb, etc.) be generated by prepare, or provided beforehand?
3. Is there a script or helper function to initialize $LEPHAREDIR properly?
4. Is this expected behavior when using catalog_utils.apply("cardinal_roman_rubin")?

---

Environment

• Platform: CCIN2P3 cluster (remote JupyterLab)

• Python: 3.11 (conda env)

• Packages:

  • rail
  • lephare

---

Additional note

The issue might be related to missing initialization of LePhare data (filters/templates), rather than a runtime bug in RAIL itself.

---

Thanks for your help!

[johannct]

Can you specify how you installe RAIL and what there is in the catalogs.yaml file? If I am not mistaken the LSST files are shared in RAIL ecosystem, so that rail_lephare expect them to be provided by the RAIL ecosystem. I cannot reproduce the error as I do not have the catalogs.yaml file. You can find the lsst filters in the lephare-data package or Rubin throughput package : https://github.com/lephare-photoz/lephare-data and https://github.com/lsst/throughputs/tree/main

[anat-dsht]

I installed RAIL via pip install pz-rail-lephare.
Here is the content of the catalogs.yaml file:

---

# Information about a filter bands
# For each band this is just a name and a reddening
# parameter: a_env.   

# The name is used for lookup and should be unique.
# It should match the name of the associated filter file.
#
# These are typically found in:
# rail_base/src/rail/examples_data/estimation_data/data/FILTER
#
# The a_env parameter can be computed from filter 
# curves using code in `rail_astro_tools`:
#
# https://github.com/LSSTDESC/rail_astro_tools/blob/main/src/rail/tools/filter_tools.py
Bands:
  - Band:
      name: comcam_u
      a_env: 4.81
  - Band:
      name: comcam_g
      a_env: 3.64
  - Band:
      name: comcam_r
      a_env: 2.70
  - Band:
      name: comcam_i
      a_env: 2.06
  - Band:
      name: comcam_z
      a_env: 1.58
  - Band:
      name: comcam_y
      a_env: 1.31
  - Band:
      name: DC2LSST_u
      a_env: 4.81
  - Band:
      name: DC2LSST_g
      a_env: 3.64
  - Band:
      name: DC2LSST_r
      a_env: 2.70
  - Band:
      name: DC2LSST_i
      a_env: 2.06
  - Band:
      name: DC2LSST_z
      a_env: 1.58
  - Band:
      name: DC2LSST_y
      a_env: 1.31
  - Band:
      name: euclid_H
      a_env: 0.47
  - Band:
      name: euclid_J
      a_env: 0.76
  - Band:
      name: euclid_Y
      a_env: 1.10
  - Band:
      name: euclid_vis
      a_env: 2.24
  - Band:
      name: hsc_g
      a_env: 3.64
  - Band:
      name: hsc_r
      a_env: 2.70
  - Band:
      name: hsc_i
      a_env: 2.06
  - Band:
      name: hsc_z
      a_env: 1.58
  - Band:
      name: hsc_y
      a_env: 1.31
  - Band:
      name: roman_K213
      a_env: 0.37
  - Band:
      name: roman_F184
      a_env: 0.47
  - Band:
      name: roman_H158
      a_env: 0.60
  - Band:
      name: roman_W146
      a_env: 0.68
  - Band:
      name: roman_J129
      a_env: 0.83
  - Band:
      name: roman_Y106
      a_env: 1.14
  - Band:
      name: roman_Z087
      a_env: 1.57
# These are just copied from the comcam values and should be updated
  - Band:
      name: DECam_g
      a_env: 3.64
  - Band:
      name: DECam_r
      a_env: 2.70
  - Band:
      name: DECam_i
      a_env: 2.06
  - Band:
      name: DECam_z
      a_env: 1.58
  - Band:
      name: DECam_y
      a_env: 1.31

---
Thanks for your help

[johannct]

Ok can you try to run rail_lephare/examples/small_demo.ipynb ? If it works, then you probably can find the proper way to get the auxiliary data you need from lephare-data

[raphaelshirley]

Hi sorry late getting to this. this is because you need to download the files using

from rail.estimation.algos.lephare import lsst_default_config
import lephare as lp
lp.data_retrieval.get_auxiliary_data(keymap=config)

Should we run this by default after checking if all files are present? Or at least write a more informative warning in a try except loop? We don't want to redownload files every time it is run

[raphaelshirley]

We could add a param to the informer stage download_auxiliary_data perhaps. With default True?