Best References Basics
======================
The primary function of CRDS is to assign the names of calibration reference files required to calibrate datasets to their metadata headers.
Operating on site using the CRDS cache at */grp/crds/cache* by default, CRDS "just works" for bestrefs with no extra configuration required.
.. tabs::
.. group-tab:: HST
CRDS provides the *crds bestrefs* program for updating dataset headers for HST with the current best references. Running bestrefs for HST is accomplished via:
.. code-block:: bash
$ crds bestrefs --files dataset*.fits --update-bestrefs
This command updates the FITS headers of the files specified by `dataset*.fits` with the names of the latest best reference files known to */grp/crds/cache*.
.. group-tab:: JWST
For JWST calibrations, CRDS is fully integrated with the CAL code and operates transparently as a consequence of running pipelines:
.. code-block:: bash
$ strun calwebb_sloper.cfg dataset.fits
The above command will transparently update the reference files specified in the metadata of `dataset.fits`.
.. group-tab:: ROMAN
For Roman calibrations, CRDS is fully integrated with the CAL code and operates transparently as a consequence of running pipelines:
.. code-block:: bash
$ strun roman_elp dataset.asdf
The above command will transparently update the reference files specified in the metadata of `dataset.asdf`.
Default Onsite Use
------------------
The CRDS default configuration permits CRDS to operate onsite with no explicit environment settings.
By default, CRDS operates using */grp/crds/cache* with no connection to any CRDS
server.
Files and settings in */grp/crds/cache* define the references that CRDS will
assign to a given dataset.
Offsite and Pipeline Use
------------------------
CRDS can be configured to operate from private/local CRDS caches. See the instructions below for setting `CRDS_PATH` and `CRDS_SERVER_URL`.
A private cache reduces the level of network i/o required for offsite use as
well as eliminating constant dependence on CRDS web servers required to run a
pipeline. A private cache can also contain writable files suitable for
experimentation.
- Onsite pipelines use private caches to reduce file system contention.
- Offsite pipelines use private caches to achieve more independence from STScI.
Setup for Offsite Use
---------------------
CRDS has been designed to (optionally) automatically fetch and cache references
you need to process your datasets to a personal CRDS cache. You can create a
small personal cache of rules and references supporting only the datasets you
care about:
.. tabs::
.. group-tab:: HST
To fetch the references required to process some FITS datasets:
.. code-block:: bash
$ export CRDS_SERVER_URL=https://hst-crds.stsci.edu # or similar
$ export CRDS_PATH=${HOME}/crds_cache
$ crds bestrefs --files dataset*.fits --sync-references=1 --update-bestrefs
.. group-tab:: JWST
CRDS is directly integrated with the calibration step code and will automatically download rules and references. The only extra setup needed is to define `CRDS_PATH` and `CRDS_SERVER_URL` appropriately prior to *strun*:
.. code-block:: bash
$ export CRDS_SERVER_URL=https://jwst-crds.stsci.edu # or similar
$ export CRDS_PATH=${HOME}/crds_cache
$ strun calwebb_sloper.cfg dataset.fits
.. group-tab:: ROMAN
CRDS is directly integrated with the calibration step code and will automatically download rules and references. The only extra setup needed is to define `CRDS_PATH` and `CRDS_SERVER_URL` appropriately prior to *strun*:
.. code-block:: bash
$ export CRDS_SERVER_URL=https://roman-crds.stsci.edu # or similar
$ export CRDS_PATH=${HOME}/crds_cache
$ strun roman_elp dataset.asdf
Overriding the Default Context
------------------------------
It's possible to use past or future/experimental CRDS contexts rather than the pipeline's default latest context as follows:
.. tabs::
.. group-tab:: HST
To fetch the references required to process some FITS datasets:
.. code-block:: bash
$ crds bestrefs --files dataset*.fits --update-bestrefs --new-context hst_0001.pmap
.. group-tab:: JWST
By setting an environment variable:
.. code-block:: bash
$ export CRDS_CONTEXT=jwst_0001.pmap
$ strun calwebb_sloper.cfg dataset.fits
.. group-tab:: ROMAN
By setting an environment variable:
.. code-block:: bash
$ export CRDS_CONTEXT=roman_0001.pmap
$ strun roman_elp dataset.asdf
Bestrefs by Dataset ID
----------------------
Ensure the appropriate CRDS environment variables are set:
.. code-block: bash
$ export CRDS_SERVER_URL=https://roman-crds-test.stsci.edu
$ export CRDS_PATH=path/to/crds_cache
.. tabs::
.. group-tab:: python
Let's say you want to download best references for a dataset with ID: 'R0000101001001001001_01101_0001.WFI16' from context `roman_0042.pmap`.
.. code-block:: python
import crds
from crds.client import api
context = "roman_0042.pmap"
instrument = "wfi"
datasetid = 'R0000101001001001001_01101_0001.WFI16'
refs = api.get_best_references_by_ids(context, [datasetid])
The results are now stored in `refs`
.. code-block:: python
print(refs)
{
'R0000101001001001001_01101_0001.WFI16:R0000101001001001001_01101_0001.WFI16': [
True, {
'area': 'NOT FOUND No match found.',
'dark': 'roman_wfi_dark_0469.asdf',
'distortion': 'roman_wfi_distortion_0016.asdf',
'flat': 'roman_wfi_flat_0231.asdf',
'gain': 'roman_wfi_gain_0142.asdf',
'inverselinearity': 'NOT FOUND No match found.',
'ipc': 'NOT FOUND No match found.',
'linearity': 'roman_wfi_linearity_0195.asdf',
'mask': 'roman_wfi_mask_0066.asdf',
'photom': 'roman_wfi_photom_0054.asdf',
'readnoise': 'roman_wfi_readnoise_0381.asdf',
'refpix': 'NOT FOUND No match found.',
'saturation': 'roman_wfi_saturation_0191.asdf'
}
]
}
Store matches in a list and then download:
.. code-block:: python
reflist = [v for k,v in refs[datasetid][1].items() if v.split('_')[0] == "roman"]
# download them to local crds cache:
api.dump_references(context, reflist)
# or if you only want a specific reference:
api.dump_references(context, ['roman_wfi_saturation_0191.asdf'])
CRDS - INFO - Fetching /home/developer/crds-cache/references/roman/roman_wfi_saturation_0191.asdf 134.2 M bytes (1 / 1 files) (0 / 134.2 M bytes)
# You can also specify which reftypes you want
refs = api.get_best_references_by_ids(
context, [dataset_id], reftypes=["dark","distortion","gain"]
)
View the header information first with `include_headers=True`
.. code-block:: python
refs = api.get_best_references_by_ids(
context,[datasetid], reftypes=["dark"],include_headers=True
)
print(refs)
{'headers': {
'R0000101001001001001_01101_0001.WFI16': {
'ROMAN.META.INSTRUMENT.DETECTOR': 'WFI16',
'ROMAN.META.EXPOSURE.MA_TABLE_NUMBER': '1',
'ROMAN.META.EXPOSURE.TYPE': 'WFI_IMAGE',
'productLevel': '2',
'ROMAN.META.INSTRUMENT.NAME': 'WFI',
'ROMAN.META.EXPOSURE.START_TIME': '2021-01-01T00:00:00.0',
'ROMAN.META.INSTRUMENT.OPTICAL_ELEMENT': 'F158',
'PARAMS_SOURCE': 'http://tlrdmsarc1.stsci.edu:8888/crds',
'PARAMS_DATE': '2023-06-07T10:45:16.150628',
'PARAMS_CTX': 'roman_0042.pmap'
}
}, 'R0000101001001001001_01101_0001.WFI16': [
True, {
'dark': 'roman_wfi_dark_0469.asdf'
}
]}
.. group-tab:: Command-Line
You can specify files to download using crds.sync on the command line:
.. code-block:: bash
$ crds sync --files 'roman_wfi_saturation_0191.asdf'