Using the Programmatic Python Interface to submit reference files
As an alternative to command-line and web submissions of reference files, there is a programmatic interface that allows users to perform the tasks of reference file submission using Python. It requires crds version 7.3.1 or higher.
Required CRDS Account
Submitting files to CRDS requires obtaining a CRDS account authorizing you to do so. These permissions are managed by CRDS and checked automatically after your MyST identity is verified.
To obtain a CRDS account, please file a ticket with the CRDS Jira project requesting authorization so submit files.
IMPORTANT Your MyST identity is connected to your CRDS permissions via your MyST e-mail. You should specify your MyST e-mail when applying for a CRDS account.
Authenticating
To utilize the programmatic API you are required to have a CRDS account and to authenticate via MyST and auth.mast. MyST will ask you to enter your AD password unless you have already authenticated. auth.mast supports the creation and revocation of secret token values.
The string value of this token must be inserted into the environment variable MAST_API_TOKEN:
import os os.environ['MAST_API_TOKEN'] = 'LongStringObtainedFromAuth.mast.stsciPage'
A typical auth.mast token looks something like:
1b73d91d0db55b1800847529a3b6f91e
IMPORTANT: Since CRDS now uses MyST and auth.mast to identify you, never never directly enter your AD password (MyST password) on the CRDS server or in response to a CRDS prompt.
NOTE: CRDS passwords for existing CRDS file submitters and operators are obsolete and no longer work.
API Description
To use this functionality, import the Submission class from crds:
from crds.submit import Submission
The primary object that a user interacts with is a Submission object. This can be instantiated using the following syntax:
s = Submission(observatory, server, context=None)
where
observatory: String of the observatory to modify. Examples: ‘jwst’, ‘hst’, or ‘roman’
server: The server string to be delivered to. Examples: ‘ops’, ‘test’, or ‘dev’
context: The derive-from context. If None, the current edit context is used. Examples: ‘jwst_775.pmap’
For example, to instantiate a JWST submission to be delivered to the OPS server deriving from the edit context, use:
s = Submission('jwst', 'ops')
The Submission object has several methods, not the least a help()
method that gives the full information on all the fields to be filled:
s.help()
deliverer (str)
---------
Name of deliverer
Who are you?
other_email (str, optional)
-----------
Other e-mail adresses to send notifications
Comma-delimited list (optional)
instrument (str)
----------
Instrument (All submitted files should match this instrument. This
instrument will be locked for your submission exclusively)
Valid choices:
{'fgs', 'miri', 'nircam', 'niriss', 'nirspec', 'system'}
file_type (str)
---------
Type of files (Bias, Dark, etc.)
history_updated (bool, optional)
---------------
Has HISTORY section in the primary header been updated to describe in
detail the reason for delivery and how the files were created?
Valid choices:
{'False', 'True'}
pedigree_updated (bool, optional)
----------------
Has PEDIGREE keyword been checked and updated as necessary?
Valid choices:
{'False', 'True'}
keywords_checked (bool, optional)
----------------
Have REFTYPE and AUTHOR been checked and updated as necessary?
Valid choices:
{'False', 'True'}
descrip_updated (bool, optional)
---------------
Was the DESCRIP keyword updated with a summary of why the files were
updated or created?
Valid choices:
{'False', 'True'}
useafter_updated (bool, optional)
----------------
Has the USEAFTER keyword been checked, and if necessary, updated?
Valid choices:
{'False', 'True'}
useafter_matches (str)
----------------
If the reference files are replacing previous versions, do the new
USEAFTER dates exactly match the old ones?
N/A for ETC Files Only
Valid choices:
{'N/A', 'No', 'Yes'}
compliance_verified (str)
-------------------
Verification for compliance complete (fits, json, asdf compliant,
certify, etc. or N/A)
optional
Valid choices:
{'N/A', 'No', 'Yes'}
ingest_files (bool, optional)
------------
Should the affected files be reprocessed?
Valid choices:
{'False', 'True'}
etc_delivery (bool, optional)
------------
Should the files be submitted to the ETC?
Valid choices:
{'False', 'True'}
jwst_etc (bool, optional)
--------
Are these JWST ETC files?
Valid choices:
{'False', 'True'}
calpipe_version (str)
---------------
Files run through the current version of the calibration software
being used by the pipeline or PYSYNPHOT and ETC (yes/no and version
number)
replacement_files (bool, optional)
-----------------
Are any files replacing old reference files (deliveries can be a mix
of files that are or are not replacing old files) (yes/no)
Valid choices:
{'False', 'True'}
old_reference_files (str, optional)
-------------------
If yes, list them here
replacing_badfiles (str)
------------------
If the files being replaced are bad (resulting in scientifically
invalid results), and should not be used with any data, please
indicate this here
... or crashing the system.
Valid choices:
{'N/A', 'No', 'Yes'}
jira_issue (str, optional)
----------
Any JIRA issues filed in regard to the references being delivered
(e.g. "REDCAT-25")
Comma-delimited list (optional)
table_rows_changed (str, optional)
------------------
If files are tables, please indicate exactly which rows have changed
modes_affected (str)
--------------
Please indicate which modes (e.g. all the STIS, FUVMAMA, E140L modes)
are affected by the changes in the files
change_level (str)
------------
Degree that new files are expected to impact science results.
Valid choices:
{'TRIVIAL', 'SEVERE', 'MODERATE'}
correctness_testing (str)
-------------------
Description of how the files were tested for correctness
additional_considerations (str, optional)
-------------------------
Additional considerations
description (str)
-----------
Information about file changes and expected impacts, include
instrument and type. Formatting note: only alphanumeric, periods,
commas, dashes, and underscores are allowed
The fields of the submission object can be filled using (key, value) assignments as is done with Python dictionaries:
s['deliverer'] = 'Scarlet Feline'
s['other_email'] = 'redcat@stsci.edu'
s['instrument'] = 'miri'
The value of s[‘instrument’] is limited to the set of valid instruments for the observatory specified in the constructor for s. If you try to set s[‘instrument’] to something other than one of these instruments, an exception will occur. Similarly, if you try to set s[‘chicken’], or some other keyword not in the set of allowed keywords, and exception will occur:
s['file_type'] = 'DARK'
s['history_updated'] = True
s['pedigree_updated'] = True
s['keywords_checked'] = True
s['descrip_updated'] = True
s['useafter_updated'] = True
s['useafter_matches'] = True
s['compliance_verified'] = 'N/A'
s['ingest_files'] = False
s['etc_delivery'] = False
s['jwst_etc'] = False
s['calpipe_version'] = '0.9.1'
s['replacement_files'] = False
#s['old_reference_files'] = ''
s['replacing_badfiles'] = 'N/A'
#s['jira_issue'] = ''
s['table_rows_changed'] = 'All rows'
s['modes_affected'] = 'All MIRI observations starting 2017-Jan-03'
s['correctness_testing'] = 'DARK was run on all MIRI data and verified '
#s['additional_considerations'] = ''
s['change_level'] = 'MODERATE'
s['description'] = 'Updating MIRI DARK starting 2017-Jan-03.'
The assignments that are commented out are unnecessary as the default values are empty strings. To add files to the submission, use the add_file() method:
s.add_file('miri_dark_file.fits')
You can also remove files:
s.remove_file('miri_dark_file.fits')
And finally, when the files to be submitted have been added and the fields of the Submission have been filled in, the Submission can be submitted:
result = s.submit()
2019-04-24 12:24:50,823 - CRDS - INFO - =============================== setting up ===============================
2019-04-24 12:24:51,038 - CRDS - INFO - Symbolic context 'jwst-edit' resolves to 'jwst_0511.pmap'
2019-04-24 12:24:51,038 - CRDS - INFO - ########################################
2019-04-24 12:24:51,038 - CRDS - INFO - Certifying './miri_dark_test.fits' (1/1) as 'FITS' relative to context 'jwst_0511.pmap'
2019-04-24 12:24:55,166 - CRDS - INFO - FITS file 'miri_dark_test.fits' conforms to FITS standards.
2019-04-24 12:24:56,219 - CRDS - WARNING - Missing suggested keyword 'META.MODEL_TYPE [DATAMODL]'
2019-04-24 12:24:56,298 - CRDS - INFO - [0] DETECTOR MIRIMAGE
2019-04-24 12:24:56,298 - CRDS - INFO - [0] EXP_TYPE MIR_IMAGE
2019-04-24 12:24:56,302 - CRDS - INFO - META.AUTHOR [AUTHOR] = 'JANE MORRISON'
2019-04-24 12:24:56,302 - CRDS - INFO - META.DESCRIPTION [DESCRIP] = 'MIRI Dark Correction for MIRI'
2019-04-24 12:24:56,302 - CRDS - INFO - META.EXPOSURE.GROUPGAP [GROUPGAP] = 0
2019-04-24 12:24:56,303 - CRDS - INFO - META.EXPOSURE.NFRAMES [NFRAMES] = 1
2019-04-24 12:24:56,303 - CRDS - INFO - META.EXPOSURE.NGROUPS [NGROUPS] = 200
2019-04-24 12:24:56,303 - CRDS - INFO - META.EXPOSURE.READPATT [READPATT] = 'FAST'
2019-04-24 12:24:56,303 - CRDS - INFO - META.EXPOSURE.TYPE [EXP_TYPE] = 'MIR_IMAGE'
2019-04-24 12:24:56,303 - CRDS - INFO - META.HISTORY [HISTORY] = 'SEE MIRI-TN-00007-UA-Mean-Dark.PDF for details\nFit was done correcting for nonlinearity and RSCD Correction\nOne solution rejected first 10 frames and last frame\nSecond solution corrected all the frames for nonlinearity and RSCD\nBecause of drifting zero points, first solution was used to find frame v\nframe value at time = 0 was subtracted off all the data\nresulting residuals were binned and average\n a linear fits was performed on the residuals to extrapolate the dark re\n the number of frames used in the fit is defined by NGVALID\n The mean dark residuals were used for frames with valid data, out side\nDATA USED: to create dark CV3 IMG_RAD_01, IMG_RAD_13\nDOCUMENT: MIRI-TR-00007-UA-Dark_06.00.pdf\nSOFTWARE: idl code create_dark_CDP6.pro merge_int_CDP6\nDIFFERENCES: darks now have two integrations\nCreated from: MiriDarkReferenceModel'
2019-04-24 12:24:56,303 - CRDS - INFO - META.INSTRUMENT.BAND [BAND] = 'UNDEFINED'
2019-04-24 12:24:56,303 - CRDS - INFO - META.INSTRUMENT.CHANNEL [CHANNEL] = 'UNDEFINED'
2019-04-24 12:24:56,303 - CRDS - INFO - META.INSTRUMENT.CORONAGRAPH [CORONMSK] = 'UNDEFINED'
2019-04-24 12:24:56,303 - CRDS - INFO - META.INSTRUMENT.DETECTOR [DETECTOR] = 'MIRIMAGE'
2019-04-24 12:24:56,303 - CRDS - INFO - META.INSTRUMENT.FILTER [FILTER] = 'ANY'
2019-04-24 12:24:56,303 - CRDS - INFO - META.INSTRUMENT.GRATING [GRATING] = 'UNDEFINED'
2019-04-24 12:24:56,303 - CRDS - INFO - META.INSTRUMENT.NAME [INSTRUME] = 'MIRI'
2019-04-24 12:24:56,303 - CRDS - INFO - META.INSTRUMENT.PUPIL [PUPIL] = 'UNDEFINED'
2019-04-24 12:24:56,303 - CRDS - INFO - META.MODEL_TYPE [DATAMODL] = 'UNDEFINED'
2019-04-24 12:24:56,303 - CRDS - INFO - META.PEDIGREE [PEDIGREE] = 'GROUND'
2019-04-24 12:24:56,303 - CRDS - INFO - META.REFTYPE [REFTYPE] = 'DARK'
2019-04-24 12:24:56,303 - CRDS - INFO - META.SUBARRAY.FASTAXIS [FASTAXIS] = 1
2019-04-24 12:24:56,303 - CRDS - INFO - META.SUBARRAY.NAME [SUBARRAY] = 'FULL'
2019-04-24 12:24:56,303 - CRDS - INFO - META.SUBARRAY.SLOWAXIS [SLOWAXIS] = 2
2019-04-24 12:24:56,304 - CRDS - INFO - META.SUBARRAY.XSIZE [SUBSIZE1] = 1032
2019-04-24 12:24:56,304 - CRDS - INFO - META.SUBARRAY.XSTART [SUBSTRT1] = 1
2019-04-24 12:24:56,304 - CRDS - INFO - META.SUBARRAY.YSIZE [SUBSIZE2] = 1024
2019-04-24 12:24:56,304 - CRDS - INFO - META.SUBARRAY.YSTART [SUBSTRT2] = 1
2019-04-24 12:24:56,304 - CRDS - INFO - META.TELESCOPE [TELESCOP] = 'JWST'
2019-04-24 12:24:56,304 - CRDS - INFO - META.USEAFTER [USEAFTER] = '2015-08-02T00:00:00'
2019-04-24 12:24:56,304 - CRDS - INFO - Running fitsverify.
2019-04-24 12:24:56,315 - CRDS - INFO - >>
2019-04-24 12:24:56,315 - CRDS - INFO - >> fitsverify 4.18 (CFITSIO V3.440)
2019-04-24 12:24:56,315 - CRDS - INFO - >> --------------------------------
2019-04-24 12:24:56,315 - CRDS - INFO - >>
2019-04-24 12:24:56,315 - CRDS - INFO - >>
2019-04-24 12:24:56,315 - CRDS - INFO - >> File: ./miri_dark_test.fits
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> 7 Header-Data Units in this file.
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> =================== HDU 1: Primary Array ===================
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> 57 header keywords
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> Null data array; NAXIS = 0
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> =================== HDU 2: Image Exten. ====================
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> 15 header keywords
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> SCI 32-bit floating point pixels, 4 axes (1032 x 1024 x 200 x 2),
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> =================== HDU 3: Image Exten. ====================
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> 15 header keywords
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> ERR 32-bit floating point pixels, 4 axes (1032 x 1024 x 200 x 2),
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,316 - CRDS - INFO - >> =================== HDU 4: Image Exten. ====================
2019-04-24 12:24:56,316 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> 16 header keywords
2019-04-24 12:24:56,317 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> DQ 32-bit integer pixels, 4 axes (1032 x 1024 x 1 x 2),
2019-04-24 12:24:56,317 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> =================== HDU 5: BINARY Table ====================
2019-04-24 12:24:56,317 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> 21 header keywords
2019-04-24 12:24:56,317 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> DQ_DEF (4 columns x 2 rows)
2019-04-24 12:24:56,317 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> Col# Name (Units) Format
2019-04-24 12:24:56,317 - CRDS - INFO - >> 1 BIT J
2019-04-24 12:24:56,317 - CRDS - INFO - >> 2 VALUE J
2019-04-24 12:24:56,317 - CRDS - INFO - >> 3 NAME 40A
2019-04-24 12:24:56,317 - CRDS - INFO - >> 4 DESCRIPTION 80A
2019-04-24 12:24:56,317 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> =================== HDU 6: Image Exten. ====================
2019-04-24 12:24:56,317 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> 8 header keywords
2019-04-24 12:24:56,317 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> METADATA 8-bit integer pixels, 1 axes (1605),
2019-04-24 12:24:56,317 - CRDS - INFO - >>
2019-04-24 12:24:56,317 - CRDS - INFO - >> =================== HDU 7: Image Exten. ====================
2019-04-24 12:24:56,318 - CRDS - INFO - >>
2019-04-24 12:24:56,318 - CRDS - INFO - >> 9 header keywords
2019-04-24 12:24:56,318 - CRDS - INFO - >>
2019-04-24 12:24:56,318 - CRDS - INFO - >> FITERR 32-bit floating point pixels, 2 axes (1032 x 1024),
2019-04-24 12:24:56,318 - CRDS - INFO - >>
2019-04-24 12:24:56,318 - CRDS - INFO - >> ++++++++++++++++++++++ Error Summary ++++++++++++++++++++++
2019-04-24 12:24:56,318 - CRDS - INFO - >>
2019-04-24 12:24:56,318 - CRDS - INFO - >> HDU# Name (version) Type Warnings Errors
2019-04-24 12:24:56,318 - CRDS - INFO - >> 1 Primary Array 0 0
2019-04-24 12:24:56,318 - CRDS - INFO - >> 2 SCI Image Array 0 0
2019-04-24 12:24:56,318 - CRDS - INFO - >> 3 ERR Image Array 0 0
2019-04-24 12:24:56,318 - CRDS - INFO - >> 4 DQ Image Array 0 0
2019-04-24 12:24:56,318 - CRDS - INFO - >> 5 DQ_DEF Binary Table 0 0
2019-04-24 12:24:56,318 - CRDS - INFO - >> 6 METADATA Image Array 0 0
2019-04-24 12:24:56,318 - CRDS - INFO - >> 7 FITERR Image Array 0 0
2019-04-24 12:24:56,318 - CRDS - INFO - >>
2019-04-24 12:24:56,318 - CRDS - INFO - >> **** Verification found 0 warning(s) and 0 error(s). ****
2019-04-24 12:24:56,318 - CRDS - INFO - Checking JWST datamodels.
2019-04-24 12:24:56,347 - CRDS - WARNING - NoTypeWarning : jwst.datamodels.util : model_type not found. Opening ./miri_dark_test.fits as a ReferenceQuadModel
2019-04-24 12:24:56,396 - CRDS - INFO - ########################################
2019-04-24 12:24:56,436 - CRDS - INFO - Logging in aquiring lock.
2019-04-24 12:24:57,489 - CRDS - INFO - =============================== wipe files ===============================
2019-04-24 12:24:57,613 - CRDS - INFO - Preparing server logging.
2019-04-24 12:24:57,723 - CRDS - INFO - ============================== ingest files ==============================
2019-04-24 12:24:57,724 - CRDS - INFO - Uploading 1 file(s) totalling 3.4 G bytes
2019-04-24 12:24:57,825 - CRDS - INFO - Upload started './miri_dark_test.fits' [ 1 / 1 files ] [ 3.4 G / 3.4 G bytes ]
2019-04-24 12:26:07,683 - CRDS - INFO - ===========================================================================
2019-04-24 12:26:07,683 - CRDS - INFO - Posting web request for '/submission_form/redcat_submit/'
CRDS - INFO - ======= monitoring server on 'e8e0f1d3-45d4-44cd-a2b8-1aa279f5dd33' =======
CRDS - INFO - >> Starting submission processing.
CRDS - INFO - >> Certifying 'miri_dark_test.fits'
CRDS - INFO - >> Resolved old rmap as 'jwst_miri_dark_0033.rmap' based on context 'jwst_0511.pmap'
CRDS - INFO - >> Doing trial insertion of 1 references into 'jwst_miri_dark_0033.rmap'
CRDS - INFO - >> Processing 'miri_dark_test.fits' [1 / 1 files] [ 3.4 G / 3.4 G / 3.4 G bytes]
CRDS - INFO - >> Renaming 'miri_dark_test.fits' --> 'jwst_miri_dark_0075.fits'
CRDS - INFO - >> Linking miri_dark_test.fits --> jwst_miri_dark_0075.fits
CRDS - INFO - >> Adding file 'miri_dark_test.fits' to database.
CRDS - INFO - >> Generating new rmap 'jwst_miri_dark_0034.rmap' from 'jwst_miri_dark_0033.rmap'.
CRDS - INFO - >> Adding file 'jwst_miri_dark_0034.rmap' to database.
CRDS - INFO - >> Certifying 'jwst_miri_dark_0034.rmap'
CRDS - INFO - >> Checking for derivation collisions.
CRDS - INFO - >> Computing file differences.
CRDS - INFO - >> Differencing 'jwst_miri_dark_0033.rmap' vs. 'jwst_miri_dark_0034.rmap'
CRDS - INFO - >> COMPLETED: https://jwst-crds-test.stsci.edu//display_result/f01bdb8b-6d70-46a8-95e1-e2bdb6ce1f9c
CRDS - INFO - ========================= monitoring server done =========================
CRDS - INFO - 0 errors
CRDS - INFO - 2 warnings
CRDS - INFO - 132 infos
CRDS - INFO - STARTED 2019-04-24 12:24:50.81
CRDS - INFO - STOPPED 2019-04-24 12:27:41.93
CRDS - INFO - ELAPSED 0:02:51.11
This will validate the submission by making sure all of the fields that need values have them and by ensuring that there is at least 1 file to submit before performing the submission.
The return value of submit() includes error and warning counts, as well as
a link to the Review/Confirm page for the submission:
result.error_count
result.warning_count
result.ready_url
The open_ready_url() function will attempt to open the Review/Confirm page in
your system’s default browser:
result.open_ready_url()
Note that the page’s confirm, cancel, and force buttons will not be available unless authenticated. If they seem to be missing, try logging in with the Login button at the upper right-hand corner of the page.
Submission
file_mapAttributeAs of
v11.16.12, the submission object includes afile_mapproperty which returns a dictionary of uploaded reference filenames, “old” mappings (keys) paired with automatically renamed references, newly generated mappings (values).s.file_map {'miri_dark_test.fits': 'jwst_miri_dark_0075.fits', 'jwst_miri_dark_0033.rmap': 'jwst_miri_dark_0034.rmap', 'jwst_0511.pmap': 'jwst_0512.pmap', 'jwst_miri_0513.imap': 'jwst_miri_0514.imap'}
Note that before a new context has been set, the new mapping names are hypothetical. If the submission has not yet been confirmed, the same is true of the new reference filenames.