Upload of a product to the gallery (oda api v1.2.15)#
This notebooks documents the functionality for uploading a data product over the data-product gallery.
Two examples are hereby described: one for creating and uploading an image product and one for creating and uploading a lighte-curve one. For each of these products the functionalities from the oda_api.ploot_tools
( Show and Save Products) are used.
All the possible input parameters, as well as the logic and rules applied, are described.
author: Gabriele Barni (gabriele.barni@unige.ch)
[1]:
from oda_api.api import GalleryDispatcherAPI
from oda_api.plot_tools import OdaImage, OdaLightCurve
from oda_api.data_products import BinaryData
import os
from importlib import reload
from oda_api.token import discover_token
from cdci_data_analysis.analysis import plot_tools, catalog
import logging
logging.getLogger('oda_api').addHandler(logging.StreamHandler())
logging.getLogger().setLevel(logging.INFO)
Connection to the dispatcher
[2]:
disp=GalleryDispatcherAPI(url='https://www.astro.unige.ch/mmoda/dispatch-data',instrument='mock')
Parameters for a data product#
For the data product we wish to upload over the gallerty, a number of arguments can be passed: these are, for instance, the images we wish to visualize and/or the parameters used for its generation. A comprehensive list of these, with their relative explanation, can be found below. It is worth to mention that such list will grow as more functionalities will be added to the product gallery.
Period of observation#
An important parameter that characterize a data product is the period of observation: this is defined by a time interval in which it is performed, as there might be several pointing directions and multiple targets. The period of observation for the data product can be indicated in two different ways: by an id or directly specifying a time interval. More explanations will be provided in the list below.
The following parameters can be set:
product_title: title to assign to the product, in case this is not provided one is automatically built using the name of the source and the type of product
product_id: identifier of a data product assigned by the user, this can be used during the creation of a new data-product, as well as to identify an already existing one and update it with the arguments provided by the user
period of observation: in the gallery, for each data product a period of observation can be assigned in two different ways:
observation_id: by specifying the id of an already present observation (eg ‘test observation’)
T1 and T2: by specifying the time range, in particular the value of
T1
andT2
in the following ISOT format'2003-03-15T23:27:40.0'
produced_by: direct link to the notebook used to generated the product
gallery_image_path: path of the generated image and to be uploaded over the gallery
fits_file_path: a list of fits file links used for the generation of the product to upload over the gallery
token: user token
insert_new_source: a boolean value to specify if, in case the sources that are passed as parameters and are not available on the product gallery, will be created and then used for the new data product
validate_source: a boolean value to specify if, in case the sources that are passed as parameters will be validated against an online service. In case the validation fails the source won’t be inserted as a parameter for the data product and a warning for the user will be generated (unless this is intentionally specified setting to
True
the boolean parameter force_insert_not_valid_new_source described below)force_insert_not_valid_new_source: a boolean value to specify if, in case the sources that are passed as parameters and its validation fails, those should be in any case provided as a parameter for the data product
apply_fields_source_resolution: a boolean value to specify if, in case only a single source is passed within the parameters and then successfully validated, to apply the parameters values returned from the validation (an example of these parameters are RA and DEC)
html_image: field used to upload an image encapsulated within an html block generated using external tools (e.g. bokeh)
kwargs: keyword arguments representing the parameters values used to generate the product. Amongst them it is important to mention the following
instrument: name of the instrument used for the generated product (e.g. isgri, jemx1)
product_type: type of product generated (e.g. isgri_lc, jemx_image)
src_names: name of a single, or a list of, known sources (eg Crab, Cyg X-1)
others: other parameters used for the product (e.g. RA=25). Not all the parameters are currently supported, but the list of the supported ones will be extended
Let’s get the token
[3]:
token = discover_token()
found token in TokenLocation.FILE_HOME your token payload: {
"email": "Gabriele.Barni@unige.ch",
"exp": 1674915001,
"msdone": false,
"mssub": false,
"name": "gbarni",
"roles": "authenticated user, administrator, user manager, general, integral-private-qla, magic, unige-hpc-full, public-pool-hpc, antares, sdss, apc, bitp, renku contributor, gallery contributor, job manager, refresh-tokens",
"sub": "Gabriele.Barni@unige.ch"
}
token expires in 28.4 h
Example of an image product#
Let’s get a dummy image product
[6]:
T1='2003-03-15T23:27:40'
T2='2023-03-16T00:03:12'
source_name='OAO 1657-415'
E1_keV=28.
E2_keV=50.
RA=50
DEC=45
detection_threshold=5.0
instrument='isgri'
product_type='isgri_image'
data_collection=disp.get_product(instrument=instrument,
product=product_type,
T1=T1,
T2=T2,
integral_data_rights="public",
osa_version='OSA11.2',
RA=50,
DEC=45,
max_pointings=1,
detection_threshold=detection_threshold,
radius=15.,
product_type='Dummy',
token=token
)
please beware that by default, in a typical setup, oda_api will not output much. To learn how to increase the verbosity, please refer to the documentation: https://oda-api.readthedocs.io/en/latest/user_guide/ScienceWindowList.html?highlight=logging#Let's-get-some-logging .
To disable this message you can pass `.get_product(..., silent=True)`
- waiting for remote response (since 2023-01-27 10:49:14), please wait for https://www.astro.unige.ch/mmoda/dispatch-data/run_analysis
session: SEJIEY4MNR455ZKG job: 59a4d18b65b40a0f
... query status prepared => done
... assigned job id: 59a4d18b65b40a0f
query COMPLETED SUCCESSFULLY (state done)
query complete: terminating
Validation of the source for an image product#
For the generation of the Image product below, a source has been provided as argument. The parameter validate_source has been set to True
, so the provided source will be validated. Once the validation is complete, and this is succesfull, a number of parameters are usually returned (eg RA
and DEC
): these will be applied to the data product we are uploading over the gallery by setting the parameter apply_fields_source_resolution to True
, as we are doing in this example. In
case the validation is not succesfull we can anyway decide to have our source argument used for our new data product, depending on how we set the parameter force_insert_not_valid_new_source, in the example below we will not use the source in case of unsuccessful validation.
Finally, in case the source of our data product was not already created over the gallery, this can be created by setting to True
the insert_new_source parameter. As for this specifc example, we will not create any new source.
Let’s upload the data product over the gallery
[7]:
image_product = OdaImage(data_collection)
img_fn = image_product.get_image_for_gallery()
# generate two dummy fits files
fits_file_fn_1 = image_product.write_fits(file_prefix='first')
fits_file_fn_2 = image_product.write_fits(file_prefix='second')
# source validation related parameters
insert_new_source=False
force_insert_not_valid_new_source=False
validate_source=True
apply_fields_source_resolution=True
d = disp.post_data_product_to_gallery(token=token,
RA=55, DEC=46, e1_kev=20, e2_kev=40,
T1=T1, T2=T2,
instrument=instrument, product_type=product_type,
product_title="very nice title",
src_name=source_name,
validate_source=validate_source,
insert_new_source=insert_new_source,
apply_fields_source_resolution=apply_fields_source_resolution,
force_insert_not_valid_new_source=force_insert_not_valid_new_source,
gallery_image_path=img_fn,
fits_file_path=[fits_file_fn_1, fits_file_fn_2]
)
WARNING: FITSFixedWarning: RADECSYS= 'FK5 '
the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]
WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATEREF to '2000-01-01' from MJDREF.
Set MJD-OBS to 54735.165023 from DATE-OBS.
Set MJD-END to 54735.638125 from DATE-END'. [astropy.wcs.wcs]
/home/gabriele/Workspace/oda_api/oda_api/plot_tools.py:132: MatplotlibDeprecationWarning: You are modifying the state of a globally registered colormap. This has been deprecated since 3.3 and in 3.6, you will not be able to modify a registered colormap in-place. To remove this warning, you can make a copy of the colormap first. cmap = mpl.cm.get_cmap("gist_earth").copy()
self.cs.cmap.set_under('k')
Applying the policy for the product isgri_image
A policy for the product_type isgri_image could not be applied
Searching the object OAO 1657-415
name: OAO 1657-415,
OAO 1657-415 successfully resolved
Source OAO 1657-415 was successfully validated
Posting a product on the gallery
Product successfully posted on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/very-nice-title-44
Using the above link you can modify the newly created product in the future.
For example, you will be able to change the instrument as well as the product type.
Example of a light curve product#
[8]:
T1='2021-02-01T00:00:00'
T2='2021-02-27T23:59:59'
source_name='OAO 1657-415'
E1_keV=28.
E2_keV=50.
RA=263
DEC=-24.7456
detection_threshold=7.0
product_type='isgri_lc'
data_collection=disp.get_product(instrument=instrument,
product=product_type,
T1=T1,
T2=T2,
osa_version='OSA11.2',
src_name=source_name,
RA=RA,
DEC=DEC,
E1_keV=E1_keV,
E2_keV=E2_keV,
integral_data_rights='public',
detection_threshold=detection_threshold,
off_line='False',
radius=8.,
product_type='Dummy'
)
please beware that by default, in a typical setup, oda_api will not output much. To learn how to increase the verbosity, please refer to the documentation: https://oda-api.readthedocs.io/en/latest/user_guide/ScienceWindowList.html?highlight=logging#Let's-get-some-logging .
To disable this message you can pass `.get_product(..., silent=True)`
/home/gabriele/Workspace/oda_api/oda_api/api.py:1683: UserWarning:
----------------------------------------------------------------------------
the parameter: detection_threshold is not among valid ones:
['src_name', 'RA', 'DEC', 'T1', 'T_format', 'T2', 'token', 'scw_list', 'selected_catalog', 'radius', 'max_pointings', 'osa_version', 'integral_data_rights', 'E1_keV', 'E2_keV', 'time_bin', 'time_bin_format']
----------------------------------------------------------------------------
warnings.warn(msg)
- waiting for remote response (since 2023-01-27 10:49:53), please wait for https://www.astro.unige.ch/mmoda/dispatch-data/run_analysis
session: SEJIEY4MNR455ZKG job: 1858c24330e39a04
... query status prepared => done
... assigned job id: 1858c24330e39a04
query COMPLETED SUCCESSFULLY (state done)
query complete: terminating
Application of a policy for a light curve product#
This requires a source to be specified for a product of this kind. If we try to create a light curve data product over the gallery, without specifying the source, an error will be generated, describing the policy applied.
[9]:
image_product = OdaLightCurve(data_collection)
d = disp.post_data_product_to_gallery(token=token,
RA=RA, DEC=DEC, e1_kev=E1_keV, e2_kev=E2_keV,
product_title=source_name,
product_type=product_type
)
Applying the policy for the product isgri_lc
The src_name parameter is mandatory for a light-curve product
---------------------------------------------------------------------------
UserError Traceback (most recent call last)
/tmp/ipykernel_340308/3365275367.py in <module>
1 image_product = OdaLightCurve(data_collection)
2
----> 3 d = disp.post_data_product_to_gallery(token=token,
4 RA=RA, DEC=DEC, e1_kev=E1_keV, e2_kev=E2_keV,
5 product_title=source_name,
~/Workspace/oda_api/oda_api/api.py in post_data_product_to_gallery(self, product_title, product_id, observation_id, gallery_image_path, fits_file_path, yaml_file_path, token, insert_new_source, validate_source, force_insert_not_valid_new_source, apply_fields_source_resolution, html_image, in_evidence, observation_time_format, **kwargs)
1309 # apply policy for the specific data product
1310 # use the product_type, if provided, and apply the policy, if applicable
-> 1311 self.check_gallery_data_product_policy(token=token, **kwargs)
1312
1313 copied_kwargs = kwargs.copy()
~/Workspace/oda_api/oda_api/api.py in check_gallery_data_product_policy(self, token, **kwargs)
1576 and hasattr(c, 'name') and c.name is not None and c.name in parents_term_list \
1577 and hasattr(c, 'check_product_for_gallery'):
-> 1578 return c.check_product_for_gallery(**kwargs)
1579 logger.info(f"A policy for the product_type {product_type} could not be applied\n")
1580 else:
~/Workspace/oda_api/oda_api/plot_tools.py in check_product_for_gallery(**kwargs)
658 if 'src_name' not in kwargs:
659 logger.warning('The src_name parameter is mandatory for a light-curve product\n')
--> 660 raise api.UserError('the src_name parameter is mandatory for a light-curve product')
661 logger.info('Policy for a light-curve product successfully verified\n')
662 return True
UserError: the src_name parameter is mandatory for a light-curve product
[10]:
image_product = OdaLightCurve(data_collection)
img_fn = image_product.get_image_for_gallery( in_source_name=source_name)
fits_file_fn = image_product.write_fits(source_name=source_name)[0]
notebook_link='https://github.com/oda-hub/oda_api/blob/master/doc/source/user_guide/UploadToGallery.ipynb'
insert_new_source=False
force_insert_not_valid_new_source=False
validate_source=True
apply_fields_source_resolution=True
d = disp.post_data_product_to_gallery(token=token,
RA=RA, DEC=DEC, e1_kev=E1_keV, e2_kev=E2_keV,
product_title=source_name,
gallery_image_path=img_fn,
fits_file_path=[fits_file_fn],
src_name=source_name,
validate_source=validate_source,
insert_new_source=insert_new_source,
force_insert_not_valid_new_source=force_insert_not_valid_new_source,
apply_fields_source_resolution=apply_fields_source_resolution,
instrument=instrument,
product_type=product_type,
produced_by=notebook_link,
observation_id=source_name + '_' + T1 + '_' + T2
)
Applying the policy for the product isgri_lc
Policy for a light-curve product successfully verified
Searching the object OAO 1657-415
name: OAO 1657-415,
OAO 1657-415 successfully resolved
Source OAO 1657-415 was successfully validated
Posting a product on the gallery
Product successfully posted on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/oao-1657-415-12
Using the above link you can modify the newly created product in the future.
For example, you will be able to change the instrument as well as the product type.
Example of the update of a data product#
Let’s get a dummy product
[11]:
source_name='OAO 1657-415'
RA=263
DEC=-24.7456
instrument='isgri'
product_type='isgri_image'
data_collection=disp.get_product(instrument=instrument,
product=product_type,
osa_version='OSA11.2',
RA=RA,
DEC=DEC,
product_type='Dummy'
)
please beware that by default, in a typical setup, oda_api will not output much. To learn how to increase the verbosity, please refer to the documentation: https://oda-api.readthedocs.io/en/latest/user_guide/ScienceWindowList.html?highlight=logging#Let's-get-some-logging .
To disable this message you can pass `.get_product(..., silent=True)`
- waiting for remote response (since 2023-01-27 10:50:58), please wait for https://www.astro.unige.ch/mmoda/dispatch-data/run_analysis
session: SEJIEY4MNR455ZKG job: 67f6cef5a7214dd6
... query status prepared => done
... assigned job id: 67f6cef5a7214dd6
query COMPLETED SUCCESSFULLY (state done)
query complete: terminating
We will generate a product_id, that will be used as an identifier for the data product over the gallery, anytime we wish to update that data product. For its generation we will use a dedicated functionality that takes as an input a dictionary of parameters and output a fixed length string.
[12]:
request_job_id = data_collection.request_job_id
params_dic_product_id = {
'instrument': instrument,
'product': product_type,
'osa_version': 'OSA11.2',
'RA': RA,
'DEC': DEC,
'product_type': 'Dummy'
}
request_product_id = DispatcherAPI.calculate_param_dict_id(params_dic_product_id)
d = disp.post_data_product_to_gallery(token=token,
RA=RA, DEC=DEC,
src_name=source_name,
product_id=request_product_id,
product_title=source_name,
instrument=instrument,
)
A product_type has not been provided for the given data product, therefore no policy will be verified
Posting a product with product_id 996b954bac4a4bf8 on the gallery
List of terms from the group products successfully returned
We noticed no product type has been specified,
for the instrument isgri, the following products are available:
['isgri_image', 'isgri_spectrum', 'isgri_lc', 'spectral_fit']
Please remember that this can be set at a later stage by editing the newly created data product.
Product successfully posted on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/oao-1657-415-13
Using the above link you can modify the newly created product in the future.
For example, you will be able to change the instrument as well as the product type.
[13]:
notebook_link='https://github.com/oda-hub/oda_api/blob/master/doc/source/user_guide/UploadToGallery.ipynb'
insert_new_source=False
force_insert_not_valid_new_source=False
validate_source=True
apply_fields_source_resolution=True
source_names = ['Orion', 'Mrk 421', 'Crab']
image_product = OdaImage(data_collection)
img_fn = image_product.get_image_for_gallery()
E1_keV=28.
E2_keV=55.
d = disp.post_data_product_to_gallery(token=token,
RA=RA, DEC=DEC, e1_kev=E1_keV, e2_kev=E2_keV,
product_title=source_name,
product_id=request_product_id,
src_name=source_names,
validate_source=validate_source,
insert_new_source=insert_new_source,
force_insert_not_valid_new_source=force_insert_not_valid_new_source,
apply_fields_source_resolution=apply_fields_source_resolution,
instrument=instrument,
produced_by=notebook_link,
gallery_image_path=img_fn
)
WARNING: FITSFixedWarning: RADECSYS= 'FK5 '
the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]
WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATEREF to '2000-01-01' from MJDREF.
Set MJD-OBS to 54735.165023 from DATE-OBS.
Set MJD-END to 54735.638125 from DATE-END'. [astropy.wcs.wcs]
/home/gabriele/Workspace/oda_api/oda_api/plot_tools.py:132: MatplotlibDeprecationWarning: You are modifying the state of a globally registered colormap. This has been deprecated since 3.3 and in 3.6, you will not be able to modify a registered colormap in-place. To remove this warning, you can make a copy of the colormap first. cmap = mpl.cm.get_cmap("gist_earth").copy()
self.cs.cmap.set_under('k')
A product_type has not been provided for the given data product, therefore no policy will be verified
Searching the object Orion
name: Orion,
Orion successfully resolved
Source Orion was successfully validated
Searching the object Mrk 421
name: Mrk 421,
Mrk 421 successfully resolved
Source Mrk 421 was successfully validated
Searching the object Crab
name: Crab,
Crab successfully resolved
Source Crab was successfully validated
Posting a product with product_id 996b954bac4a4bf8 on the gallery
List of terms from the group products successfully returned
We noticed no product type has been specified,
for the instrument isgri, the following products are available:
['isgri_image', 'isgri_spectrum', 'isgri_lc', 'spectral_fit']
Please remember that this can be set at a later stage by editing the newly created data product.
Product successfully updated on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/oao-1657-415-13
Using the above link you can modify the newly created product in the future.
For example, you will be able to change the instrument as well as the product type.
Posting an image as HTML#
In the case an image generated using an external library (eg bokeh) and encapsulated within a large piece of html code, is to be added to a data product on the gallery, this can be passed as parameter via the parameter html_image.
Let’s update the previously generated product.
[14]:
html_image = None
with open('bokeh_html_image.html') as fh:
html_image = fh.read()
d = disp.post_data_product_to_gallery(token=token,
product_id=request_product_id,
html_image=html_image
)
A product_type has not been provided for the given data product, therefore no policy will be verified
Posting a product with product_id 996b954bac4a4bf8 on the gallery
Product successfully updated on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/oao-1657-415-13
Using the above link you can modify the newly created product in the future.
For example, you will be able to change the instrument as well as the product type.
removing tmp_path_html_folder_path=/tmp/tmpy_2fkwgzgallery_temp_files created for temporary files
[ ]: