{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Upload of a product to the gallery (oda api v1.3.5)\n", "\n", "This notebooks documents the functionality for uploading a data product over the data-product gallery.\n", "\n", "Two examples are hereby described: one for creating and uploading an image product and one for creating and uploading a lighte-curve one. For each of these products the functionalities from the `oda_api.ploot_tools` ( [Show and Save Products](Show_and_Save_Products.html)) are used.\n", "\n", "All the possible input parameters, as well as the logic and rules applied, are described.\n", "\n", "*author: Gabriele Barni (gabriele.barni@unige.ch)*" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from oda_api.api import GalleryDispatcherAPI\n", "from oda_api.plot_tools import OdaImage, OdaLightCurve\n", "from oda_api.data_products import BinaryData\n", "import os\n", "\n", "from importlib import reload\n", "\n", "from oda_api.token import discover_token\n", "\n", "from cdci_data_analysis.analysis import plot_tools, catalog\n", "\n", "import logging\n", "logging.getLogger('oda_api').addHandler(logging.StreamHandler())\n", "logging.getLogger().setLevel(logging.INFO)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Connection to the dispatcher**" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "disp=GalleryDispatcherAPI(url='https://www.astro.unige.ch/mmoda/dispatch-data',instrument='mock')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Parameters for a data product\n", "\n", "For the data product we wish to upload over the gallerty, a number of arguments can be passed: these are, for instance, the images we wish to visualize and/or the parameters used for its generation. A comprehensive list of these, with their relative explanation, can be found below. It is worth to mention that such list will grow as more functionalities will be added to the product gallery.\n", "\n", "#### Period of observation\n", "An important parameter that characterize a data product is the **period of observation**: this is defined by a time interval in which it is performed, as there might be several pointing directions and multiple targets. The period of observation for the data product can be indicated in two different ways: by an id or directly specifying a time interval. More explanations will be provided in the list below.\n", "\n", "**The following parameters can be set:**\n", "\n", "* **product_title**: title to assign to the product, in case this is not provided one is automatically built using the name of the source and the type of product\n", "* **product_id**: identifier of a data product assigned by the user, this can be used during the creation of a new data-product, as well as to identify an already existing one and update it with the arguments provided by the user\n", "* **period of observation**: in the gallery, for each data product a period of observation can be assigned in two different ways:\n", " * **observation_id**: by specifying the id of an already present observation (eg 'test observation')\n", " * **T1** and **T2**: by specifying the time range, in particular the value of `T1` and `T2` in the following ISOT format `'2003-03-15T23:27:40.0'`\n", "* **produced_by**: direct link to the notebook used to generated the product\n", "* **gallery_image_path**: path of the generated image and to be uploaded over the gallery\n", "* **fits_file_path**: a list of fits file links used for the generation of the product to upload over the gallery\n", "* **token**: user token\n", "* **insert_new_source**: a boolean value to specify if, in case the sources that are passed as parameters and are not available on the product gallery, will be created and then used for the new data product\n", "* **validate_source**: a boolean value to specify if, in case the sources that are passed as parameters will be validated against an online service. In case the validation fails the source won't be inserted as a parameter for the data product and a warning for the user will be generated (unless this is intentionally specified setting to `True` the boolean parameter **force_insert_not_valid_new_source** described below)\n", "* **force_insert_not_valid_new_source**: a boolean value to specify if, in case the sources that are passed as parameters and its validation fails, those should be in any case provided as a parameter for the data product\n", "* **apply_fields_source_resolution**: a boolean value to specify if, in case only a single source is passed within the parameters and then successfully validated, to apply the parameters values returned from the validation (an example of these parameters are RA and DEC)\n", "* **html_image**: field used to upload an image encapsulated within an html block generated using external tools (e.g. bokeh)\n", "* **kwargs**: keyword arguments representing the parameters values used to generate the product. Amongst them it is important to mention the following\n", " * **instrument**: name of the instrument used for the generated product (e.g. isgri, jemx1)\n", " * **product_type**: type of product generated (e.g. isgri_lc, jemx_image)\n", " * **src_names**: name of a single, or a list of, known sources (eg Crab, Cyg X-1)\n", " * **others**: other parameters used for the product (e.g. RA=25). Not all the parameters are currently supported, but the list of the supported ones will be extended" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Let's get the token**" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "found token in TokenLocation.FILE_HOME your token payload: {\n", " \"email\": \"Gabriele.Barni@unige.ch\",\n", " \"exp\": 1674915001,\n", " \"msdone\": false,\n", " \"mssub\": false,\n", " \"name\": \"gbarni\",\n", " \"roles\": \"authenticated user, administrator, user manager, general, integral-private-qla, magic, unige-hpc-full, public-pool-hpc, antares, sdss, apc, bitp, renku contributor, gallery contributor, job manager, refresh-tokens\",\n", " \"sub\": \"Gabriele.Barni@unige.ch\"\n", "}\n", "token expires in 28.4 h\n" ] } ], "source": [ "token = discover_token()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of an image product\n", "**Let's get a dummy image product**" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "please beware that by default, in a typical setup, oda_api will not output much. To learn how to increase the verbosity, please refer to the documentation: https://oda-api.readthedocs.io/en/latest/user_guide/ScienceWindowList.html?highlight=logging#Let's-get-some-logging . \n", "To disable this message you can pass `.get_product(..., silent=True)`\n", "- waiting for remote response (since 2023-01-27 10:49:14), please wait for https://www.astro.unige.ch/mmoda/dispatch-data/run_analysis\n", "session: SEJIEY4MNR455ZKG job: 59a4d18b65b40a0f\n", "\n", "... query status \u001b[35mprepared\u001b[0m => \u001b[35mdone\u001b[0m\n", "... assigned job id: \u001b[33m59a4d18b65b40a0f\u001b[0m\n", "\u001b[32mquery COMPLETED SUCCESSFULLY (state done)\u001b[0m\n", "query complete: terminating\n" ] } ], "source": [ "T1='2003-03-15T23:27:40'\n", "T2='2023-03-16T00:03:12'\n", "source_name='OAO 1657-415'\n", "E1_keV=28.\n", "E2_keV=50.\n", "RA=50\n", "DEC=45\n", "detection_threshold=5.0\n", "\n", "instrument='isgri'\n", "product_type='isgri_image'\n", "\n", "data_collection=disp.get_product(instrument=instrument,\n", " product=product_type,\n", " T1=T1,\n", " T2=T2,\n", " integral_data_rights=\"public\",\n", " osa_version='OSA11.2',\n", " RA=50,\n", " DEC=45,\n", " max_pointings=1,\n", " detection_threshold=detection_threshold,\n", " radius=15.,\n", " product_type='Dummy',\n", " token=token\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Validation of the source for an image product\n", "\n", "For the generation of the Image product below, a source has been provided as argument. The parameter **validate_source** has been set to `True`, so the provided source will be validated. Once the validation is complete, and this is succesfull, a number of parameters are usually returned (eg `RA` and `DEC`): these will be applied to the data product we are uploading over the gallery by setting the parameter **apply_fields_source_resolution** to `True`, as we are doing in this example. In case the validation is not succesfull we can anyway decide to have our source argument used for our new data product, depending on how we set the parameter **force_insert_not_valid_new_source**, in the example below we will not use the source in case of unsuccessful validation.\n", "\n", "Finally, in case the source of our data product was not already created over the gallery, this can be created by setting to `True` the **insert_new_source** parameter. As for this specifc example, we will not create any new source.\n", "\n", "**Let's upload the data product over the gallery**" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING: FITSFixedWarning: RADECSYS= 'FK5 ' \n", "the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]\n", "WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATEREF to '2000-01-01' from MJDREF.\n", "Set MJD-OBS to 54735.165023 from DATE-OBS.\n", "Set MJD-END to 54735.638125 from DATE-END'. [astropy.wcs.wcs]\n", "/home/gabriele/Workspace/oda_api/oda_api/plot_tools.py:132: MatplotlibDeprecationWarning: You are modifying the state of a globally registered colormap. This has been deprecated since 3.3 and in 3.6, you will not be able to modify a registered colormap in-place. To remove this warning, you can make a copy of the colormap first. cmap = mpl.cm.get_cmap(\"gist_earth\").copy()\n", " self.cs.cmap.set_under('k')\n", "Applying the policy for the product isgri_image\n", "\n", "A policy for the product_type isgri_image could not be applied\n", "\n", "Searching the object OAO 1657-415\n", "\n", " name: OAO 1657-415, \n", "OAO 1657-415 successfully resolved\n", "\n", "Source OAO 1657-415 was successfully validated\n", "\n", "Posting a product on the gallery\n", "Product successfully posted on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/very-nice-title-44\n", "Using the above link you can modify the newly created product in the future.\n", "For example, you will be able to change the instrument as well as the product type.\n", "\n" ] } ], "source": [ "image_product = OdaImage(data_collection)\n", "img_fn = image_product.get_image_for_gallery()\n", "\n", "# generate two dummy fits files\n", "fits_file_fn_1 = image_product.write_fits(file_prefix='first')\n", "fits_file_fn_2 = image_product.write_fits(file_prefix='second')\n", "\n", "# source validation related parameters\n", "insert_new_source=False\n", "force_insert_not_valid_new_source=False\n", "validate_source=True\n", "apply_fields_source_resolution=True\n", "\n", "d = disp.post_data_product_to_gallery(token=token,\n", " RA=55, DEC=46, e1_kev=20, e2_kev=40,\n", " T1=T1, T2=T2,\n", " instrument=instrument, product_type=product_type,\n", " product_title=\"very nice title\",\n", " src_name=source_name,\n", " validate_source=validate_source,\n", " insert_new_source=insert_new_source,\n", " apply_fields_source_resolution=apply_fields_source_resolution,\n", " force_insert_not_valid_new_source=force_insert_not_valid_new_source,\n", " gallery_image_path=img_fn,\n", " fits_file_path=[fits_file_fn_1, fits_file_fn_2]\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of a light curve product" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "please beware that by default, in a typical setup, oda_api will not output much. To learn how to increase the verbosity, please refer to the documentation: https://oda-api.readthedocs.io/en/latest/user_guide/ScienceWindowList.html?highlight=logging#Let's-get-some-logging . \n", "To disable this message you can pass `.get_product(..., silent=True)`\n", "/home/gabriele/Workspace/oda_api/oda_api/api.py:1683: UserWarning: \n", "----------------------------------------------------------------------------\n", "the parameter: detection_threshold is not among valid ones:\n", "['src_name', 'RA', 'DEC', 'T1', 'T_format', 'T2', 'token', 'scw_list', 'selected_catalog', 'radius', 'max_pointings', 'osa_version', 'integral_data_rights', 'E1_keV', 'E2_keV', 'time_bin', 'time_bin_format']\n", "----------------------------------------------------------------------------\n", "\n", " warnings.warn(msg)\n", "- waiting for remote response (since 2023-01-27 10:49:53), please wait for https://www.astro.unige.ch/mmoda/dispatch-data/run_analysis\n", "session: SEJIEY4MNR455ZKG job: 1858c24330e39a04\n", "\n", "... query status \u001b[35mprepared\u001b[0m => \u001b[35mdone\u001b[0m\n", "... assigned job id: \u001b[33m1858c24330e39a04\u001b[0m\n", "\u001b[32mquery COMPLETED SUCCESSFULLY (state done)\u001b[0m\n", "query complete: terminating\n" ] } ], "source": [ "T1='2021-02-01T00:00:00'\n", "T2='2021-02-27T23:59:59'\n", "source_name='OAO 1657-415'\n", "E1_keV=28.\n", "E2_keV=50.\n", "RA=263\n", "DEC=-24.7456\n", "detection_threshold=7.0\n", "\n", "product_type='isgri_lc'\n", "\n", "data_collection=disp.get_product(instrument=instrument,\n", " product=product_type,\n", " T1=T1,\n", " T2=T2,\n", " osa_version='OSA11.2',\n", " src_name=source_name,\n", " RA=RA,\n", " DEC=DEC,\n", " E1_keV=E1_keV,\n", " E2_keV=E2_keV,\n", " integral_data_rights='public',\n", " detection_threshold=detection_threshold,\n", " off_line='False',\n", " radius=8.,\n", " product_type='Dummy'\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Application of a policy for a light curve product\n", "This requires a source to be specified for a product of this kind. If we try to create a light curve data product over the gallery, without specifying the source, an error will be generated, describing the policy applied." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Applying the policy for the product isgri_lc\n", "\n", "The src_name parameter is mandatory for a light-curve product\n", "\n" ] }, { "ename": "UserError", "evalue": "the src_name parameter is mandatory for a light-curve product", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mUserError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m/tmp/ipykernel_340308/3365275367.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mimage_product\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOdaLightCurve\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata_collection\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m d = disp.post_data_product_to_gallery(token=token,\n\u001b[0m\u001b[1;32m 4\u001b[0m \u001b[0mRA\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mRA\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mDEC\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mDEC\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0me1_kev\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mE1_keV\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0me2_kev\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mE2_keV\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mproduct_title\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msource_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/Workspace/oda_api/oda_api/api.py\u001b[0m in \u001b[0;36mpost_data_product_to_gallery\u001b[0;34m(self, product_title, product_id, observation_id, gallery_image_path, fits_file_path, yaml_file_path, token, insert_new_source, validate_source, force_insert_not_valid_new_source, apply_fields_source_resolution, html_image, in_evidence, observation_time_format, **kwargs)\u001b[0m\n\u001b[1;32m 1309\u001b[0m \u001b[0;31m# apply policy for the specific data product\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1310\u001b[0m \u001b[0;31m# use the product_type, if provided, and apply the policy, if applicable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1311\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcheck_gallery_data_product_policy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtoken\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtoken\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1312\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1313\u001b[0m \u001b[0mcopied_kwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/Workspace/oda_api/oda_api/api.py\u001b[0m in \u001b[0;36mcheck_gallery_data_product_policy\u001b[0;34m(self, token, **kwargs)\u001b[0m\n\u001b[1;32m 1576\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'name'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mparents_term_list\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1577\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'check_product_for_gallery'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1578\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcheck_product_for_gallery\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1579\u001b[0m \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"A policy for the product_type {product_type} could not be applied\\n\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1580\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/Workspace/oda_api/oda_api/plot_tools.py\u001b[0m in \u001b[0;36mcheck_product_for_gallery\u001b[0;34m(**kwargs)\u001b[0m\n\u001b[1;32m 658\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;34m'src_name'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 659\u001b[0m \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarning\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'The src_name parameter is mandatory for a light-curve product\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 660\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mapi\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mUserError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'the src_name parameter is mandatory for a light-curve product'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 661\u001b[0m \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Policy for a light-curve product successfully verified\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 662\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mUserError\u001b[0m: the src_name parameter is mandatory for a light-curve product" ] } ], "source": [ "image_product = OdaLightCurve(data_collection)\n", "\n", "d = disp.post_data_product_to_gallery(token=token,\n", " RA=RA, DEC=DEC, e1_kev=E1_keV, e2_kev=E2_keV,\n", " product_title=source_name,\n", " product_type=product_type\n", " )" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Applying the policy for the product isgri_lc\n", "\n", "Policy for a light-curve product successfully verified\n", "\n", "Searching the object OAO 1657-415\n", "\n", " name: OAO 1657-415, \n", "OAO 1657-415 successfully resolved\n", "\n", "Source OAO 1657-415 was successfully validated\n", "\n", "Posting a product on the gallery\n", "Product successfully posted on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/oao-1657-415-12\n", "Using the above link you can modify the newly created product in the future.\n", "For example, you will be able to change the instrument as well as the product type.\n", "\n" ] } ], "source": [ "image_product = OdaLightCurve(data_collection)\n", "img_fn = image_product.get_image_for_gallery( in_source_name=source_name)\n", "\n", "fits_file_fn = image_product.write_fits(source_name=source_name)[0]\n", "\n", "notebook_link='https://github.com/oda-hub/oda_api/blob/master/doc/source/user_guide/UploadToGallery.ipynb'\n", "\n", "insert_new_source=False\n", "force_insert_not_valid_new_source=False\n", "validate_source=True\n", "apply_fields_source_resolution=True\n", " \n", "d = disp.post_data_product_to_gallery(token=token,\n", " RA=RA, DEC=DEC, e1_kev=E1_keV, e2_kev=E2_keV,\n", " product_title=source_name,\n", " gallery_image_path=img_fn,\n", " fits_file_path=[fits_file_fn],\n", " src_name=source_name,\n", " validate_source=validate_source,\n", " insert_new_source=insert_new_source,\n", " force_insert_not_valid_new_source=force_insert_not_valid_new_source,\n", " apply_fields_source_resolution=apply_fields_source_resolution,\n", " instrument=instrument,\n", " product_type=product_type,\n", " produced_by=notebook_link,\n", " observation_id=source_name + '_' + T1 + '_' + T2\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of the update of a data product\n", "**Let's get a dummy product**" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "please beware that by default, in a typical setup, oda_api will not output much. To learn how to increase the verbosity, please refer to the documentation: https://oda-api.readthedocs.io/en/latest/user_guide/ScienceWindowList.html?highlight=logging#Let's-get-some-logging . \n", "To disable this message you can pass `.get_product(..., silent=True)`\n", "- waiting for remote response (since 2023-01-27 10:50:58), please wait for https://www.astro.unige.ch/mmoda/dispatch-data/run_analysis\n", "session: SEJIEY4MNR455ZKG job: 67f6cef5a7214dd6\n", "\n", "... query status \u001b[35mprepared\u001b[0m => \u001b[35mdone\u001b[0m\n", "... assigned job id: \u001b[33m67f6cef5a7214dd6\u001b[0m\n", "\u001b[32mquery COMPLETED SUCCESSFULLY (state done)\u001b[0m\n", "query complete: terminating\n" ] } ], "source": [ "source_name='OAO 1657-415'\n", "\n", "RA=263\n", "DEC=-24.7456\n", "\n", "instrument='isgri'\n", "product_type='isgri_image'\n", "\n", "data_collection=disp.get_product(instrument=instrument,\n", " product=product_type,\n", " osa_version='OSA11.2',\n", " RA=RA,\n", " DEC=DEC,\n", " product_type='Dummy'\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will generate a **product_id**, that will be used as an identifier for the data product over the gallery, anytime we wish to update that data product. For its generation we will use a dedicated functionality that takes as an input a dictionary of parameters and output a fixed length string." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "A product_type has not been provided for the given data product, therefore no policy will be verified\n", "\n", "Posting a product with product_id 996b954bac4a4bf8 on the gallery\n", "List of terms from the group products successfully returned\n", "\n", "We noticed no product type has been specified,\n", "for the instrument isgri, the following products are available:\n", "['isgri_image', 'isgri_spectrum', 'isgri_lc', 'spectral_fit']\n", "Please remember that this can be set at a later stage by editing the newly created data product.\n", "\n", "Product successfully posted on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/oao-1657-415-13\n", "Using the above link you can modify the newly created product in the future.\n", "For example, you will be able to change the instrument as well as the product type.\n", "\n" ] } ], "source": [ "request_job_id = data_collection.request_job_id\n", "\n", "params_dic_product_id = {\n", " 'instrument': instrument,\n", " 'product': product_type,\n", " 'osa_version': 'OSA11.2',\n", " 'RA': RA,\n", " 'DEC': DEC,\n", " 'product_type': 'Dummy'\n", "}\n", "\n", "request_product_id = DispatcherAPI.calculate_param_dict_id(params_dic_product_id)\n", " \n", "d = disp.post_data_product_to_gallery(token=token,\n", " RA=RA, DEC=DEC,\n", " src_name=source_name,\n", " product_id=request_product_id,\n", " product_title=source_name,\n", " instrument=instrument,\n", " )" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING: FITSFixedWarning: RADECSYS= 'FK5 ' \n", "the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]\n", "WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATEREF to '2000-01-01' from MJDREF.\n", "Set MJD-OBS to 54735.165023 from DATE-OBS.\n", "Set MJD-END to 54735.638125 from DATE-END'. [astropy.wcs.wcs]\n", "/home/gabriele/Workspace/oda_api/oda_api/plot_tools.py:132: MatplotlibDeprecationWarning: You are modifying the state of a globally registered colormap. This has been deprecated since 3.3 and in 3.6, you will not be able to modify a registered colormap in-place. To remove this warning, you can make a copy of the colormap first. cmap = mpl.cm.get_cmap(\"gist_earth\").copy()\n", " self.cs.cmap.set_under('k')\n", "A product_type has not been provided for the given data product, therefore no policy will be verified\n", "\n", "Searching the object Orion\n", "\n", " name: Orion, \n", "Orion successfully resolved\n", "\n", "Source Orion was successfully validated\n", "\n", "Searching the object Mrk 421\n", "\n", " name: Mrk 421, \n", "Mrk 421 successfully resolved\n", "\n", "Source Mrk 421 was successfully validated\n", "\n", "Searching the object Crab\n", "\n", " name: Crab, \n", "Crab successfully resolved\n", "\n", "Source Crab was successfully validated\n", "\n", "Posting a product with product_id 996b954bac4a4bf8 on the gallery\n", "List of terms from the group products successfully returned\n", "\n", "We noticed no product type has been specified,\n", "for the instrument isgri, the following products are available:\n", "['isgri_image', 'isgri_spectrum', 'isgri_lc', 'spectral_fit']\n", "Please remember that this can be set at a later stage by editing the newly created data product.\n", "\n", "Product successfully updated on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/oao-1657-415-13\n", "Using the above link you can modify the newly created product in the future.\n", "For example, you will be able to change the instrument as well as the product type.\n", "\n" ] } ], "source": [ "notebook_link='https://github.com/oda-hub/oda_api/blob/master/doc/source/user_guide/UploadToGallery.ipynb'\n", "\n", "insert_new_source=False\n", "force_insert_not_valid_new_source=False\n", "validate_source=True\n", "apply_fields_source_resolution=True\n", "source_names = ['Orion', 'Mrk 421', 'Crab']\n", "\n", "image_product = OdaImage(data_collection)\n", "img_fn = image_product.get_image_for_gallery()\n", "\n", "E1_keV=28.\n", "E2_keV=55.\n", "\n", "d = disp.post_data_product_to_gallery(token=token,\n", " RA=RA, DEC=DEC, e1_kev=E1_keV, e2_kev=E2_keV,\n", " product_title=source_name,\n", " product_id=request_product_id,\n", " src_name=source_names,\n", " validate_source=validate_source,\n", " insert_new_source=insert_new_source,\n", " force_insert_not_valid_new_source=force_insert_not_valid_new_source,\n", " apply_fields_source_resolution=apply_fields_source_resolution,\n", " instrument=instrument,\n", " produced_by=notebook_link,\n", " gallery_image_path=img_fn\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Posting an image as HTML\n", "\n", "In the case an image generated using an external library (eg **bokeh**) and encapsulated within a large piece of html code, is to be added to a data product on the gallery, this can be passed as parameter via the parameter **html_image**.\n", "\n", "Let's update the previously generated product." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "A product_type has not been provided for the given data product, therefore no policy will be verified\n", "\n", "Posting a product with product_id 996b954bac4a4bf8 on the gallery\n", "Product successfully updated on the gallery, at the link http://cdciweb02.isdc.unige.ch/mmoda/gallery/data-products/oao-1657-415-13\n", "Using the above link you can modify the newly created product in the future.\n", "For example, you will be able to change the instrument as well as the product type.\n", "\n", "removing tmp_path_html_folder_path=/tmp/tmpy_2fkwgzgallery_temp_files created for temporary files\n" ] } ], "source": [ "html_image = None\n", "with open('bokeh_html_image.html') as fh:\n", " html_image = fh.read()\n", " \n", "d = disp.post_data_product_to_gallery(token=token,\n", " product_id=request_product_id,\n", " \n", " html_image=html_image\n", " )" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernel_info": { "name": "python2" }, "kernelspec": { "display_name": "oda-api", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.7" }, "nteract": { "version": "0.15.0" } }, "nbformat": 4, "nbformat_minor": 2 }