Download and MACCS problems for Sentinel 2A and B

brentf · April 18, 2018, 10:17am

Hi,

Upon upgrading to v 1.8, I found out that there were no downloaded sentinel images in my disk. But they are showed as downloaded in the downloader_historytable:

                                 product_name                                        | status_id |      product_date      |                                                           full_path                                              
          
-------------------------------------------------------------------------------------+-----------+------------------------+------------------------------------------------------------------------------------------------------------------
 S2B_MSIL1C_20180417T021339_N0206_R060_T51NYJ_20180417T034328.SAFE                   |         2 | 2018-04-17 02:13:39+08 | https://scihub.copernicus.eu/apihub/odata/v1/Products('c2de60e0-b78e-45b7-8b4b-03f69f8b4a10')/$value
 S2B_MSIL1C_20180417T021339_N0206_R060_T51PYK_20180417T034328.SAFE                   |         2 | 2018-04-17 02:13:39+08 | https://scihub.copernicus.eu/apihub/odata/v1/Products('4e636f04-573c-45ba-b8f5-02a01d6e27a9')/$value
 S2B_MSIL1C_20180417T021339_N0206_R060_T51PXK_20180417T034328.SAFE                   |         2 | 2018-04-17 02:13:39+08 | https://scihub.copernicus.eu/apihub/odata/v1/Products('8cb098e5-0588-488c-9a51-91bddbdedf32')/$value
 S2B_MSIL1C_20180417T021339_N0206_R060_T51NXJ_20180417T034328.SAFE                   |         2 | 2018-04-17 02:13:39+08 | https://scihub.copernicus.eu/apihub/odata/v1/Products('5c867213-f48e-4773-b7dc-24f4e6f64fdb')/$value
 S2A_MSIL1C_20180412T021351_N0206_R060_T51PYK_20180412T034726.SAFE                   |         2 | 2018-04-12 02:13:51+08 | https://scihub.copernicus.eu/apihub/odata/v1/Products('20b9664e-4f7c-459b-9312-9b66ad5515b1')/$value
 S2A_MSIL1C_20180412T021351_N0206_R060_T51NXJ_20180412T034726.SAFE                   |         2 | 2018-04-12 02:13:51+08 | https://scihub.copernicus.eu/apihub/odata/v1/Products('c112ce18-a1fa-4df0-a61f-1e3fa98c13b1')/$value
 S2A_MSIL1C_20180412T021351_N0206_R060_T51PXK_20180412T034726.SAFE                   |         2 | 2018-04-12 02:13:51+08 | https://scihub.copernicus.eu/apihub/odata/v1/Products('4135638b-cb1c-4c75-a78a-a6a4462cd7ad')/$value
 S2A_MSIL1C_20180412T021351_N0206_R060_T51NYJ_20180412T034726.SAFE                   |         2 | 2018-04-12 02:13:51+08 | https://scihub.copernicus.eu/apihub/odata/v1/Products('45f707a2-0b6c-44a8-a9f3-4fa2a2db9f67')/$value

I suppose this is the same reason sen2agri-services and MACCS doesn’t work for both Landsat and Sentinel images.

[brentf@ESSCGeo-Sen2Agri PH_BUK_LC_PI_20180413_perennials_only]$ systemctl status -l sen2agri-services
● sen2agri-services.service - Services for Sen2Agri
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-04-18 16:07:01 +08; 2h 5min ago
 Main PID: 7573 (start.sh)
   CGroup: /system.slice/sen2agri-services.service
           ├─7573 /bin/bash /usr/share/sen2agri/sen2agri-services/bin/start.sh
           └─7584 java -cp ../modules/*:../lib/* org.esa.sen2agri.ServicesStartup

Apr 18 18:12:05 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:05.129  INFO 7584 --- [pool-4-thread-1] o.esa.sen2agri.services.ScheduleManager  : Scheduled job 'Lookup.Landsat8' (site info changed, next run: 2018-04-18T18:12:15.129)
Apr 18 18:12:05 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:05.131  INFO 7584 --- [pool-4-thread-1] o.esa.sen2agri.services.ScheduleManager  : Scheduled job 'Retry.Sentinel2' (site info changed, next run: 2018-04-18T18:12:15.130)
Apr 18 18:12:05 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:05.133  INFO 7584 --- [pool-4-thread-1] o.esa.sen2agri.services.ScheduleManager  : Sensor 'Sentinel1' is globally disabled
Apr 18 18:12:05 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:05.134  INFO 7584 --- [pool-4-thread-1] o.esa.sen2agri.services.ScheduleManager  : Scheduled job 'Retry.Landsat8' (site info changed, next run: 2018-04-18T18:12:15.134)
Apr 18 18:12:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:15.127  INFO 7584 --- [eduler_Worker-9] org.esa.sen2agri.scheduling.LookupJob    : Job 'Lookup.Sentinel2' started
Apr 18 18:12:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:15.127  INFO 7584 --- [eduler_Worker-5] org.esa.sen2agri.scheduling.LookupJob    : Job 'Lookup.Landsat8' started
Apr 18 18:12:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:15.130  INFO 7584 --- [eduler_Worker-8] org.esa.sen2agri.scheduling.RetryJob     : Job 'Retry.Sentinel2' started
Apr 18 18:12:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:15.132  INFO 7584 --- [icesSentinel2-1] org.esa.sen2agri.scheduling.RetryJob     : [site 'pantaronv2',sensor 'Sentinel2'] No products to retry
Apr 18 18:12:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:15.134  INFO 7584 --- [eduler_Worker-7] org.esa.sen2agri.scheduling.RetryJob     : Job 'Retry.Landsat8' started
Apr 18 18:12:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-18 18:12:15.136  INFO 7584 --- [ USGSLandsat8-1] org.esa.sen2agri.scheduling.RetryJob     : [site 'pantaronv2',sensor 'Landsat8'] No products to retry

[brentf@ESSCGeo-Sen2Agri PH_BUK_LC_PI_20180413_perennials_only]$ systemctl status -l sen2agri-demmaccs
● sen2agri-demmaccs.service - Runs MACCS on L1C products
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-demmaccs.service; static; vendor preset: disabled)
   Active: inactive (dead) since Wed 2018-04-18 18:15:00 +08; 47s ago
  Process: 18651 ExecStart=/usr/share/sen2agri/sen2agri-demmaccs/demmaccs_launcher.py (code=exited, status=0/SUCCESS)
 Main PID: 18651 (code=exited, status=0/SUCCESS)

Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: Checking ROOT for valid symlink = https://scihub.copernicus.eu/apihub/odata/v1/Products('4e636f04-573c-45ba-b8f5-02a01d6e27a9')/$value
Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: 2018-04-18 18:15:00.200240:[18657]:The L1C product name is bad: $value
Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: Found key s2.enabled for all sites with value true
Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: --
Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: Checking ROOT for valid symlink = https://scihub.copernicus.eu/apihub/odata/v1/Products('8cb098e5-0588-488c-9a51-91bddbdedf32')/$value
Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: 2018-04-18 18:15:00.204971:[18657]:The L1C product name is bad: $value
Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: Found key s2.enabled for all sites with value true
Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: --
Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: Checking ROOT for valid symlink = https://scihub.copernicus.eu/apihub/odata/v1/Products('5c867213-f48e-4773-b7dc-24f4e6f64fdb')/$value
Apr 18 18:15:00 ESSCGeo-Sen2Agri demmaccs_launcher.py[18651]: 2018-04-18 18:15:00.209672:[18657]:The L1C product name is bad: $value

cudroiu · April 18, 2018, 3:12pm

Hello,
This seems to be an abnormal situation as the product is marked as downloaded (status 2) but actually the full_path column was not updated to a disk location (it remained the original query product location).
Could you :

send me the content of the “datasource” table?
clear the downloader_history table for the products having status 2 and full_path starting with “https://” and in the same time check the logs of the sen2agri-services with “sudo journalctl -f -u sen2agri-services”?
check if the /mnt/archive/ allows to the user sen2agri-service to write/create folders?

Thanks,
Cosmin

brentf · April 19, 2018, 10:04am

You mean I will delete them in the downloader_history table?

-- Logs begin at Sun 2018-01-14 20:56:16 +08. --
Apr 19 17:57:05 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:05.117  INFO 7584 --- [pool-4-thread-1] o.esa.sen2agri.services.ScheduleManager  : Sensor 'Sentinel1' is globally disabled
Apr 19 17:57:05 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:05.118  INFO 7584 --- [pool-4-thread-1] o.esa.sen2agri.services.ScheduleManager  : Rescheduled job 'Retry.Landsat8' (next run: 2018-04-19T17:57:15.118)
Apr 19 17:57:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:15.111  INFO 7584 --- [eduler_Worker-6] org.esa.sen2agri.scheduling.LookupJob    : Job 'Lookup.Sentinel2' started
Apr 19 17:57:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:15.114  INFO 7584 --- [eduler_Worker-4] org.esa.sen2agri.scheduling.LookupJob    : Job 'Lookup.Landsat8' started
Apr 19 17:57:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:15.115  INFO 7584 --- [eduler_Worker-9] org.esa.sen2agri.scheduling.RetryJob     : Job 'Retry.Sentinel2' started
Apr 19 17:57:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:15.115  WARN 7584 --- [eduler_Worker-6] org.esa.sen2agri.scheduling.LookupJob    : [site 'Pantaronv2',sensor 'Sentinel2'] No season defined or season not started
Apr 19 17:57:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:15.117  WARN 7584 --- [eduler_Worker-4] org.esa.sen2agri.scheduling.LookupJob    : [site 'Pantaronv2',sensor 'Landsat8'] No season defined or season not started
Apr 19 17:57:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:15.118  INFO 7584 --- [eduler_Worker-5] org.esa.sen2agri.scheduling.RetryJob     : Job 'Retry.Landsat8' started
Apr 19 17:57:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:15.119  INFO 7584 --- [icesSentinel2-1] org.esa.sen2agri.scheduling.RetryJob     : [site 'pantaronv2',sensor 'Sentinel2'] No products to retry
Apr 19 17:57:15 ESSCGeo-Sen2Agri start.sh[7573]: 2018-04-19 17:57:15.122  INFO 7584 --- [ USGSLandsat8-1] org.esa.sen2agri.scheduling.RetryJob     : [site 'pantaronv2',sensor 'Landsat8'] No products to retry

drwxrwxrwx   12 sen2agri-service sen2agri-service    4096 Apr 19 16:18 .
drwxr-xr-x    4 sen2agri-service sen2agri-service    4096 Jan 24 17:26 ..
drwxr-xr-x 1310 sen2agri-service sen2agri-service   40960 Apr 18 19:09 demmaccs_tmp
drwxr-xr-x    4 sen2agri-service sen2agri-service    4096 Feb 22 11:37 dwn_def
drwx------    4 sen2agri-service sen2agri-service    4096 Feb 22 11:32 gipp
drwxr-xr-x    3 apache           apache              4096 Apr 19 16:18 insitu
drwxr-xr-x    3 sen2agri-service sen2agri-service    4096 Feb 22 14:34 maccs_def
drwxr-xr-x    3 sen2agri-service sen2agri-service    4096 Apr 11 19:33 orchestrator_temp
drwxr-xr-x    3 sen2agri-service sen2agri-service    4096 Apr 12 02:53 pantaronv2
drwxr-xr-x    2 sen2agri-service sen2agri-service    4096 Apr 11 19:01 reference_data
drwxr-xr-x    2 sen2agri-service sen2agri-service   36864 Feb 22 11:31 srtm
drwxr-xr-x    2 sen2agri-service sen2agri-service 1052672 Feb 22 11:32 swbd

cudroiu · April 19, 2018, 11:26am

Hello,

Yes, you can delete the products having status_id = 2 and full_path starting with “https” from the downloader_history table.
Also, I noticed an error in the log messages :

org.esa.sen2agri.scheduling.LookupJob : [site ‘Pantaronv2’,sensor ‘Sentinel2’] No season defined or season not started

Do you have a season defined and enabled for that site?
Could you send me the content of the site and season tables, please?

brentf · April 19, 2018, 11:41am

Hi,

Now I get it, since I’m extremely wary of my disk space, I frequently turn disable the sites through GUI. I will enable them now and update you about it.

Brent

brentf · April 19, 2018, 12:23pm

Hello again,

Like what you have suggested, I just deleted the entries in downloader_history with full path starting with ‘https…’. And the downloaded files were now in the disk.

product_name                                                       |      product_date      | status_id |                                                  full_path                                                   
-------------------------------------------------------------------+------------------------+-----------+--------------------------------------------------------------------------------------------------------------
 S2B_MSIL1C_20180417T021339_N0206_R060_T51NYJ_20180417T034328.SAFE | 2018-04-17 02:13:39+08 |         2 | /mnt/archive/dwn_def/s2/default/pantaronv2/S2B_MSIL1C_20180417T021339_N0206_R060_T51NYJ_20180417T034328.SAFE
 S2B_MSIL1C_20180417T021339_N0206_R060_T51NXJ_20180417T034328.SAFE | 2018-04-17 02:13:39+08 |         2 | /mnt/archive/dwn_def/s2/default/pantaronv2/S2B_MSIL1C_20180417T021339_N0206_R060_T51NXJ_20180417T034328.SAFE
 S2B_MSIL1C_20180417T021339_N0206_R060_T51PXK_20180417T034328.SAFE | 2018-04-17 02:13:39+08 |         2 | /mnt/archive/dwn_def/s2/default/pantaronv2/S2B_MSIL1C_20180417T021339_N0206_R060_T51PXK_20180417T034328.SAFE
 S2B_MSIL1C_20180417T021339_N0206_R060_T51PYK_20180417T034328.SAFE | 2018-04-17 02:13:39+08 |         2 | /mnt/archive/dwn_def/s2/default/pantaronv2/S2B_MSIL1C_20180417T021339_N0206_R060_T51PYK_20180417T034328.SAFE
(4 rows)

Cheers.

cudroiu · April 19, 2018, 12:29pm

That’s great!
I was just testing using your seasons and products were correctly inserted into the database. Nevertheless, it would be interesting know how those products got in the database with status 2 but without having the full_path updated.
Please let me know if you encounter again that problem and the operations you made (maybe there is still an issue somewhere that occurs only after a specific combination of operations).

Best regards,
Cosmin