Sentinel-2 - not downloading ver. 1.8.3

jan.semrad · September 22, 2018, 12:03pm

Hello,

after upgrade to verison 1.8.3 I created new site. The site tile is correctly recongnised (for Sentinel 2 - 33UWR) by the web interface, but the downloader is returning “Query returned 0 products” - see attached log. Pls. could you help me what´s wrong. And how to configure to download the products for Sentinel 2.

Thank you in advance

Rgsd

Jan

zář 22 13:42:19 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:42:19.777 INFO [XNIO-2 task-1] o.s.w.s.DispatcherServlet - FrameworkServlet zář 22 13:42:19 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:42:19.844 INFO [XNIO-2 task-1] o.s.w.s.DispatcherServlet - FrameworkServlet zář 22 13:56:59 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:56:59.276 DEBUG [pool-4-thread-1] o.e.s.services.ScheduleManager - zář 22 13:56:59 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:56:59.306 INFO [pool-4-thread-1] o.e.s.services.ScheduleManager - zář 22 13:56:59 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:56:59.326 INFO [pool-4-thread-1] o.e.s.services.ScheduleManager - zář 22 13:56:59 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:56:59.350 INFO [pool-4-thread-1] o.e.s.services.ScheduleManager - zář 22 13:56:59 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:56:59.405 INFO [pool-4-thread-1] o.e.s.services.ScheduleManager - zář 22 13:56:59 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:56:59.423 INFO [pool-4-thread-1] o.e.s.services.ScheduleManager - zář 22 13:56:59 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:56:59.456 INFO [pool-4-thread-1] o.e.s.services.ScheduleManager - zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.303 INFO [DefaultQuartzScheduler_Worker-5] o.e.s.scheduling.LookupJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.326 INFO [DefaultQuartzScheduler_Worker-6] o.e.s.scheduling.LookupJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.328 DEBUG [DefaultQuartzScheduler_Worker-5] o.e.s.scheduling.LookupJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.344 INFO [DefaultQuartzScheduler_Worker-6] o.e.s.scheduling.LookupJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.406 INFO [DefaultQuartzScheduler_Worker-7] o.e.s.scheduling.RetryJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.423 INFO [DefaultQuartzScheduler_Worker-8] o.e.s.scheduling.RetryJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.428 DEBUG [Scientific Data HubSentinel2-1] o.e.s.scheduling.RetryJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.445 INFO [DefaultQuartzScheduler_Worker-8] o.e.s.scheduling.RetryJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.450 DEBUG [DefaultQuartzScheduler_Worker-5] o.e.s.scheduling.LookupJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.456 INFO [Scientific Data HubSentinel2-1] o.e.s.scheduling.RetryJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.468 DEBUG [DefaultQuartzScheduler_Worker-5] o.e.s.scheduling.LookupJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.478 INFO [DefaultQuartzScheduler_Worker-5] o.e.s.scheduling.LookupJob zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.535 INFO [Scientific Data HubSentinel2-1] r.c.t.d.r.s.SciHubDataQuery - [] zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.536 DEBUG [Scientific Data HubSentinel2-1] r.c.t.d.r.s.SciHubDataQuery zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.949 INFO [Scientific Data HubSentinel2-1] r.c.t.d.r.s.SciHubDataQuery - [] zář 22 13:57:09 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:09.950 DEBUG [Scientific Data HubSentinel2-1] r.c.t.d.r.s.SciHubDataQuery zář 22 13:57:10 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:10.085 INFO [Scientific Data HubSentinel2-1] r.c.t.d.r.s.SciHubDataQuery - [] zář 22 13:57:10 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:10.086 DEBUG [Scientific Data HubSentinel2-1] r.c.t.d.r.s.SciHubDataQuery zář 22 13:57:10 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:10.172 INFO [Scientific Data HubSentinel2-1] r.c.t.d.r.s.SciHubDataQuery zář 22 13:57:10 localhost.localdomain start.sh[4323]: 2018-Zář-22 13:57:10.173 INFO [DefaultQuartzScheduler_Worker-5] o.e.s.scheduling.LookupJob ‘dispatcherServlet’: initialization started
‘dispatcherServlet’: initialization completed in 66 ms
Configuration refreshed from database
Scheduled job ‘Lookup.Sentinel2’ (site info changed, next run: 2018-09-22T13:57:09.302)
Scheduled job ‘Lookup.Landsat8’ (site info changed, next run: 2018-09-22T13:57:09.325)
Sensor ‘Sentinel1’ is globally disabled
Scheduled job ‘Retry.Sentinel2’ (site info changed, next run: 2018-09-22T13:57:09.404)
Scheduled job ‘Retry.Landsat8’ (site info changed, next run: 2018-09-22T13:57:09.421)
Sensor ‘Sentinel1’ is globally disabled
- Job ‘Lookup.Sentinel2’ started
- Job ‘Lookup.Landsat8’ started
- [site ‘slovec_new’,sensor ‘Sentinel2’] Seasons defined: 2018 [2018-04-01,2018-07-01,2018-09-30]
- [site ‘slovec_new’,sensor ‘Landsat8’] Download disabled
- Job ‘Retry.Sentinel2’ started
- Job ‘Retry.Landsat8’ started
- [site ‘slovec_new’,sensor ‘Sentinel2’] Retry failed products
- [site ‘slovec_new’,sensor ‘Landsat8’] Download disabled
- [site ‘slovec_new’,sensor ‘Sentinel2’] Using start date: 2018-01-01T00:00 and end date: 2018-09-22T13:57:09.450
- [site ‘slovec_new’,sensor ‘Sentinel2’] No products to retry
- [site ‘slovec_new’,sensor ‘Sentinel2’] Lookup for new products in range 2018-01-01 00:00:00 - 2018-09-22 13:57:09
- [site ‘Slovec_new’,sensor ‘Sentinel2’] Performing query for interval 2018-01-01T00:00 - 2018-09-22T13:57:09.450
POLYGON((14,9997 49,5548,16,5493 49,5548,16,5493 50,5523,14,9997 50,5523,14,9997 49,5548)) footprint:“Intersects(POLYGON((9997,0 49,0,5493,0 49,0,5493,0 50,0,9997,0 50,0,9997,0 49,0)))” AND beginPosition:[2018-01-01T00:00:00.000Z TO 2018-09-22T13:57:09.000Z] AND endPosition:[2018-01-01T00:00:00.000Z TO 2018-0
- Executing query: https://scihub.copernicus.eu/apihub/search?rows=1&start=0&q=footprint%3A"Intersects(POLYGON((9997%2C0%2049%2C0%2C5493%2C0%2049%2C0%2C5493%2C0%2050%2C0%2C9997%2C0%2050%2C0%2C9997%2C0%2049%2C0)))"%20AND%20beginPosition%3A[2018-01-01T00%3A00%3A00.000Z%20TO%202018-09-22T13%3A57%3A09.000Z]%20AN
POLYGON((14,9997 49,5548,16,5493 49,5548,16,5493 50,5523,14,9997 50,5523,14,9997 49,5548)) footprint:“Intersects(POLYGON((9997,0 49,0,5493,0 49,0,5493,0 50,0,9997,0 50,0,9997,0 49,0)))” AND beginPosition:[2018-01-01T00:00:00.000Z TO 2018-09-22T13:57:09.000Z] AND endPosition:[2018-01-01T00:00:00.000Z TO 2018-0
- Executing query: https://scihub.copernicus.eu/apihub/search?rows=1&start=0&q=footprint%3A"Intersects(POLYGON((9997%2C0%2049%2C0%2C5493%2C0%2049%2C0%2C5493%2C0%2050%2C0%2C9997%2C0%2050%2C0%2C9997%2C0%2049%2C0)))"%20AND%20beginPosition%3A[2018-01-01T00%3A00%3A00.000Z%20TO%202018-09-22T13%3A57%3A09.000Z]%20AN
POLYGON((14,9997 49,5548,16,5493 49,5548,16,5493 50,5523,14,9997 50,5523,14,9997 49,5548)) footprint:“Intersects(POLYGON((9997,0 49,0,5493,0 49,0,5493,0 50,0,9997,0 50,0,9997,0 49,0)))” AND beginPosition:[2018-01-01T00:00:00.000Z TO 2018-09-22T13:57:09.000Z] AND endPosition:[2018-01-01T00:00:00.000Z TO 2018-0
- Executing query: https://scihub.copernicus.eu/apihub/search?rows=50&start=1&q=footprint%3A"Intersects(POLYGON((9997%2C0%2049%2C0%2C5493%2C0%2049%2C0%2C5493%2C0%2050%2C0%2C9997%2C0%2050%2C0%2C9997%2C0%2049%2C0)))"%20AND%20beginPosition%3A[2018-01-01T00%3A00%3A00.000Z%20TO%202018-09-22T13%3A57%3A09.000Z]%20A
- Query returned 0 products
- [site ‘Slovec_new’,sensor ‘Sentinel2’] Query returned 0 products

buckoli · September 24, 2018, 11:57am

I am experiencing the same problem. No S2 downloads. The Scihub query always returns 0 products.

If I configure to use also Landsat USGS, the Landsat downloads are working.

So I guess the S2 download bug from 1.8.2 is still not properly fixed, but I don’t know the solution, I am afraid.

cudroiu · September 24, 2018, 1:26pm

Hello,
Could you upload the log from sen2agri-services as plain text in order to investigate, please?
It seems that you have some locale that make the logs not to be well displayed in forum.
Also, could you provide the shapefile that you use for creating your site, please? The POLYGON((9997,0 49,0,5493,0 49,0,5493,0 50,0,9997,0 50,0,9997,0 49,0)) does not seem to be a valid one.

Best regards,
Cosmin

buckoli · September 24, 2018, 1:55pm

Dear Cosmin,

please find attached the log-file and the Site-shape I use

S2_site_th_pilot_v1.zip (1.0 KB)
log.txt (28.6 KB)

jan.semrad · September 24, 2018, 2:03pm

Dear Cosmin,

see attached log and shape.

Rgds

Janlog.txt (30.9 KB)
Slovec.zip (1.8 KB)

cudroiu · September 25, 2018, 7:33am

Hello,
It seems that the system is using in one case your locales for formatting the numbers.
Thus, it formats the number using “,” as decimal separator instead of “.” (as expected by SciHub).
Would it be possible for you, until we come with a correction in a next version, to use the US locales (“en_US.UTF-8”) instead your current locale?
You can try this using :

localectl set-locale LANG=en_US.UTF-8

Please let me know.

Best regards,
Cosmin

buckoli · September 25, 2018, 9:56am

Not sure what went wrong, but it seems that some units are not longer available. I just tried to re-run the 1.8.3 update.sh script and it gives the following errors:

Failed to stop sen2agri-sentinel-downloader.service: Unit sen2agri-sentinel-downloader.service not loaded.
Failed to stop sen2agri-landsat-downloader.service: Unit sen2agri-landsat-downloader.service not loaded.
Failed to stop sen2agri-sentinel-downloader.timer: Unit sen2agri-sentinel-downloader.timer not loaded.
Failed to stop sen2agri-landsat-downloader.timer: Unit sen2agri-landsat-downloader.timer not loaded.

How could I load these services? I have tried to restart the sen2agri services, but this did not change anything.

THanks
Oliver

cudroiu · September 25, 2018, 11:15am

Hello Olivier,

You should not worry about these failures. These services were removed in version 1.8 and are no longer available. Instead, since version 1.8, sen2agri-services are used for downloading S2 and L8 products.
Since you are upgrading from version 1.8/1.8.x to 1.8.3, these services are not available and hence these failure. We will update in the next version the script to avoid these misleading messages.
You should only check that the sen2agri-services are running ok.

Best regards,
Cosmin

jan.semrad · September 25, 2018, 12:54pm

Thank you,

now it seems to be working. “Query returned 160 products”.

Rgds

Jan

buckoli · September 25, 2018, 3:14pm

Still not working. On the WebGUI the sites and datasources tabs show no information other than “Cannot get site lists! Please check that the sen2agri-services are started!”

If I check the service status with

systemctl status sen2agri-services

I get the attached log.

log_services.txt (3.1 KB)

cudroiu · September 26, 2018, 7:15am

Hello,
Could you provide the content of the table datasource, please?

sudo -u postgres psql sen2agri -c “select * from datasource;” > datasource.txt

And send us the file datasource.txt.

Best regards,
Cosmin

buckoli · September 26, 2018, 7:39am

Please find attached the datasource table content. Please note that I have replaced the pwd/credentials with XXX.
We use a local store for the S2 data. I have also tried to delete the datasource entries. It is then automatically rebuild (after sen2agri service restart), but does not contain the information from my service.properties configuration located under

/usr/share/sen2agri/sen2agri-services/config

Maybe this causes the problems. Then how would I properly configure the service.properties and/or the datasource table?

THanks
Oliver

datasource.txt (3.8 KB)

cudroiu · September 26, 2018, 7:51am

Hello Olivier,

First, you will have to delete the entry with id 16 from the datasource table:

sudo -u postgres psql sen2agri -c “delete from datasource where id = 16;”

This is actually blocking the starting of the application (not sure how it got as duplicate there but we will investigate).
You can also run the following (but not absolutely necessary):

sudo -u postgres psql sen2agri -c “update datasource set enabled = false where satellite_id = 3;”
sudo -u postgres psql sen2agri -c “update datasource set scope = 2 where satellite_id = 1 and name = ‘Amazon Web Services’;”

Normally, no other change should be needed in the services.properties file from /usr/share/sen2agri/sen2agri-services/config directory.
After the restart of the sen2agri-services, there should be no problem (assuming that your /media/raid0_0/sen2agri/eo_data/S2/2016_ESA folders structure respect the specifications from SUM).

Please let me know.

Best regards,
Cosmin

buckoli · September 26, 2018, 8:28am

Thanks Cosmin,

I have delete the duplicate entry and after after a restart of the sen2agri-services it seems to work now.

Thanks for the very fast help.
Oliver

cudroiu · September 26, 2018, 8:46am

One more precision (I missed that on my first response).
I noticed that you have in the datasource table set the fetch_mode = 3 for the “Scientific Data Hub” (ID = 14) for the local repository. This value means that the products are actually copied from your local data source. In the case your local repository is permanently accessible and you have good transfer speed from it, you might want to set this value to 4 (which means symlinks will be created to the products from your local repository). In this case L1C products will not be copied again (so you will gain disk space) but will be accessed directly from your external media via the created symbolic links.
In order to switch to this mode (as your transfer might already started after the successful start of the application) you will have to do:

sudo systemctl stop sen2agri-services
sudo -u postgres psql sen2agri -c “update datasource set fetch_mode = 4 where satellite_id = 1 and name = ‘Scientific Data Hub’;”

Optionally (but recommended for the products already copied):

delete the entries in the downloader_history table for the products already copied

sudo -u postgres psql sen2agri -c “delete from downloader_history where site_id in (select id from site where short_name = ‘<your_site_short_name>’) and created_timestamp >= ‘2018-09-26’ ";

delete from the disk the folders created today from /mnt/archive/dwn_def/s2/default/<your_site_short_name>

After all these, restart the sen2agri-services.

I would recommend the optional steps to be done with care, in order to avoid any loss of data (products that are older).

Hope this helps.

Best regards,
Cosmin

buckoli · September 26, 2018, 9:51am

Thanks for the hint. I overlooked this. Saves me disk space.

resnow92 · September 26, 2018, 10:48am

Hi,
I have also a problem with the download in the new version. All downloads fails. The error message is : Cannot download http://sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/products/2016/12/4/S2A_MSIL1C_20161204T104422_N0204_R008_T31UGR_20161204T104538/metadata.xml: Server returned HTTP response code: 403 for URL: http://sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/products/2016/12/4/S2A_MSIL1C_20161204T104422_N0204_R008_T31UGR_20161204T104538/metadata.xml

Thanks

cudroiu · September 26, 2018, 11:16am

Hello,

Apparently, your system is still using somehow the AWS as downloading data source but unfortunately AWS introduces requester pays so it does not available anymore for free downloads.
You will have to switch to the SciHub download datasource. Normally, this is done automatically when the system is upgraded to version 1.8.3.
Could you confirm that you have the system upgraded to 1.8.3, please?
In this case, could you provide the database “datasource” table , please?
See above in this thread the command for exporting it in a text file.

Best regards,
Cosmin

resnow92 · September 26, 2018, 11:31am

Thanks,
Yes I know Amazon is not working. I downloaded the new version this morning and I made a fresh install.
here is a capture of the data source table.

buckoli · September 26, 2018, 11:37am

Doing so does not change the setting as shown via the Web GUI Tab “Data sources”. May be something to look at in the future?