Automated download do not start

System Installation and Configuration
1

We started an experiment over Azerbaijan with Sen2-Agri system with release 1.8.
After installation procedure we performed site configuration.
However we have some problems with automated execution of data processing workflow.

Corresponding sysctl logs attached.
[sysctl] https://drive.google.com/open?id=1f6TrkOg4NFUdvs0v4HChX_yKsaGALPQy
[sysctl for sen2agri] https://drive.google.com/open?id=1SrvTiMaE75qaVS4KThe5dTLgU9V3Ujn5

2

Hi,

It looks like the Landsat-8 download doesn’t start because it is disabled. For Sentinel, it already tried to download them and either finished, or gave up. I suggest truncating the downloader_history table while monitoring the logs:

sudo -u postgres psql sen2agri -c \
    "truncate table downloader_history;"

@cudroiu might be able to follow up with the steps for enabling the Landsat download.

3

downloader_history table was truncated.
Sysctl after service restart is attached in pdf-format (as the only on appropriate format from available).
sen2agri_sysctl_v2.pdf (53.1 KB)

4

I enabled attaching txt and zip files.

I don’t see anything useful in your log, are you sure it’s complete. Also, what’s the time on the server?

кві 18 14:57:32 ip-10-2-0-220.eu-central-1.compute.internal start.sh[3110]: 2018-04-18 14:57:32.826 INFO 3116 --- [pool-4-thread-1] o.esa.sen2agri.services.ScheduleManager : Scheduled job 'Lookup.Landsat8' (site info changed, next run: 2018-04-18T14:57:42.826)

@cudroiu Any ideas?

5

Hi Andreii,

Could you check if there is any entry in the downloader_history table?
Also, could you provide the values in the datasources table, please?

6

hi,

The output is following

sen2agri=# select * from downloader_history;
id | site_id | satellite_id | product_name | full_path | created_timestamp | status_id | no_of_retries | product_date | orbit_id | status_reason | tiles
----±--------±-------------±-------------±----------±------------------±----------±--------------±-------------±---------±--------------±------
(0 rows)

sen2agri=# select * from datasource;
id | satellite_id | name | scope | username | passwrd | fetch_mode | max_retries | retry_interval_minutes | download_path |
specific_params
| max_connections | local_root | enabled
----±-------------±--------------------±------±-----------±-----------±-----------±------------±-----------------------±--------------------------------±--------

---------------------------------------------------------------------±----------------±-----------±--------
3 | 3 | Scientific Data Hub | 3 | login | password | 1 | 3 | 1440 | /mnt/archive | {“parame
ters”:{“dsParameter”:[{“name”:“platformName”,“type”:“java.lang.String”,“value”:“Sentinel-1”},{“name”:“productType”,“type”:“java.lang.String”,“value”:“SLC”}]}}
| 1 | | t
2 | 2 | Amazon Web Services | 2 | | | 1 | 3 | 1440 | /mnt/archive/dwn_def/l8/default | {“parame
ters”:{“dsParameter”:[{“name”:“cloudcoverpercentage”,“type”:“java.lang.Double”,“value”:“100.0”},{“name”:“collection”,“type”:“java.lang.String”,“value”:“COLLECTION_1”},{“na
me”:“platformName”,“type”:“java.lang.String”,“value”:“Landsat-8”}]}} | 1 | | f
4 | 1 | Scientific Data Hub | 1 | dan1234567 | Delete0312 | 1 | 3 | 1440 | /mnt/archive/dwn_def/s2/default | {“parame
ters”:{“dsParameter”:[{“name”:“cloudcoverpercentage”,“type”:“java.lang.Double”,“value”:“100.0”},{“name”:“platformName”,“type”:“java.lang.String”,“value”:“Sentinel-2”},{“na
me”:“productType”,“type”:“java.lang.String”,“value”:“S2MSI1C”}]}} | 1 | | t
5 | 1 | Amazon Web Services | 2 | | | 1 | 3 | 1440 | /mnt/archive/dwn_def/s2/default | {“parame
ters”:{“dsParameter”:[{“name”:“cloudcoverpercentage”,“type”:“java.lang.Double”,“value”:“100.0”},{“name”:“platformName”,“type”:“java.lang.String”,“value”:“Sentinel-2”}]}}
| 1 | | t
1 | 2 | USGS | 3 | | | 1 | 3 | 1440 | /mnt/archive/dwn_def/l8/default | {“parame
ters”:{“dsParameter”:[{“name”:“cloud_from”,“type”:“java.lang.Double”,“value”:“0.0”},{“name”:“cloud_to”,“type”:“java.lang.Double”,“value”:“100.0”},{“name”:“satellite_name”,
“type”:“java.lang.String”,“value”:“landsat-8”}]}} | 1 | | t
(5 rows)

7

Hi Andrii,

The datasources table seems ok. I notices from the logs that you sent non mail that the L8 download is disabled for the site “azerbaidjan_18” and also the user and password for USGS are not set. Nevertheless, this doesn’t explain why the S2 downloads are not functioning. Could you send me also the information about the season for the site that you are using (“sites” and “season” tables but also the “config” table if possible), please? I assume the shapefile of the “azerbaidjan_18” site is the one that you sent by mail.
I would like to try simulating on our side the context you have and see if we get the same behavior here.

Thanks,
Cosmin

8

Hi, Cosmin
L-8 is disabled.
Scihub account configured put I replaced login and password with dummy login-password pair.

bash-4.2$ psql sen2agri -c “select * from season;”
id | site_id | name | start_date | end_date | mid_date | enabled
----±--------±-------------±-----------±-----------±-----------±--------
1 | 1 | Vegetation17 | 2017-02-01 | 2017-10-31 | 2017-06-30 | t
(1 row)

Config and site are in attachment
site.pdf (15.4 KB)
config.pdf (32.8 KB)

9

Thanks for the infos. I will try simulate the site and the download on our side and I will let you know ASAP about the conclusions.

10

Seems that the problem is the shapefile of the site that contains two areas.
SciHub is returning 0 products when providing such polygon. Could you split the site in two sites, corresponding to the two areas? I guess this will solve your problem.

11

Thank you for help!
At the moment data processing has been started.

12

Hi,

I have similar problem, the 1.8 version stop to download tiles. On the Sentinel web there are for more than one day granules available, but in downloader_history I can not se them. 4 days before the download was working without problems and nothing changes on the system. Any suggestion, how to strat the downloader?

Thank you

Jan Semrad

13

Hello,

Could you provide us some log files for the sen2agri-services service and also some information about the season you are using?
You can restart the downloader using :

sudo systemctl restart sen2agri-services

Please note that the system is querying SciHub to extract the list of tiles and Amazon to download the files. Between SciHub and Amazon there might be some delay in products availability (products on Amazon can appear later than on SciHub) so if you are using a real-time processing it might be a delay in downloading products, even if they are available on SciHub.

Best regards,
Cosmin

14

Hi,

  1. Where (path) can I found logs you need?
  2. Season info attached as screenshot.
    season
    season.PNG699×380 11.5 KB
  3. Normaly the data was available in couple of hours, now is already more than 24 hours. (so I don´t think it is a problem SciHub/Amazon).

Thank you

Jan

15

Dear Jan,

I have been working with Scihub for SAr data - most of the previous day it was offline, unfortunately

Andrii.

16

By the my one :slight_smile:
https://scihub.copernicus.eu/dhus/odata/v1/Products(‘da862640-acac-45c7-969f-7c485119afa8’)/$value

is available.

Jan

17

Hi,

after 2 days the tiles from 24.4. was not recognized nor downloaded and another tiile from the same granule from yesterday was rocognized but not downloaded - in downloader_history is “Could not download or no tiles”. Could you pls tell me where exactly find the log files to find the problem. I need relibale system, that I don´t need to control that every tile is downloaded.

Kind regards

Thank you

Jan

18

Hello,

Could you first try to perform the execute the following command and then, after some time (1-2 hours) check if your products appeared:
NOTE: Please pay attention to the double quotes and single quotes if you are copying the commands from the forum to be the correct ones otherwise the command will fail.

sudo -u postgres psql sen2agri -c “delete from downloader_history where status_id in (3, 4) and product_date > ‘2018-04-01’;”

Could you also please provide me :

  • the shapefile that you use for the site
  • the season that you defined for the site
  • the tiles and the products that you expect to be downloaded but they were not
  • the sen2agri-services log exported using “sudo journalctl -f -u sen2agri-services --since yesterday”
  • the content of the downloader_history table that you can export using the command:

sudo -u postgres psql sen2agri -c “select * from downloader_history;” > downloader_history.txt

  • the content of the datasource table using the command

sudo -u postgres psql sen2agri -c “select * from datasource;” > datasource.txt

Please send me the created files “downloader_history.txt” and “datasource.txt” in order to make an analysis on our side and see of it is a problem in the system or a normal situation.

Please note that some configurations are needed and there are also external factors that can influence the functioning of the system like the availability of the datasources on which we cannot have control.

Best regards,
Cosmin

19

Hello,

thank you for information, I am collecting all needed information. But could you provide me some e-mail address where to send it, because in the exported data are sensitive data like login etc. So I don´t want to put the files on forum.

Kind regards

Jan

20

Hello,

You can send them to cosmin@c-s.ro.

Best regards,
Cosmin