Still not working. On the WebGUI the sites and datasources tabs show no information other than “Cannot get site lists! Please check that the sen2agri-services are started!”
If I check the service status with
systemctl status sen2agri-services
I get the attached log.
log_services.txt (3.1 KB)
Hello,
Could you provide the content of the table datasource, please?
sudo -u postgres psql sen2agri -c “select * from datasource;” > datasource.txt
And send us the file datasource.txt.
Best regards,
Cosmin
Please find attached the datasource table content. Please note that I have replaced the pwd/credentials with XXX.
We use a local store for the S2 data. I have also tried to delete the datasource entries. It is then automatically rebuild (after sen2agri service restart), but does not contain the information from my service.properties configuration located under
/usr/share/sen2agri/sen2agri-services/config
Maybe this causes the problems. Then how would I properly configure the service.properties and/or the datasource table?
THanks
Oliver
datasource.txt (3.8 KB)
Hello Olivier,
First, you will have to delete the entry with id 16 from the datasource table:
sudo -u postgres psql sen2agri -c “delete from datasource where id = 16;”
This is actually blocking the starting of the application (not sure how it got as duplicate there but we will investigate).
You can also run the following (but not absolutely necessary):
sudo -u postgres psql sen2agri -c “update datasource set enabled = false where satellite_id = 3;”
sudo -u postgres psql sen2agri -c “update datasource set scope = 2 where satellite_id = 1 and name = ‘Amazon Web Services’;”
Normally, no other change should be needed in the services.properties file from /usr/share/sen2agri/sen2agri-services/config directory.
After the restart of the sen2agri-services, there should be no problem (assuming that your /media/raid0_0/sen2agri/eo_data/S2/2016_ESA folders structure respect the specifications from SUM).
Please let me know.
Best regards,
Cosmin
Thanks Cosmin,
I have delete the duplicate entry and after after a restart of the sen2agri-services it seems to work now.
Thanks for the very fast help.
Oliver
One more precision (I missed that on my first response).
I noticed that you have in the datasource table set the fetch_mode = 3 for the “Scientific Data Hub” (ID = 14) for the local repository. This value means that the products are actually copied from your local data source. In the case your local repository is permanently accessible and you have good transfer speed from it, you might want to set this value to 4 (which means symlinks will be created to the products from your local repository). In this case L1C products will not be copied again (so you will gain disk space) but will be accessed directly from your external media via the created symbolic links.
In order to switch to this mode (as your transfer might already started after the successful start of the application) you will have to do:
sudo systemctl stop sen2agri-services
sudo -u postgres psql sen2agri -c “update datasource set fetch_mode = 4 where satellite_id = 1 and name = ‘Scientific Data Hub’;”
Optionally (but recommended for the products already copied):
sudo -u postgres psql sen2agri -c “delete from downloader_history where site_id in (select id from site where short_name = ‘<your_site_short_name>’) and created_timestamp >= ‘2018-09-26’ ";
After all these, restart the sen2agri-services.
I would recommend the optional steps to be done with care, in order to avoid any loss of data (products that are older).
Hope this helps.
Best regards,
Cosmin
Thanks for the hint. I overlooked this. Saves me disk space. 
Hi,
I have also a problem with the download in the new version. All downloads fails. The error message is : Cannot download http://sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/products/2016/12/4/S2A_MSIL1C_20161204T104422_N0204_R008_T31UGR_20161204T104538/metadata.xml: Server returned HTTP response code: 403 for URL: http://sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/products/2016/12/4/S2A_MSIL1C_20161204T104422_N0204_R008_T31UGR_20161204T104538/metadata.xml
Thanks
Hello,
Apparently, your system is still using somehow the AWS as downloading data source but unfortunately AWS introduces requester pays so it does not available anymore for free downloads.
You will have to switch to the SciHub download datasource. Normally, this is done automatically when the system is upgraded to version 1.8.3.
Could you confirm that you have the system upgraded to 1.8.3, please?
In this case, could you provide the database “datasource” table , please?
See above in this thread the command for exporting it in a text file.
Best regards,
Cosmin
Thanks,
Yes I know Amazon is not working. I downloaded the new version this morning and I made a fresh install.
here is a capture of the data source table.
Doing so does not change the setting as shown via the Web GUI Tab “Data sources”. May be something to look at in the future?
Dear Olivier,
Thank you for notifying us about this display issue in the Web GUI Tab “Data sources”. Indeed, seems that the value of the “Fetch mode” combo box remains to the “Overwrite” value instead of selecting the “Symbolic link” value (as it is specified in the database).
We will correct this in the next version.
Best regards,
Cosmin
Hello,
It seems strange that the data source for AWS was not disabled during the upgrade.
Did you encountered any errors during the update of the system?
Is your system displaying in the IHM the version 1.8.3?
Anyway, you should do the following operations:
sudo systemctl stop sen2agri-services
sudo -u postgres psql sen2agri -c "update datasource set enabled = false where satellite_id = 1 and name = ‘Amazon Web Services’; "
sudo -u postgres psql sen2agri -c "update datasource set scope=3 where satellite_id = 1 and name = ‘Scientific Data Hub’; "
sudo systemctl start sen2agri-services
This should normally solve your issue.
Please let me know.
Best regards,
Cosmin
Thanks,
I made your modification on the datasource table and I created a new site. The error in "status_reason’ is now “Operation timed out”
For precision, i didn’t make an update of the previous version but a new install (on a machine where sen2agri have never been installed). I didn’t have any problem for the installation.
Hello,
Seems that when fresh install is made, the AWS is still enabled by default for download and not SciHub.
Sorry for this inconvenience, it will be corrected in the next version.
Nevertheless, we also encountered the “Operation timed out” from SciHub due to some problems in ESA internal network. Apparently, depending from the point where you are, you might be routed to some servers that have problems and hence you can get “Operation timeout” or “Gateway timeout” errors.
We experienced situations when from one server the same request was working in terms of mili-seconds while from another server, located in another place, we encountered timeout.
In this case, you can try to increase the key “network.connexions.timeout” from the file
/usr/share/sen2agri/sen2agri-services/config/services.properties
to a greater value (for example 60 or even more, let’s say 300 => that is 5 minutes)
network.connexions.timeout = 300
After that, you will have to restart the sen2agri-services.
Please let me know.
Best regards,
Cosmin
Oh it is working ! One download have started.
Thanks for your help !
I have a problem with my local storage that I can not explain. I have use the local storage previously and it worked fine. The data folders are structured according to the SUM. No once I set up a new site the journalctl -f output states:
Oct 04 15:54:18 localhost.localdomain start.sh[1422]: 2018-Oct-04 15:54:18.306 WARN [Scientific Data HubSentinel2-1] r.c.t.d.r.s.d.Sentinel2DownloadStrategy - Product link failed
Oct 04 15:54:18 localhost.localdomain start.sh[1422]: 2018-Oct-04 15:54:18.309 INFO [Scientific Data HubSentinel2-1] o.e.s.scheduling.LookupJob - Download Completed [S2A_MSIL1C_20161029T102132_N0204_R065_T32UPU_20161029T102445] - remaining products to complete for site ‘Bfn_sm_2016’ and satellite ‘S2’ = ‘1’
Oct 04 15:54:18 localhost.localdomain start.sh[1422]: 2018-Oct-04 15:54:18.313 WARN [Scientific Data HubSentinel2-1] r.c.t.d.r.s.d.Sentinel2DownloadStrategy - S2A_MSIL1C_20161029T102132_N0204_R065_T32TPT_20161029T102445 not found locally
Oct 04 15:54:18 localhost.localdomain start.sh[1422]: 2018-Oct-04 15:54:18.313 WARN [Scientific Data HubSentinel2-1] r.c.t.d.r.s.d.Sentinel2DownloadStrategy - Product link failed
Oct 04 15:54:18 localhost.localdomain start.sh[1422]: 2018-Oct-04 15:54:18.316 INFO [Scientific Data HubSentinel2-1] o.e.s.scheduling.LookupJob - Download Completed [S2A_MSIL1C_20161029T102132_N0204_R065_T32TPT_20161029T102445] - remaining products to complete for site ‘Bfn_sm_2016’ and satellite ‘S2’ = ‘0’
The status in the downloader_history is set to “3”. The folders under
sen2agri/archive/dwn_def/s2/default/
<site-id>
are all created but empty
What else could I check?
Oliver
Hello,
Could you check also the table “datasource” from the database and see if the location is correctly set in there? Aparently, the datasource is configured for a local location but the products are not found in the configured location.
Best regards,
Cosmin
Hello,
I’m also encountering the same problem. Here’s the status of my sen2agri-services:
[brentf@nirvana-ESSC_MNL ~]$ systemctl status -l sen2agri-services.service
● sen2agri-services.service - Services for Sen2Agri
Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2018-10-22 18:32:50 +08; 4min 22s ago
Main PID: 11455 (start.sh)
Tasks: 56
CGroup: /system.slice/sen2agri-services.service
├─11455 /bin/bash /usr/share/sen2agri/sen2agri-services/bin/start.sh
└─11467 java -cp ../modules/*:../lib/* org.esa.sen2agri.ServicesStartup
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: Caused by: ro.cs.tao.datasource.QueryException: The request was not successful. Reason: Not Found
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.getCount(SciHubDataQuery.java:158)
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: at org.esa.sen2agri.services.internal.DownloadServiceImpl.count(DownloadServiceImpl.java:119)
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: at org.esa.sen2agri.scheduling.LookupJob.lambda$lookupAndDownload$1(LookupJob.java:167)
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: at java.lang.Thread.run(Thread.java:748)
Oct 22 18:37:05 nirvana-ESSC_MNL start.sh[11455]: 2018-10-22 18:37:05.914 ERROR 11467 --- [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : ro.cs.tao.datasource.QueryException: The request was not successful. Reason: Not FoundCould you please provide the datasource table content? Are you sure it wasn’t a temporary problem? From what I see the query to scihub failed.