Hello,
We recently downloaded Sen2Agri v1.8 to observe its capabilities for a national project. We tried to download from sentinel2 and landsat8 products but at this point we cannot get the sentinel product download from our sites. Log messages donot seem to give much information and below is the recent files after the services are restarted. ‘datasources’ table are attached for your reference. We are a newbie in sen2agri operations and would very much appreciate your assistance.
Please note that datasources are working in the following manner:
scope = 1 => this means that the this datasource is only used to query the availability of the products but no download is actually made from here. For example, SciHub is used by default for the querying as this location is the one where the products normally appear first.
scope = 2 => this means that this datasource is used only for downloading the products that were previously queried from another datasource. The download datasource can be SciHub, Amazon or a local repository. By default Amazon is used as it is much faster than the SciHub.
scope = 3 => this means that the datasource is used both for querying and for downloading (if both operations are supported as for some datasources the querying was not implemented due to the inefficiency). So, for example, the USGS is used both for asking for products and also downloading from there.
From what I see, you have :
enabled in the “datasources” only the “query” from the SciHub (scope = 1). This means that there is no datasource configured for downloading. You must :
either set to true the enable flag of the the datasource with id = 5 (the Amazon Web Services, with scope 2 and satellite = 1 (S2) )
either set to 3 the scope for the already enabled datasource with id = 4 (SciHub for satellite = 1 (S2)). Nevertheless, I would not recommend you due to the limited speed but you can try.
Also, I noticed that the USGS datasource for L8 is disabled. If you want to download also L8 you should enable the datasource with id = 1 (USGS scope = 3).
Please note that these flags should be changed from the sen2agri-services.properties and after that restart the application (and then check if they changed also in the database).
Also, if not working, send me again the “datasources” table content and the content of the “config” table.
First of all, thank you very much for taking the time to explain the case in detail.
I assumed that the sen2agri-services.properties file was mainly responsible for the Sentinel/Landsat download operations, however some info in this file does not match the records in the datasources table. Hence should we go mainly with the datasources table instead? sen2agri-services.properties file is attached for your reference.
I followed your instructions and modified the datasources table accordingly yesterday, restarted the sen2agri services and left the system running for a day. This morning I could see about 80 records from scihub in the downloader_history table which were either in aborted or failed status. The journal info seems to freeze after couple of minutes. I tried clearing the dowloader_history table and restarting the services but to no avail. All related files are attached for your reference. Thank you for the help in advance.
Best regards,
Asaf sen2agri-services.properties.txt (1.8 KB) sen2agri_downloader_history.txt (27.1 KB) Sen2Agri_Services_Datasources.txt (3.2 KB) sen2agri_config.txt (41.6 KB) Sen2Agri_Services_Journal.txt (32.9 KB)
Normally, the changes in the sen2agri-services.properties should reflect automatically in the datasources table with no other change in this table. Nevertheless, as this new implementation is new, there might be situations that were not tested well and some fields might not be updated automatically.
What I noticed in the “sen2agri-services.properties” is that you configured the key:
If you set this key in this manner, the system will assume that you already downloaded manually the products from SciHub or other source of products and you have all the needed products in local. But I have doubts that this applies in your case. This normally applies when you have a virtual machine in a cloud like IPT or Amazon and you have access to a clone of the SciHub L1C products repository so you don’t have to download the products again as you already have them in local, as a network share or object storage.
What I suggest is to :
Stop the application using :
sudo systemctl stop sen2agri-services
Change the sen2agri-services.properties file in order to comment the lines like below:
Thank you for detailed explanation.
I managed to resolve the issue somehow like this. I disabled Landsat download from the Sen2agri-services.properties file and the datasource table. I gave full access to everyone to the specified directories on the system and the system started downloading sentinel images from Amazon succesfully.
I read at the forum that there might be a problem with Landsat downloads
and I think this might halt the overall download process.
The download has been going on since yesterday for Sentinel images and I will apply and try your recommendations after the download is complete.
For troubleshooting purposes, also I wanted to access the source code for Sen2agri-services module in Java , hence do you provide the source code for this module?
Unfortunately, the java code is not yet available on GitHub as it depends on another library developed by us for another ESA project that is also ongoing but for that project was not yet defined the licensing method (it is also open source code but are still discussions if Apache or GPL).
We are working on this and we hope the code will be available soon.
I also have downloading issues I cannot explain.
I upgraded from 181. to 1.8.2 and wanted to reprocess some site for the better L2A Processing.
Therefore I deleted my sites and set them up from scratch (e.g. also deleting all existing products, but kept the locally stored downloads)…
After the 1.8.2 upgrade I restarted the sen2agri.services but the L2A processing is not starting. There are no entries in the downloader_history table for the new site.
The log give me the following:
Jul 27 21:27:29 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:29.988 DEBUG 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : [site ‘bfn_sm_2016’,sensor ‘Sentinel2’] Seasons defined: 2016_summer [2015-10-01,2016-05-31,2016-10-31]
Jul 27 21:27:29 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:29.996 DEBUG 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : [site ‘bfn_sm_2016’,sensor ‘Sentinel2’] Using start date: 2015-10-01T00:00 and end date: 2016-10-31T23:59:59
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:30.002 DEBUG 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : [site ‘bfn_sm_2016’,sensor ‘Sentinel2’] Lookup for new products in range 2015-10-01 00:00:00 - 2016-10-31 23:59:59
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:30.003 INFO 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : [site ‘Bfn_sm_2016’,sensor ‘Sentinel2’] Performing query for interval 2015-10-01T00:00 - 2016-10-31T23:59:59
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: com.vividsolutions.jts.io.ParseException: Expected number but found ‘,’ (line 1)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.parseErrorWithLine(WKTReader.java:427)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.parseErrorExpected(WKTReader.java:421)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.getNextNumber(WKTReader.java:303)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.getPreciseCoordinate(WKTReader.java:258)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.getCoordinates(WKTReader.java:230)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.readLinearRingText(WKTReader.java:549)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.readPolygonText(WKTReader.java:635)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.readGeometryTaggedText(WKTReader.java:481)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.read(WKTReader.java:205)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.read(WKTReader.java:174)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.splitMultiPolygon(SciHubDataQuery.java:264)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.buildQueriesParams(SciHubDataQuery.java:187)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.getCount(SciHubDataQuery.java:146)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.services.internal.DownloadServiceImpl.count(DownloadServiceImpl.java:122)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.LookupJob.lambda$lookupAndDownload$1(LookupJob.java:183)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.lang.Thread.run(Thread.java:748)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: java.util.concurrent.ExecutionException: java.lang.NullPointerException
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.FutureTask.report(FutureTask.java:122)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.FutureTask.get(FutureTask.java:192)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.LookupJob.lookupAndDownload(LookupJob.java:184)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.LookupJob.executeImpl(LookupJob.java:109)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.AbstractJob.execute(AbstractJob.java:48)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: Caused by: java.lang.NullPointerException
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.buildQueriesParams(SciHubDataQuery.java:190)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.getCount(SciHubDataQuery.java:146)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.services.internal.DownloadServiceImpl.count(DownloadServiceImpl.java:122)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.LookupJob.lambda$lookupAndDownload$1(LookupJob.java:183)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.lang.Thread.run(Thread.java:748)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:30.010 ERROR 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : java.lang.NullPointerException @ org.esa.sen2agri.scheduling.LookupJob.executeImpl(LookupJob.java:109) < org.quartz.core.JobRunShell.run(JobRunShell.java:202) < java.util.concurrent.FutureTask.get(FutureTask.java:192) < org.esa.sen2agri.scheduling.AbstractJob.execute(AbstractJob.java:48) < org.esa.sen2agri.scheduling.LookupJob.lookupAndDownload(LookupJob.java:184) < java.util.concurrent.FutureTask.report(FutureTask.java:122) < org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
To add I have my S2 images stored locally on a mounted drive. In the new service.properties I set:
Hello,
A folder will be created with your site name as soon as L3* or L4* products are produced. Before that, downloaded (L1*) and pre-processed (L2A) products will be respectively located under /archive/dwn_def/ and /archive/maccs_def/.
i make a fresh install of 1.8.2 version. I have 2 sites configured, but the sites are downloading and processing just Landsat data. Could you pls help me how to start the download of Sentinel 2 data as well. Attached see my config file and datasource table.datasource.csv.txt (1.8 KB) sen2agri-services.properties.txt (2.0 KB)
I case you need any additional data pls. let me know.