Sentinel2 /landsat8 product download problem

Hello,
We recently downloaded Sen2Agri v1.8 to observe its capabilities for a national project. We tried to download from sentinel2 and landsat8 products but at this point we cannot get the sentinel product download from our sites. Log messages donot seem to give much information and below is the recent files after the services are restarted. ‘datasources’ table are attached for your reference. We are a newbie in sen2agri operations and would very much appreciate your assistance.

Best regards
sen2agri_Datasources.txt (3.1 KB)
sen2agri_Services_startup.txt (14.3 KB)

Hello,

Please note that datasources are working in the following manner:

  • scope = 1 => this means that the this datasource is only used to query the availability of the products but no download is actually made from here. For example, SciHub is used by default for the querying as this location is the one where the products normally appear first.
  • scope = 2 => this means that this datasource is used only for downloading the products that were previously queried from another datasource. The download datasource can be SciHub, Amazon or a local repository. By default Amazon is used as it is much faster than the SciHub.
  • scope = 3 => this means that the datasource is used both for querying and for downloading (if both operations are supported as for some datasources the querying was not implemented due to the inefficiency). So, for example, the USGS is used both for asking for products and also downloading from there.

From what I see, you have :

  • enabled in the “datasources” only the “query” from the SciHub (scope = 1). This means that there is no datasource configured for downloading. You must :
    • either set to true the enable flag of the the datasource with id = 5 (the Amazon Web Services, with scope 2 and satellite = 1 (S2) )
    • either set to 3 the scope for the already enabled datasource with id = 4 (SciHub for satellite = 1 (S2)). Nevertheless, I would not recommend you due to the limited speed but you can try.

Also, I noticed that the USGS datasource for L8 is disabled. If you want to download also L8 you should enable the datasource with id = 1 (USGS scope = 3).

Please note that these flags should be changed from the sen2agri-services.properties and after that restart the application (and then check if they changed also in the database).

Also, if not working, send me again the “datasources” table content and the content of the “config” table.

Please let me know.

Best regards,
Cosmin

Hello Cosmin,

First of all, thank you very much for taking the time to explain the case in detail.
I assumed that the sen2agri-services.properties file was mainly responsible for the Sentinel/Landsat download operations, however some info in this file does not match the records in the datasources table. Hence should we go mainly with the datasources table instead? sen2agri-services.properties file is attached for your reference.

I followed your instructions and modified the datasources table accordingly yesterday, restarted the sen2agri services and left the system running for a day. This morning I could see about 80 records from scihub in the downloader_history table which were either in aborted or failed status. The journal info seems to freeze after couple of minutes. I tried clearing the dowloader_history table and restarting the services but to no avail. All related files are attached for your reference. Thank you for the help in advance.
Best regards,
Asaf
sen2agri-services.properties.txt (1.8 KB)
sen2agri_downloader_history.txt (27.1 KB)
Sen2Agri_Services_Datasources.txt (3.2 KB)
sen2agri_config.txt (41.6 KB)
Sen2Agri_Services_Journal.txt (32.9 KB)

Hello,

Normally, the changes in the sen2agri-services.properties should reflect automatically in the datasources table with no other change in this table. Nevertheless, as this new implementation is new, there might be situations that were not tested well and some fields might not be updated automatically.
What I noticed in the “sen2agri-services.properties” is that you configured the key:

AWSDataSource.Sentinel2.local_archive_path = /mnt/archive/dwn_def/s2/default

If you set this key in this manner, the system will assume that you already downloaded manually the products from SciHub or other source of products and you have all the needed products in local. But I have doubts that this applies in your case. This normally applies when you have a virtual machine in a cloud like IPT or Amazon and you have access to a clone of the SciHub L1C products repository so you don’t have to download the products again as you already have them in local, as a network share or object storage.

What I suggest is to :

  • Stop the application using :

sudo systemctl stop sen2agri-services

  • Change the sen2agri-services.properties file in order to comment the lines like below:
## AWSDataSource.Sentinel2.local_archive_path=/mnt/archive/dwn_def/s2/default
## AWSDataSource.Sentinel2.fetch_mode=3
  • I would suggest, just to be sure, to update also the corresponding columns in the datasources table using :

sudo -u postgres psql sen2agri -c “update datasource set fetch_mode = 1”
sudo -u postgres psql sen2agri -c “update datasource set local_root = null”

  • Delete the entries from the downloader_history table using the command :

sudo -u postgres psql sen2agri -c “delete from downloader_history”

  • Start de sen2agri-services using:

sudo systemctl start sen2agri-services

Hope this helps.

Best regards,
Cosmin

Hello,

Thank you for detailed explanation.
I managed to resolve the issue somehow like this. I disabled Landsat download from the Sen2agri-services.properties file and the datasource table. I gave full access to everyone to the specified directories on the system and the system started downloading sentinel images from Amazon succesfully.

I read at the forum that there might be a problem with Landsat downloads


and I think this might halt the overall download process.
The download has been going on since yesterday for Sentinel images and I will apply and try your recommendations after the download is complete.

For troubleshooting purposes, also I wanted to access the source code for Sen2agri-services module in Java , hence do you provide the source code for this module?

Thank you again for your time and effort.

Best regards
Asaf

Hello,

Unfortunately, the java code is not yet available on GitHub as it depends on another library developed by us for another ESA project that is also ongoing but for that project was not yet defined the licensing method (it is also open source code but are still discussions if Apache or GPL).
We are working on this and we hope the code will be available soon.

Best regards,
Cosmin

Hello Cosmin,

We would appreciate if you could inform it on the site, when available. Thanks for the good job and your efforts.

Best regards
Asaf

I also have downloading issues I cannot explain.
I upgraded from 181. to 1.8.2 and wanted to reprocess some site for the better L2A Processing.

Therefore I deleted my sites and set them up from scratch (e.g. also deleting all existing products, but kept the locally stored downloads)…

After the 1.8.2 upgrade I restarted the sen2agri.services but the L2A processing is not starting. There are no entries in the downloader_history table for the new site.

The log give me the following:

Jul 27 21:27:29 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:29.988 DEBUG 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : [site ‘bfn_sm_2016’,sensor ‘Sentinel2’] Seasons defined: 2016_summer [2015-10-01,2016-05-31,2016-10-31]
Jul 27 21:27:29 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:29.996 DEBUG 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : [site ‘bfn_sm_2016’,sensor ‘Sentinel2’] Using start date: 2015-10-01T00:00 and end date: 2016-10-31T23:59:59
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:30.002 DEBUG 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : [site ‘bfn_sm_2016’,sensor ‘Sentinel2’] Lookup for new products in range 2015-10-01 00:00:00 - 2016-10-31 23:59:59
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:30.003 INFO 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : [site ‘Bfn_sm_2016’,sensor ‘Sentinel2’] Performing query for interval 2015-10-01T00:00 - 2016-10-31T23:59:59
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: com.vividsolutions.jts.io.ParseException: Expected number but found ‘,’ (line 1)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.parseErrorWithLine(WKTReader.java:427)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.parseErrorExpected(WKTReader.java:421)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.getNextNumber(WKTReader.java:303)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.getPreciseCoordinate(WKTReader.java:258)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.getCoordinates(WKTReader.java:230)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.readLinearRingText(WKTReader.java:549)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.readPolygonText(WKTReader.java:635)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.readGeometryTaggedText(WKTReader.java:481)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.read(WKTReader.java:205)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at com.vividsolutions.jts.io.WKTReader.read(WKTReader.java:174)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.splitMultiPolygon(SciHubDataQuery.java:264)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.buildQueriesParams(SciHubDataQuery.java:187)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.getCount(SciHubDataQuery.java:146)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.services.internal.DownloadServiceImpl.count(DownloadServiceImpl.java:122)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.LookupJob.lambda$lookupAndDownload$1(LookupJob.java:183)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.lang.Thread.run(Thread.java:748)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: java.util.concurrent.ExecutionException: java.lang.NullPointerException
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.FutureTask.report(FutureTask.java:122)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.FutureTask.get(FutureTask.java:192)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.LookupJob.lookupAndDownload(LookupJob.java:184)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.LookupJob.executeImpl(LookupJob.java:109)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.AbstractJob.execute(AbstractJob.java:48)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: Caused by: java.lang.NullPointerException
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.buildQueriesParams(SciHubDataQuery.java:190)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at ro.cs.tao.datasource.remote.scihub.SciHubDataQuery.getCount(SciHubDataQuery.java:146)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.services.internal.DownloadServiceImpl.count(DownloadServiceImpl.java:122)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at org.esa.sen2agri.scheduling.LookupJob.lambda$lookupAndDownload$1(LookupJob.java:183)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: at java.lang.Thread.run(Thread.java:748)
Jul 27 21:27:30 wonderwoman.eftasgmbh.local start.sh[3887]: 2018-07-27 21:27:30.010 ERROR 3892 — [eduler_Worker-3] org.esa.sen2agri.scheduling.LookupJob : java.lang.NullPointerException @ org.esa.sen2agri.scheduling.LookupJob.executeImpl(LookupJob.java:109) < org.quartz.core.JobRunShell.run(JobRunShell.java:202) < java.util.concurrent.FutureTask.get(FutureTask.java:192) < org.esa.sen2agri.scheduling.AbstractJob.execute(AbstractJob.java:48) < org.esa.sen2agri.scheduling.LookupJob.lookupAndDownload(LookupJob.java:184) < java.util.concurrent.FutureTask.report(FutureTask.java:122) < org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

To add I have my S2 images stored locally on a mounted drive. In the new service.properties I set:

AWSDataSource.Sentinel2.local_archive_path=/media/raid0_0/sen2agri/eo_data/S2/2016_ESA
AWSDataSource.Sentinel2.fetch_mode=4

I also tried setting the later to “=3”, but this did not change anything.

Any ideas?

Thanks
Oli

Jus tried to create a new site via the GUI. It is correctyl added there and shown, but if I look in
/media/raid0_0/sen2agri/sen2agri/archive

there is no folder created with the name of my new site.

If I check with
sudo -u postgres psql sen2agri -c “select id, name from site;”

the site is listed.

Hello,
A folder will be created with your site name as soon as L3* or L4* products are produced. Before that, downloaded (L1*) and pre-processed (L2A) products will be respectively located under /archive/dwn_def/ and /archive/maccs_def/.

Nicolas

Hello,

i make a fresh install of 1.8.2 version. I have 2 sites configured, but the sites are downloading and processing just Landsat data. Could you pls help me how to start the download of Sentinel 2 data as well. Attached see my config file and datasource table.datasource.csv.txt (1.8 KB)
sen2agri-services.properties.txt (2.0 KB)

I case you need any additional data pls. let me know.

Thank you in advance
Rgds

Dear Nicolas,

sorry for my late reply (mangaged to get some holiday ;).

The issue still remains, for the L2A processing is not starting, and therefore there are also no folders under /archive/dwn_def and /arcive/maccs_def

I had again a look at the journalctl output and found the following error:

Aug 20 13:19:35 wonderwoman.eftasgmbh.local start.sh[28662]: 2018-08-20 13:19:35.295 ERROR 28667 — [eduler_Worker-2] org.esa.sen2agri.scheduling.LookupJob : java.lang.NullPointerException @ org.esa.sen2agri.scheduling.LookupJob.executeImpl(LookupJob.java:109) < org.quartz.core.JobRunShell.run(JobRunShell.java:202) < java.util.concurrent.FutureTask.get(FutureTask.java:192) < org.esa.sen2agri.scheduling.AbstractJob.execute(AbstractJob.java:48) < org.esa.sen2agri.scheduling.LookupJob.lookupAndDownload(LookupJob.java:184) < java.util.concurrent.FutureTask.report(FutureTask.java:122) < org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Could this cause the issue? And if so, what causes this error?
Regards
Oliver