Download not working (latest Sen2Agri installation), privilege issues?

Use of Graphical User Interface
1

Hello,

I’ve successfully uploaded the my site on my Sen2agri GUI after following the advice from my previous post:

image
image1090×382 136 KB

After adding my desired products and seasons, and eventually restart sen2agri-services, downloading of sentinel images still fail. I made sure that I had right permissions in /mnt/upload and /mnt/archive directories.

image
image895×81 16.8 KB

I’ve also chown the root dir / to sen2agri-services to make sure the system will have full privileges.

image
image1342×66 14 KB

Is there something I missed out? @cudroiu @admins @lnicola
By the way, I still have 631GB available to my system. My site has two tiles with six months duration of season.

2

Dear Brent,

Are you running your system on a DIAS or locally?
If you are running it locally, you might have issues due to the availability of data from SciHub.
From what I noticed, you are using a season in 2020 so if you are trying to retrieve data from Scihub on a local machine you will probably face the long term archiving issues introduced by ESA some time ago (only the products in the last month are kept in the online repository).
What are the messages from the sen2agri-services?

Best regards,
Cosmin

3

Hi Cosmin,

I see, that’s why I cannot download any sentinel images in /mnt/archive.
Anyway, here are the error messages I got from my sen2agri-service.

image
image1367×377 104 KB 

Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: at ro.cs.tao.datasource.DataSourceComponent.doFetch(DataSourceComponent.java:430)
Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: at org.esa.sen2agri.services.internal.DownloadServiceImpl.download(DownloadServiceImpl.java:335)
Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: at org.esa.sen2agri.scheduling.LookupJob.lambda$lookupAndDownload$5(LookupJob.java:424)
Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: at org.esa.sen2agri.scheduling.DownloadTask.run(DownloadTask.java:52)
Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: at java.lang.Thread.run(Thread.java:750)
Feb 10 18:23:23 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-10 18:23:23.083 DEBUG [Scientific Data Hub-S2-1] o.e.s.scheduling.LookupJob     - Download completed [S2A_MSIL1C_20200531T021351_N0209_R060_T51NXJ_20200531T045217] - remaining products for {site:'Agus_site',satellite:'S2'}: 0

Cheers,
Brent

4

Hi Brent,

In the messages before the ones above the ones you mentioned, do you have some messages like “The product is not online” or something like this or other messages?

Best regards,
Cosmin

5

Hello @cudroiu

I didn’t encounter any messages that says “The product is not online” or any similar messages.

It just says that downloads were completed but it’s different from what is shown in Sen2Agri GUI.

Regards,
Brent

6

Hi @brentf,

I am asking because I noticed in your logs an exception thrown during download but you did not copied the full stack trace. Could you provide more lines above the ones you provided already, before the “Download completed” message?

Cosmin

7

Hello @cudroiu,

Yes, I found the specific part in the log messages where it says that “XXXX is not online” (at the ending of the shown log below).

Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.273 TRACE [DefaultQuartzScheduler_Worker-4] o.e.s.scheduling.LookupJob     - 75 S2 products were discarded.
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.357 INFO  [DefaultQuartzScheduler_Worker-4] o.e.s.scheduling.LookupJob     - [site 'Agus_site',sensor 'Sentinel2'] Found 123 products for site agus_site and satellite S2
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.360 DEBUG [DefaultQuartzScheduler_Worker-4] o.e.s.scheduling.LookupJob     - Actual products to download for site agus_site and satellite S2: 123
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.360 TRACE [DefaultQuartzScheduler_Worker-4] o.esa.sen2agri.commons.Config  - Requested worker for Scientific Data Hub,S2 (active workers: 0, queued tasks: 0)
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.360 TRACE [Scientific Data Hub-S2-1] o.e.s.scheduling.LookupJob     - A task for 'agus_site-S2' has been dequeued for execution
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.360 INFO  [DefaultQuartzScheduler_Worker-4] o.e.s.scheduling.LookupJob     - Job 'Lookup.Lookup-s2-agus_site' completed
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.360 DEBUG [DefaultQuartzScheduler_Worker-4] o.e.s.services.ScheduleManager - Trigger 'Lookup.Lookup-s2-agus_site' completed with code 'NOOP'
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.374 INFO  [Scientific Data Hub-S2-1] o.e.s.s.ProductDownloadListener - Download started [S2A_MSIL1C_20191004T015651_N0208_R060_T51PXK_20191004T053007]
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.375 TRACE [thread_pool_executor-1] o.e.s.s.i.DownloadServiceImpl  - Received start of download for S2A_MSIL1C_20191004T015651_N0208_R060_T51PXK_20191004T053007
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.375 DEBUG [thread_pool_executor-1] r.c.t.s.p.i.ProgressReportServiceImpl - Started S2A_MSIL1C_20191004T015651_N0208_R060_T51PXK_20191004T053007
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.520 TRACE [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - GET https://scihub.copernicus.eu/dhus/odata/v1/Products('10192669-c641-47ff-9690-23463eb9a257')/Online/$value received status code 200 and content
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.520 TRACE [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Content: false
Feb 11 09:21:06 ESSCGeo-Sen2Agri start.sh[11639]: 2022-Feb-11 09:21:06.520 ERROR [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - java.io.IOException: Product S2A_MSIL1C_20191004T015651_N0208_R060_T51PXK_20191004T053007 is not online

I’m sharing the whole log file for reference.

Much thanks,
Brent

8

Hello all,
I am facing the exact same issue as @brentf, and i do also have a “the product is not online” issue.
Does that globally mean we won’t be able to do local reprocessing and subsequent work based on these datasets ? Are you aware of any workaround for this ?
Thanks in advance,
Best,
J.

9

Hello @jlavarenne ,

I’m still thinking of a workaround on this issue. If we can only download the old images in https://scihub.copernicus.eu/dhus/#/home or from usgs. Maybe we can ingest the images in the system for it to be processed by MAJA, and other higher level products. If not possible, the only way is to purchase the services of these cloud data storage companies.

Regards,
Brent