Dowloads not working on v2 and v2.1 versions

JoaoUrbano · July 30, 2019, 6:44pm

Hi Guys,

Started with a fresh 2.0 installation with 2 sites, everything was working fine until a few days ago when files stopped downloading.

I have updated to 2.1 but the problem persists, I have checked the forum but no help to my problem.

I am currently getting the following java exceptions when I run systemctl status sen2agri-services

● sen2agri-services.service - Services for Sen2Agri
Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2019-07-30 19:25:02 +01; 13min ago
Main PID: 10363 (start.sh)
Tasks: 100
CGroup: /system.slice/sen2agri-services.service
├─10363 /bin/bash /usr/share/sen2agri/sen2agri-services/bin/start.sh
└─10369 java -cp …/modules/:…/lib/:…/config/* org.esa.sen2agri.ServicesStartup

Jul 30 19:35:25 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:25.282 WARN [DefaultQuartzScheduler_Worker-5] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect’s prepareConnection flag to false
Jul 30 19:35:27 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:27.898 INFO [Scientific Data Hub-S2-3] r.c.t.d.r.s.SciHubDataQuery - Query returned 0 products
Jul 30 19:35:28 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:28.451 INFO [Scientific Data Hub-S2-3] r.c.t.d.r.s.SciHubDataQuery - Query returned 0 products
Jul 30 19:35:28 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:28.451 INFO [DefaultQuartzScheduler_Worker-8] o.e.s.scheduling.LookupJob - [site ‘Chaouia’,sensor ‘Sentinel2’] Found 0 products for site chaouia
Jul 30 19:35:28 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:28.459 WARN [DefaultQuartzScheduler_Worker-8] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect’s prepareConnection flag to false
Jul 30 19:35:29 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:29.097 ERROR [DefaultQuartzScheduler_Worker-10] o.e.s.scheduling.LookupJob - ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: java.lang.NullPointerException @ org.esa.sen2agri.scheduling.LookupJob.lookupAndDownload(LookupJob.java:258) < org.esa.sen2agri.scheduling.AbstractJob.execute(AbstractJob.java:65) < org.quartz.core.JobRunShell.run(JobRunShell.java:202) < java.util.concurrent.FutureTask.get(FutureTask.java:192) < java.util.concurrent.FutureTask.report(FutureTask.java:122) < org.esa.sen2agri.scheduling.LookupJob.executeImpl(LookupJob.java:154) < org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Jul 30 19:35:31 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:31.618 INFO [USGS-L8-3 ] r.c.t.d.usgs.Landsat8Query - Query returned 68 products
Jul 30 19:35:33 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:33.347 INFO [USGS-L8-3 ] r.c.t.d.usgs.Landsat8Query - Query returned 70 products
Jul 30 19:35:34 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:34.992 INFO [USGS-L8-3 ] r.c.t.d.usgs.Landsat8Query - Query returned 69 products
Jul 30 19:35:37 localhost.localdomain start.sh[10363]: 2019-Jul-30 19:35:37.859 ERROR [DefaultQuartzScheduler_Worker-9] o.e.s.scheduling.LookupJob - ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: java.lang.NullPointerException @ org.esa.sen2agri.scheduling.LookupJob.lookupAndDownload(LookupJob.java:258) < org.esa.sen2agri.scheduling.AbstractJob.execute(AbstractJob.java:65) < org.quartz.core.JobRunShell.run(JobRunShell.java:202) < java.util.concurrent.FutureTask.get(FutureTask.java:192) < java.util.concurrent.FutureTask.report(FutureTask.java:122) < org.esa.sen2agri.scheduling.LookupJob.executeImpl(LookupJob.java:154) < org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Can someone help?

Thanks in advance

JoaoUrbano · July 31, 2019, 2:51pm

Hi guys, quick update.

After some fiddling I am currently getting the following in the web console:

A lot of products listed but the downloads are stopped, not progressing after a couple of hours.

I get the following from systemctl status sen2agri-services -l

sen2agri-services.service - Services for Sen2Agri
Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-07-31 14:56:13 +01; 43min ago
Main PID: 3418 (start.sh)
Tasks: 102
CGroup: /system.slice/sen2agri-services.service
├─3418 /bin/bash /usr/share/sen2agri/sen2agri-services/bin/start.sh
└─3423 java -cp …/modules/:…/lib/:…/config/* org.esa.sen2agri.ServicesStartup

Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.009 WARN [thread_pool_executor-2] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect’s prepareConnection flag to false
Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.009 DEBUG [thread_pool_executor-2] r.c.t.s.p.i.ProgressReportServiceImpl - Completed S2B_MSIL1C_20170815T110619_N0205_R137_T29SQT_20170815T111826:S2B_MSIL1C_20170815T110619_N0205_R137_T29SQT_20170815T111826.zip
Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.016 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Begin download for S2A_MSIL1C_20170820T110651_N0205_R137_T29SQT_20170820T111220.zip
Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.016 WARN [thread_pool_executor-1] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect’s prepareConnection flag to false
Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.016 DEBUG [thread_pool_executor-1] r.c.t.s.p.i.ProgressReportServiceImpl - Started S2A_MSIL1C_20170820T110651_N0205_R137_T29SQT_20170820T111220
Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.017 WARN [thread_pool_executor-2] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect’s prepareConnection flag to false
Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.017 DEBUG [thread_pool_executor-2] r.c.t.s.p.i.ProgressReportServiceImpl - Started S2A_MSIL1C_20170820T110651_N0205_R137_T29SQT_20170820T111220:S2A_MSIL1C_20170820T110651_N0205_R137_T29SQT_20170820T111220.zip
Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.543 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - (S2A_MSIL1C_20170820T110651_N0205_R137_T29SQT_20170820T111220,Archive) S2A_MSIL1C_20170820T110651_N0205_R137_T29SQT_20170820T111220.zip [size: 810789kB]
Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.543 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Local temporary file /mnt/archive/dwn_def/s2/default/zaers/S2A_MSIL1C_20170820T110651_N0205_R137_T29SQT_20170820T111220.zip created
Jul 31 15:38:51 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:38:51.543 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Begin reading from input stream
[jonaszed@localhost mnt]$

from journalctl I repeatedly get the following:

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.073 WARN [thread_pool_executor-1] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.073 DEBUG [thread_pool_executor-1] r.c.t.s.p.i.ProgressReportServiceImpl - Completed S2A_MSIL1C_20171029T111201_N0206_R137_T29SQT_20171029T131241:S2A_MSIL1C_20171029T111201_N0206_R137_T29SQT_

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.081 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Begin download for S2B_MSIL1C_20171103T111209_N0206_R137_T29SQT_20171106T185400.zip

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.081 DEBUG [thread_pool_executor-1] r.c.t.s.p.i.ProgressReportServiceImpl - Started S2B_MSIL1C_20171103T111209_N0206_R137_T29SQT_20171106T185400:S2B_MSIL1C_20171103T111209_N0206_R137_T29SQT_20

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.083 WARN [thread_pool_executor-2] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.083 DEBUG [thread_pool_executor-2] r.c.t.s.p.i.ProgressReportServiceImpl - Started S2B_MSIL1C_20171103T111209_N0206_R137_T29SQT_20171106T185400

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.723 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - (S2B_MSIL1C_20171103T111209_N0206_R137_T29SQT_20171106T185400,Archive) S2B_MSIL1C_20171103T111209_N0206_R

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.723 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Local temporary file /mnt/archive/dwn_def/s2/default/zaers/S2B_MSIL1C_20171103T111209_N0206_R137_T29SQT_2

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: java.io.IOException: Server returned HTTP response code: 403 for URL: https://scihub.copernicus.eu/apihub/odata/v1/Products(‘abd89a62-8fe6-4a72-88ac-87cd676b0634’)/$value

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.reflect.GeneratedConstructorAccessor93.newInstance(Unknown Source)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1944)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1939)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at java.security.AccessController.doPrivileged(Native Method)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1938)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1508)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at ro.cs.tao.datasource.remote.DownloadStrategy.downloadFile(DownloadStrategy.java:483)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at ro.cs.tao.datasource.remote.DownloadStrategy.downloadFile(DownloadStrategy.java:341)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at ro.cs.tao.datasource.remote.scihub.download.SentinelDownloadStrategy.fetchImpl(SentinelDownloadStrategy.java:97)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at ro.cs.tao.datasource.remote.scihub.download.Sentinel2ArchiveDownloadStrategy.fetchImpl(Sentinel2ArchiveDownloadStrategy.java:30)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at ro.cs.tao.datasource.remote.DownloadStrategy.fetch(DownloadStrategy.java:247)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at ro.cs.tao.datasource.DataSourceComponent.doFetch(DataSourceComponent.java:413)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at org.esa.sen2agri.services.internal.DownloadServiceImpl.download(DownloadServiceImpl.java:267)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at org.esa.sen2agri.scheduling.RetryJob.retryDownloads(RetryJob.java:131)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at org.esa.sen2agri.scheduling.RetryJob.lambda$executeImpl$0(RetryJob.java:81)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at java.lang.Thread.run(Thread.java:748)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: https://scihub.copernicus.eu/apihub/odata/v1/Products(‘abd89a62-8fe6-4a72-88ac-87cd676b0634’)/$value

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3051)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at java.net.URLConnection.getHeaderFieldLong(URLConnection.java:629)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at java.net.URLConnection.getContentLengthLong(URLConnection.java:501)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at sun.net.www.protocol.https.HttpsURLConnectionImpl.getContentLengthLong(HttpsURLConnectionImpl.java:411)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at ro.cs.tao.datasource.remote.DownloadStrategy.downloadFile(DownloadStrategy.java:453)

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: … 13 more

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.724 ERROR [Scientific Data Hub-S2-1] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Cannot download https://scihub.copernicus.eu/apihub/odata/v1/Products('abd89a62-8fe6-4a72-88ac-87cd676b06

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.725 DEBUG [thread_pool_executor-2] r.c.t.s.p.i.ProgressReportServiceImpl - Completed S2B_MSIL1C_20171103T111209_N0206_R137_T29SQT_20171106T185400

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: 2019-Jul-31 15:47:47.725 WARN [Scientific Data Hub-S2-1] r.c.t.d.DataSourceComponent - Fetching product ‘S2B_MSIL1C_20171103T111209_N0206_R137_T29SQT_20171106T185400’ failed: java.io.IOException: Cannot downl

Jul 31 15:47:47 localhost.localdomain start.sh[3418]: at ro.cs.tao.datasource.remote.DownloadStrategy.downloadFile(DownloadStrategy.java:516)