Hello.
I’m still having issues to download any product.
@henryhein, how did you update to 2.0.2? Using update.sh script?
Thanks,
Hello,
Could you please provide more details on the issue you had when running custom jobs (the processor you executed and parameters)?
Also, could you try executing again the processor but prior of that, execute:
sudo journalctl -fu sen2agri-orchestrator
sudo journalctl -fu sen2agri-executor
After that, once the processor finished, could you please share the results of the logs for the orchestrator and executor?
Best regards,
Cosmin
Hello @cudroiu. Thanks for taking care of the issue.
[kserrade@bscsen2agri ~]$ sudo journalctl -fu sen2agri-orchestrator
-- Logs begin at dt 2020-02-18 13:57:14 CET. --
mar 13 08:45:23 bscsen2agri.bsc.es systemd[1]: Stopping Orchestrator for Sen2Agri...
mar 13 08:45:23 bscsen2agri.bsc.es systemd[1]: Stopped Orchestrator for Sen2Agri.
mar 13 08:45:55 bscsen2agri.bsc.es systemd[1]: Starting Orchestrator for Sen2Agri...
mar 13 08:45:56 bscsen2agri.bsc.es sen2agri-orchestrator[7496]: Reading settings from /etc/sen2agri/sen2agri.conf
mar 13 08:45:56 bscsen2agri.bsc.es systemd[1]: Started Orchestrator for Sen2Agri.
mar 13 08:54:09 bscsen2agri.bsc.es systemd[1]: Stopping Orchestrator for Sen2Agri...
mar 13 08:54:09 bscsen2agri.bsc.es systemd[1]: Stopped Orchestrator for Sen2Agri.
mar 13 08:54:22 bscsen2agri.bsc.es systemd[1]: Starting Orchestrator for Sen2Agri...
mar 13 08:54:22 bscsen2agri.bsc.es sen2agri-orchestrator[8352]: Reading settings from /etc/sen2agri/sen2agri.conf
mar 13 08:54:22 bscsen2agri.bsc.es systemd[1]: Started Orchestrator for Sen2Agri.
[kserrade@bscsen2agri ~]$ sudo journalctl -fu sen2agri-executor
-- Logs begin at dt 2020-02-18 13:57:14 CET. --
mar 18 18:37:52 bscsen2agri.bsc.es sen2agri-executor[8351]: [23B blob data]
mar 18 18:37:52 bscsen2agri.bsc.es sen2agri-executor[8351]: [42B blob data]
mar 19 04:56:23 bscsen2agri.bsc.es sen2agri-executor[8351]: Invalid JSON message:
mar 19 04:56:23 bscsen2agri.bsc.es sen2agri-executor[8351]: An error occurred during parsing message
mar 19 04:56:24 bscsen2agri.bsc.es sen2agri-executor[8351]: Invalid JSON message: GET / HTTP/1.1
mar 19 04:56:24 bscsen2agri.bsc.es sen2agri-executor[8351]: Host: 84.88.186.32:7777
mar 19 04:56:24 bscsen2agri.bsc.es sen2agri-executor[8351]: An error occurred during parsing message GET / HTTP/1.1
mar 19 04:56:24 bscsen2agri.bsc.es sen2agri-executor[8351]: Host: 84.88.186.32:7777
mar 19 04:56:25 bscsen2agri.bsc.es sen2agri-executor[8351]: [25B blob data]
mar 19 04:56:25 bscsen2agri.bsc.es sen2agri-executor[8351]: [44B blob data]
Mar 19 05:30:26 bscsen2agri start.sh: NOT STARTED.
Mar 19 05:30:26 bscsen2agri start.sh: Currently in standby mode.
Mar 19 05:30:26 bscsen2agri start.sh: Number of jobs executed: 0
Mar 19 05:30:27 bscsen2agri start.sh: Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Mar 19 05:30:27 bscsen2agri start.sh: Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:26.997 INFO [main ] o.q.impl.StdSchedulerFactory - Quartz scheduler 'quartzScheduler' initialized from an externally provided properties instance.
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:26.997 INFO [main ] o.q.impl.StdSchedulerFactory - Quartz scheduler version: 2.3.0
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:26.998 INFO [main ] o.quartz.core.QuartzScheduler - JobFactory set to: org.springframework.boot.autoconfigure.quartz.AutowireCapableBeanJobFactory@e0d9e3f
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.248 INFO [main ] o.s.b.a.s.s.UserDetailsServiceAutoConfiguration -
Mar 19 05:30:27 bscsen2agri start.sh: Using generated security password: f7ff3db9-267a-494b-ab78-8250958b99fa
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.771 INFO [main ] o.s.s.w.DefaultSecurityFilterChain - Creating filter chain: org.springframework.security.web.util.matcher.AnyRequestMatcher@1, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@58720eb9, org.springframework.security.web.context.SecurityContextPersistenceFilter@2d2afc39, org.springframework.security.web.header.HeaderWriterFilter@2f8ab988, org.springframework.security.web.authentication.logout.LogoutFilter@2b30b627, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@26c2f767, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@453e1bba, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@24749ad3, org.springframework.security.web.session.SessionManagementFilter@558ed473, org.springframework.security.web.access.ExceptionTranslationFilter@304e1e4e, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@32430075]
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.904 INFO [main ] o.s.j.e.a.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.914 INFO [main ] o.s.c.s.DefaultLifecycleProcessor - Starting beans in phase 2147483647
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.914 INFO [main ] o.s.s.q.SchedulerFactoryBean - Starting Quartz Scheduler now
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.915 INFO [main ] o.quartz.core.QuartzScheduler - Scheduler quartzScheduler_$_NON_CLUSTERED started.
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.927 INFO [main ] org.esa.sen2agri.CoreLauncher - Network connection timeout initialized at 30 seconds
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.928 INFO [main ] org.esa.sen2agri.CoreLauncher - Database configuration polling initialized at 10 minutes
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.945 WARN [main ] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect's prepareConnection flag to false
Mar 19 05:30:27 bscsen2agri start.sh: 2020-Mar-19 10:30:27.946 INFO [main ] org.esa.sen2agri.CoreLauncher - Batch notification initialized at 60 minutes with message limit of 0
Mar 19 05:30:28 bscsen2agri start.sh: 2020-Mar-19 10:30:28.012 INFO [main ] o.s.b.w.e.t.TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path ''
Mar 19 05:30:28 bscsen2agri start.sh: 2020-Mar-19 10:30:28.016 INFO [main ] o.esa.sen2agri.ServicesStartup - Started ServicesStartup in 13.388 seconds (JVM running for 15.176)
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.008 INFO [DefaultQuartzScheduler_Worker-1] o.e.s.scheduling.RetryJob - Job 'Retry.lleidaarago-Landsat8' started
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.035 INFO [DefaultQuartzScheduler_Worker-2] o.e.s.scheduling.RetryJob - Job 'Retry.lleidaarago-Sentinel2' started
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.069 INFO [DefaultQuartzScheduler_Worker-3] o.e.s.scheduling.LookupJob - Job 'Lookup.lleidaarago-Landsat8' started
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.069 INFO [DefaultQuartzScheduler_Worker-4] o.e.s.scheduling.LookupJob - Job 'Lookup.lleidaarago-Sentinel2' started
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.126 INFO [USGS-L8-1 ] o.e.s.scheduling.RetryJob - [site 'lleidaarago',sensor 'Landsat8'] No products to retry
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.128 INFO [Scientific Data Hub-S2-1] o.e.s.scheduling.RetryJob - [site 'lleidaarago',sensor 'Sentinel2'] No products to retry
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.131 INFO [Scientific Data Hub-S2-1] o.e.s.scheduling.RetryJob - [site 'lleidaarago',sensor 'Sentinel2'] No products to retry
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.134 INFO [USGS-L8-1 ] o.e.s.scheduling.RetryJob - [site 'lleidaarago',sensor 'Landsat8'] No products to retry
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.168 WARN [DefaultQuartzScheduler_Worker-4] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect's prepareConnection flag to false
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.169 WARN [DefaultQuartzScheduler_Worker-3] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect's prepareConnection flag to false
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.301 INFO [USGS-L8-1 ] - Loading Landsat8 tiles extents
Mar 19 05:30:36 bscsen2agri start.sh: 2020-Mar-19 10:30:36.301 INFO [Scientific Data Hub-S2-1] - Loading Sentinel-2 tiles extents
Mar 19 05:30:38 bscsen2agri start.sh: 2020-Mar-19 10:30:38.382 WARN [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: The supplied credentials are invalid!
Mar 19 05:30:38 bscsen2agri start.sh: 2020-Mar-19 10:30:38.564 WARN [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: The supplied credentials are invalid!
Mar 19 05:30:38 bscsen2agri start.sh: 2020-Mar-19 10:30:38.778 WARN [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: The supplied credentials are invalid!
Mar 19 05:30:38 bscsen2agri start.sh: 2020-Mar-19 10:30:38.958 WARN [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: The supplied credentials are invalid!
Mar 19 05:30:39 bscsen2agri start.sh: 2020-Mar-19 10:30:39.143 ERROR [DefaultQuartzScheduler_Worker-3] o.e.s.scheduling.LookupJob - ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: The supplied credentials are invalid! @ org.esa.sen2agri.scheduling.LookupJob.lookupAndDownload(LookupJob.java:258) < org.esa.sen2agri.scheduling.AbstractJob.execute(AbstractJob.java:65) < org.quartz.core.JobRunShell.run(JobRunShell.java:202) < java.util.concurrent.FutureTask.get(FutureTask.java:192) < java.util.concurrent.FutureTask.report(FutureTask.java:122) < org.esa.sen2agri.scheduling.LookupJob.executeImpl(LookupJob.java:154) < org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Mar 19 05:31:00 bscsen2agri systemd: Started Runs MACCS on L1C products.
Mar 19 05:31:03 bscsen2agri start.sh: 2020-Mar-19 10:31:03.469 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 48 products
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: CompressTiffs = 0, CogTiffs = 0, RemoveSRE = true, RemoveFre = 0
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: CompressTiffs = False, CogTiffs = False, RemoveSRE = True, RemoveFre = False
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: Starting thread dmworker_0
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: Starting thread dmworker_1
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: Waiting for queue to join...
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: dmworker_1 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: 2020-03-19 10:31:05.724192:[18365]:dmworker_1: No tile to process. Gracefully closing...
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: 2020-03-19 10:31:05.724479:[18365]:dmworker_1: Exit thread
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: dmworker_0 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: 2020-03-19 10:31:05.724787:[18365]:dmworker_0: No tile to process. Gracefully closing...
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: 2020-03-19 10:31:05.725800:[18365]:dmworker_0: Exit thread
Mar 19 05:31:05 bscsen2agri demmaccs_launcher.py: All the workers finished their job. Exiting...
Mar 19 05:31:06 bscsen2agri start.sh: 2020-Mar-19 10:31:06.739 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 24 products
Mar 19 05:31:10 bscsen2agri start.sh: 2020-Mar-19 10:31:10.142 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 50 products
Mar 19 05:31:13 bscsen2agri start.sh: 2020-Mar-19 10:31:13.139 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 0 products
Mar 19 05:31:16 bscsen2agri start.sh: 2020-Mar-19 10:31:16.626 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 50 products
Mar 19 05:31:16 bscsen2agri start.sh: 2020-Mar-19 10:31:16.702 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 0 products
Mar 19 05:31:19 bscsen2agri start.sh: 2020-Mar-19 10:31:19.973 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 48 products
Mar 19 05:31:22 bscsen2agri sen2agri-scheduler: Loading tasks from database ...
Mar 19 05:31:22 bscsen2agri sen2agri-scheduler: No tasks defined in the database yet!
Mar 19 05:31:23 bscsen2agri start.sh: 2020-Mar-19 10:31:23.617 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 24 products
Mar 19 05:31:23 bscsen2agri start.sh: 2020-Mar-19 10:31:23.620 INFO [DefaultQuartzScheduler_Worker-4] o.e.s.scheduling.LookupJob - [site 'LleidaArago',sensor 'Sentinel2'] Found 244 products for site lleidaarago
Mar 19 05:31:23 bscsen2agri start.sh: 2020-Mar-19 10:31:23.635 WARN [DefaultQuartzScheduler_Worker-4] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect's prepareConnection flag to false
Mar 19 05:31:48 bscsen2agri su: (to root) kserrade on pts/0
Mar 19 05:32:00 bscsen2agri systemd: Started Runs MACCS on L1C products.
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: CompressTiffs = 0, CogTiffs = 0, RemoveSRE = true, RemoveFre = 0
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: CompressTiffs = False, CogTiffs = False, RemoveSRE = True, RemoveFre = False
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: Starting thread dmworker_0
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: Starting thread dmworker_1
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: Waiting for queue to join...
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: dmworker_1 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: 2020-03-19 10:32:05.799086:[18427]:dmworker_1: No tile to process. Gracefully closing...
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: 2020-03-19 10:32:05.799335:[18427]:dmworker_1: Exit thread
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: dmworker_0 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: 2020-03-19 10:32:05.799606:[18427]:dmworker_0: No tile to process. Gracefully closing...
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: 2020-03-19 10:32:05.799768:[18427]:dmworker_0: Exit thread
Mar 19 05:32:05 bscsen2agri demmaccs_launcher.py: All the workers finished their job. Exiting...
For the error on USGS credentials, mention that I followed your previous reply and I have the right for downloading the products (my request has been accepted two weeks ago).
Thanks!
Yes. update.sh script. Custom jobs are also working after restarting slurm and here is command to restart.
sudo systemctl restart slurmd slurmctld slurmdbd munge