Paging in sentinel product downloads is not correct

t.lygutas · October 17, 2019, 2:29pm

In log I see many entries like this:

Oct 17 14:08:42 s84036 start.sh: 2019-Oct-17 14:08:42.107 INFO  [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Query returned 0 products
Oct 17 14:08:42 s84036 start.sh: 2019-Oct-17 14:08:42.107 DEBUG [Scientific Data Hub-S2-1] o.e.s.s.i.DownloadServiceImpl  - Query page #1 for site id 1
Oct 17 14:08:42 s84036 start.sh: 2019-Oct-17 14:08:42.107 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Executing query: https://scihub.copernicus.eu/apihub/search?rows=50&start=1&q=footprint%3A%22Intersects%28POLYGON%28%2822.862%2054.334%2C22.782%2054.583%2C22.758%2054.857%2C22.614%2055.075%2C22.074%2055.067%2C21.268%2055.19%2C21.064%2056.053%2C21.439%2056.311%2C22.173%2056.474%2C24.338%2056.341%2C24.943%2056.483%2C25.17%2056.245%2C25.739%2056.147%2C26.773%2055.67%2C26.649%2055.47%2C26.777%2055.303%2C26.736%2055.118%2C26.375%2055.068%2C26.008%2054.878%2C25.733%2054.398%2C25.726%2054.228%2C25.556%2054.175%2C25.27%2054.229%2C25.068%2054.098%2C24.8%2054.006%2C24.396%2053.926%2C23.664%2053.952%2C23.365%2054.178%2C22.862%2054.334%29%29%29%22%20AND%20endPosition%3A%5B2019-10-15T00%3A00%3A00.000Z%20TO%202019-10-17T14%3A07%3A51.000Z%5D%20AND%20filename%3A*34UFG*%20AND%20platformName%3ASentinel-2%20AND%20beginPosition%3A%5B2019-10-15T00%3A00%3A00.000Z%20TO%202019-10-17T14%3A07%3A51.000Z%5D%20AND%20productType%3AS2MSI1C&orderby=beginposition%20asc

The issue I see is the part
&start=1

First page should be 0. Now results on 1 page are empty.
Am i the only one with this problem?

Can it be fixed with some parameter?

@cudroiu ?

cudroiu · October 17, 2019, 3:09pm

Hello,

Indeed, this is an issue (the SciHub API changed at some point) and was already corrected. We’ll release a new version of the services soon, including this correction.

Best regards,
Cosmin

t.lygutas · October 17, 2019, 3:18pm

Could I get some hot fix, or make it myself?

t.lygutas · October 21, 2019, 12:29pm

@cudroiu when can we expect the update ?

cudroiu · October 23, 2019, 7:55am

Hello,

If you want to try it, you can find the latest version of the sen2agri-services (unofficial) here:

https://we.tl/t-tUzGJsYaVz

In order to update it, you should perform the following operations:

Stop the sen2agri-services with:

sudo systemctl stop sen2agri-services

Backup the existing sen2agri-services in case of issues:

sudo mv /usr/share/sen2agri/sen2agri-services/lib /usr/share/sen2agri/sen2agri-services/lib_bkp
sudo mv /usr/share/sen2agri/sen2agri-services/modules /usr/share/sen2agri/sen2agri-services/modules_bkp

Unzip the downloaded archive into a directory on disk ex. ~/sen2agri-services.
Copy the extracted application.properties, lib and modules files in /usr/share/sen2agri/sen2agri-services:

sudo cp -fR ~/sen2agri-services/lib /usr/share/sen2agri/sen2agri-services
sudo cp -fR ~/sen2agri-services/modules /usr/share/sen2agri/sen2agri-services
sudo cp -fR ~/sen2agri-services/config/application.properties /usr/share/sen2agri/sen2agri-services/config

Restart the sen2agri-services:

sudo systemctl start sen2agri-services

Best regards,
Cosmin

P.S. Please note that this version of the services is not the official one and was not fully tested. If any issues, please let me know.

EDIT : After you copy the lib directory to the /usr/share/sen2agri/sen2agri-services, it is highly recommended to remove the tao-datasources-aws-1.0.jar and the tao-datasources-mundi-1.0.jar from this folder. If the services were already started, after the removal of these jar files you could do:

sudo -u postgres psql sen2agri -c “delete from datasource where name like ‘%Mundi DIAS%’;’”
sudo -u postgres psql sen2agri -c “delete from datasource where name like ‘%Amazon Web Services%’;’”

And then restart the services …

ss.cullen · November 6, 2019, 8:50pm

Do you have an update on this Cosmin? The download link for the newer sen2agri-services is no longer valid. Is there a time frame for when these changes will be added to an official version of sen2agri?

frzambra · November 7, 2019, 6:57pm

Same issue, same request!! Landsat 8 downloading is not working for me too.

Nooredin · November 15, 2019, 6:00am

Hi Cosmin,

The link is not working. Can you please share again or use other cloud drives to share.

Thank you,
Nuredin

cudroiu · November 26, 2019, 3:06pm

Hello,

The sen2agri-services archive can be found here:

https://we.tl/t-tUzGJsYaVz

To update, please follow the steps from this post:

Best regards,
Cosmin

Nuredin · November 26, 2019, 3:38pm

Many thanks Cosmin. I will check it.

Innocturion · December 13, 2019, 1:47pm

Dear Cosmin,

after patch application I cannot start the service. I restored the backup files but again… no success. Any ideas?

indent preformatted text Dez 13 14:45:15 gwdg.novalocal systemd[1]: Started Services for Sen2Agri.
[root@gwdg lib]# systemctl status sen2agri-services.service 
● sen2agri-services.service - Services for Sen2Agri
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; enabled; vendor preset: disabled)
   Active: active (running) since Fr 2019-12-13 14:45:15 CET; 3s ago
 Main PID: 30090 (start.sh)
   CGroup: /system.slice/sen2agri-services.service
           ├─30090 /bin/bash /usr/share/sen2agri/sen2agri-services/bin/start.sh
           └─30095 java -cp ../modules/*:../lib/*:../config/* org.esa.sen2agri.ServicesStartup

Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.context.event.AbstractApplicationEventMulticaster.retrieveApplicationListeners(AbstractApplicationEventMulticaster.java:225)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:196)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:133)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:76)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at ro.cs.tao.services.commons.StartupBase.run(StartupBase.java:106)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.esa.sen2agri.ServicesStartup.main(ServicesStartup.java:37)
[root@gwdg lib]# systemctl status sen2agri-services.service 
● sen2agri-services.service - Services for Sen2Agri
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Fr 2019-12-13 14:45:19 CET; 2s ago
  Process: 30090 ExecStart=/usr/share/sen2agri/sen2agri-services/bin/start.sh (code=exited, status=0/SUCCESS)
 Main PID: 30090 (code=exited, status=0/SUCCESS)

Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.context.event.AbstractApplicationEventMulticaster.retrieveApplicationListeners(AbstractApplicationEventMulticaster.java:225)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:196)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:133)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:76)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at ro.cs.tao.services.commons.StartupBase.run(StartupBase.java:106)
Dez 13 14:45:18 gwdg.novalocal start.sh[30090]: at org.esa.sen2agri.ServicesStartup.main(ServicesStartup.java:37)
by 4 spaces

cudroiu · December 16, 2019, 9:26am

Hello,

Are you sure you moved the lib and modules directories before copying the new services files?
This error is given normally if some jar files were kept from the previous version of the services.

Best regards,
Cosmin

Innocturion · December 16, 2019, 11:49am

Dear Cosmin,

many thanks for the hint. This could be the reason. Unfortunately I restored the snapshot and patched again… Everything is fine now.

I tested the patch and its runs without any issues. Thank you and your team! This patch solves many issues and Landsat is working!

Innocturion · January 22, 2020, 3:42pm

Dear cosmin,

since I patched the files you provided, an issue with sen2agri-config occured. The GUI opens, but always reads: “An error occured while loading the configuration: Missing global value for parameter s1.preprocessing.enabled”.
How can we fix this issue?

Is there any possibility to raise the number of parallel processes for maja using sen2agri-config or directly in the program files?