Errors in sen2agri-services dowloading

Hello,

I’m currently encountering some errors with sen2agri-services and its downloading:

When Landsat is enabled:

● sen2agri-services.service - Services for Sen2Agri
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-05-23 11:19:29 PST; 16min ago
 Main PID: 10741 (start.sh)
    Tasks: 70
   CGroup: /system.slice/sen2agri-services.service
           ├─10741 /bin/bash /usr/share/sen2agri/sen2agri-services/bin/start.sh
           └─12020 java -cp ../modules/*:../lib/* org.esa.sen2agri.ServicesStartup

May 23 11:35:00 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:35:00.233 WARN  [DefaultQuartzScheduler_Worker-5] o.s.o.j.v.HibernateJpaDialect  - JDBC Conne
ction to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hib
ernate 4.2+ (or switch HibernateJpaDialect's prepareConnection flag to false
May 23 11:35:00 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:35:00.674 WARN  [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query     - ro.cs.tao.datasource.remote.res
ult.ParseException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "cloudCover" (class ro.cs.tao.datasource.usgs.json.re
sponses.SearchResult), not marked as ignorable (17 known properties: "metadataUrl", "sceneBounds", "browseUrl", "summary", "downloadUrl", "endTime", "ordered"
, "startTime", "modifiedDate", "acquisitionDate", "displayId", "entityId", "dataAccessUrl", "spatialFootprint", "fgdcMetadataUrl", "orderUrl", "bulkOrdered"])
May 23 11:35:00 nirvana-ESSC_MNL start.sh[10741]: at [Source: (String)"{"errorCode":null,"error":"","data":{"numberReturned":1,"totalHits":14,"firstRecord":1,
"lastRecord":1,"nextRecord":2,"results":[{"acquisitionDate":"2018-10-16","startTime":"2018-10-16","endTime":"2018-10-16","spatialFootprint":{"type":"Polygon",
"coordinates":[[[120.43185,16.65639],[122.17253,16.28907],[122.57422,18.02446],[120.8167,18.38972],[120.43185,16.65639]]]},"sceneBounds":"120.43185,16.28907,1
22.57422,18.38972","browseUrl":"https:\/\/ims.cr.usgs.gov\/browse\/landsat_8_c1\/2018\/116\/048\/"[truncated 1016 chars]; line: 1, column: 914] (through refer
ence chain: ro.cs.tao.datasource.usgs.json.responses.SearchResponse["data"]->ro.cs.tao.datasource.usgs.json.responses.SearchResults["results"]->java.util.Arra
yList[0]->ro.cs.tao.datasource.usgs.json.responses.SearchResult["cloudCover"])
May 23 11:35:02 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:35:02.141 WARN  [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query     - ro.cs.tao.datasource.remote.res
ult.ParseException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "cloudCover" (class ro.cs.tao.datasource.usgs.json.re
sponses.SearchResult), not marked as ignorable (17 known properties: "metadataUrl", "sceneBounds", "browseUrl", "summary", "downloadUrl", "endTime", "ordered"
, "startTime", "modifiedDate", "acquisitionDate", "displayId", "entityId", "dataAccessUrl", "spatialFootprint", "fgdcMetadataUrl", "orderUrl", "bulkOrdered"])
May 23 11:35:02 nirvana-ESSC_MNL start.sh[10741]: at [Source: (String)"{"errorCode":null,"error":"","data":{"numberReturned":1,"totalHits":13,"firstRecord":1,
"lastRecord":1,"nextRecord":2,"results":[{"acquisitionDate":"2018-10-09","startTime":"2018-10-09","endTime":"2018-10-09","spatialFootprint":{"type":"Polygon",
"coordinates":[[[121.98231,16.65649],[123.72301,16.28909],[124.12479,18.02447],[122.36724,18.3898],[121.98231,16.65649]]]},"sceneBounds":"121.98231,16.28909,1
24.12479,18.3898","browseUrl":"https:\/\/ims.cr.usgs.gov\/browse\/landsat_8_c1\/2018\/115\/048\/L"[truncated 1016 chars]; line: 1, column: 913] (through refer
ence chain: ro.cs.tao.datasource.usgs.json.responses.SearchResponse["data"]->ro.cs.tao.datasource.usgs.json.responses.SearchResults["results"]->java.util.Arra
yList[0]->ro.cs.tao.datasource.usgs.json.responses.SearchResult["cloudCover"])
May 23 11:35:03 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:35:03.923 WARN  [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query     - ro.cs.tao.datasource.remote.res
ult.ParseException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "cloudCover" (class ro.cs.tao.datasource.usgs.json.re
sponses.SearchResult), not marked as ignorable (17 known properties: "metadataUrl", "sceneBounds", "browseUrl", "summary", "downloadUrl", "endTime", "ordered"
, "startTime", "modifiedDate", "acquisitionDate", "displayId", "entityId", "dataAccessUrl", "spatialFootprint", "fgdcMetadataUrl", "orderUrl", "bulkOrdered"])
May 23 11:35:03 nirvana-ESSC_MNL start.sh[10741]: at [Source: (String)"{"errorCode":null,"error":"","data":{"numberReturned":1,"totalHits":13,"firstRecord":1,
"lastRecord":1,"nextRecord":2,"results":[{"acquisitionDate":"2018-10-09","startTime":"2018-10-09","endTime":"2018-10-09","spatialFootprint":{"type":"Polygon",
"coordinates":[[[121.66475,15.21208],[123.39333,14.84567],[123.79028,16.58166],[122.04643,16.94589],[121.66475,15.21208]]]},"sceneBounds":"121.66475,14.84567,

When Landsat is disabled:

 sen2agri-services.service - Services for Sen2Agri
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-05-23 11:19:29 PST; 11min ago
 Main PID: 10741 (start.sh)
    Tasks: 69
   CGroup: /system.slice/sen2agri-services.service
           ├─10741 /bin/bash /usr/share/sen2agri/sen2agri-services/bin/start.sh
           └─12020 java -cp ../modules/*:../lib/* org.esa.sen2agri.ServicesStartup

May 23 11:28:41 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:41.572 WARN  [DefaultQuartzScheduler_Worker-7] o.s.o.j.v.HibernateJpaDialect  - JDBC Conne
ction to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hib
ernate 4.2+ (or switch HibernateJpaDialect's prepareConnection flag to false
May 23 11:28:41 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:41.954 INFO  [DefaultQuartzScheduler_Worker-8] o.e.s.scheduling.RetryJob	  - Job 'Retry
.isabela19-Sentinel2' started
May 23 11:28:46 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:46.981 INFO  [Scientific Data Hub-S2-1] o.e.s.scheduling.RetryJob	   - [site 'isabela19'
,sensor 'Sentinel2'] No products to retry
May 23 11:28:46 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:46.982 INFO  [Scientific Data Hub-S2-1] o.e.s.scheduling.RetryJob	   - [site 'isabela19'
,sensor 'Sentinel2'] No products to retry
May 23 11:28:51 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:51.185 INFO  [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Query returned 0
products
May 23 11:28:51 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:51.904 INFO  [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Query returned 0
products
May 23 11:28:52 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:52.618 INFO  [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Query returned 0
products
May 23 11:28:53 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:53.336 INFO  [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Query returned 0
products
May 23 11:28:53 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:53.337 INFO  [DefaultQuartzScheduler_Worker-7] o.e.s.scheduling.LookupJob     - [site 'Isa
bela19',sensor 'Sentinel2'] Found 0 products for site isabela19
May 23 11:28:53 nirvana-ESSC_MNL start.sh[10741]: 2019-May-23 11:28:53.344 WARN  [DefaultQuartzScheduler_Worker-7] o.s.o.j.v.HibernateJpaDialect  - JDBC Conne
ction to reset not identical to originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hib
ernate 4.2+ (or switch HibernateJpaDialect's prepareConnection flag to false

I’ve checked that all my credentials to USGS and ESA Scihub are right. (I can log in through their websites.)

Regards to all,
Brent

Hello Brent,

Seems that right after our release of the system USGS changed the response of the requests for queries as they added some new fields. We noticed that at the beginning of the week and I think you encountered the same issue. We already corrected this and we’ll probably release a new version of the sen2agri-services (probably will be a minor release of the full system in the next days including also corrections for the problems you encountered during consecutive migrations from 1.7 to 2.0).
Concerning the S2 downloads, could you provide us the shapefile that you use for your site? We made also a correction when the sites had extent close to prime meridian or equator.

Please let me know.

Best regards,
Cosmin

Hello,

Here is my shapefile. My country of interest is near the equator.

Isaebla_UTM.zip (1.5 KB)

Brent

Hello Cosmin,

For an unknown reason, the sen2agri-services suddenly resumed and it started downloading on May. I’ll force restart (from the config) the site for those images with this error: (‘failed_reason’ column in downloader_history) Cannot download https://scihub.copernicus.eu/apihub/odata/v1/Products('02e8e925-5bf5-4a4e-b58c-c3f30c9b9617') /$value: Server returned HTTP response code: 503 for URL: https://scihub.copernicus.eu/apihub/odata/v1/Products('02e8e925-5bf5-4a4e-b58c-c3f30c9b9617')/$value

I’ll keep you posted on this.

Regards.

Hello Cosmin,

After setting the partially downloaded images to status_id =3, I got this error:

May 29 12:59:05 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:05.301 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Executing query: https://scihub.copernicus.eu/api
hub/search?rows=50&start=1&q=footprint%3A%22Intersects%28POLYGON%28%28121.514%2017.547%2C122.499%2017.547%2C122.498%2016.518%2C121.489%2016.524%2C121.514%2017.547%29%29%29%22%20AND%20endPosi
tion%3A%5B2019-05-29T00%3A00%3A00.000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20filename%3A*51QVU*%20AND%20platformName%3ASentinel-2%20AND%20beginPosition%3A%5B2019-05-29T00%3A00%3A00.
000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20productType%3AS2MSI1C&orderby=beginposition%20asc
May 29 12:59:06 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:06.485 INFO  [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Query returned 0 products
May 29 12:59:06 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:06.486 DEBUG [Scientific Data Hub-S2-1] o.e.s.s.i.DownloadServiceImpl  - Query page #1 for site id 6
May 29 12:59:06 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:06.486 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Executing query: https://scihub.copernicus.eu/api
hub/search?rows=50&start=1&q=footprint%3A%22Intersects%28POLYGON%28%28121.514%2017.547%2C122.499%2017.547%2C122.498%2016.518%2C121.489%2016.524%2C121.514%2017.547%29%29%29%22%20AND%20endPosi
tion%3A%5B2019-05-29T00%3A00%3A00.000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20filename%3A*51QVV*%20AND%20platformName%3ASentinel-2%20AND%20beginPosition%3A%5B2019-05-29T00%3A00%3A00.
000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20productType%3AS2MSI1C&orderby=beginposition%20asc
May 29 12:59:07 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:07.853 INFO  [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Query returned 0 products
May 29 12:59:07 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:07.854 DEBUG [Scientific Data Hub-S2-1] o.e.s.s.i.DownloadServiceImpl  - Query page #1 for site id 6
May 29 12:59:07 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:07.854 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Executing query: https://scihub.copernicus.eu/api
hub/search?rows=50&start=1&q=footprint%3A%22Intersects%28POLYGON%28%28121.514%2017.547%2C122.499%2017.547%2C122.498%2016.518%2C121.489%2016.524%2C121.514%2017.547%29%29%29%22%20AND%20endPosi
tion%3A%5B2019-05-29T00%3A00%3A00.000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20filename%3A*51QUU*%20AND%20platformName%3ASentinel-2%20AND%20beginPosition%3A%5B2019-05-29T00%3A00%3A00.
000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20productType%3AS2MSI1C&orderby=beginposition%20asc
May 29 12:59:09 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:09.183 INFO  [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery    - Query returned 0 products
May 29 12:59:09 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:09.183 INFO  [DefaultQuartzScheduler_Worker-4] o.e.s.scheduling.LookupJob     - [site 'Isabela19',sensor 'Sentinel2'] Foun
d 0 products for site isabela19
May 29 12:59:09 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:09.189 WARN  [DefaultQuartzScheduler_Worker-4] o.s.o.j.v.HibernateJpaDialect  - JDBC Connection to reset not identical to
originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect's prepareConnection f
lag to false

I also applied this work around, but nothing happened:

Regards.

Hi Cosmin,

I assume that the sen2agri-services resumed its downlloading since my MAJA is continuously processing some images. But I can’t access the journalctl or systemctl status, it says:

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Besides, the downloader interface from the web GUI is stuck at 0%.

Regards.

Hello @cudroiu,

I have found out by setting the value = ‘true’ of key = downloader.s2.forcestart and site_id = 6 (my new and current site) from config table helped in forcing to redownload the images which has status_id of either 3 or 6.

However, I had the same journalctl and systemctl status outputs as described from above. I have no plans to restart sen2agri-services since the web interface already shows the current downloading status.

Hello Brent,

You can find a new version of sen2agri-services here:


You should copy the content of lib/ and modules/ into the /usr/share/sen2agri/sen2agri-services corresponding subfolders and restart the sen2agri-services.
This should solve the following issues:

  • the sites having sub-unitary coordinates
  • the L8 download from USGS (it seems that right after our delivery they changed a little bit the API).

After that you should force again the re-downloading, as you already did.

Hope this helps,
Cosmin

Hello @cudroiu

This is the current status of sen2agri-services after copying the new lib/ and modules/ to /usr/share/sen2agri/sen2agri-services. I typed this command sudo cp -a /sen2agri-services-2.0-20190530/modules/. /usr/share/sen2agri/sen2agri-services/

● sen2agri-services.service - Services for Sen2Agri
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Jun 06 10:36:07 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:36:10 nirvana-ESSC_MNL start.sh[27029]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup
Jun 06 10:36:21 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:36:21 nirvana-ESSC_MNL start.sh[27105]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup
Jun 06 10:36:26 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:36:26 nirvana-ESSC_MNL start.sh[27173]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup
Jun 06 10:39:28 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:39:29 nirvana-ESSC_MNL start.sh[27663]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup
Jun 06 10:42:09 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:42:09 nirvana-ESSC_MNL start.sh[30661]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup

Is there something wrong on how I copied specific folder contents, or should I need to wait first for about 30 minutes? I’ll update you in this post. (As of this writing, I still have the same error).

Regards.

Regards.

Hello Brent,

Did you deleted previously the /usr/share/sen2agri/sen2agri-services/lib and /usr/share/sen2agri/sen2agri-services/modules?
Please note that you should copy the lib/ and modules/ from the archive to /usr/share/sen2agri/sen2agri-services/ into /usr/share/sen2agri/sen2agri-services and not their content into /usr/share/sen2agri/sen2agri-services. The command you provided copies the jar files from modules into the /usr/share/sen2agri/sen2agri-services.
Please check that you have the new files from the archive in the /usr/share/sen2agri/sen2agri-services/lib and /usr/share/sen2agri/sen2agri-services/modules.
You should better do (assuming you have unzipped the services archive into /sen2agri-services-2.0-20190530):

sudo rm -fr /usr/share/sen2agri/sen2agri-services/lib
sudo rm -fr /usr/share/sen2agri/sen2agri-services/modules
sudo cp -fR /sen2agri-services-2.0-20190530/lib /usr/share/sen2agri/sen2agri-services/
sudo cp -fR /sen2agri-services-2.0-20190530/modules /usr/share/sen2agri/sen2agri-services/

Please let me know.

Best regards,
Cosmin

Ho @cudroiu,

Yes my mistake, I copied the the contents of the folder, instead of copying the folder itself.
Now, it’s resuming it’s Sentinel download. I’ll update you if the system is already downloading Landsat images.

Here’s the Landsat query:

USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query     - Executing query for product : https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/search?jsonRequest=%7B%22a
  [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query     - Query returned 27 products
G [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query     - Executing query for product : https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/search?jsonRequest=%7B%22a
  [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query     - Query returned 32 products
G [USGS-L8-1 ] o.e.s.s.i.DownloadServiceImpl  - Query page #1 for site id 6
G [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query     - Executing query for product : https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/search?jsonRequest=%7B%22a

Thanks a lot.

Regards.

Hi Cosmin, even with the patches applied according to your instructions, the L8 download still doesn’t work.

Here’s an error I’ve found in the log:

Jun 10 12:22:47 wfp.sen2agri start.sh[25744]: 2019-Jun-10 12:22:47.366 ERROR [DefaultQuartzScheduler_Worker-5] o.e.s.scheduling.LookupJob - ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: java.lang.NullPointerException @ org.esa.sen2agri.scheduling.LookupJob.lookupAndDownload(LookupJob.java:258) < org.esa.sen2agri.scheduling.AbstractJob.execute(AbstractJob.java:65) < org.quartz.core.JobRunShell.run(JobRunShell.java:202) < java.util.concurrent.FutureTask.get(FutureTask.java:192) < java.util.concurrent.FutureTask.report(FutureTask.java:122) < org.esa.sen2agri.scheduling.LookupJob.executeImpl(LookupJob.java:154) < org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Val

is the L8 download working for anyone? Maybe @brentf ?

@val as of this time, my Landsat 8 downloading is not working.

Thanks for your answer. Hopefully this will be looked at soon.