Seems that right after our release of the system USGS changed the response of the requests for queries as they added some new fields. We noticed that at the beginning of the week and I think you encountered the same issue. We already corrected this and we’ll probably release a new version of the sen2agri-services (probably will be a minor release of the full system in the next days including also corrections for the problems you encountered during consecutive migrations from 1.7 to 2.0).
Concerning the S2 downloads, could you provide us the shapefile that you use for your site? We made also a correction when the sites had extent close to prime meridian or equator.
For an unknown reason, the sen2agri-services suddenly resumed and it started downloading on May. I’ll force restart (from the config) the site for those images with this error: (‘failed_reason’ column in downloader_history) Cannot download https://scihub.copernicus.eu/apihub/odata/v1/Products('02e8e925-5bf5-4a4e-b58c-c3f30c9b9617') /$value: Server returned HTTP response code: 503 for URL: https://scihub.copernicus.eu/apihub/odata/v1/Products('02e8e925-5bf5-4a4e-b58c-c3f30c9b9617')/$value
After setting the partially downloaded images to status_id =3, I got this error:
May 29 12:59:05 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:05.301 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Executing query: https://scihub.copernicus.eu/api
hub/search?rows=50&start=1&q=footprint%3A%22Intersects%28POLYGON%28%28121.514%2017.547%2C122.499%2017.547%2C122.498%2016.518%2C121.489%2016.524%2C121.514%2017.547%29%29%29%22%20AND%20endPosi
tion%3A%5B2019-05-29T00%3A00%3A00.000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20filename%3A*51QVU*%20AND%20platformName%3ASentinel-2%20AND%20beginPosition%3A%5B2019-05-29T00%3A00%3A00.
000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20productType%3AS2MSI1C&orderby=beginposition%20asc
May 29 12:59:06 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:06.485 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 0 products
May 29 12:59:06 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:06.486 DEBUG [Scientific Data Hub-S2-1] o.e.s.s.i.DownloadServiceImpl - Query page #1 for site id 6
May 29 12:59:06 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:06.486 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Executing query: https://scihub.copernicus.eu/api
hub/search?rows=50&start=1&q=footprint%3A%22Intersects%28POLYGON%28%28121.514%2017.547%2C122.499%2017.547%2C122.498%2016.518%2C121.489%2016.524%2C121.514%2017.547%29%29%29%22%20AND%20endPosi
tion%3A%5B2019-05-29T00%3A00%3A00.000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20filename%3A*51QVV*%20AND%20platformName%3ASentinel-2%20AND%20beginPosition%3A%5B2019-05-29T00%3A00%3A00.
000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20productType%3AS2MSI1C&orderby=beginposition%20asc
May 29 12:59:07 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:07.853 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 0 products
May 29 12:59:07 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:07.854 DEBUG [Scientific Data Hub-S2-1] o.e.s.s.i.DownloadServiceImpl - Query page #1 for site id 6
May 29 12:59:07 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:07.854 DEBUG [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Executing query: https://scihub.copernicus.eu/api
hub/search?rows=50&start=1&q=footprint%3A%22Intersects%28POLYGON%28%28121.514%2017.547%2C122.499%2017.547%2C122.498%2016.518%2C121.489%2016.524%2C121.514%2017.547%29%29%29%22%20AND%20endPosi
tion%3A%5B2019-05-29T00%3A00%3A00.000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20filename%3A*51QUU*%20AND%20platformName%3ASentinel-2%20AND%20beginPosition%3A%5B2019-05-29T00%3A00%3A00.
000Z%20TO%202019-05-29T12%3A58%3A58.000Z%5D%20AND%20productType%3AS2MSI1C&orderby=beginposition%20asc
May 29 12:59:09 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:09.183 INFO [Scientific Data Hub-S2-1] r.c.t.d.r.s.SciHubDataQuery - Query returned 0 products
May 29 12:59:09 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:09.183 INFO [DefaultQuartzScheduler_Worker-4] o.e.s.scheduling.LookupJob - [site 'Isabela19',sensor 'Sentinel2'] Foun
d 0 products for site isabela19
May 29 12:59:09 nirvana-ESSC_MNL start.sh[24305]: 2019-May-29 12:59:09.189 WARN [DefaultQuartzScheduler_Worker-4] o.s.o.j.v.HibernateJpaDialect - JDBC Connection to reset not identical to
originally prepared Connection - please make sure to use connection release mode ON_CLOSE (the default) and to run against Hibernate 4.2+ (or switch HibernateJpaDialect's prepareConnection f
lag to false
I also applied this work around, but nothing happened:
I assume that the sen2agri-services resumed its downlloading since my MAJA is continuously processing some images. But I can’t access the journalctl or systemctl status, it says:
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
Besides, the downloader interface from the web GUI is stuck at 0%.
I have found out by setting the value = ‘true’ of key = downloader.s2.forcestart and site_id = 6 (my new and current site) from config table helped in forcing to redownload the images which has status_id of either 3 or 6.
However, I had the same journalctl and systemctl status outputs as described from above. I have no plans to restart sen2agri-services since the web interface already shows the current downloading status.
You can find a new version of sen2agri-services here: https://we.tl/t-GkAiT6JcWP
You should copy the content of lib/ and modules/ into the /usr/share/sen2agri/sen2agri-services corresponding subfolders and restart the sen2agri-services.
This should solve the following issues:
the sites having sub-unitary coordinates
the L8 download from USGS (it seems that right after our delivery they changed a little bit the API).
After that you should force again the re-downloading, as you already did.
This is the current status of sen2agri-services after copying the new lib/ and modules/ to /usr/share/sen2agri/sen2agri-services. I typed this command sudo cp -a /sen2agri-services-2.0-20190530/modules/. /usr/share/sen2agri/sen2agri-services/
● sen2agri-services.service - Services for Sen2Agri
Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Jun 06 10:36:07 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:36:10 nirvana-ESSC_MNL start.sh[27029]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup
Jun 06 10:36:21 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:36:21 nirvana-ESSC_MNL start.sh[27105]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup
Jun 06 10:36:26 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:36:26 nirvana-ESSC_MNL start.sh[27173]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup
Jun 06 10:39:28 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:39:29 nirvana-ESSC_MNL start.sh[27663]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup
Jun 06 10:42:09 nirvana-ESSC_MNL systemd[1]: Started Services for Sen2Agri.
Jun 06 10:42:09 nirvana-ESSC_MNL start.sh[30661]: Error: Could not find or load main class org.esa.sen2agri.ServicesStartup
Is there something wrong on how I copied specific folder contents, or should I need to wait first for about 30 minutes? I’ll update you in this post. (As of this writing, I still have the same error).
Did you deleted previously the /usr/share/sen2agri/sen2agri-services/lib and /usr/share/sen2agri/sen2agri-services/modules?
Please note that you should copy the lib/ and modules/ from the archive to /usr/share/sen2agri/sen2agri-services/ into /usr/share/sen2agri/sen2agri-services and not their content into /usr/share/sen2agri/sen2agri-services. The command you provided copies the jar files from modules into the /usr/share/sen2agri/sen2agri-services.
Please check that you have the new files from the archive in the /usr/share/sen2agri/sen2agri-services/lib and /usr/share/sen2agri/sen2agri-services/modules.
You should better do (assuming you have unzipped the services archive into /sen2agri-services-2.0-20190530):
Yes my mistake, I copied the the contents of the folder, instead of copying the folder itself.
Now, it’s resuming it’s Sentinel download. I’ll update you if the system is already downloading Landsat images.
Here’s the Landsat query:
USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - Executing query for product : https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/search?jsonRequest=%7B%22a
[USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - Query returned 27 products
G [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - Executing query for product : https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/search?jsonRequest=%7B%22a
[USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - Query returned 32 products
G [USGS-L8-1 ] o.e.s.s.i.DownloadServiceImpl - Query page #1 for site id 6
G [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - Executing query for product : https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/search?jsonRequest=%7B%22a
The previous versions of the Sen2Agri system were using the api.developmentseed.org to download the products that required no operations/right to download products. Nevertheless, the repository data seem to not be updated anymore since July 2018 so we had to switch to the downloading directly from USGS.
But for this, you need to have activated at USGS the right for downloading the products. To do this, you need to perform a request to USGS in order to activate the downloads for your account:
“Scenes required” – In your case you can fill 2 monthly or 24 yearly (depends on your site dimension). You can also put more if you consider you will have additional sites to monitor
o
“Scripting Capabilities”: python, java
o
For the other fields, you will have to fill the requested information
After that, you should receive after several hours an email from USGS confirming the activation. Once this is done, the L8 should start also being downloaded.
Hi, I’m having the same problem trying download landsat data, I get the follow error
nov 07 15:04:57 localhost.localdomain start.sh[17573]: 2019-nov-07 15:04:57.055 WARN [USGS-L8-1 ] r.c.t.d.usgs.Landsat8Query - ro.cs.tao.datasource.QueryException: ro.cs.tao.datasource.QueryException: The supplied credentials are invalid!
I requested acces to the linkt that you provided and made all the steps you indicate, but still can’t I download landsat data. Also, I have another problem, in the GUI the ‘products’ is not showing the list of products available.