Processing stops after a few L2A products

General Discussions
1

Hi,

I’m running sen2agri version 2.0.2. I have tried several times running the service for a small 25km2 area in Australia using different season dates. Every time I run for a new site the system downloads a few dates and produces a few L2A products but then seems to stop and not proceed any further. The seasons have all been 6 months or above but I only get for instance 4 L2A products but no NDVI or LAI products produced.

The journal file just seems to repeat the following log. I feel that it should have downloaded more data than it has. Is there anything in the log below that indicates what might be wrong or is there something in the database that might give me a clue to the issue?

Many thanks,

Kevin

Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Loading tasks from database …
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Computing next run time and updating status in database…
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Extracting ready list…
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Ready list extracted!
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Getting job definition for processor 4 and site 3
Nov 05 13:00:30 localhost.localdomain sen2agri-orchestrator[2209]: Scheduled job for L3E and site ID 3 with start date Sat May 16 00:00:00 2020 and end date Sat May 16 00:00:00 2020 will not be executed (no products)!
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Job definition got 4
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: The job for processor: 4, siteId: 3 cannot be started now as is invalid
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Getting job definition for processor 2 and site 3
Nov 05 13:00:30 localhost.localdomain sen2agri-orchestrator[2209]: Scheduler L3A: Getting season dates for site 3 for scheduled date Wed Sep 30 00:00:00 2020!
Nov 05 13:00:30 localhost.localdomain sen2agri-orchestrator[2209]: Scheduler L3A: Extracted season dates: Start: Sat May 16 00:00:00 2020, End: Sat Nov 21 00:00:00 2020!
Nov 05 13:00:30 localhost.localdomain sen2agri-orchestrator[2209]: Scheduled job for L3A and site ID 3 with start date Thu Aug 6 00:00:00 2020 and end date Fri Sep 25 00:00:00 2020 will not be executed (no products)!
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Job definition got 2
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: The job for processor: 2, siteId: 3 cannot be started now as is invalid
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Getting job definition for processor 2 and site 3
Nov 05 13:00:30 localhost.localdomain sen2agri-orchestrator[2209]: Scheduler L3A: Getting season dates for site 3 for scheduled date Fri May 31 00:00:00 2019!
Nov 05 13:00:30 localhost.localdomain sen2agri-orchestrator[2209]: IsInSeason: Date not in season (start = Sat May 16 2020, end = Sat Nov 21 2020, current=Fri May 31 2019)
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Job definition got 2
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: The job for processor: 2, siteId: 3 cannot be started now as is invalid
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Getting job definition for processor 4 and site 3
Nov 05 13:00:30 localhost.localdomain sen2agri-orchestrator[2209]: IsInSeason: Date not in season (start = Sat May 16 2020, end = Sat Nov 21 2020, current=Sat May 18 2019)
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Job definition got 4
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: The job for processor: 4, siteId: 3 cannot be started now as is invalid
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Updating ready list to database …
Nov 05 13:00:30 localhost.localdomain sen2agri-scheduler[767]: Updating ready list to database … Done!
Nov 05 13:01:01 localhost.localdomain systemd[1]: Started Runs MACCS on L1C products.
Nov 05 13:01:01 localhost.localdomain systemd[1]: Created slice User Slice of root.
Nov 05 13:01:01 localhost.localdomain systemd[1]: Started Session 4 of user root.
Nov 05 13:01:01 localhost.localdomain CROND[4310]: (root) CMD (run-parts /etc/cron.hourly)
Nov 05 13:01:01 localhost.localdomain run-parts(/etc/cron.hourly)[4319]: starting 0anacron
Nov 05 13:01:01 localhost.localdomain run-parts(/etc/cron.hourly)[4325]: finished 0anacron
Nov 05 13:01:01 localhost.localdomain systemd[1]: Removed slice User Slice of root.
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: CompressTiffs = 0, CogTiffs = 0, RemoveSRE = 1, RemoveFre = 0
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: CompressTiffs = False, CogTiffs = False, RemoveSRE = True, RemoveFre = False
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: Starting thread dmworker_0
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: Starting thread dmworker_1
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: Waiting for queue to join…
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: dmworker_0 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: 2020-11-05 13:01:06.999265:[4298]:dmworker_0: No tile to process. Gracefully closing…
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: dmworker_1 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: 2020-11-05 13:01:06.999765:[4298]:dmworker_1: No tile to process. Gracefully closing…
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: 2020-11-05 13:01:07.000485:[4298]:dmworker_1: Exit thread
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: 2020-11-05 13:01:07.000611:[4298]:dmworker_0: Exit thread
Nov 05 13:01:07 localhost.localdomain demmaccs_launcher.py[4298]: All the workers finished their job. Exiting…

2

Hello,
please did you get any answer on your issue from the Sen2Agri authors? Cosmin … or?
Because I think after reinstalling my system (v.2.0.3) I am dealing with the same issue.
thanks a lot for any feedback,
eva

3

Hi Eva,

No, unfortunately I haven’t had a reply to the issue.

Cheers,

Kevin

4

This is not unusual, because sen2agri sometimes behaves unpredictably. We do many tests with sen2agri and had the same problem.
To me it looks like not enough satellite data has been downloaded to be processed with MAJA. A look into the database may help. There you can find the information about downloaded tiles

select product_name, status_id, no_of_retries, orbit_id, status_description, created_timestamp from downloader_history inner join downloader_status on downloader_status.id = status_id order by status_id DESC;

and MAJA processing.

sudo -u postgres psql sen2agri -c “select downloader_history.status_id, product_name, failed_reason, retry_count from l1_tile_history inner join downloader_history on downloader_history.id = l1_tile_history.downloader_history_id;”

You can find more commands in the sen2agri faq on the website.

Please check if there is enough data available for MAJA and if there are any error messages from MAJA .

5

Hello Eva,

unfortunately it is difficult to get answers. Is the sen2agri project still active?
Interesting that you also report this error. We also have sen2agri 2.0.3 installed and at least MAJA works. But no level3 data.

If you are interested in sharing your experience on sen2agri, we can talk or write. These days we will publish our ansible installation script on git.

6

Hello,

I am part of the team which is still working at UCLouvain and CS Romania on the development of Sen2-Agri. The maintenance and develoment of the system is ensured through the Agricultural Virtual Lab project (even if less intensive than during the project phase). The version 2.0.3 was fully tested and I confirmed that the level 3 and level 4 processors were all working. We will try to go through your different demands in the next / weeks.

Best regards,

Philippe

7

As mentioned, we developed an ansible installation for sen2agri: https://github.com/Open-Forecast/Sen2Agri-ansible. It is still under construction and will be further developed by OpenForecast project.
You are invited to fork and pull.