L3A - Failed to start Orchestrator for Sen2Agri

[anouville]

Hi,
I launched processes on Sen2agri on last Friday. My shapefile is covering 3 tiles, all were downloaded and well processed with MAACS.
When I created the site, I defined the season between April 2017 and December 2017 and I enabled the processors L3A, L3B and L4A.
I through everything was ok, because MAACS was running and producing outputs. But this morning, I saw that MAACS was finished and no other process have been started …
You will find some capture of my system.

Thanks for your help.!
Screenshot%20from%202018-08-06%2009-47-48|690x431

[anouville]

I have new informations: I tested the L3A processor on 3 L2A images by command line in a first time and Snap interface in a second time, and both seems to work. Scripts are running without error and I have outputs. So the problem is coming from jobs on the Sen2agri web interface.
Maybe it is related to the capacity of the machine ? I have 4 CPU, 16 GB RAM and 3 TB of storage, but, as i want to process 3 tiles (total of 260 images downloaded and corrected with MAACS), I trough that these capacities were sufficient.
I don’t know if these informations may help.
Thanks

[cudroiu]

Hello,
Normally, the configuration that you have should be enough for the 3 tiles.
The problem seems to be with the orchestrator application that from some reason does not start.
Did you made any manual changes in the tables “processor” and “product_type”?
If not, could you stop the orchestrator (with a systemctl stop) and clean up (truncate) the following tables:

• event
• job
• config_job
• scheduled_task
• scheduled_task_status
• step
• step_resource_log
• task

Please let me know if the orchestrator starts after this operation (after a systemctl start).
Also, after this operation you will have to add again scheduled jobs or execute custom jobs for what you need.

[anouville]

Hi,
thaks for your response. I didn’t made any manual changes. I followed your instructions:

systemctl stop sen2agri-orchestrator
In Pgadmin : TRUNCATE TABLE event job config_job …
systemctl start sen2agri-orchestrator

this last command displays this :

I added a new custom job, but nothing happened

[lnicola]

This is strange. I think I’ve seen it before, and restarting the system fixed it. Can you please try the following?

$ sudo systemctl stop sen2agri-orchestrator
$ sudo yum -y install gdb
$ sudo su -l sen2agri-service
$ strace -f senagri-orchestrator # wait until it crashes, copy the console output
$ gdb --args sen2agri-orchestrator 
(gdb) r
(gdb) bt
(gdb) thread apply all bt
(gdb) quit
# copy the gdb output from above
$ exit

[anouville]

here you will find the copy of the console output for the 2 steps :

log1.txt (62.9 KB)
log2.txt (5.2 KB)

[lnicola]

I don’t know if it will help, but you missed part of the output here:

#21 0x00007ffff7bd0abc in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5
#22 0x00007ffff7b80deb in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5
#23 0x00007ffff79d46c8 in QThread::exec() () from /usr/lib64/libQt5Core.so.5
#24 0x00007ffff79d8b71 in QThreadPrivate::start(void*) ()
---Type <return> to continue, or q <return> to quit---q
 from /usr/lib64/libQt5Quit
(gdb) quit

[anouville]

ah ok i will try to have the end of the message, but for now, after the manipulations, i can’t download anymore. When I create a new site all the downloads have immediately the Failed status. I don’t know what happened it is embarrassing. I tried to uninstall all the system and re install but nothing to do… I can’t download.
But, something has changed, the orchestrator starts because in the “Monitoring” page jobs are created (in “job history”), it was not the case yesterday.

[anouville]

any idea why i can’t download anymore ?

[cudroiu]

It seems that Amazon stopped providing free access to the Sentinel 2 products and we will be forced to switch the downloading to SciHub (which is slower). Support exists in Sen2Agri system and normally it is a matter of a changing a configuration key but unfortunately there is an issue with some metadata that are not correctly downloaded from SciHub causing MACCS to fail. We are working on this and will come with a new version of the system in several days. Sorry for the inconvenience.

[anouville]

Thanks for your response

MACCS fails because there is no L1C product as all downloads fails.
you mean that the problem is general and started since two days ?
So there is no issue on my computer for the download, and I just have to wait the new version ?

thanks a lot for your help.

[cudroiu]

Yes, we will release soon a new version of the system in order to use another data source (SciHub) instead Amazon (which is not free anymore).

[anouville]

Great. Thanks you for the information!