Hi,
Sorry, please ignore the original post and consider this one as it has more information.
Your help is needed, getting desperate…
First of all, I have been through the FAQ on this topic and failed to find an answer.
I currently have S2 and LS level 1 products downloaded (around 192). The site name is Zar3imoroco with one single season from 2018-09-10 to 2019-03-15
The first strange thing is that I get the following from the web interface in the monitoring window:
Site Product ProductType
Zar3iMorocco S2A_MSIL1C_20180907T112111_N0206_R037_T29SNT_20180907T133316 S2
S2A_MSIL1C_20180907T112111_N0206_R037_T29RMP_20180907T133316
That is, two downloads, the first one normal but the second one without Site and Product type and 0% progress, it happens always and also with duplicate when downloading Landsat products too.
demmacs is running ok:
jonaszed@localhost ~]$ systemctl status sen2agri-demmaccs.{timer,service}
● sen2agri-demmaccs.timer - Sentinel-2 DEM/MACCS runner
Loaded: loaded (/usr/lib/systemd/system/sen2agri-demmaccs.timer; enabled; vendor preset: disabled)
Active: active (waiting) since Wed 2019-04-10 15:59:14 GMT; 24h ago
Apr 10 15:59:14 localhost.localdomain systemd[1]: Started Sentinel-2 DEM/MACCS runner.
● sen2agri-demmaccs.service - Runs MACCS on L1C products
Loaded: loaded (/usr/lib/systemd/system/sen2agri-demmaccs.service; static; vendor preset: disabled)
Active: inactive (dead) since Thu 2019-04-11 15:59:03 GMT; 12s ago
Process: 2134 ExecStart=/usr/share/sen2agri/sen2agri-demmaccs/demmaccs_launcher.py (code=exited, status=0/SUCCESS)
Main PID: 2134 (code=exited, status=0/SUCCESS)
Apr 11 15:59:03 localhost.localdomain systemd[1]: Started Runs MACCS on L1C products.
Apr 11 15:59:03 localhost.localdomain demmaccs_launcher.py[2134]: CompressTiffs = 0, CogTiffs = 0, RemoveSRE = 1, RemoveFre = 0
Apr 11 15:59:03 localhost.localdomain demmaccs_launcher.py[2134]: CompressTiffs = False, CogTiffs = False, RemoveSRE = True, RemoveFre = False
Apr 11 15:59:03 localhost.localdomain demmaccs_launcher.py[2134]: 2019-04-11 15:59:03.326453:[2134]:No unprocessed L1C found in DB
I understand that it has not found unprocessed L1C products in the DB
but if run
sudo -u postgres psql sen2agri -c “select downloader_history.id, product_name, status_description from downloader_history inner join downloader_status on downloader_status.id = downloader_history.status_id;”
I get a list of failed processes:
id | product_name | status_description
------±------------------------------------------------------------------±-------------------
1 | LC08_L1TP_203038_20180611_20180615_01_T1 | processing_failed
2 | LC08_L1TP_203039_20180611_20180615_01_T1 | processing_failed
3 | LC08_L1TP_203037_20180611_20180615_01_T1 | processing_failed
4 | LC08_L1TP_201037_20180613_20180703_01_T1 | processing_failed
5 | LC08_L1TP_201036_20180613_20180703_01_T1 | processing_failed
6 | LC08_L1TP_201038_20180613_20180703_01_T1 | processing_failed
7 | LC08_L1TP_202039_20180620_20180703_01_T1 | processing_failed
8 | LC08_L1TP_202037_20180620_20180703_01_T1 | processing_failed
9 | LC08_L1TP_202038_20180620_20180703_01_T1 | processing_failed
10 | LC08_L1TP_202036_20180620_20180703_01_T1 | processing_failed
11 | LC08_L1TP_200036_20180622_20180703_01_T1 | processing_failed
12 | LC08_L1TP_200038_20180622_20180703_01_T1 | processing_failed
13 | LC08_L1TP_200037_20180622_20180703_01_T1 | processing_failed
14 | LC08_L1TP_203038_20180627_20180704_01_T1 | processing_failed
15 | LC08_L1TP_203039_20180627_20180704_01_T1 | processing_failed
16 | LC08_L1TP_203037_20180627_20180704_01_T1 | processing_failed
17 | LC08_L1TP_201037_20180629_20180716_01_T1 | processing_failed
18 | LC08_L1TP_201036_20180629_20180716_01_T1 | processing_failed
19 | LC08_L1TP_201038_20180629_20180716_01_T1 | processing_failed
20 | LC08_L1TP_202039_20180706_20180717_01_T1 | processing_failed
21 | LC08_L1TP_202037_20180706_20180717_01_T1 | processing_failed
24 | LC08_L1TP_200036_20180708_20180717_01_T1 | processing_failed
425 | S2B_MSIL1C_20180721T110619_N0206_R137_T30STB_20180721T150056.SAFE | failed
455 | S2A_MSIL1C_20180723T105621_N0206_R094_T29RPQ_20180723T131758.SAFE | failed
25 | LC08_L1TP_200038_20180708_20180717_01_T1 | processing_failed
23 | LC08_L1TP_202036_20180706_20180717_01_T1 | processing_failed
445 | S2A_MSIL1C_20180723T105621_N0206_R094_T29RQQ_20180723T131758.SAFE | failed
440 | S2A_MSIL1C_20180723T105621_N0206_R094_T30SUD_20180723T131758.SAFE | failed
448 | S2A_MSIL1C_20180723T105621_N0206_R094_T29SQR_20180723T131758.SAFE | failed
…
(3172 rows)
My question is at this stage, why are they failing?
[jonaszed@localhost archive]$ sudo -u postgres psql sen2agri -c “select id, name from product;”
id | name
----±-----
(0 rows)
confirmation of no processed L2A products available.
another thing I have tried it to go to a product folder tp check its log file:
mnt/archive/maccs_def/…/PRODUCT_NAME/
[jonaszed@localhost S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE]$ cat demmaccs.log
I get the following:
2019-04-11 14:11:06.421937:[25529]:Starting demmaccs
2019-04-11 14:11:06.424057:[25529]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 14:11:06.656061:[25541]:working_dir = /mnt/archive/demmaccs_tmp/25541
2019-04-11 14:11:06.657575:[25541]:dem_working_dir = /mnt/archive/demmaccs_tmp/25541_DEM_TMP
2019-04-11 14:11:06.659079:[25541]:dem_output_dir = /mnt/archive/demmaccs_tmp/25541_DEM_OUT
2019-04-11 14:11:06.663615:[25541]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/25541_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/25541_DEM_OUT
2019-04-11 14:11:08.827343:[25541]:Command finished NOK (res = 1) in 0:00:02.163178 : /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/25541_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/25541_DEM_OUT
2019-04-11 14:11:08.893099:[25541]:DEM failed
2019-04-11 14:11:08.915152:[25529]:Command finished NOK (res = 255) in 0:00:02.491420 : /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 14:11:08.916632:[25529]:demmaccs.py script didn’t work!
2019-04-11 14:11:08.918119:[25529]:Only set the state as processed in downloader_history (no l2a tiles found after maccs) for product /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 15:57:45.777278:[31318]:Starting demmaccs
2019-04-11 15:57:45.779105:[31318]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 15:57:46.018883:[1733]:working_dir = /mnt/archive/demmaccs_tmp/1733
2019-04-11 15:57:46.020792:[1733]:dem_working_dir = /mnt/archive/demmaccs_tmp/1733_DEM_TMP
2019-04-11 15:57:46.022966:[1733]:dem_output_dir = /mnt/archive/demmaccs_tmp/1733_DEM_OUT
2019-04-11 15:57:46.052756:[1733]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/1733_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/1733_DEM_OUT
2019-04-11 15:57:48.143362:[1733]:Command finished NOK (res = 1) in 0:00:02.090216 : /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/1733_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/1733_DEM_OUT
2019-04-11 15:57:48.145410:[1733]:DEM failed
2019-04-11 15:57:48.169416:[31318]:Command finished NOK (res = 255) in 0:00:02.389979 : /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 15:57:48.171002:[31318]:demmaccs.py script didn’t work!
2019-04-11 15:57:48.172542:[31318]:Only set the state as processed in downloader_history (no l2a tiles found after maccs) for product /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 16:20:01.890066:[8270]:Starting demmaccs
2019-04-11 16:20:01.891567:[8270]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 16:20:02.160014:[18961]:working_dir = /mnt/archive/demmaccs_tmp/18961
2019-04-11 16:20:02.161540:[18961]:dem_working_dir = /mnt/archive/demmaccs_tmp/18961_DEM_TMP
2019-04-11 16:20:02.163016:[18961]:dem_output_dir = /mnt/archive/demmaccs_tmp/18961_DEM_OUT
2019-04-11 16:20:02.166481:[18961]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/18961_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/18961_DEM_OUT
2019-04-11 16:20:04.164770:[18961]:Command finished NOK (res = 1) in 0:00:01.998114 : /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/18961_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/18961_DEM_OUT
2019-04-11 16:20:04.166415:[18961]:DEM failed
2019-04-11 16:20:04.182715:[8270]:Command finished NOK (res = 255) in 0:00:02.291063 : /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 16:20:04.184415:[8270]:demmaccs.py script didn’t work!
2019-04-11 16:20:04.185846:[8270]:Only set the state as processed in downloader_history (no l2a tiles found after maccs) for product /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
Can anyone help me?
Thank you in advance,
Joao