MACCS processing error DEM?


#1

Hi,

Your help is needed, getting desperate…

First of all, I have been through the FAQ on this topic and failed to find an answer.

I currently have S2 and LS level 1 products downloaded (around 192). The site name is Zar3imoroco with one single season from 2018-09-10 to 2019-03-15

The first strange thing is that I get the following from the web interface in the monitoring window:

Site Product Product Type

demmacs is running ok:

jonaszed@localhost ~]$ systemctl status sen2agri-demmaccs.{timer,service}
● sen2agri-demmaccs.timer - Sentinel-2 DEM/MACCS runner
Loaded: loaded (/usr/lib/systemd/system/sen2agri-demmaccs.timer; enabled; vendor preset: disabled)
Active: active (waiting) since Wed 2019-04-10 15:59:14 GMT; 24h ago

Apr 10 15:59:14 localhost.localdomain systemd[1]: Started Sentinel-2 DEM/MACCS runner.

● sen2agri-demmaccs.service - Runs MACCS on L1C products
Loaded: loaded (/usr/lib/systemd/system/sen2agri-demmaccs.service; static; vendor preset: disabled)
Active: inactive (dead) since Thu 2019-04-11 15:59:03 GMT; 12s ago
Process: 2134 ExecStart=/usr/share/sen2agri/sen2agri-demmaccs/demmaccs_launcher.py (code=exited, status=0/SUCCESS)
Main PID: 2134 (code=exited, status=0/SUCCESS)

Apr 11 15:59:03 localhost.localdomain systemd[1]: Started Runs MACCS on L1C products.
Apr 11 15:59:03 localhost.localdomain demmaccs_launcher.py[2134]: CompressTiffs = 0, CogTiffs = 0, RemoveSRE = 1, RemoveFre = 0
Apr 11 15:59:03 localhost.localdomain demmaccs_launcher.py[2134]: CompressTiffs = False, CogTiffs = False, RemoveSRE = True, RemoveFre = False
Apr 11 15:59:03 localhost.localdomain demmaccs_launcher.py[2134]: 2019-04-11 15:59:03.326453:[2134]:No unprocessed L1C found in DB

I understand that it has not found unprocessed L1C products in the DB

but if run

sudo -u postgres psql sen2agri -c “select downloader_history.id, product_name, status_description from downloader_history inner join downloader_status on downloader_status.id = downloader_history.status_id;”

I get a list of failed processes: but none from the

id | product_name | status_description
------±------------------------------------------------------------------±-------------------
1 | LC08_L1TP_203038_20180611_20180615_01_T1 | processing_failed
2 | LC08_L1TP_203039_20180611_20180615_01_T1 | processing_failed
3 | LC08_L1TP_203037_20180611_20180615_01_T1 | processing_failed
4 | LC08_L1TP_201037_20180613_20180703_01_T1 | processing_failed
5 | LC08_L1TP_201036_20180613_20180703_01_T1 | processing_failed
6 | LC08_L1TP_201038_20180613_20180703_01_T1 | processing_failed
7 | LC08_L1TP_202039_20180620_20180703_01_T1 | processing_failed
8 | LC08_L1TP_202037_20180620_20180703_01_T1 | processing_failed
9 | LC08_L1TP_202038_20180620_20180703_01_T1 | processing_failed
10 | LC08_L1TP_202036_20180620_20180703_01_T1 | processing_failed
11 | LC08_L1TP_200036_20180622_20180703_01_T1 | processing_failed
12 | LC08_L1TP_200038_20180622_20180703_01_T1 | processing_failed
13 | LC08_L1TP_200037_20180622_20180703_01_T1 | processing_failed
14 | LC08_L1TP_203038_20180627_20180704_01_T1 | processing_failed
15 | LC08_L1TP_203039_20180627_20180704_01_T1 | processing_failed
16 | LC08_L1TP_203037_20180627_20180704_01_T1 | processing_failed
17 | LC08_L1TP_201037_20180629_20180716_01_T1 | processing_failed
18 | LC08_L1TP_201036_20180629_20180716_01_T1 | processing_failed
19 | LC08_L1TP_201038_20180629_20180716_01_T1 | processing_failed
20 | LC08_L1TP_202039_20180706_20180717_01_T1 | processing_failed
21 | LC08_L1TP_202037_20180706_20180717_01_T1 | processing_failed
24 | LC08_L1TP_200036_20180708_20180717_01_T1 | processing_failed
425 | S2B_MSIL1C_20180721T110619_N0206_R137_T30STB_20180721T150056.SAFE | failed
455 | S2A_MSIL1C_20180723T105621_N0206_R094_T29RPQ_20180723T131758.SAFE | failed
25 | LC08_L1TP_200038_20180708_20180717_01_T1 | processing_failed
23 | LC08_L1TP_202036_20180706_20180717_01_T1 | processing_failed
445 | S2A_MSIL1C_20180723T105621_N0206_R094_T29RQQ_20180723T131758.SAFE | failed
440 | S2A_MSIL1C_20180723T105621_N0206_R094_T30SUD_20180723T131758.SAFE | failed
448 | S2A_MSIL1C_20180723T105621_N0206_R094_T29SQR_20180723T131758.SAFE | failed

(3172 rows)

My question is at this stage, why are they failing?

[jonaszed@localhost archive]$ sudo -u postgres psql sen2agri -c “select id, name from product;”
id | name
----±-----
(0 rows)

confirmation of no processed L2A products available.

Thank you in advance,

Joao


#2

Hi,

Sorry, please ignore the original post and consider this one as it has more information.

Your help is needed, getting desperate…

First of all, I have been through the FAQ on this topic and failed to find an answer.

I currently have S2 and LS level 1 products downloaded (around 192). The site name is Zar3imoroco with one single season from 2018-09-10 to 2019-03-15

The first strange thing is that I get the following from the web interface in the monitoring window:

Site Product ProductType
Zar3iMorocco S2A_MSIL1C_20180907T112111_N0206_R037_T29SNT_20180907T133316 S2
S2A_MSIL1C_20180907T112111_N0206_R037_T29RMP_20180907T133316

That is, two downloads, the first one normal but the second one without Site and Product type and 0% progress, it happens always and also with duplicate when downloading Landsat products too.

demmacs is running ok:

jonaszed@localhost ~]$ systemctl status sen2agri-demmaccs.{timer,service}
● sen2agri-demmaccs.timer - Sentinel-2 DEM/MACCS runner
Loaded: loaded (/usr/lib/systemd/system/sen2agri-demmaccs.timer; enabled; vendor preset: disabled)
Active: active (waiting) since Wed 2019-04-10 15:59:14 GMT; 24h ago

Apr 10 15:59:14 localhost.localdomain systemd[1]: Started Sentinel-2 DEM/MACCS runner.

● sen2agri-demmaccs.service - Runs MACCS on L1C products
Loaded: loaded (/usr/lib/systemd/system/sen2agri-demmaccs.service; static; vendor preset: disabled)
Active: inactive (dead) since Thu 2019-04-11 15:59:03 GMT; 12s ago
Process: 2134 ExecStart=/usr/share/sen2agri/sen2agri-demmaccs/demmaccs_launcher.py (code=exited, status=0/SUCCESS)
Main PID: 2134 (code=exited, status=0/SUCCESS)

Apr 11 15:59:03 localhost.localdomain systemd[1]: Started Runs MACCS on L1C products.
Apr 11 15:59:03 localhost.localdomain demmaccs_launcher.py[2134]: CompressTiffs = 0, CogTiffs = 0, RemoveSRE = 1, RemoveFre = 0
Apr 11 15:59:03 localhost.localdomain demmaccs_launcher.py[2134]: CompressTiffs = False, CogTiffs = False, RemoveSRE = True, RemoveFre = False
Apr 11 15:59:03 localhost.localdomain demmaccs_launcher.py[2134]: 2019-04-11 15:59:03.326453:[2134]:No unprocessed L1C found in DB

I understand that it has not found unprocessed L1C products in the DB

but if run

sudo -u postgres psql sen2agri -c “select downloader_history.id, product_name, status_description from downloader_history inner join downloader_status on downloader_status.id = downloader_history.status_id;”

I get a list of failed processes:

id | product_name | status_description
------±------------------------------------------------------------------±-------------------
1 | LC08_L1TP_203038_20180611_20180615_01_T1 | processing_failed
2 | LC08_L1TP_203039_20180611_20180615_01_T1 | processing_failed
3 | LC08_L1TP_203037_20180611_20180615_01_T1 | processing_failed
4 | LC08_L1TP_201037_20180613_20180703_01_T1 | processing_failed
5 | LC08_L1TP_201036_20180613_20180703_01_T1 | processing_failed
6 | LC08_L1TP_201038_20180613_20180703_01_T1 | processing_failed
7 | LC08_L1TP_202039_20180620_20180703_01_T1 | processing_failed
8 | LC08_L1TP_202037_20180620_20180703_01_T1 | processing_failed
9 | LC08_L1TP_202038_20180620_20180703_01_T1 | processing_failed
10 | LC08_L1TP_202036_20180620_20180703_01_T1 | processing_failed
11 | LC08_L1TP_200036_20180622_20180703_01_T1 | processing_failed
12 | LC08_L1TP_200038_20180622_20180703_01_T1 | processing_failed
13 | LC08_L1TP_200037_20180622_20180703_01_T1 | processing_failed
14 | LC08_L1TP_203038_20180627_20180704_01_T1 | processing_failed
15 | LC08_L1TP_203039_20180627_20180704_01_T1 | processing_failed
16 | LC08_L1TP_203037_20180627_20180704_01_T1 | processing_failed
17 | LC08_L1TP_201037_20180629_20180716_01_T1 | processing_failed
18 | LC08_L1TP_201036_20180629_20180716_01_T1 | processing_failed
19 | LC08_L1TP_201038_20180629_20180716_01_T1 | processing_failed
20 | LC08_L1TP_202039_20180706_20180717_01_T1 | processing_failed
21 | LC08_L1TP_202037_20180706_20180717_01_T1 | processing_failed
24 | LC08_L1TP_200036_20180708_20180717_01_T1 | processing_failed
425 | S2B_MSIL1C_20180721T110619_N0206_R137_T30STB_20180721T150056.SAFE | failed
455 | S2A_MSIL1C_20180723T105621_N0206_R094_T29RPQ_20180723T131758.SAFE | failed
25 | LC08_L1TP_200038_20180708_20180717_01_T1 | processing_failed
23 | LC08_L1TP_202036_20180706_20180717_01_T1 | processing_failed
445 | S2A_MSIL1C_20180723T105621_N0206_R094_T29RQQ_20180723T131758.SAFE | failed
440 | S2A_MSIL1C_20180723T105621_N0206_R094_T30SUD_20180723T131758.SAFE | failed
448 | S2A_MSIL1C_20180723T105621_N0206_R094_T29SQR_20180723T131758.SAFE | failed

(3172 rows)

My question is at this stage, why are they failing?

[jonaszed@localhost archive]$ sudo -u postgres psql sen2agri -c “select id, name from product;”
id | name
----±-----
(0 rows)

confirmation of no processed L2A products available.

another thing I have tried it to go to a product folder tp check its log file:

mnt/archive/maccs_def/…/PRODUCT_NAME/

[jonaszed@localhost S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE]$ cat demmaccs.log

I get the following:

2019-04-11 14:11:06.421937:[25529]:Starting demmaccs
2019-04-11 14:11:06.424057:[25529]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 14:11:06.656061:[25541]:working_dir = /mnt/archive/demmaccs_tmp/25541
2019-04-11 14:11:06.657575:[25541]:dem_working_dir = /mnt/archive/demmaccs_tmp/25541_DEM_TMP
2019-04-11 14:11:06.659079:[25541]:dem_output_dir = /mnt/archive/demmaccs_tmp/25541_DEM_OUT
2019-04-11 14:11:06.663615:[25541]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/25541_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/25541_DEM_OUT
2019-04-11 14:11:08.827343:[25541]:Command finished NOK (res = 1) in 0:00:02.163178 : /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/25541_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/25541_DEM_OUT
2019-04-11 14:11:08.893099:[25541]:DEM failed
2019-04-11 14:11:08.915152:[25529]:Command finished NOK (res = 255) in 0:00:02.491420 : /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 14:11:08.916632:[25529]:demmaccs.py script didn’t work!
2019-04-11 14:11:08.918119:[25529]:Only set the state as processed in downloader_history (no l2a tiles found after maccs) for product /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 15:57:45.777278:[31318]:Starting demmaccs
2019-04-11 15:57:45.779105:[31318]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 15:57:46.018883:[1733]:working_dir = /mnt/archive/demmaccs_tmp/1733
2019-04-11 15:57:46.020792:[1733]:dem_working_dir = /mnt/archive/demmaccs_tmp/1733_DEM_TMP
2019-04-11 15:57:46.022966:[1733]:dem_output_dir = /mnt/archive/demmaccs_tmp/1733_DEM_OUT
2019-04-11 15:57:46.052756:[1733]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/1733_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/1733_DEM_OUT
2019-04-11 15:57:48.143362:[1733]:Command finished NOK (res = 1) in 0:00:02.090216 : /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/1733_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/1733_DEM_OUT
2019-04-11 15:57:48.145410:[1733]:DEM failed
2019-04-11 15:57:48.169416:[31318]:Command finished NOK (res = 255) in 0:00:02.389979 : /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 15:57:48.171002:[31318]:demmaccs.py script didn’t work!
2019-04-11 15:57:48.172542:[31318]:Only set the state as processed in downloader_history (no l2a tiles found after maccs) for product /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 16:20:01.890066:[8270]:Starting demmaccs
2019-04-11 16:20:01.891567:[8270]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 16:20:02.160014:[18961]:working_dir = /mnt/archive/demmaccs_tmp/18961
2019-04-11 16:20:02.161540:[18961]:dem_working_dir = /mnt/archive/demmaccs_tmp/18961_DEM_TMP
2019-04-11 16:20:02.163016:[18961]:dem_output_dir = /mnt/archive/demmaccs_tmp/18961_DEM_OUT
2019-04-11 16:20:02.166481:[18961]:Starting command: /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/18961_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/18961_DEM_OUT
2019-04-11 16:20:04.164770:[18961]:Command finished NOK (res = 1) in 0:00:01.998114 : /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/18961_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/18961_DEM_OUT
2019-04-11 16:20:04.166415:[18961]:DEM failed
2019-04-11 16:20:04.182715:[8270]:Command finished NOK (res = 255) in 0:00:02.291063 : /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maccs/core/5.1/bin/maccs --delete-temp True /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/
2019-04-11 16:20:04.184415:[8270]:demmaccs.py script didn’t work!
2019-04-11 16:20:04.185846:[8270]:Only set the state as processed in downloader_history (no l2a tiles found after maccs) for product /mnt/archive/maccs_def/zar3imorocco/l2a/S2B_MSIL2A_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE/

Can anyone help me?

Thank you in advance,

Joao


#3

Hello,

Could you try to do the following:

sudo su -l sen2agri-service

And then execute:

/usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/18961_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/18961_DEM_OUT

Please check if you can see any error that can help finding the problem (access right, disk space issues, unavailability of some files etc.).

Please let me know.

Best regards,
Cosmin


#4

Hi,

I am getting the following:

[sen2agri-service@localhost ~]$ /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/18961_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/18961_DEM_OUT
bounding box -5 40 0 30
The following SRTM tiles are missing: [‘srtm_36_06.tif’, ‘srtm_36_05.tif’]. Please provide them in the SRTM directory (/mnt/archive/srtm). Will try to continue, but unreliable results
2019-04-12 09:27:25.404662:[15740]:Starting command: gdalbuildvrt -q /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.vrt /mnt/archive/srtm/srtm_36_06.tif /mnt/archive/srtm/srtm_36_05.tif
ERROR 4: `/mnt/archive/srtm/srtm_36_06.tif’ does not exist in the file system,
and is not recognised as a supported dataset name.

Warning 1: Can’t open /mnt/archive/srtm/srtm_36_06.tif. Skipping it
ERROR 4: `/mnt/archive/srtm/srtm_36_05.tif’ does not exist in the file system,
and is not recognised as a supported dataset name.

Warning 1: Can’t open /mnt/archive/srtm/srtm_36_05.tif. Skipping it
2019-04-12 09:27:25.447206:[15740]:Command finished OK (res = 0) in 0:00:00.042565 : gdalbuildvrt -q /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.vrt /mnt/archive/srtm/srtm_36_06.tif /mnt/archive/srtm/srtm_36_05.tif
2019-04-12 09:27:25.447359:[15740]:Starting command: otbcli_BandMath -il /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.vrt -out /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.tif -exp ‘im1b1 == -32768 ? 0 : im1b1’ -progress false

ImageFileReader (0x12b4950)
RTTI typeinfo: otb::ImageFileReader<otb::VectorImage<float, 2u>, otb::DefaultConvertPixelTraits >
Reference Count: 1
Modified Time: 1776
Debug: Off
Object Name:
Observers:
none
Inputs:
Primary: (0)
Indexed Inputs:
0: Primary (0)
No Required Input Names
NumberOfRequiredInputs: 0
Outputs:
Primary: (0x12b50d0)
Indexed Outputs:
0: Primary (0x12b50d0)
NumberOfRequiredOutputs: 1
Number Of Threads: 24
ReleaseDataFlag: Off
ReleaseDataBeforeUpdateFlag: Off
AbortGenerateData: Off
Progress: 0
Multithreader:
RTTI typeinfo: itk::MultiThreader
Reference Count: 1
Modified Time: 1765
Debug: Off
Object Name:
Observers:
none
Thread Count: 24
Global Maximum Number Of Threads: 128
Global Default Number Of Threads: 24
ImageIO: (null)
UserSpecifiedImageIO flag: 0
m_FileName: /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.vrt
m_UseStreaming flag: 1
m_ActualIORegion: ImageIORegion (0x12b4ad0)
Dimension: 2
Index: 0 0
Size: 0 0

m_AdditionalNumber: 0
2019 Apr 12 09:27:26 : Application.logger (FATAL) The following error occurred during application execution : /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/Applications/AppMathParser/app/otbBandMath.cxx:168:
itk::ERROR: BandMath(0x1065790): No input Image set…; please set at least one input image
2019-04-12 09:27:26.672761:[15740]:Command finished NOK (res = 1) in 0:00:01.225415 : otbcli_BandMath -il /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.vrt -out /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.tif -exp ‘im1b1 == -32768 ? 0 : im1b1’ -progress false
2019-04-12 09:27:26.672931:[15740]:Starting command: gdalwarp -q -overwrite -multi -r cubic -t_srs EPSG:32630 -tr 10.0 -10.0 -te 399960.0 3790200.0 509760.0 3900000.0 /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.tif /mnt/archive/demmaccs_tmp/18961_DEM_OUT/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/S2_TEST_AUX_REFDE2_30SVD_0001_ALT_R1.TIF
ERROR 4: `/mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.tif’ does not exist in the file system,
and is not recognised as a supported dataset name.

2019-04-12 09:27:26.764237:[15740]:Command finished NOK (res = 1) in 0:00:00.091290 : gdalwarp -q -overwrite -multi -r cubic -t_srs EPSG:32630 -tr 10.0 -10.0 -te 399960.0 3790200.0 509760.0 3900000.0 /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.tif /mnt/archive/demmaccs_tmp/18961_DEM_OUT/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/S2_TEST_AUX_REFDE2_30SVD_0001_ALT_R1.TIF
/mnt/archive/demmaccs_tmp/18961_DEM_OUT/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/S2_TEST_AUX_REFDE2_30SVD_0001_ALT_R1.TIF|/mnt/archive/demmaccs_tmp/18961_DEM_OUT/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/S2_TEST_AUX_REFDE2_30SVD_0001_ALT_R2.TIF|20|-20
ERROR 4: `/mnt/archive/demmaccs_tmp/18961_DEM_OUT/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/S2_TEST_AUX_REFDE2_30SVD_0001_ALT_R1.TIF’ does not exist in the file system,
and is not recognised as a supported dataset name.

Traceback (most recent call last):
File “/usr/share/sen2agri/sen2agri-demmaccs/dem.py”, line 520, in
res.get(1)
File “/usr/lib64/python2.7/multiprocessing/pool.py”, line 554, in get
raise self._value
AttributeError: ‘NoneType’ object has no attribute ‘RasterXSize’

Apparently some of the SRTM files are missing,

What can I do?

Regards,

Joao


#5

Hello,

Perhaps you should check if the SRTM.zip were properly copied into your system (/mnt/archive/srtm)? I don’t know if the srtm files were corrupted or missing?

Regards,
Brent


#6

Hi Brentf,

Thanks for the quick reply, Some of the files were missing, I am currently extracting then from the zip file and copying then to mnt/archive/srtm

Is this step usually done during the installation or do one needs to extract them manually (together with the SDBM?

Thanks,

Joao


#7

Hello,

Actually it is included in the installation folder: http://www.esa-sen2agri.org/operational-system/system-download/

Regards,
Brent


#8

Hello,

Yes, I have downloaded them but do we have to extract them manually of is it part of the installation scrip? I can’t find the instructions regarding SRTM and SWBD…

I am copying manually the SRTM files now, do I need to copy the SWDB files too?

Thanks


#9

Hello again,

I am getting another strange this:

● sen2agri-monitor-agent.service - Monitoring Agent for Sen2Agri
Loaded: loaded (/usr/lib/systemd/system/sen2agri-monitor-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-04-11 17:26:31 GMT; 16h ago
Main PID: 4205 (sen2agri-monito)
Tasks: 4
CGroup: /system.slice/sen2agri-monitor-agent.service
└─4205 /usr/bin/sen2agri-monitor-agent

Apr 12 10:14:04 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory
Apr 12 10:15:05 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory
Apr 12 10:16:06 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory
Apr 12 10:17:07 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory
Apr 12 10:18:08 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory
Apr 12 10:19:09 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory
Apr 12 10:20:10 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory
Apr 12 10:21:11 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory
Apr 12 10:22:12 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory
Apr 12 10:23:13 localhost.localdomain sen2agri-monitor-agent[4205]: No such file or directory

Is this OK?

Thanks


#10

Yes, you should copy also the swbd files too.
Concerning the archive files the reason why they are not embedded in the package is that:

  • there were some complains from the users that the package is too big if included especially the SRTMs. The remark was reasonable as the SRTM and SWBD do not usually change between the versions and in this case, when passing to a new version, there is no need to download them again. This is why we decided to extract them in separate archives.
  • At the first installation, if you copy the srtm.zip and swbd.zip in the installation package on the same level with the folders (docs, gipp, install_script, etc). then the installer will use them automatically.
  • If the system already finds some folders /mnt/archive/srtm and /mnt/archive/swbd, then it will use these ones and will perform no other extraction.

Cosmin


#11

Done, what shall I do now?

BR


#12

Could you launch again the dem.py command you launched above?


#13

here it goes:

[jonaszed@localhost ~]$ sudo su -l sen2agri-service
Last login: Fri Apr 12 09:27:14 GMT 2019 on pts/0
[sen2agri-service@localhost ~]$ /usr/share/sen2agri/sen2agri-demmaccs/dem.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd -p 20 -w /mnt/archive/demmaccs_tmp/18961_DEM_TMP /mnt/archive/dwn_def/s2/default/zar3imorocco/S2B_MSIL1C_20180906T105639_N0206_R094_T30SVD_20180906T145718.SAFE /mnt/archive/demmaccs_tmp/18961_DEM_OUT
bounding box -5 40 0 30
2019-04-12 11:02:49.054755:[24279]:Starting command: gdalbuildvrt -q /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.vrt /mnt/archive/srtm/srtm_36_06.tif /mnt/archive/srtm/srtm_36_05.tif
2019-04-12 11:02:49.111972:[24279]:Command finished OK (res = 0) in 0:00:00.057273 : gdalbuildvrt -q /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.vrt /mnt/archive/srtm/srtm_36_06.tif /mnt/archive/srtm/srtm_36_05.tif
2019-04-12 11:02:49.112128:[24279]:Starting command: otbcli_BandMath -il /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.vrt -out /mnt/archive/demmaccs_tmp/18961_DEM_TMP/S2_TEST_AUX_REFDE2_30SVD_20180906T105639_0001.DBL.DIR/dem.tif -exp ‘im1b1 == -32768 ? 0 : im1b1’ -progress false

2019 Apr 12 11:02:50 : Application.logger (INFO) Image #1 has 1 components

it continues to run…


#14

Sounds good … once this is done (I assume it will not take too long and it will be successful) you should do the following:

  • stop the services “demmaccs” and “demmaccs.timer”

sudo systemctl stop sen2agri-demmaccs sen2agri-demmaccs.timer

  • Reset the statuses of the failed products in the database such that they will be re-processed (they probably have status 6 in this moment)

sudo -u postgres psql sen2agri -c “update downloader_history set status_id = 2 where status_id = 6”

  • Restart the services “demmaccs” and “demmaccs.timer”

sudo systemctl stop sen2agri-demmaccs sen2agri-demmaccs.timer

Normally, now it should be OK and you should have soon some L2A products.

Cosmin


#15

Thanks,

What is the password for the sen2agri-service user?

I am getting
[jonaszed@localhost ~]$ sudo -u postgres psql sen2agri -c “update downloader_history set status_id = 2 where status_id = 6”
could not change directory to “/home/jonaszed”: Permission denied
psql: warning: extra command-line argument “set” ignored
psql: warning: extra command-line argument “status_id” ignored
psql: warning: extra command-line argument “=” ignored
psql: warning: extra command-line argument “2” ignored
psql: warning: extra command-line argument “where” ignored
psql: warning: extra command-line argument “status_id” ignored
psql: warning: extra command-line argument “=” ignored
psql: warning: extra command-line argument “6”” ignored
psql: FATAL: Peer authentication failed for user “downloader_history”
[jonaszed@localhost ~]$ sudo su -l sen2agri-service
Last login: Fri Apr 12 11:02:36 GMT 2019 on pts/1
[sen2agri-service@localhost ~]$ sudo -u postgres psql sen2agri -c “update downloader_history set status_id = 2 where status_id = 6”

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.

[sudo] password for sen2agri-service:


#16

Hi,
Regarding the status of the monitor-agent, Is it OK to read:

● sen2agri-monitor-agent.service - Monitoring Agent for Sen2Agri
Loaded: loaded (/usr/lib/systemd/system/sen2agri-monitor-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-04-12 10:34:41 GMT; 2h 11min ago
Main PID: 28183 (sen2agri-monito)
Tasks: 4
CGroup: /system.slice/sen2agri-monitor-agent.service
└─28183 /usr/bin/sen2agri-monitor-agent

Apr 12 12:35:41 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory
Apr 12 12:36:42 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory
Apr 12 12:37:43 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory
Apr 12 12:38:44 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory
Apr 12 12:39:45 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory
Apr 12 12:40:46 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory
Apr 12 12:41:47 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory
Apr 12 12:42:48 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory
Apr 12 12:43:49 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory
Apr 12 12:44:50 localhost.localdomain sen2agri-monitor-agent[28183]: No such file or directory

Regards,

Joao


#17

The sen2agri-monitoring is just for grabbing information about the load of the system and providing them to the GUI (in the “system monitoring” tab) so if cannot read something is not blocking the system processing.
Concerning the command that you tried:

  • you should not run the “sudo -u postgres psql sen2agri -c …” commands as “sen2agri-service” user but as another user (the jonaszed user).
  • the errors you get are because you copied from the Web page and tried to execute the commands with some invalid characters (i.e. " are copied incorrectly). What I suggest is to copy the command and replace the " with the characters inserted directly from keyboard and execute it under the jonaszed user, as you did in the first row.

#18

Thanks cudroiu,

You are a star!!!

Fingers crossed now…

Regards


#19

Nothing is showing yet, will give it a couple of days.

Thank you so much