Sen2Agri doesn't work after upload site

Do the system logs (journalctl like above) contain anything after reprocessing?

Hi,

I installed sen2agri system in server with 2,7TB, RAM 64GB and 2 Intel E-5520 Quadcore 2,16 GHz. I uploaded one site in Spain and started to download but when I check monitoring:
image

I have errors in L3B product and I don’t know why.

Thanks,
Manuel.

Hello,

I can see in monitoring:

Also, the errror that system shows:
L3B
image

L4A
image

For the L4A you can click the three [Copy to clipboard] links and paste them here.

For L3B you could try to use journalctl -u sen2agri-orchestrator to see why it failed.

Hi,

For L4A I can see

Command:
CropMaskFused.py -ratio 0.75 -nbtrsample 40000 -classifier rf -rseed 0 -pixsize 10 -rfnbtrees 100 -rfmax 25 -rfmin 25 -window 6 -lmbd 2 -eroderad 1 -alpha 0.01 -nbcomp 6 -spatialr 10 -ranger 0.65 -minsize 10 -minarea 20 -siteid 1 -outdir /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/ -targetfolder /mnt/archive/carm/l4a/ -outprops /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/product_properties.txt -refr /mnt/archive/reference_data/ESACCI-LC-L4-LCCS-Map-300m-P5Y-2010-v1.6.1.tif -input /mnt/archive/maccs_def/carm/l2a/S2A_MSIL2A_20170422T110651_N0204_R137_T29SQB_20170422T111300.SAFE/S2A_OPER_SSC_L2VALD_29SQB____20170422.HDR -lut /usr/share/sen2agri/crop-mask.lut

Output:
2018 May 04 22:42:58 : Application.logger (DEBUG) Starting trimming
2018 May 04 22:42:59 : Application.logger (DEBUG) Splitting pixels into classes!
2018 May 04 22:50:55 : Application.logger (DEBUG) Splitting done!
2018 May 04 22:50:55 : Application.logger (DEBUG) Performing Trimming!
2018 May 04 22:50:55 : Application.logger (DEBUG) Trimming done!
2018 May 04 22:51:00 : Application.logger (INFO) Input dataset extent is (-6.75468, 36.907) (-5.47745, 37.9256)
2018 May 04 22:51:00 : Application.logger (INFO) Input dataset projection reference system is: GEOGCS[“GCS_WGS_1984”,DATUM[“WGS_1984”,SPHEROID[“WGS_84”,6378137,298.257223563]],PRIMEM[“Greenwich”,0],UNIT[“Degree”,0.017453292519943295]]
2018 May 04 22:51:00 : Application.logger (INFO) Output projection reference system is: PROJCS[“WGS 84 / UTM zone 29N”,
GEOGCS[“WGS 84”,
DATUM[“WGS_1984”,
SPHEROID[“WGS 84”,6378137,298.257223563,
AUTHORITY[“EPSG”,“7030”]],
AUTHORITY[“EPSG”,“6326”]],
PRIMEM[“Greenwich”,0],
UNIT[“degree”,0.0174532925199433],
AUTHORITY[“EPSG”,“4326”]],
PROJECTION[“Transverse_Mercator”],
PARAMETER[“latitude_of_origin”,0],
PARAMETER[“central_meridian”,-9],
PARAMETER[“scale_factor”,0.9996],
PARAMETER[“false_easting”,500000],
PARAMETER[“false_northing”,0],
UNIT[“metre”,1,
AUTHORITY[“EPSG”,“9001”]],
AUTHORITY[“EPSG”,“32629”]]
2018 May 04 22:51:00 : Application.logger (INFO) Output origin: [699965, 4.2e+06]
2018 May 04 22:51:00 : Application.logger (INFO) Output size: [10980, 10980]
2018 May 04 22:51:00 : Application.logger (INFO) Output spacing: [10, -10]
2018 May 04 22:51:07 : Application.logger (INFO) Image #1 has 1 components

2018 May 04 22:51:07 : Application.logger (INFO) Image #2 has 1 components

Main mission: SENTINEL
Tiles: 1
29SQB 1
Single stratum: True
Increasing NOFILE soft limit from 1024 to 512000
Processing 1 tiles at once
Using 16 threads for each tile
Performing tile preparation (1/2) for tile: 29SQB
Executing PrepareReference_29SQB at 2018-05-04 22:31:13.629294
gdalwarp -q -dstnodata 0 -t_srs ‘PROJCS[“WGS 84 / UTM zone 29N”,
GEOGCS[“WGS 84”,
DATUM[“WGS_1984”,
SPHEROID[“WGS 84”,6378137,298.257223563,
AUTHORITY[“EPSG”,“7030”]],
AUTHORITY[“EPSG”,“6326”]],
PRIMEM[“Greenwich”,0],
UNIT[“degree”,0.0174532925199433],
AUTHORITY[“EPSG”,“4326”]],
PROJECTION[“Transverse_Mercator”],
PARAMETER[“latitude_of_origin”,0],
PARAMETER[“central_meridian”,-9],
PARAMETER[“scale_factor”,0.9996],
PARAMETER[“false_easting”,500000],
PARAMETER[“false_northing”,0],
UNIT[“metre”,1,
AUTHORITY[“EPSG”,“9001”]],
AUTHORITY[“EPSG”,“32629”]]’ -te 699960.0 4090200.0 809760.0 4200000.0 -tr 10 10 -overwrite /mnt/archive/reference_data/ESACCI-LC-L4-LCCS-Map-300m-P5Y-2010-v1.6.1.tif /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/reference-29SQB.tif
PrepareReference_29SQB done at 2018-05-04 22:31:18.343074. Duration: 0:00:04.713780
Executing Erosion_29SQB at 2018-05-04 22:31:18.343227
otbcli Erosion ‘’ -progress false -radius 1 -in /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/reference-29SQB.tif -out /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/reference-eroded-29SQB.tif
Erosion_29SQB done at 2018-05-04 22:37:13.635760. Duration: 0:05:55.292533
Executing SpectralFeatures_29SQB at 2018-05-04 22:37:13.635956
otbcli SpectralFeaturesExtraction ‘’ -progress false -mission SENTINEL -pixsize 10 -lambda 2 -out /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/spectral-features-29SQB.tif -rededge true -il /mnt/archive/maccs_def/carm/l2a/S2A_MSIL2A_20170422T110651_N0204_R137_T29SQB_20170422T111300.SAFE/S2A_OPER_SSC_L2VALD_29SQB____20170422.HDR
SpectralFeatures_29SQB done at 2018-05-04 22:42:56.561200. Duration: 0:05:42.925244
Performing tile preparation (2/2) for tile: 29SQB
Executing Trimming_29SQB at 2018-05-04 22:42:56.610220
otbcli Trimming ‘’ -progress false -alpha 0.01 -nbsamples 0 -seed 0 -feat /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/spectral-features-29SQB.tif -ref /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/reference-eroded-29SQB.tif -out /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/reference-trimmed-29SQB.tif
Trimming_29SQB done at 2018-05-04 22:50:57.706061. Duration: 0:08:01.095841
Building model for stratum: 0
Reference raster for tile: /mnt/archive/maccs_def/carm/l2a/S2A_MSIL2A_20170422T110651_N0204_R137_T29SQB_20170422T111300.SAFE/S2A_OPER_SSC_L2VALD_29SQB____20170422.DBL.DIR/S2A_OPER_SSC_PDTIMG_L2VALD_29SQB____20170422_FRE_R1.DBL.TIF
Executing Rasterize mask at 2018-05-04 22:50:57.785004
otbcli_Rasterization -progress false -in /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/tile-mask-0-29SQB.shp -im /mnt/archive/maccs_def/carm/l2a/S2A_MSIL2A_20170422T110651_N0204_R137_T29SQB_20170422T111300.SAFE/S2A_OPER_SSC_L2VALD_29SQB____20170422.DBL.DIR/S2A_OPER_SSC_PDTIMG_L2VALD_29SQB____20170422_FRE_R1.DBL.TIF -out ‘/mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/classification-mask-0-29SQB.tif?gdal:co:COMPRESS=DEFLATE’ uint8 -mode binary
Rasterize mask done at 2018-05-04 22:51:05.368532. Duration: 0:00:07.583528
Executing BandMath_29SQB at 2018-05-04 22:51:05.368719
otbcli_BandMath -progress false -exp ‘im1b1 > 0 ? im2b1 : -10000’ -il /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/classification-mask-0-29SQB.tif /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/reference-trimmed-29SQB.tif -out ‘/mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/reference-trimmed-0-29SQB.tif?gdal:co:COMPRESS=DEFLATE’ int16
BandMath_29SQB done at 2018-05-04 22:51:14.301648. Duration: 0:00:08.932929
Executing TrainImagesClassifier at 2018-05-04 22:51:14.301886
otbcli TrainImagesClassifierNew ‘’ -nodatalabel -10000 -rand 0 -sample.bm 0 -io.confm2018 May 04 22:51:16 : Application.logger (DEBUG) Entering DoExecute
2018 May 04 22:51:17 : Application.logger (INFO) Processing input 0

2018 May 04 22:51:17 : Application.logger (FATAL) The following error occurred during application execution : /home/cudroiu/sen2agri/sen2agri-processors/Common/TrainImagesClassifierNew/otbListSampleGeneratorRaster.txx:315:
itk::ERROR: No training sample found inside image
atout /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/confusion-matrix-training-0.csv -io.out /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/model-0.txt -sample.mt 40000 -sample.mv 1000 -sample.vfn CROP -sample.vtr 0.01 -classifier rf -classifier.rf.nbtrees 100 -classifier.rf.min 25 -classifier.rf.max 25 -io.rs /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/reference-trimmed-0-29SQB.tif -io.il /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/spectral-features-29SQB.tif
Error running TrainImagesClassifier at 2018-05-04 22:51:17.674916. The call returned 1
Processor finished in 0:20:04.059105

Can you compare the reference map /mnt/archive/reference_data/ESACCI-LC-L4-LCCS-Map-300m-P5Y-2010-v1.6.1.tif against your tile extent?

Or even better, you can take a look at /mnt/archive/orchestrator_temp/l4a/10/3-crop-mask-fused/reference-trimmed-0-29SQB.tif.

It’s possible that the processor didn’t extract enough training samples over your tile.

By one hand, if I compare the ESACCI-LC-L4-LCCS-Map-300m-P5Y-2010-v1.6.1.tif and my file extent (CARM):

By other, in /mnt/archive/orchestrator_temp/l4a/ I don’t have anything:
image

Anyway, for L4A product Error also:

Computing class counts

Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/sen2agri_common.py”, line 581, in execute
pool.map(self.internal_train_stratum, self.strata)
File “/usr/lib64/python2.7/multiprocessing/pool.py”, line 250, in map
return self.map_async(func, iterable, chunksize).get()
File “/usr/lib64/python2.7/multiprocessing/pool.py”, line 554, in get
raise self._value
Exception: (‘Error running TrainImagesClassifier’, 1)

Finally, if the processor didn’t extract enough training samples over your tile, which is it a possible solution?

For L3B:

It is repeat a lot of times…

It doesn’t actually matter, but can you also overlay the 29SQB tile extent? (or send my the coordinates of CARM)? It seems to be a little far from your area, the following is 29SQB:

Also, note that the processor is only looking for pixels with values 10, 11 and 20 in that reference map. I’ve highlighted those in the image above. There seem to be a lot of green them in that area.

There is, however, a filtering step which uses a Mahalanobis trimming process to exclude some of those pixels. I noticed you’re only using two acquisitions, which isn’t a lot, although I would be surprised if every last one was removed.

I would suggest either:

  • keeping the temporary files so that we can check the trimming output (run sudo -u postgres sen2agri psql and in there run update config set value = '1' where key = 'executor.processor.l4a.keep_job_folders';)
  • or using more acquisitions and maybe more tiles (30STG and maybe 30SUG?) as input

CARM has a 200.000ha and the coordinates are:
image
I don’t know why is 29SQB…Yes, it is far from my area.

Can you check the site extent and tile filter in the database? The site is in the site table, and the tiles are in the site_tiles table. You can connect with sudo -u postgres psql sen2agri. Feel free to ask if you need help with that.

CARM has a 200.000ha and the coordinates are:

Is that EPSG:32630?

I noticed you’re only using two acquisitions, which isn’t a lot

I misread the logs, you only have one. You really need more acquisitions for the processor to work.

Sen2agri downloaded automatic all acquisitons…

the projection is ETRS89 (30N) and EPSG:3042

Ok… but now what I do?
image

Thanks for all

Anyway, if the problem is the size of site I’m going to add one shapefile of Sevilla province and I will check if all products can be generated.

So it’s something like this:

You could check:

-- mage the output easier to read
\x
\pset pager off

-- the site extent, as defined in the database
select id, name from site;
select st_astext(geog) from site where id = 72; -- replace 72 with the id from above

-- check the site tiles, if filtered
select * from site_tiles where site_id = 72; -- replace 72 with the site id

-- look at the available L2A products
select name from product where site_id = 72 and processor_id = 1;

DId the job start automatically? It’s possible that not enough L2A were available at the time.

I got now:


**(Sin filas) = (No rows)

Do you see any strange?

By other hand, I uploaded 4 provinces of Spain and I am waitiing to check what the system is going to do.

Many thanks for all.

Hi Inicola,

Here is the system logs after forcing reprocessing

Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: 2018-06-01 13:50:41.732431:[4279]:Starting command: /data/maccs/ --input /mnt/archive/demmaccs_tmp/4015/169078 --TileId 169078 --output /mnt/archive/maccs_def/test_11/l2a/LC08_L2A_169078_20161114_20170318_01_T1//maccs_169078 --mode L2INIT --loglevel DEBUG --enableTest false --CheckXMLFilesWithSchema false
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: Traceback (most recent call last):
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: File “/usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py”, line 456, in
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: pool_outputs = pool.map(maccs_launcher, demmaccs_contexts)
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: File “/usr/lib64/python2.7/multiprocessing/pool.py”, line 250, in map
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: return self.map_async(func, iterable, chunksize).get()
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: File “/usr/lib64/python2.7/multiprocessing/pool.py”, line 554, in get
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: raise self._value
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: OSError: [Errno 13] Permission denied
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: 2018-06-01 13:50:41.756148:[4006]:Command finished NOK (res = 1) in 0:02:41.446013 : /usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 20 --processes-number-maccs 2 --gipp-dir /mnt/archive/gipp/ --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /data/maccs/ --delete-temp True /mnt/archive/dwn_def/l8/default/test_11/LC08_L1TP_169078_20161114_20170318_01_T1 /mnt/archive/maccs_def/test_11/l2a/LC08_L2A_169078_20161114_20170318_01_T1/
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: 2018-06-01 13:50:41.757405:[4006]:demmaccs.py script didn’t work!
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[3999]: 2018-06-01 13:50:41.758218:[4006]:Only set the state as processed in downloader_history (no l2a tiles found after maccs) for product /mnt/archive/maccs_def/test_11/l2a/LC08_L2A_169078_20161114_20170318_01_T1/
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za systemd[1]: Started Runs MACCS on L1C products.
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za systemd[1]: Starting Runs MACCS on L1C products…
Jun 01 13:50:41 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[4304]: 2018-06-01 13:50:41.958993:[4304]:No unprocessed L1C found in DB
Jun 01 13:51:00 mitkldsa01.ict.my-it.co.za systemd[1]: Started Runs MACCS on L1C products.
Jun 01 13:51:00 mitkldsa01.ict.my-it.co.za systemd[1]: Starting Runs MACCS on L1C products…
Jun 01 13:52:00 mitkldsa01.ict.my-it.co.za systemd[1]: Started Runs MACCS on L1C products.
Jun 01 13:52:00 mitkldsa01.ict.my-it.co.za systemd[1]: Starting Runs MACCS on L1C products…
Jun 01 13:53:00 mitkldsa01.ict.my-it.co.za systemd[1]: Started Runs MACCS on L1C products.
Jun 01 13:53:00 mitkldsa01.ict.my-it.co.za systemd[1]: Starting Runs MACCS on L1C products…
Jun 01 13:53:00 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[4401]: 2018-06-01 13:53:00.583977:[4401]:No unprocessed L1C found in DB
Jun 01 13:54:00 mitkldsa01.ict.my-it.co.za systemd[1]: Started Runs MACCS on L1C products.
Jun 01 13:54:00 mitkldsa01.ict.my-it.co.za systemd[1]: Starting Runs MACCS on L1C products…
Jun 01 13:54:00 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[4490]: 2018-06-01 13:54:00.650697:[4490]:No unprocessed L1C found in DB
Jun 01 13:55:00 mitkldsa01.ict.my-it.co.za systemd[1]: Started Runs MACCS on L1C products.
Jun 01 13:55:00 mitkldsa01.ict.my-it.co.za systemd[1]: Starting Runs MACCS on L1C products…
Jun 01 13:55:00 mitkldsa01.ict.my-it.co.za demmaccs_launcher.py[4538]: 2018-06-01 13:55:00.777881:[4538]:No unprocessed L1C found in DB

Hi,

can you check the permissions on /mnt/archive/demmaccs_tmp/ and /mnt/archive/maccs_def/? They should be owned by sen2agri-service.