Error in processing L4B

Hello,

I’m using the version 2.0 of Sen2Agri. I used the custom jobs for L4B and here’s the error log based on the monitoring.

Number of inputs 4
Computing class counts
Image 0 vector length 219
Image 1 vector length 219
Image 2 vector length 219
Image 3 vector length 219
12 155820
13 86707.8
53 29825.4
82 204572

Processing input 0
Image 0 vector length 219
Tile pixels of class 12: 4676
Tile pixels of class 13: 3638
Tile pixels of class 53: 208
Tile pixels of class 82: 10165
Training samples: 18687
Validation samples: 22
Processing input 1
Image 1 vector length 219
Tile pixels of class 12: 9329
Tile pixels of class 13: 30769
Tile pixels of class 53: 3519
Tile pixels of class 82: 26665
Training samples: 70282
Validation samples: 99
Processing input 2
Image 2 vector length 219
Tile pixels of class 12: 12093
Tile pixels of class 13: 795
Tile pixels of class 53: 13690
Training samples: 26578
Validation samples: 140
Processing input 3
Image 3 vector length 219
Tile pixels of class 12: 13347
Tile pixels of class 13: 4049
Tile pixels of class 53: 11737
Tile pixels of class 82: 2630
Training samples: 31763
Validation samples: 132
Total pixels of class 12: 39445
Total pixels of class 13: 39251
Total pixels of class 53: 29154
Total pixels of class 82: 39460
Total training samples: 147310
Total validation samples: 393


Traceback (most recent call last):
  File "/usr/local/bin/gdal_edit.py", line 328, in <module>
    sys.exit(gdal_edit(sys.argv))
  File "/usr/local/bin/gdal_edit.py", line 227, in gdal_edit
    ds = gdal.OpenEx(datasetname, gdal.OF_RASTER | gdal.OF_UPDATE, open_options=open_options)
AttributeError: 'module' object has no attribute 'OpenEx'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sen2agri_common.py", line 609, in execute
    pool.map(self.internal_postprocess_tile, self.tiles)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
Exception: ('Error running Nodata_51NYJ', 1)
Traceback (most recent call last):
  File "/usr/local/bin/gdal_edit.py", line 328, in <module>
    sys.exit(gdal_edit(sys.argv))
  File "/usr/local/bin/gdal_edit.py", line 227, in gdal_edit
    ds = gdal.OpenEx(datasetname, gdal.OF_RASTER | gdal.OF_UPDATE, open_options=open_options)
AttributeError: 'module' object has no attribute 'OpenEx'

I tried another custom job using the same training data (I rechecked it using check_shp.py), and waiting if I will encounter the error again.

Regards to all.

Unfortunately, still encountering the error.

Any ideas?

Hi,

Same problem here. For some reason gdal_edit.py fails to set the nodata value for (random) crop_type_map_masked_*.tif files at the very end of the “postprocess-tiles” step.

(As a workaround, I set the no data value manually to -10000 using a more recent version of GDAL and continue with CropTypeFused.py -mode compute-quality-flags … and CropTypeFused.py -mode validate …. Didn’t encounter any side-effects so far…)

Best regards

thanks. I’ll try your suggestion.

Hi @sbo,

I am also having the same problem. I am new to this system so could you please elaborate on the steps you mentioned.

Kind Regards

Vidya

Hello,

I was not able to try @sbo 's suggestion. I wasn’t able to figure out how to do that.
As a work around, I applied a separate mask (perennial crops from a separate L4B product) using QGIS raster calculator.

Regards.

Has anyone else been able to implement this solution? From the above solution, it seems that the problem is a bug in GDAL version 2.3.3. Is this correct?

Hi @RPA_GI_Services,

I basically run the L4B processor manually from the command line until CropTypeFused.py eventually fails in the postprocess-tiles stage. As far as I understand it, postprocess-tiles merely applies the crop mask to the crop type map, i.e. setting all pixels not identified as crop in the L4A product to 0. Subsequently, postprocess-tiles attempts to set the NoData value of the masked maps with gdal_edit.py to -10000 and for some reason fails (for some tiles). In my case the crop_type_map_masked…tif files are produced without a problem. I just take the crop_type_map_mask files and ‘manually’ set the NoData value to -10000 (i.e. gdal_edit.py -a_nodata -10000 crop_type_map_masked...tif) using a dockerized GDAL 3.1.0 on the same machine. After that, I proceed with the remaining steps by ‘manually’ running CropTypeFused.py -mode compute-quality-flags … and CropTypeFused.py -mode validate ....

Hope that helps (and even more I hope this has been fixed in Version 2.0.1!?)

Best regards

1 Like

Hi @brentf,

Apologies for the late response, was away on leave. Thanks for the suggestion and would try the same.

Regards

Hi @sbo,

Apologies for the late reply, was away on leave. Thanks for taking time in explaining the error and the remedy. I will try to work on it.

Regards