Some errors on automated and custom job processing

brentf · January 12, 2018, 2:58am

Hi,

I’m posting here some errors I found out in the crop type map processing upon inspecting the output link in the monitoring tab of the web interface.

This is the error when I ran the processor in automated mode:

Number of inputs 4
Computing class counts
Image 0 vector length 137
Image 1 vector length 137
Image 2 vector length 137
Image 3 vector length 137
12 28227.7
13 18683.7
53 75.7222
82 28762
99 10767.3
100 40901.6
230 886.84
440 9528.15
1000 3933.03
1003 9333.57
1004 7120.94
1005 10394.6
2001 9640.04
4001 5318.47
5001 19137.5
5555 6104.4
6666 24170.3
7001 21821.8
8001 629.155

Processing input 0
Image 0 vector length 137
Tile pixels of class 12: 5425
Tile pixels of class 13: 2988
Tile pixels of class 53: 71
Tile pixels of class 82: 5458
Tile pixels of class 99: 750
Tile pixels of class 100: 3971
Tile pixels of class 230: 384
Tile pixels of class 440: 720
Tile pixels of class 1003: 532
Tile pixels of class 1004: 911
Tile pixels of class 1005: 3103
Tile pixels of class 2001: 19
Tile pixels of class 4001: 134
Tile pixels of class 5001: 409
Tile pixels of class 5555: 1418
Tile pixels of class 6666: 1350
Tile pixels of class 7001: 9875
Training samples: 37518
Validation samples: 321
Processing input 1
Image 1 vector length 137
Tile pixels of class 12: 10131
Tile pixels of class 13: 11047
Tile pixels of class 82: 22663
Tile pixels of class 99: 5559
Tile pixels of class 100: 19188
Tile pixels of class 230: 449
Tile pixels of class 440: 1033
Tile pixels of class 1000: 1380
Tile pixels of class 1003: 5303
Tile pixels of class 1004: 5162
Tile pixels of class 1005: 6540
Tile pixels of class 2001: 4510
Tile pixels of class 4001: 4258
Tile pixels of class 5001: 17216
Tile pixels of class 5555: 2405
Tile pixels of class 6666: 19936
Tile pixels of class 7001: 11688
Tile pixels of class 8001: 454
Training samples: 148922
Validation samples: 1317
Processing input 2
Image 2 vector length 137
Tile pixels of class 12: 1621
Tile pixels of class 13: 275
Tile pixels of class 99: 1661
Tile pixels of class 100: 4280
Tile pixels of class 230: 42
Tile pixels of class 440: 513
Tile pixels of class 1000: 857
Tile pixels of class 1003: 3057
Tile pixels of class 1004: 573
Tile pixels of class 1005: 606
Tile pixels of class 2001: 70
Tile pixels of class 4001: 209
Tile pixels of class 5001: 674
Tile pixels of class 5555: 1155
Tile pixels of class 6666: 1092
Training samples: 16685
Validation samples: 127
Processing input 3
Image 3 vector length 137
Tile pixels of class 12: 10335
Tile pixels of class 13: 3475
Tile pixels of class 99: 2677
Tile pixels of class 100: 6058
Tile pixels of class 440: 7098
Tile pixels of class 1000: 1638
Tile pixels of class 1003: 293
Tile pixels of class 1004: 353
Tile pixels of class 2001: 4445
Tile pixels of class 4001: 562
Tile pixels of class 5001: 468
Tile pixels of class 5555: 828
Tile pixels of class 6666: 1331
Tile pixels of class 8001: 170
Training samples: 39731
Validation samples: 343
Total pixels of class 12: 27512
Total pixels of class 13: 17785
Total pixels of class 53: 71
Total pixels of class 82: 28121
Total pixels of class 99: 10647
Total pixels of class 100: 33497
Total pixels of class 230: 875
Total pixels of class 440: 9364
Total pixels of class 1000: 3875
Total pixels of class 1003: 9185
Total pixels of class 1004: 6999
Total pixels of class 1005: 10249
Total pixels of class 2001: 9044
Total pixels of class 4001: 5163
Total pixels of class 5001: 18767
Total pixels of class 5555: 5806
Total pixels of class 6666: 23709
Total pixels of class 7001: 21563
Total pixels of class 8001: 624
Total training samples: 242856
Total validation samples: 2108

    WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/Wrappers/ApplicationEngine/src/otbWrapperOutputImageParameter.cxx, line 311
    OutputImageParameter (0x1dba020): UInt8RGBImageType will be saved in UInt8 format.

    WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 543
    ImageFileWriter (0x3ac6c70): The ImageFactory selected for the image file </mnt/archive/bukidnon_pantaron/l4b//S2AGRI_L4B_PRD_S1_20180108T215956_V20170602_20170915/TILES/S2AGRI_L4B_V20170602_20170915_T51NXJ/S2AGRI_L4B_PVI_V20170602_20170915_T51NXJ.jpg> does not support streaming.

    WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/Wrappers/ApplicationEngine/src/otbWrapperOutputImageParameter.cxx, line 311
    OutputImageParameter (0x229e020): UInt8RGBImageType will be saved in UInt8 format.

    WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 543
    ImageFileWriter (0x422cce0): The ImageFactory selected for the image file </mnt/archive/bukidnon_pantaron/l4b//S2AGRI_L4B_PRD_S1_20180108T215956_V20170602_20170915/TILES/S2AGRI_L4B_V20170602_20170915_T51NYJ/S2AGRI_L4B_PVI_V20170602_20170915_T51NYJ.jpg> does not support streaming.

    WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/Wrappers/ApplicationEngine/src/otbWrapperOutputImageParameter.cxx, line 311
    OutputImageParameter (0x1a95020): UInt8RGBImageType will be saved in UInt8 format.

    WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 543
    ImageFileWriter (0x33e6ce0): The ImageFactory selected for the image file </mnt/archive/bukidnon_pantaron/l4b//S2AGRI_L4B_PRD_S1_20180108T215956_V20170602_20170915/TILES/S2AGRI_L4B_V20170602_20170915_T51PXK/S2AGRI_L4B_PVI_V20170602_20170915_T51PXK.jpg> does not support streaming.

    WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/Wrappers/ApplicationEngine/src/otbWrapperOutputImageParameter.cxx, line 311
    OutputImageParameter (0x17b0020): UInt8RGBImageType will be saved in UInt8 format.

    WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 543
    ImageFileWriter (0x3f6ace0): The ImageFactory selected for the image file </mnt/archive/bukidnon_pantaron/l4b//S2AGRI_L4B_PRD_S1_20180108T215956_V20170602_20170915/TILES/S2AGRI_L4B_V20170602_20170915_T51PYK/S2AGRI_L4B_PVI_V20170602_20170915_T51PYK.jpg> does not support streaming.

WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/Wrappers/ApplicationEngine/src/otbWrapperOutputImageParameter.cxx, line 311
OutputImageParameter (0x299c130): UInt8RGBImageType will be saved in UInt8 format.

WARNING: In /home/cudroiu/sen2agri/packaging/Sen2AgriPlatform/build/OTB-5.0/Modules/IO/ImageIO/include/otbImageFileWriter.txx, line 543
ImageFileWriter (0x277f420): The ImageFactory selected for the image file </mnt/archive/bukidnon_pantaron/l4b//S2AGRI_L4B_PRD_S1_20180108T215956_V20170602_20170915/LEGACY_DATA/S2AGRI_L4B_PVI_LY_V20170602_20170915.jpg> does not support streaming.

Warning 6: JPEG driver doesn't support data type Float32. Only eight bit byte bands supported.

On the other hand, this is the error I found for the custom job processing.

Number of inputs 4
Computing class counts
Image 0 vector length 477
Image 1 vector length 477
Image 2 vector length 477
Image 3 vector length 477
12 72911
13 64163.6
53 19318.9
82 106414
99 46391.9
100 294273
230 3251.78
440 26260.2
1000 9737.71
1003 31510.9
1004 13666.9
1005 11416.6
2001 31962.9
4001 19597.5
5001 1.14825e+06
5555 15604.6
6666 27390.1
7001 71726.1
8001 54661.6

Processing input 0
Image 0 vector length 477
Tile pixels of class 12: 4038
Tile pixels of class 13: 2714
Tile pixels of class 53: 709
Tile pixels of class 82: 2388
Tile pixels of class 99: 1227
Tile pixels of class 100: 566
Tile pixels of class 230: 615
Tile pixels of class 440: 4995
Tile pixels of class 1003: 892
Tile pixels of class 1004: 1568
Tile pixels of class 1005: 3792
Tile pixels of class 2001: 3106
Tile pixels of class 4001: 3120
Tile pixels of class 5001: 8114
Tile pixels of class 5555: 1391
Tile pixels of class 6666: 2483
Tile pixels of class 7001: 7256
Training samples: 48974
Validation samples: 270
Processing input 1
Image 1 vector length 477
Tile pixels of class 12: 14901
Tile pixels of class 13: 30601
Tile pixels of class 53: 4182
Tile pixels of class 82: 32924
Tile pixels of class 99: 27588
Tile pixels of class 100: 9642
Tile pixels of class 230: 1067
Tile pixels of class 440: 5657
Tile pixels of class 1000: 2931
Tile pixels of class 1003: 20233
Tile pixels of class 1004: 9054
Tile pixels of class 1005: 6325
Tile pixels of class 2001: 14435
Tile pixels of class 4001: 8795
Tile pixels of class 5001: 15142
Tile pixels of class 5555: 4253
Tile pixels of class 6666: 20630
Tile pixels of class 7001: 17997
Tile pixels of class 8001: 39509
Training samples: 285866
Validation samples: 1197
Processing input 2
Image 2 vector length 477
Tile pixels of class 12: 4122
Tile pixels of class 13: 1139
Tile pixels of class 53: 8014
Tile pixels of class 82: 3
Tile pixels of class 99: 3253
Tile pixels of class 100: 18823
Tile pixels of class 230: 1077
Tile pixels of class 440: 2223
Tile pixels of class 1000: 2111
Tile pixels of class 1003: 5726
Tile pixels of class 1004: 1571
Tile pixels of class 1005: 1107
Tile pixels of class 2001: 3311
Tile pixels of class 4001: 2021
Tile pixels of class 5001: 2716
Tile pixels of class 5555: 7762
Tile pixels of class 6666: 1232
Tile pixels of class 7001: 11191
Tile pixels of class 8001: 360
Training samples: 77762
Validation samples: 370
Processing input 3
Image 3 vector length 477
Tile pixels of class 12: 16069
Tile pixels of class 13: 4951
Tile pixels of class 53: 5894
Tile pixels of class 82: 4194
Tile pixels of class 99: 7670
Tile pixels of class 100: 10163
Tile pixels of class 230: 429
Tile pixels of class 440: 12913
Tile pixels of class 1000: 4520
Tile pixels of class 1003: 4098
Tile pixels of class 1004: 1257
Tile pixels of class 1005: 12
Tile pixels of class 2001: 9919
Tile pixels of class 4001: 5143
Tile pixels of class 5001: 13231
Tile pixels of class 5555: 1710
Tile pixels of class 6666: 2447
Tile pixels of class 7001: 3320
Tile pixels of class 8001: 189
Training samples: 108129
Validation samples: 565
Total pixels of class 12: 39130
Total pixels of class 13: 39405
Total pixels of class 53: 18799
Total pixels of class 82: 39509
Total pixels of class 99: 39738
Total pixels of class 100: 39194
Total pixels of class 230: 3188
Total pixels of class 440: 25788
Total pixels of class 1000: 9562
Total pixels of class 1003: 30949
Total pixels of class 1004: 13450
Total pixels of class 1005: 11236
Total pixels of class 2001: 30771
Total pixels of class 4001: 19079
Total pixels of class 5001: 39203
Total pixels of class 5555: 15116
Total pixels of class 6666: 26792
Total pixels of class 7001: 39764
Total pixels of class 8001: 40058
Total training samples: 520731
Total validation samples: 2402

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sen2agri_common.py", line 587, in execute
    pool.map(self.internal_classify_tile, self.tiles)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
Exception: ('Error running Rasterize model mask', 1)

lnicola · January 12, 2018, 1:36pm

This is the error when I ran the processor in automated mode:

Those are just some warnings from a library the processor is using. You can safely ignore them. From that log fragment, I assume the processor finished successfully.

On the other hand, this is the error I found for the custom job processing.

If that’s from the “error” log, does the “output” section contain anything else? The error comes from the strata shapefile (see my other reply from yesterday). I suspect it doesn’t have an ID field, although I haven’t tried to reproduce the error.

brentf · January 17, 2018, 3:44am

Hello,

I decided not to include anymore the strata shapefile in the processing based from your advice in this thread:

Coverage of shapefile for strata/multi tile processing

Anyway, it seemed that I don’t get any products in running Sen2Agri automatically, so I always resorted to custom jobs. I’m hoping to generate products automatically because of the multi-tile processing, however, I went back again to custom jobs in the web interface since I have an area which are geographically homogeneous.

Regards,

brentf · January 18, 2018, 3:35am

Hi again,

I realized that I didn’t have my desire products in automated mode because I attempted to cancel the jobs using this commands in FAQs.

Thank you very much.

lnicola · January 19, 2018, 3:04pm

It’s good to hear you’ve figured out the issue. As a note, you can do multi-tile products in custom jobs. It’s enough to select all the products available on the site.