L3B multi-date processing hangs up

I tried running Custom Job via Web UI
L3B Vegetation Status
=> selected “Reprocessing with the last N-Days”,
selected L2A products for all site for duration of one month.

Job never finishes, it just hangs partly done. !
job|690x232

What am I doing wrong? Can it be that one month of data is not enough?
How can I debug this issue?
@lnicola , @cudroiu, anyone ?

Overall I’m trying to find description of how multi-date processing works but I’m not able to find clear description. Does it help to fill in information missing due to clouds by approximating values ?

I’m also having a similar issue with L3B End of Season re-processing.

I have run L3B monodates successfully for my site, and when I run L3B End of Season re-processing, it hangs up on 2/6 tasks being run.

Looking at the commands and error outputs for the 2 tasks, I can see the Error time series build step has an error.
Command:

/usr/bin/otbcli TimeSeriesBuilder -out /mnt/archive/orchestrator_temp/l3b/177/1301-lai-err-time-series-builder/Err_time_series.tif&gdal:co:BIGTIFF=YES -main /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191113T170134_A20190504T161043/TILES/S2AGRI_L3B_A20190504T161043_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190504T161043_T18TVR.TIF -il /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191113T170134_A20190504T161043/TILES/S2AGRI_L3B_A20190504T161043_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190504T161043_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191113T202950_A20190506T160042/TILES/S2AGRI_L3B_A20190506T160042_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190506T160042_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191113T235652_A20190519T161039/TILES/S2AGRI_L3B_A20190519T161039_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190519T161039_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191114T032322_A20190529T161038/TILES/S2AGRI_L3B_A20190529T161038_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190529T161038_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191114T065013_A20190608T161038/TILES/S2AGRI_L3B_A20190608T161038_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190608T161038_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191114T101703_A20190610T160046/TILES/S2AGRI_L3B_A20190610T160046_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190610T160046_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191114T134319_A20190618T161039/TILES/S2AGRI_L3B_A20190618T161039_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190618T161039_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191114T171004_A20190623T161043/TILES/S2AGRI_L3B_A20190623T161043_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190623T161043_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191114T203719_A20190703T161044/TILES/S2AGRI_L3B_A20190703T161044_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190703T161044_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191115T000400_A20190705T160043/TILES/S2AGRI_L3B_A20190705T160043_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190705T160043_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191115T033052_A20190708T161040/TILES/S2AGRI_L3B_A20190708T161040_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190708T161040_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191115T065801_A20190710T160047/TILES/S2AGRI_L3B_A20190710T160047_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190710T160047_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191115T102444_A20190713T161044/TILES/S2AGRI_L3B_A20190713T161044_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190713T161044_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191115T135136_A20190715T160043/TILES/S2AGRI_L3B_A20190715T160043_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190715T160043_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191115T171841_A20190718T161040/TILES/S2AGRI_L3B_A20190718T161040_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190718T161040_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191115T204526_A20190720T160047/TILES/S2AGRI_L3B_A20190720T160047_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190720T160047_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191116T001149_A20190723T161044/TILES/S2AGRI_L3B_A20190723T161044_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190723T161044_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191116T033833_A20190725T160043/TILES/S2AGRI_L3B_A20190725T160043_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190725T160043_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191116T070503_A20190728T161040/TILES/S2AGRI_L3B_A20190728T161040_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190728T161040_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191116T103140_A20190802T161043/TILES/S2AGRI_L3B_A20190802T161043_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190802T161043_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191116T135835_A20190804T160043/TILES/S2AGRI_L3B_A20190804T160043_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190804T160043_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191116T172517_A20190814T160042/TILES/S2AGRI_L3B_A20190814T160042_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190814T160042_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191116T205134_A20190819T160044/TILES/S2AGRI_L3B_A20190819T160044_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190819T160044_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191117T001745_A20190822T161040/TILES/S2AGRI_L3B_A20190822T161040_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190822T161040_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191117T034400_A20190824T160040/TILES/S2AGRI_L3B_A20190824T160040_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190824T160040_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191117T071010_A20190829T160042/TILES/S2AGRI_L3B_A20190829T160042_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190829T160042_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191117T103627_A20190901T161039/TILES/S2AGRI_L3B_A20190901T161039_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190901T161039_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191117T140252_A20190911T161036/TILES/S2AGRI_L3B_A20190911T161036_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190911T161036_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191117T172908_A20190913T160037/TILES/S2AGRI_L3B_A20190913T160037_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190913T160037_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191117T205601_A20190916T161035/TILES/S2AGRI_L3B_A20190916T161035_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190916T161035_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191118T002316_A20190918T160037/TILES/S2AGRI_L3B_A20190918T160037_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190918T160037_T18TVR.TIF /mnt/archive/cfia_try2/l3b_lai//S2AGRI_L3B_PRD_S9_20191118T035005_A20190921T161035/TILES/S2AGRI_L3B_A20190921T161035_T18TVR/QI_DATA/S2AGRI_L3B_MLAIERR_A20190921T161035_T18TVR.TIF

Here is the error output:

ERROR: Missing mandatory parameter: il  0
ERROR: Troubles loading parameter, please check your line argument...
This is the TimeSeriesBuilder application, version 5.0.0
Creates one image from all input images.

Complete documentation: http://www.orfeo-toolbox.org/Applications/TimeSeriesBuilder.html

Parameters: 
        -progress <boolean>        Report progress 
MISSING -il       <string list>    The image files list  (mandatory)
        -out      <string> [pixel] Image containing all bands from the image files list  [pixel=uint8/uint16/int16/uint32/int32/float/double] (default value is float) (mandatory)
        -deqval   <float>          The de-quantification value to be used  (optional, on by default, default value is -1)
        -isflg    <int32>          Specifies if the time series is built of flags  (optional, on by default, default value is 0)
        -main     <string>         The image from the il that is used for the cutting other images  (optional, off by default)
        -inxml    <string>         Load otb application from xml file  (optional, off by default)

Examples: 
otbcli_TimeSeriesBuilder -il image1.tif image2.tif -out result.tif

What I think is happening is the ampersand in the command is being interpreted by bash and the rest of the command past the ampersand is not sent as arguments to the original OTB command.

Running the command manually and completely omitting the ampersand and part after it works as expected. I also had a permission error when running the command manually (due to running as a user other than sen2agri-service) but after adding read and write access to all users to the orchestrator_temp folder, the command started working.

If you can check the OUTPUT for your custom job and see if there is a specific step that has a similar error output. They might be related to the error time series not being able to be generated properly.

I have also the same problem, when trying to generate L3C products witth L3B LAI processor from INRA “activated”. Althoug the mono-date products are generated properly, the process to generate N-days products (multi-date products) fails since it never finishes.

On the other hand, when INRA processor is “not activated” the process for the L3B mono-date generation fails (L3B mono-date is necessary to compute L3C multidate, isn’t it?)

It is not much information about how to solve the L3C generation problems. The sen2agri version I’m using is 2.0.1

I would appreciate any suggestions regarding the aforementioned problem of generating L3C products.