How to generate vegetation status data set?

Akash_Verma · September 8, 2020, 9:35pm

Hello,

I am trying to generate vegetation status (L3B) for my site, L2A Atmospheric correction and L3B LAI mono-date product generates successfully but not L3B vegetation, I am also getting this in logs

Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: Found a number of 59 L2A products not processed in L3B
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190510T153601_N0212_R111_T19TFN_20190510T194648.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190517T152631_N0212_R068_T19TFN_20190517T193710.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190520T153601_N0212_R111_T19TFN_20190520T193604.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190527T152631_N0212_R068_T19TFN_20190527T211208.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190530T153601_N0212_R111_T19TFN_20190530T194254.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190606T152631_N0212_R068_T19TFN_20190606T192859.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190609T153601_N0212_R111_T19TFN_20190609T194321.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190616T152631_N0212_R068_T19TFN_20190616T212543.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190619T153601_N0212_R111_T19TFN_20190619T213201.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190626T152631_N0212_R068_T19TFN_20190626T193400.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190629T153601_N0212_R111_T19TFN_20190629T194342.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190706T152631_N0212_R068_T19TFN_20190706T193003.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190709T153601_N0213_R111_T19TFN_20190709T194214.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190716T152631_N0213_R068_T19TFN_20190716T193341.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190719T153601_N0213_R111_T19TFN_20190719T194639.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190726T152631_N0213_R068_T19TFN_20190726T193334.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190729T153601_N0213_R111_T19TFN_20190729T212732.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190805T152631_N0213_R068_T19TFN_20190805T193249.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190808T153601_N0213_R111_T19TFN_20190808T194734.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190815T152631_N0213_R068_T19TFN_20190815T231911.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190818T153601_N0213_R111_T19TFN_20190818T194832.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190825T152631_N0213_R068_T19TFN_20190825T193154.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190828T153601_N0213_R111_T19TFN_20190828T212036.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190904T152631_N0213_R068_T19TFN_20190904T193011.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190907T153601_N0213_R111_T19TFN_20190907T194501.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190914T152631_N0213_R068_T19TFN_20190914T193624.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190917T153601_N0213_R111_T19TFN_20190917T194355.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190924T152631_N0213_R068_T19TFN_20190924T192150.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2A_MSIL2A_20190927T153601_N0213_R111_T19TFN_20190927T194244.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190512T152639_N0212_R068_T19TFN_20190512T191027.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190515T153609_N0212_R111_T19TFN_20190515T193119.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190522T152639_N0212_R068_T19TFN_20190522T190015.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190525T153609_N0212_R111_T19TFN_20190525T192013.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190601T152639_N0212_R068_T19TFN_20190601T191827.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190604T153609_N0212_R111_T19TFN_20190604T193920.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190611T152639_N0212_R068_T19TFN_20190611T204626.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190614T153609_N0212_R111_T19TFN_20190614T193118.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190621T152639_N0212_R068_T19TFN_20190621T193047.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190624T153609_N0212_R111_T19TFN_20190624T194152.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190701T152639_N0212_R068_T19TFN_20190701T190517.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190704T153609_N0212_R111_T19TFN_20190704T194526.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190711T152639_N0213_R068_T19TFN_20190711T204647.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190714T153609_N0213_R111_T19TFN_20190714T193209.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190721T152639_N0213_R068_T19TFN_20190721T185756.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190724T153609_N0213_R111_T19TFN_20190724T192351.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190731T152639_N0213_R068_T19TFN_20190731T205613.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190803T153609_N0213_R111_T19TFN_20190803T193951.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190810T152639_N0213_R068_T19TFN_20190810T204947.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190813T153609_N0213_R111_T19TFN_20190813T193234.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190820T152629_N0213_R068_T19TFN_20190820T193409.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190823T153559_N0213_R111_T19TFN_20190823T193748.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190830T152629_N0213_R068_T19TFN_20190830T193430.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190902T153559_N0213_R111_T19TFN_20190902T210732.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190909T152629_N0213_R068_T19TFN_20190909T205938.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190912T153559_N0213_R111_T19TFN_20190912T194129.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190919T152629_N0213_R068_T19TFN_20190919T193224.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190922T153559_N0213_R111_T19TFN_20190922T191841.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190922T153559_N0213_R111_T19TFN_20190922T193101.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: ==> Missing L2A from L3B: /mnt/archive/dwn_def/s2/default/field_50388_test/S2B_MSIL2A_20190929T152629_N0213_R068_T19TFN_20190929T193439.SAFE
Sep 08 21:20:40 ’ sen2agri-orchestrator[1388]: A number of 0 L2A products needs to be processed in L3B after checking already launched products
Sep 08 21:31:40 ’ sen2agri-orchestrator[1388]: L3B Scheduled job with id 3468 for site 14 marked as done as no products are available for now to process
Sep 08 21:32:04 ’ systemd[1]: Started Runs MACCS on L1C products.
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: CompressTiffs = 0, CogTiffs = 0, RemoveSRE = 1, RemoveFre = 0
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: CompressTiffs = False, CogTiffs = False, RemoveSRE = True, RemoveFre = False
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: global name ‘file_path’ is not defined
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: Starting thread dmworker_0
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: Starting thread dmworker_1
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: Waiting for queue to join…
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: dmworker_1 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: 2020-09-08 21:32:09.769776:[26709]:dmworker_1: No tile to process. Gracefully closing…
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: 2020-09-08 21:32:09.769986:[26709]:dmworker_1: Exit thread
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: dmworker_0 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: 2020-09-08 21:32:09.771936:[26709]:dmworker_0: No tile to process. Gracefully closing…
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: 2020-09-08 21:32:09.772091:[26709]:dmworker_0: Exit thread
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: All the workers finished their job. Exiting…
Sep 08 21:32:09 ’ demmaccs_launcher.py[26709]: global name ‘file_path’ is not defined
Sep 08 21:32:21 ’ sen2agri-executor[5682]: HandleProcessorEndedMsg: Received message from job name TSKID_6700_STEPNAME_BVLaiNewProcessorFCOVER_0 with status OK and exit code 0

How I can generate vegetation status ?

cudroiu · September 10, 2020, 8:16am

Hello,

From what I see in the logs, the L3B is already running (this is why, in the messages above the orchestrator finds no products to be processed as all 59 L2A products are currently processing, or are in the queue to be processed) .
This is indicated by the last message that specifies that an L3B step just finished successfully so you will have to wait until all steps are finished.
If you have a powerful machine, you can increase the L3B steps parallelism (it is set to 1 by default which means that only one step in the L3B processor is running at a time) using the command (you can use 3 or more):

sudo sacctmgr modify qos qoslai set GrpJobs=3

Best regards,
Cosmin

Akash_Verma · September 10, 2020, 11:35am

Only L3B LAI mono-date product generates automatically in web-interface other L3B other products needs to generate from the custom job? and my machine is having 64 GB RAM and 8 core CPU.

cudroiu · September 10, 2020, 12:42pm

Yes, the L3C and L3D should be launched manually when all the L3B products are created as these processors are using them. But you will have to wait for the L3B products to be finished.

Cosmin

Akash_Verma · September 10, 2020, 12:56pm

Thanks for your help @cudroiu, one thing more I have created L3E Pheno NDVI in custom job for my site it took long to process and when finished it is showing this error

and no product is available in Products tab, can you explain this whats wrong here?