L3-Processor step ProductFormatter crash

Cloud Free Composite Processor
1

Hi,

L3A-Processor crashes very unluckily on the last step ProductFormatter?
I started a custom Job selecting some tiles from my site because I wanted to execute a testrun for one S2-Tile.
Unfortunatley all intermediate results in /orchestrator_temp have been deleted:frowning:

grafik

Output from the Monitoring-tab:

2018 May 22 16:48:34  :  Application.logger  (FATAL) The following error occurred during application execution : /home/cudroiu/sen2agri/sen2agri-processors/MACCSMetadata/src/ProductFormatter.cpp:831:
itk::ERROR: Error creating lock file for /mnt/archive/at/l3a//S2AGRI_L3A_PRD_S2_20180522T164834_V20170710

executor_log:

● sen2agri-executor.service - Executor for Sen2Agri
   Loaded: loaded (/usr/lib/systemd/system/sen2agri-executor.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2018-05-19 21:03:15 CEST; 2 days ago
 Main PID: 29590 (sen2agri-execut)
   CGroup: /system.slice/sen2agri-executor.service
           └─29590 /usr/bin/sen2agri-executor

May 22 16:48:32 vm-sen2agri sen2agri-executor[29590]: HandleStartProcessor: Executing command srun with params --qos qoscomposite --job-name TSKID_102037_STEPNAME_ProductFormatter_0 /usr/bin/sen2agri-processor-wrapper SRV_IP_ADDR=127.0.0.1 SRV_PORT_NO=7777 WRP_SEND_RETRIES_NO=3600 WRP_TIMEOUT_BETWEEN_RETRIES=1000 WRP_EXECUTES_LOCAL=1 JOB_NAME=TSKID_102037_STEPNAME_ProductFormatter_0 PROC_PATH=/usr/bin/otbcli PROC_PARAMS ProductFormatter -destroot /mnt/archive/at/l3a/ -fileclass SVT1 -level L3A -timeperiod 20170710 -baseline 01.00 -siteid 2 -processor composite -compress 1 -gipp /mnt/archive/orchestrator_temp/l3a/266/102037-product-formatter/executionInfos.xml -outprops /mnt/archive/orchestrator_temp/l3a/266/102037-product-formatter/product_properties.txt -il /mnt/archive/maccs_def/at/l2a/S2B_MSIL2A_20170801T095029_N0205_R079_T33TWN_20170801T095712.SAFE/S2B_OPER_SSC_L2VALD_33TWN____20170801.HDR -lut /usr/share/sen2agri/composite.map -processor.composite.refls TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/101963-composite-splitter/L3AResult_refls.tif TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/102035-composite-splitter/L3AResult_refls.tif -processor.composite.weights TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/101963-composite-splitter/L3AResult_weights.tif TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/102035-composite-splitter/L3AResult_weights.tif -processor.composite.flags TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/101963-composite-splitter/L3AResult_flags.tif TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/102035-composite-splitter/L3AResult_flags.tif -processor.composite.dates TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/101963-composite-splitter/L3AResult_dates.tif TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/102035-composite-splitter/L3AResult_dates.tif -processor.composite.rgb TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/101963-composite-splitter/L3AResult_rgb.tif TILE_33TWN /mnt/archive/orchestrator_temp/l3a/266/102035-composite-splitter/L3AResult_rgb.tif
May 22 16:48:32 vm-sen2agri sen2agri-executor[29590]: HandleStartProcessor: Executing command sbatch with params --job-name TSKID_102037_STEPNAME_ProductFormatter_0 --qos qoscomposite /tmp/sen2agri-executor.t29590
May 22 16:48:32 vm-sen2agri sen2agri-executor[29590]: HandleStartProcessor: Sbatch command returned: "Submitted batch job 8630
May 22 16:48:32 vm-sen2agri sen2agri-executor[29590]: "
May 22 16:48:32 vm-sen2agri sen2agri-executor[29590]: MarkStepPendingStart took 105 ms
May 22 16:48:33 vm-sen2agri sen2agri-executor[29590]: MarkStepStarted took 415 ms
May 22 16:48:34 vm-sen2agri sen2agri-executor[29590]: HandleProcessorEndedMsg: Received message from job name TSKID_102037_STEPNAME_ProductFormatter_0 with status FAILED and exit code 1
May 22 16:48:34 vm-sen2agri sen2agri-executor[29590]: HandleProcessorEndedMsg: Executing command sacct --parsable2 --format=JobID,JobName,NodeList,AveCPU,UserCPU,SystemCPU,ExitCode,AveVMSize,MaxRSS,MaxVMSize,MaxDiskRead,MaxDiskWrite
May 22 16:48:35 vm-sen2agri sen2agri-executor[29590]: Unable to parse SACCT results for job name TSKID_102037_STEPNAME_ProductFormatter_0. Retrying ...
May 22 16:48:36 vm-sen2agri sen2agri-executor[29590]: MarkStepFinished took 120 ms

Martin

2

Can you check the permissions on the /mnt/archive/at/l3a/ directory? The sen2agri-service should have write access to it:

ls -ld /mnt/archive/at/l3a

# and also
ls -ld  /mnt/archive/at/l3a//S2AGRI_L3A_PRD_S2_20180522T164834_V20170710*
1 Like
3

Ok thanks the missing write permission was the problem!

Best,
Martin