Scientific Data Hub Update produced failed Demmaccs

Dear all,

since our download stopped working, we updated query and download of sentinel 2 to be performed by SciHub:
SciHubDataSource.Sentinel2.scope=3

but for some reason we are getting problems in processing base image. I send a attach of the error log, maybe some one can give me some clues.Error.Log (219.5 KB)

Regards,
Joao Rodrigues

Hello,

Unfortunately, there is an issue in the SciHub downloader plugin (some metadata files are not downloaded, causing MACCS to fail). We are working on this and will come with a new version in several days. Sorry for the inconvenience.

Best regards,
Cosmin

To complement Cosmin’s answer, you are right that the Sen2-Agri system is no more downloading the Sentinel-2 images. This is due to a change in the free access policy of AWS to its Sentinel-2 L1C bucket, which was configured as Requester Pays on Tuesday 7th of August, while it was previously configured as Owner pays.

By default, the Sen2-Agri system was configured to download the Sentinel-2 image on AWS, because it is significantly faster than the direct download on the ESA-SciHub. We are currently working on updating the Sentinel-2 downloader and we will deliver you with an updated version in the coming days.

Nicolas

Perfect guys, thank you for the quick feedback.
I will wait for the update 1.8.3 :slight_smile:

Regards,
Joao Rodrigues

Can you provide us a date to when is expected update? Or some other workaround? Our sen2agri is fully stopped, and we did not find any workaround.

Is this still an active issue? I have been wrestling with getting L2A Sentinel2 to be created by DEMMACCS since the AWS switch. I can verify that the L1C Sentinel2 is being downloaded through SciHub, but cannot figure out how to get L2A processing to work. Landsat still working fine. Anyone still having this issue?

Tried to find more information so I can confirm this is the same problem.

Here is the relevant part of the demmaccs log (I think):

Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: - satFilename = /mnt/archive/demmaccs_tmp/29226/23KKR/S2A_OPER_PRD_MSIL1C_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE/GR
ANULE/S2A_OPER_MSI_L1C_TL_SGS__20160808T200408_A005898_T23KKR_N02.04/QI_DATA/S2A_OPER_MSK_SATURA_SGS__20160808T200408_A005898_T23KKR_B02_MSIL1C.gml
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: - defectFilename = /mnt/archive/demmaccs_tmp/29226/23KKR/S2A_OPER_PRD_MSIL1C_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE/GR
ANULE/S2A_OPER_MSI_L1C_TL_SGS__20160808T200408_A005898_T23KKR_N02.04/QI_DATA/S2A_OPER_MSK_DEFECT_SGS__20160808T200408_A005898_T23KKR_B02_MSIL1C.gml
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: - zoneFilename = /mnt/archive/demmaccs_tmp/29226/23KKR/S2A_OPER_PRD_MSIL1C_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE/GR
ANULE/S2A_OPER_MSI_L1C_TL_SGS__20160808T200408_A005898_T23KKR_N02.04/QI_DATA/S2A_OPER_MSK_DETFOO_SGS__20160808T200408_A005898_T23KKR_B02_MSIL1C.gml
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: - nodataFilename = /mnt/archive/demmaccs_tmp/29226/23KKR/S2A_OPER_PRD_MSIL1C_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE/GR
ANULE/S2A_OPER_MSI_L1C_TL_SGS__20160808T200408_A005898_T23KKR_N02.04/QI_DATA/S2A_OPER_MSK_NODATA_SGS__20160808T200408_A005898_T23KKR_B02_MSIL1C.gml [vnsSentinel2L1ImageFileReaderBase.txx:RasterizeGmlMasks:658]
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: 2018-08-17T22:23:18.269455 sen2agri-centos7-v2 maccs-processing 05.01 [000000031992] [D] Start loading the XML file </mnt/archive/demmaccs_tmp/29226/23KKR/S2A_OPER_PRD_MSIL1C_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE/GRANULE/S2A_OPER_MSI_L1C_TL_SGS__20160808T200408_A005898_T23KKR_N02.04/QI_DATA/S2A_OPER_MSK_DETFOO_SGS__20160808T200408_A005898_T23KKR_B02_MSIL1C.gml> … (XMLFileHandler::LoadFile) [vnsXMLFileHandler.cxx:LoadFile:73]
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: 2018-08-17T22:23:18.288250 sen2agri-centos7-v2 maccs-processing 05.01 [000000031992] [E] vns::Data::ERROR: XMLFileHandler(0x4d429c0): XMLFileHandler::LoadFile: Error while loading the file [/mnt/archive/demmaccs_tmp/29226/23KKR/S2A_OPER_PRD_MSIL1C_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE/GRANULE/S2A_OPER_MSI_L1C_TL_SGS__20160808T200408_A005898_T23KKR_N02.04/QI_DATA/S2A_OPER_MSK_DETFOO_SGS__20160808T200408_A005898_T23KKR_B02_MSIL1C.gml]. Parsed with errors
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: Error description: File was not found
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: Error offset: 0 [vnsXMLFileHandler.cxx:LoadFile:88]
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: 2018-08-17T22:23:18.291012 sen2agri-centos7-v2 maccs-processing 05.01 [000000031992] [E] ****************************************************************************************************** [vnsMaccsMainProcessor.cxx:main:145]
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: 2018-08-17T22:23:18.291252 sen2agri-centos7-v2 maccs-processing 05.01 [000000031992] [E] ScientificProcessing: vns::Data::ERROR: XMLFileHandler(0x4d429c0): XMLFileHandler::LoadFile: Error while loading the file [/mnt/archive/demmaccs_tmp/29226/23KKR/S2A_OPER_PRD_MSIL1C_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE/GRANULE/S2A_OPER_MSI_L1C_TL_SGS__20160808T200408_A005898_T23KKR_N02.04/QI_DATA/S2A_OPER_MSK_DETFOO_SGS__20160808T200408_A005898_T23KKR_B02_MSIL1C.gml]. Parsed with errors
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: Error description: File was not found
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: Error offset: 0 [vnsXMLFileHandler.cxx:88] [MACCS Data Exception] [vnsMaccsMainProcessor.cxx:main:145]
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: 2018-08-17T22:23:18.291452 sen2agri-centos7-v2 maccs-processing 05.01 [000000031992] [E] [vnsMaccsMainProcessor.cxx:main:145]
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: 2018-08-17T22:23:18.291762 sen2agri-centos7-v2 maccs-processing 05.01 [000000031992] [E] ****************************************************************************************************** [vnsMaccsMainProcessor.cxx:main:145]
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: 2018-08-17 22:23:18.320242:[31927]:Command finished NOK (res = 133) in 0:03:00.050494 : /opt/maccs/core/5.1/bin/maccs --input /mnt/archive/demmaccs_tmp/29226/23KKR --TileId 23KKR --output /mnt/archive/maccs_def/brazil_onemoretime/l2a/S2A_OPER_PRD_MSIL2A_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE//maccs_23KKR --mode L2INIT --loglevel DEBUG --enableTest false --CheckXMLFilesWithSchema false --conf /usr/share/sen2agri/sen2agri-demmaccs/UserConfiguration
Aug 17 22:23:18 sen2agri-centos7-v2 demmaccs_launcher.py[29195]: 2018-08-17 22:23:18.320397:[31927]:MACCS mode L2INIT didn’t work for /mnt/archive/dwn_def/s2/default/brazil_onemoretime/S2A_OPER_PRD_MSIL1C_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE | TileID: 23KKR. Location /mnt/archive/maccs_def/brazil_onemoretime/l2a/S2A_OPER_PRD_MSIL2A_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE/

So the problem is it can’t load .gml file from the QI_DATA folder maybe?

XMLFileHandler::LoadFile: Error while loading the file [/mnt/archive/demmaccs_tmp/29226/23KKR/S2A_OPER_PRD_MSIL1C_PDMC_20160809T004644_R038_V20160808T132518_20160808T132518.SAFE/GRANULE/S2A_OPER_MSI_L1C_TL_SGS__20160808T200408_A005898_T23KKR_N02.04/QI_DATA/S2A_OPER_MSK_DETFOO_SGS__20160808T200408_A005898_T23KKR_B02_MSIL1C.gml]. Parsed with errors

Hope this helps!

Dear all,
Yes the issue is still active, please see Amazon Web Service change for Sentinel-2 access for more details.
Sorry for the inconvenience it may cause, we are intensively working on it,
Nicolas

Hi, is there maybe any news on progress with this issue?

1 Like