Version 2.0: Issues with S2 products from local storage

Hello,

I’ve recently updated one of our Sen2Agri instances from 1.8 to 2.0. The update seems to have worked just fine, at least I didn’t come across any obvious error messages.

However, for some reason there seems to be an issue with “downloading” Sentinel-2 products from a local directory.

Here are the relevant entries in the datasource table:

 sudo -u postgres psql sen2agri -c "select name, fetch_mode, download_path, local_root, enabled from datasource;"
could not change directory to "/home/centos": Permission denied
        name         | fetch_mode |          download_path          |         local_root          | enabled 
---------------------+---------------+---------------+------------+---------------------------------+-----------------------------+---------
 Amazon Web Services |               |               |          1 | /mnt/archive/dwn_def/s2/default |                             | f
 Scientific Data Hub |     4 | /mnt/archive/dwn_def/s2/default | /mnt/archive/LOCAL_DATA/S2/ | t
 Amazon Web Services |               |               |          1 | /mnt/archive/dwn_def/l8/default |                             | f
 USGS                |          1 | /mnt/archive/dwn_def/l8/default |                             | t
 Scientific Data Hub         1 | /mnt/archive                    |                             | t

As you can see, the local directory is /mnt/archive/LOCAL_DATA/S2/ and the fetchmode is 4, so only symlinks should be created.

The permissions look as follows:

1 drwxrwxr-x. 3 sen2agri-service sen2agri-service 4096 May 29 11:43 LOCAL_DATA

and the format is hierarchical as described in the SUM.

So after service restart, nothing happens and looking at the logs I can see why:

May 30 06:26:37 wfp.sen2agri start.sh[13221]: 2019-May-30 06:26:37.111 WARN [Scientific Data Hub-S2-4] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Product link failed
May 30 06:26:37 wfp.sen2agri start.sh[13221]: 2019-May-30 06:26:37.111 WARN [Scientific Data Hub-S2-8] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Product S2B_MSIL1C_20190421T072619_N0207_R049_T38SQA_20190421T102305 not found in the local archive
May 30 06:26:37 wfp.sen2agri start.sh[13221]: 2019-May-30 06:26:37.111 WARN [Scientific Data Hub-S2-8] r.c.t.d.r.s.d.Sentinel2ArchiveDownloadStrategy - Product link failed
May 30 06:26:37 wfp.sen2agri start.sh[13221]: 2019-May-30 06:26:37.112 INFO [Scientific Data Hub-S2-5] o.e.s.s.ProductDownloadListener - Product S2B_MSIL1C_20190421T072619_N0207_R049_T38SPA_20190421T102305 aborted after 72 retries

Which is odd, because the product exists:

[centos@wfp ~]$ find /mnt/archive/LOCAL_DATA/S2/ -name 'S2B_MSIL1C_20190421T072619_N0207_R049_T38SPA_20190421T102305.SAFE'

/mnt/archive/LOCAL_DATA/S2/2019/04/21/S2B_MSIL1C_20190421T072619_N0207_R049_T38SPA_20190421T102305.SAFE

and contains the imagery:

[centos@wfp ~]$ ls /mnt/archive/LOCAL_DATA/S2/2019/04/21/S2B_MSIL1C_20190421T072619_N0207_R049_T38SPA_20190421T102305.SAFE
AUX_DATA  DATASTRIP  GRANULE  HTML  INSPIRE.xml  MTD_MSIL1C.xml  manifest.safe  rep_info

Our /mnt mount is on an NFS drive with delicate permissions. But the sen2agri-service user should be able to read and write there … could there be still an issue?

Thanks,

Val

Hi Val,

I’m encountering exactly the same issue. Are there any news on it?

best,
Martin

Dear Val and Maatin,

did you find any solutions? I am running into the same issue. The v2.0.1 is not running with my local storage, which worked fine with v1.8.3.

Any one else having the same issue?
Thanks
oli

Hi @buckoli,

sorry for the delayed response.

If you still have the problem, in the file /usr/share/sen2agri/sen2agri-services/config/services.properties you should find a key SciHubDataSource.Sentinel2.path.suffix which lets you fine tune the suffix for local files.

This was not readily available from version 2.0, so it needed to be added manually via patch. Not sure if that’s already added in the newer versions, but I would assume so.

Best,
Val