Local data store does not work [1.8.3]

System Installation and Configuration
1

Hi all,

I am trying to make use of a local data store that we have on premise, and which is available on the VM on /mnt/archive/LOCAL_DATA/S2

In the web interface, I set the following parameters under “data sources”

  • Disabled Landsat8 Amazon Web Services

  • Disabled Landsat8 USGS

  • Disabled Sentinel2 - Amazon Web Services

  • Enabled Sentinel - 2 Scientific Data Hub, with the following specs:

image

…but nothing happens.

I restarted sen2agri-services with “sudo systemctl restart sen2agri-services”, but still, nothing happens.

Are my settings correct?

Thanks for your help!

2

Have a look here, it might help with your issue

Have a look what the log files tell and check the entries in the “datasource” table and the “downloader_history”

cheers
oliver

3

I finally managed to make it work with the following settings:

services.properties

################################################
## Data sources default purposes
## 1 = query, 2 = download, 3 = query & download
SciHubDataSource.Sentinel1.scope=1
SciHubDataSource.Sentinel2.scope=3
AWSDataSource.Sentinel2.scope=1
USGSDataSource.Landsat8.scope=1
AWSDataSource.Landsat8.scope=1

## If the local_archive_path is set, the specified local repository will be used for downloading products
SciHubDataSource.Sentinel2.local_archive_path=/mnt/archive/LOCAL/S2
## Possible values for fetch_mode
## 1 = OVERWRITE, 2 = RESUME, 3 = COPY, 4 = SYMLINK
## 3 or 4 to be set if local_archive_path is set to a repository like /eodata/...
SciHubDataSource.Sentinel2.fetch_mode=4

SciHubDataSource.Sentinel1.enabled=false
SciHubDataSource.Sentinel2.enabled=true
AWSDataSource.Sentinel2.enabled=false
USGSDataSource.Landsat8.enabled=false
AWSDataSource.Landsat8.enabled=false

AWSDataSource.Sentinel2.usePadding=true
AWSDataSource.Landsat8.usePadding=true
AWSDataSource.Sentinel2.local.archive.path.format = yyyy/M/d

datasource table

 satellite_id |        name         | scope | fetch_mode |          download_path          |         local_root         
--------------+---------------------+-------+------------+---------------------------------+----------------------------
            1 | Amazon Web Services |     2 |          1 | /mnt/archive/dwn_def/s2/default | 
            1 | Scientific Data Hub |     3 |          4 | /mnt/archive/dwn_def/s2/default | /mnt/archive/LOCAL_DATA/S2
            2 | Amazon Web Services |     2 |          1 | /mnt/archive/dwn_def/l8/default | 
            2 | USGS                |     1 |          1 | /mnt/archive/dwn_def/l8/default | 
            3 | Scientific Data Hub |     3 |          1 | /mnt/archive                    |

Please note, that on the web-portal, in data sources, Fetch Mode displays “Overwrite”, even though in both the properties file and in the DB table it is set to 4 (= symlink). I guess this is a bug.

Another thing I discovered: the passwords are stored in plain text in the datasource table, this should be changed!

I have an additional question: does the service.properties overrule the datasource table (or vice versa) if they differ?

4

Did you modify services.properties file only and the provided above datasource table reflects these changes or you had to modify datasource table too (how to edit this table?)?

In my case I edited services.properties to point to the local collection of S2 data but I don’t see it working and in the WEB interface / data sources TAB the Local root field stays empty. Also the User/Password info has not been updatet although I had changed this information in the services.properties.

5

Hello.

Should I create manually this folder /mnt/archive/LOCAL_DATA/S2 ?

Thanks

6

Hi Yassine,
yes, at least I had to create it manually.

7

hello,
thanks for replying, But I think that the fetch mode 4 requires that the images are already downloaded. So It won’t download them from Scihub .