Adding processed L2 files to new site

General Discussions
1

Hi,

I added previously processed L2 to a new site, expecting they would not be downloaded and processed again. Unfortunately, that does not seem to be the case:

sudo -u postgres psql sen2agri -c "select product_name, status_id from downloader_history where product_name in(select REPLACE(name, 'MSIL2A', 'MSIL1C') as name from product);"
could not change directory to "/home/centos": Permission denied
                           product_name                            | status_id 
-------------------------------------------------------------------+-----------
 S2B_MSIL1C_20170709T075609_N0205_R035_T36NYM_20170709T081038.SAFE |         3
 S2B_MSIL1C_20170709T075609_N0205_R035_T36NXK_20170709T081038.SAFE |         2
 S2B_MSIL1C_20170709T075609_N0205_R035_T36NWK_20170709T081038.SAFE |         1
(3 rows)

Similarly for Landsat-8:

sudo -u postgres psql sen2agri -c "select product_name, status_id from downloader_history where product_name in(select REPLACE(name, 'L2A', 'L1TP') as name from product);"
could not change directory to "/home/centos": Permission denied
               product_name               | status_id 
------------------------------------------+-----------
 LC08_L1TP_171058_20161011_20170320_01_T1 |         1

The already processed products are properly inserted in the products table to the respective site.

Is there any way to add these products to the downloader_history table with status_id 5?
Also, is it possible to add already downloaded files to the downloader_history table with status_id 2 ?

Many thanks.

2

We discussed this over email, but I’m posting here in case anyone else finds this thread.

In the case of Landsat 8, the issue is that the downloader is also trying to download products following the old naming convention. These should no longer be available, but it looks like they are. The next version of the system will have a different downloader implementation, and will possibly fix this issue. Otherwise there is no workaround.

Entries from the product table can be copied to downloader_history using a statement like:

truncate table downloader_history;

insert
into downloader_history(
    site_id,
    satellite_id,
    product_name ,
    full_path,
    created_timestamp,
    status_id,
    no_of_retries,
    product_date,
    orbit_id
)
select site_id,
       satellite_id,
       replace(replace(name, 'MSIL2A', 'MSIL1C'), '_L2A', ''),
       full_path,
       inserted_timestamp,
       5,
       1,
       created_timestamp,
       orbit_id
from product
where product_type_id  = 1;

However, that will only work for old-style L8 products. For new-style products (e.g. LC08_L1TP_171058_20161011_20170320_01_T1), the product table has entries containing the L2A string instead of L1TP. Since in general, L8 products use L1TP, L1GS or L1GT, there is no way to reverse that transformation, and one would need to insert three downloader_history lines for each L8 L2A product.

In the case of the S2 products, I’m not sure what’s causing them to be downloaded again.