Downloaded Landsat Images Collection Catergory

General Discussions
1

Hello,

I have two machines with Sen2Agri version 1.8.1 installed, with the same site and season dates (for year 2018).
I queried the downloader_history for both machines and I got different results:

For the first machine:

       product_name               |      product_date      | status_id 
------------------------------------------+------------------------+-----------
 LC08_L1TP_113055_20180504_20180504_01_RT | 2018-05-05 00:00:00+08 |         6
 LC08_L1TP_113054_20180504_20180504_01_RT | 2018-05-05 00:00:00+08 |         6
 LC08_L1TP_112054_20180427_20180502_01_T1 | 2018-04-28 00:00:00+08 |         5
 LC08_L1TP_112055_20180427_20180502_01_T1 | 2018-04-28 00:00:00+08 |         5
 LC08_L1TP_112055_20180427_20180427_01_RT | 2018-04-28 00:00:00+08 |         6
 LC08_L1TP_113054_20180418_20180501_01_T1 | 2018-04-19 00:00:00+08 |         5
 LC08_L1TP_113055_20180418_20180418_01_RT | 2018-04-19 00:00:00+08 |         6
 LC08_L1TP_113055_20180418_20180501_01_T1 | 2018-04-19 00:00:00+08 |         5
 LC08_L1TP_113054_20180418_20180418_01_RT | 2018-04-19 00:00:00+08 |         6
 LC08_L1TP_112055_20180411_20180417_01_T1 | 2018-04-12 00:00:00+08 |         5
 LC08_L1TP_112054_20180411_20180417_01_T1 | 2018-04-12 00:00:00+08 |         6
 LC08_L1TP_112055_20180411_20180411_01_RT | 2018-04-12 00:00:00+08 |         6
 LC08_L1TP_112054_20180411_20180411_01_RT | 2018-04-12 00:00:00+08 |         6
 LC08_L1TP_113055_20180402_20180416_01_T1 | 2018-04-03 00:00:00+08 |         5
 LC08_L1TP_113055_20180402_20180402_01_RT | 2018-04-03 00:00:00+08 |         6
 LC08_L1TP_113054_20180402_20180402_01_RT | 2018-04-03 00:00:00+08 |         6
 LC08_L1TP_113054_20180402_20180416_01_T1 | 2018-04-03 00:00:00+08 |         5
 LC08_L1TP_112054_20180326_20180326_01_RT | 2018-03-27 00:00:00+08 |         6
 LC08_L1TP_112054_20180326_20180404_01_T1 | 2018-03-27 00:00:00+08 |         6
 LC08_L1GT_112055_20180326_20180404_01_T2 | 2018-03-27 00:00:00+08 |         6
 LC08_L1TP_113054_20180317_20180402_01_T1 | 2018-03-18 00:00:00+08 |         5
 LC08_L1TP_113055_20180317_20180402_01_T1 | 2018-03-18 00:00:00+08 |         5
 LC08_L1TP_112055_20180310_20180320_01_T1 | 2018-03-11 00:00:00+08 |         5
 LC08_L1TP_112054_20180310_20180320_01_T1 | 2018-03-11 00:00:00+08 |         5
 LC08_L1TP_113054_20180301_20180308_01_T1 | 2018-03-02 00:00:00+08 |         5
 LC08_L1TP_113055_20180301_20180308_01_T1 | 2018-03-02 00:00:00+08 |         5
 LC08_L1TP_112054_20180222_20180308_01_T1 | 2018-02-23 00:00:00+08 |         5
 LC08_L1TP_112055_20180222_20180308_01_T1 | 2018-02-23 00:00:00+08 |         5
 LC08_L1GT_113054_20180213_20180222_01_T2 | 2018-02-14 00:00:00+08 |         6
 LC08_L1TP_113055_20180213_20180222_01_T1 | 2018-02-14 00:00:00+08 |         6
 LC08_L1TP_112055_20180206_20180221_01_T1 | 2018-02-07 00:00:00+08 |         5
 LC08_L1TP_112054_20180206_20180221_01_T1 | 2018-02-07 00:00:00+08 |         5
 LC08_L1TP_113054_20180128_20180207_01_T1 | 2018-01-29 00:00:00+08 |         6
 LC08_L1TP_113055_20180128_20180207_01_T1 | 2018-01-29 00:00:00+08 |         5
 LC08_L1GT_112054_20180121_20180206_01_T2 | 2018-01-22 00:00:00+08 |         6
 LC08_L1TP_112055_20180121_20180206_01_T1 | 2018-01-22 00:00:00+08 |         6
 LC08_L1TP_113055_20180112_20180119_01_T1 | 2018-01-13 00:00:00+08 |         5
 LC08_L1TP_113054_20180112_20180119_01_T1 | 2018-01-13 00:00:00+08 |         5
 LC08_L1TP_112055_20180105_20180118_01_T1 | 2018-01-06 00:00:00+08 |         5

While for the second machine:

       product_name               |      product_date      | status_id 
------------------------------------------+------------------------+-----------
 LC08_L1TP_112055_20180427_20180502_01_T1 | 2018-04-27 00:00:00+08 |         5
 LC08_L1TP_112054_20180427_20180502_01_T1 | 2018-04-27 00:00:00+08 |         5
 LC08_L1TP_112055_20180411_20180417_01_T1 | 2018-04-11 00:00:00+08 |         5
 LC08_L1TP_112054_20180411_20180417_01_T1 | 2018-04-11 00:00:00+08 |         5
 LC08_L1TP_113055_20180213_20180222_01_T1 | 2018-02-13 00:00:00+08 |         6
 LC08_L1TP_112055_20180206_20180221_01_T1 | 2018-02-06 00:00:00+08 |         5
 LC08_L1TP_112054_20180206_20180221_01_T1 | 2018-02-06 00:00:00+08 |         5
 LC08_L1TP_113055_20180128_20180207_01_T1 | 2018-01-28 00:00:00+08 |         5
 LC08_L1TP_113054_20180128_20180207_01_T1 | 2018-01-28 00:00:00+08 |         6
 LC08_L1TP_112055_20180121_20180206_01_T1 | 2018-01-21 00:00:00+08 |         6
 LC08_L1TP_113055_20180112_20180119_01_T1 | 2018-01-12 00:00:00+08 |         5
 LC08_L1TP_113054_20180112_20180119_01_T1 | 2018-01-12 00:00:00+08 |         5
 LC08_L1TP_112055_20180105_20180118_01_T1 | 2018-01-05 00:00:00+08 |         5
 LC08_L1TP_112054_20180105_20180118_01_T1 | 2018-01-05 00:00:00+08 |         5

Is there a way to make them the same? I found out that there was difference in collection category based from the naming convention of Landsat (only Tier 1 products were downloaded in the second machine). Or there were some settings that I need to change in the sen2agri-services?

Best

2

You can make the same by copying the table and products, but the system shouldn’t be using RT and T2 products. This was an bug in the old downloader, but it was fixed a while ago. Did you update one of the machines from an existing install?

3

Yes I updated the system from an existing (1.7) installation for both of the machines.

1st machine:
from 1.7 ===> 1.8.1 stable version

2nd machine:
from 1.7 ==> pre release 1.8 ==> 1.8.1 stable version

4

Hello,

Do you use different datasources on the two systems for L8?
Normally, only the T1 products should be downloaded and not the RT and T2.
Could you send us the structure of the “datasource”, “config” and “site”, please?

Cosmin

5

This is the datasource, config and site for the first machine:

datasource

 id | satellite_id |        name         | scope |       username        |  passwrd   | fetch_mode | max_retries | retry_interval_minutes |          download_path          |                                                                                                                     specific_params                                                                                                                     | max_connections | local_root | enabled 
----+--------------+---------------------+-------+-----------------------+------------+------------+-------------+------------------------+---------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+---------
  4 |            1 | Scientific Data Hub |     1 | brentf                |            |          1 |           3 |                   1440 | /mnt/archive/dwn_def/s2/default | {"parameters":{"dsParameter":[{"name":"cloudcoverpercentage","type":"java.lang.Double","value":"100.0"},{"name":"platformName","type":"java.lang.String","value":"Sentinel-2"},{"name":"productType","type":"java.lang.String","value":"S2MSI1C"}]}}    |               1 |            | t
  5 |            1 | Amazon Web Services |     2 |                       |            |          1 |           3 |                   1440 | /mnt/archive/dwn_def/s2/default | {"parameters":{"dsParameter":[{"name":"cloudcoverpercentage","type":"java.lang.Double","value":"100.0"},{"name":"platformName","type":"java.lang.String","value":"Sentinel-2"}]}}                                                                       |               1 |            | t
  1 |            2 | USGS                |     3 | brentiebark@gmail.com |            |          1 |           3 |                   1440 | /mnt/archive/dwn_def/l8/default | {"parameters":{"dsParameter":[{"name":"cloud_from","type":"java.lang.Double","value":"0.0"},{"name":"cloud_to","type":"java.lang.Double","value":"100.0"},{"name":"satellite_name","type":"java.lang.String","value":"landsat-8"}]}}                    |               1 |            | t
  3 |            3 | Scientific Data Hub |     3 | brentf                |            |          1 |           3 |                   1440 | /mnt/archive                    | {"parameters":{"dsParameter":[{"name":"platformName","type":"java.lang.String","value":"Sentinel-1"},{"name":"productType","type":"java.lang.String","value":"SLC"}]}}                                                                                  |               1 |            | t
  2 |            2 | Amazon Web Services |     2 |                       |            |          1 |           3 |                   1440 | /mnt/archive/dwn_def/l8/default | {"parameters":{"dsParameter":[{"name":"cloudcoverpercentage","type":"java.lang.Double","value":"100.0"},{"name":"collection","type":"java.lang.String","value":"COLLECTION_1"},{"name":"platformName","type":"java.lang.String","value":"Landsat-8"}]}} |               1 |            | f
(5 rows)

config

110 | downloader.start.offset                                |         | 3                                                                          | 2016-07-20 20:05:00+08
 106 | downloader.s2.write-dir                                |         | /mnt/archive/dwn_def/s2/default                                            | 2016-02-27 01:26:49.986675+08
 108 | downloader.s2.max-retries                              |         | 3                                                                          | 2016-03-15 21:44:14.118906+08
 223 | downloader.s2.enabled                                  |       1 | true                                                                       | 2018-05-10 10:14:53.295407+08
 198 | downloader.s2.enabled                                  |         | true                                                                       | 2017-10-24 20:56:57.501918+08
 197 | downloader.s1.enabled                                  |         | false                                                                      | 2017-10-24 20:56:57.501918+08
 111 | downloader.max-cloud-coverage                          |         | 100                                                                        | 2016-02-04 00:05:38.425734+08
 107 | downloader.l8.write-dir                                |         | /mnt/archive/dwn_def/l8/default                                            | 2016-02-27 01:30:06.821627+08
 109 | downloader.l8.max-retries                              |         | 3                                                                          | 2016-03-15 21:44:22.03691+08
 221 | downloader.l8.enabled                                  |       1 | true                                                                       | 2018-05-10 10:14:53.207856+08
 199 | downloader.l8.enabled                                  |         | true                                                                       | 2017-10-24 20:56:57.501918+08
 196 | downloader.enabled                                     |         | true                                                                       | 2017-10-24 20:56:57.501918+08

site

id |       name        |    short_name     |                                                                                                         geog                                                                                                         | enabled 
----+-------------------+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------
  1 | Bukidnon_Pantaron | bukidnon_pantaron | 0106000020E61000000100000001030000000100000005000000E08DC17DB11D5F4021CA5DDBCA2E2140B5550C28A05C5F40A4C20270403021408BB01C0AE75D5F405C5E4175879C1D408EE9D15FF81E5F40F26CF74B9C991D40E08DC17DB11D5F4021CA5DDBCA2E2140 | t

For the second machine:

datasource

 id | satellite_id |        name         | scope |       username        |  passwrd   | fetch_mode | max_retries | retry_interval_minutes |          download_path          |                                                                                                                     specific_params                                                                                                                     | max_connections | local_root | enabled 
----+--------------+---------------------+-------+-----------------------+------------+------------+-------------+------------------------+---------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+---------
  2 |            2 | Amazon Web Services |     2 |                       |            |          1 |           3 |                   1440 | /mnt/archive/dwn_def/l8/default | {"parameters":{"dsParameter":[{"name":"cloudcoverpercentage","type":"java.lang.Double","value":"100.0"},{"name":"collection","type":"java.lang.String","value":"COLLECTION_1"},{"name":"platformName","type":"java.lang.String","value":"Landsat-8"}]}} |               1 |            | f
  4 |            1 | Scientific Data Hub |     1 | brentf                |            |          1 |           3 |                   1440 | /mnt/archive/dwn_def/s2/default | {"parameters":{"dsParameter":[{"name":"cloudcoverpercentage","type":"java.lang.Double","value":"100.0"},{"name":"platformName","type":"java.lang.String","value":"Sentinel-2"},{"name":"productType","type":"java.lang.String","value":"S2MSI1C"}]}}    |               1 |            | t
  5 |            1 | Amazon Web Services |     2 |                       |            |          1 |           3 |                   1440 | /mnt/archive/dwn_def/s2/default | {"parameters":{"dsParameter":[{"name":"cloudcoverpercentage","type":"java.lang.Double","value":"100.0"},{"name":"platformName","type":"java.lang.String","value":"Sentinel-2"}]}}                                                                       |               1 |            | t
  1 |            2 | USGS                |     3 | brentiebark@gmail.com |            |          1 |           3 |                   1440 | /mnt/archive/dwn_def/l8/default | {"parameters":{"dsParameter":[{"name":"cloud_from","type":"java.lang.Double","value":"0.0"},{"name":"cloud_to","type":"java.lang.Double","value":"100.0"},{"name":"satellite_name","type":"java.lang.String","value":"landsat-8"}]}}                    |               1 |            | t
  3 |            3 | Scientific Data Hub |     3 | brentf                |            |          1 |           3 |                   1440 | /mnt/archive                    | {"parameters":{"dsParameter":[{"name":"platformName","type":"java.lang.String","value":"Sentinel-1"},{"name":"productType","type":"java.lang.String","value":"SLC"}]}}                                                                                  |               1 |            | t
(5 rows)

config

 110 | downloader.start.offset                                |         | 3                                                                          | 2016-07-20 20:05:00+08
 106 | downloader.s2.write-dir                                |         | /mnt/archive/dwn_def/s2/default                                            | 2016-02-27 01:26:49.986675+08
 108 | downloader.s2.max-retries                              |         | 3                                                                          | 2016-03-15 21:44:14.118906+08
 601 | downloader.s2.enabled                                  |       2 | true                                                                       | 2018-04-11 19:27:42.426585+08
 576 | downloader.s2.enabled                                  |         | true                                                                       | 2017-10-24 20:56:57.501918+08
 575 | downloader.s1.enabled                                  |         | false                                                                      | 2017-10-24 20:56:57.501918+08
 111 | downloader.max-cloud-coverage                          |         | 100                                                                        | 2016-02-04 00:05:38.425734+08
 107 | downloader.l8.write-dir                                |         | /mnt/archive/dwn_def/l8/default                                            | 2016-02-27 01:30:06.821627+08
 109 | downloader.l8.max-retries                              |         | 3                                                                          | 2016-03-15 21:44:22.03691+08
 577 | downloader.l8.enabled                                  |         | true                                                                       | 2017-10-24 20:56:57.501918+08
 599 | downloader.l8.enabled                                  |       2 | true                                                                       | 2018-04-11 19:27:42.370617+08
 574 | downloader.enabled                                     |         | true                                                                       | 2017-10-24 20:56:57.501918+08

site

 id |    name    | short_name |                                                                                                         geog                                                                                                         | enabled 
----+------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------
  2 | Pantaronv2 | pantaronv2 | 0106000020E61000000100000001030000000100000005000000B08DC17DB11D5F4021CA5DDBCA2E2140E3520C28A05C5F40C0C202704030214084AD1C0AE75D5F408B5E4175879C1D4058E9D15FF81E5F40F56CF74B9C991D40B08DC17DB11D5F4021CA5DDBCA2E2140 | t

In the config I just included the key name, most relevant about this post.

Best.

6

Hi there,

Does the different set of downloaded imagery on my first machine have a significant effect on the products, especially when choosing the L2A input images in custom jobs product processing?

7

Hello,

The two machines seem to be correctly configured (and have the same configuration). Could you check on the one that downloaded more products (including RT and T2 products) if actually the old downloaders are still running (check the sen2agri-landsat-downloader service maybe it was somehow not uninstalled).
Normally, having more products will increase the accuracy of the subsequent L2A products and the final processed products. Nevertheless, the RT and T2 products seem to not be processed by MACCS successfully (they have status 6 -> failed instead of 5 -> success).

Cosmin

8

Hi,

The old downloader processors were not there anymore, meaning they were successfully uninstalled.
Anyway, I think I’ll just ignore this minor issue since the RT and T2 images don’t have an L2A product.
(I reprocessed them many times, and they always fail, maybe due to the system fix as said by Laurentiu)

Best.