L2A-Processing Issue

Hi,

I recently started using Sen2agri 1.8 and downloaded some images for my site.
After downloading several L8 and S2-images the L2A processing started.

But now there are only L8 products ready in my GUI:

I think the downloader and l2A-processor are still running
There are already a lot of L2A-Prodocts in my archive (also S2).
Why do they not appear in the GUI-tab “products”?
And is there a possibility to speed up L2A-Processing?

I still have a lot of processing power left and but maccs can only process between 100 and 130 images per day?`

system check results in the following output:

[root@vm-sen2agri archive]# systemctl status sen2agri-services -l
● sen2agri-services.service - Services for Sen2Agri
Loaded: loaded (/usr/lib/systemd/system/sen2agri-services.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2018-04-19 10:35:31 CEST; 3 days ago
Main PID: 20380 (start.sh)
CGroup: /system.slice/sen2agri-services.service
├─20380 /bin/bash /usr/share/sen2agri/sen2agri-services/bin/start.sh
└─20386 java -cp …/modules/:…/lib/ org.esa.sen2agri.ServicesStartup

Apr 23 10:32:49 vm-sen2agri start.sh[20380]: 2018-04-23 10:32:49.285 WARN 20386 — [icesSentinel2-1] Sentinel2Strategy : Either the product S2B_MSIL1C_20170913T100019_N0205_R122_T32TQR_20170913T100017 was not found in the data bucket or the metadata file could not be downloaded
Apr 23 10:32:49 vm-sen2agri start.sh[20380]: 2018-04-23 10:32:49.285 WARN 20386 — [icesSentinel2-1] DownloadStrategy : Product download aborted
Apr 23 10:32:49 vm-sen2agri start.sh[20380]: 2018-04-23 10:32:49.339 WARN 20386 — [icesSentinel2-1] DownloadStrategy : Cannot download http://sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/products/2017/9/22/S2B_MSIL1C_20170922T103009_N0205_R108_T32UNU_20170922T103009/metadata.xml: No such file
Apr 23 10:32:49 vm-sen2agri start.sh[20380]: /mnt/archive/dwn_def/s2/default/at/S2B_MSIL1C_20170922T103009_N0205_R108_T32UNU_20170922T103009.SAFE
Apr 23 10:32:49 vm-sen2agri start.sh[20380]: 2018-04-23 10:32:49.339 WARN 20386 — [icesSentinel2-1] Sentinel2Strategy : Either the product S2B_MSIL1C_20170922T103009_N0205_R108_T32UNU_20170922T103009 was not found in the data bucket or the metadata file could not be downloaded
Apr 23 10:32:49 vm-sen2agri start.sh[20380]: 2018-04-23 10:32:49.339 WARN 20386 — [icesSentinel2-1] DownloadStrategy : Product download aborted
Apr 23 10:32:49 vm-sen2agri start.sh[20380]: 2018-04-23 10:32:49.394 WARN 20386 — [icesSentinel2-1] DownloadStrategy : Cannot download http://sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/products/2017/9/22/S2B_MSIL1C_20170922T103009_N0205_R108_T32TNT_20170922T103009/metadata.xml: No such file
Apr 23 10:32:49 vm-sen2agri start.sh[20380]: /mnt/archive/dwn_def/s2/default/at/S2B_MSIL1C_20170922T103009_N0205_R108_T32TNT_20170922T103009.SAFE
Apr 23 10:32:49 vm-sen2agri start.sh[20380]: 2018-04-23 10:32:49.394 WARN 20386 — [icesSentinel2-1] Sentinel2Strategy : Either the product S2B_MSIL1C_20170922T103009_N0205_R108_T32TNT_20170922T103009 was not found in the data bucket or the metadata file could not be downloaded
Apr 23 10:32:49 vm-sen2agri start.sh[20380]: 2018-04-23 10:32:49.394 WARN 20386 — [icesSentinel2-1] DownloadStrategy : Product download aborted

Hello,

Could you check that you have some Sentinel2 L1C products in the database, please?

sudo -u postgres psql sen2agri -c “select count(*) from downloader_history where status_id = 2 satellite_id = 1”

Also, you can check if there are some Sentinel2 products in the product table by running:

sudo -u postgres psql sen2agri -c “select count(*) from product where satellite_id = 1”

I am asking to check first this as:

  • MACCS is using a 3 months products window before the season. This means the system is downloading products 3 months before the season and process them into L2A but they since 1.8 are not displayed in the products tab as they are not used by the processors.
  • Demmaccs service is using an in-memory queue of products and it will not consider other products until it finishes the current queue. Maybe (but less probable) it created a list of L8 products that it is processing but no S2 was available at that moment.

If you have some S2 products in the downloader_history table you can try restarting the sen2agri-demmaccs service with:

sudo systemctl restart sen2agri-demmaccs

Please let me know.

Best regards,
Cosmin

Hi,

ah now I got it. All the S2-data is there in my archive.
But until now there were only S2-products processed which have been acquired in these three months before my Start of Season.
And the L8-Scenes have been acquired all inside my ‘real’ season. Thats the reason why they are already shown in the GUI.
So I just need to wait until the maccs processor starts to process the S2-Scenes inside my season.

Another question is there a way to speed up l2a-processing?
And does L3A-Processing start automatically when the l2a-products are ready if it’s enabled for the site?

Thanks for the fast answer!
Martin

Hello,

Unfortunately, the L2A-preprocessing was built initially to satisfy the old version of S2 L1C format (that contained several tiles in one product) so the processing is product oriented and not tile oriented. Due to the fact that MACCS needs the previous L2A product in the nominal mode, a parallelization is not possible in this moment. MACCS performs in parallel the processing of L8 and S2 and also sites but not products from the same satellite and the same site.
The change to a tile oriented processing is foreseen in one of the next versions (possible in version 1.9).
In this moment, to process quicker a site, you can do the following operations, if you know the tiles that cover the site:

  • create several sites (let’s say 4 sites) having exactly the same properties (season, shape etc.) called Site_1, … Site_4.
  • Split the tiles in 4 groups and set the tiles filter for each site (assign one group of tiles to each site) using the filter_site_tiles.py script.
  • Enable the sites. The processing will be performed in parallel for 4 S2 products and 4 L8 products.
  • When all the sites are done with the L2A processing, you can create a new site Site_Full that has the exact properties as the ones before but with no sites filter. Before enabling the site, you need to import the L1C and L2A products into this site by:
    - Disable sites Site_1 … Site_4;
    - Using the scripts insert_l2a_product_to_db.py for L2A products (or you can go in the table “product” and change directly the site_id for the sites Site_1, … Site_4 to be the id of Site_Full);
    - Go in the table “downloader_history” and change the site_id for the sites Site_1, … Site_4 to be the id of Site_Full (ex. update downloader_history set site_id = <SITE_FULL_ID> where site_id in (select id from site where short_name in (‘site_1’,‘site_2’,‘site_3’,‘site_4’)) ). This operation needs to be done in order to avoid redownloading and processing the products for Site_FULL
    - You can now enable the Site_FULL and all the products in previous sites will now appear only in site Site_FULL.
    You can now delete the sites Site_1 … Site_4 (by choosing not to delete products)

Best regards,
Cosmin

1 Like

Hi cosmin,

thanks a lot for the detailed description of your workflow!
I’ll try it for my next site and keep you informed!

Martin

Hi All,

Quick question…

How long does/should processing of L2A usually take for a season of about 6 months?

I set my season to start November 2017 and to end June 2018. Site area is about 17 000 hectares.

In the last 3 days since I started the process, it has created L2A products up to the end of December 2017.

Now I know the processor uses product from 3 months before season start. so this means that in the last 3 days my system created 5 months worth of L2A products (Aug to Dec 2017) and it has 6 months of products still left to process to do the full season.

Thus, am I right to assume that at the speed the processing is going now , a whole season of 6 months will take about 6-8 days?

Is this a normal time frame for this kind of processing?

Hello,

I can’t say for sure that it would really take 6 days to process a six month long season.
It largely depends on site extent, download speed and your processor (CPU) speed.
You can consult the appendix C (p. 102-103) of the Software User Manual
Sometimes, the servers are down (amazon, sci hub and usgs), and this might affect also the processing time.

The downloading of new sentinel and landsat images might have stopped and were stuck to a particular date. In my experience, I deleted the said season (through Sen2Agri web interface), and created a new one (with the same season coverage), and the downloading of new images continued. This work around might be ok to you.

As Brentf said, there are many factors that can impact the processing speed. However, 6-8 days for download and pre-processing seems to me very close to our own performance.

Nicolas

Thanks everyone. Know I know that this is the norm and that I should just be patient.