MAJA not processing downloaded images?

Hello all,

I’ve recently installed and updated my sen2agri and its new atmospheric correction MAJA.
I disabled from previous sites and added a new site to download images that will be processed by MAJA.
However, I check the downloader_history table and it seems the images were not processed:

site_id |                           product_name                            | status_id |     created_timestamp      
---------+-------------------------------------------------------------------+-----------+----------------------------
       6 | S2B_MSIL1C_20181229T022109_N0207_R003_T51QVU_20181229T040501.SAFE |         1 | 2019-05-20 08:21:58.893+08
       6 | S2B_MSIL1C_20181229T022109_N0207_R003_T51QUV_20181229T040501.SAFE |         2 | 2019-05-20 08:21:58.849+08
       6 | S2A_MSIL1C_20181224T022111_N0207_R003_T51QUU_20181224T041731.SAFE |         2 | 2019-05-20 07:36:05.712+08
       6 | S2A_MSIL1C_20181224T022111_N0207_R003_T51QUV_20181224T041731.SAFE |         2 | 2019-05-20 05:55:06.142+08
       6 | S2A_MSIL1C_20181224T022111_N0207_R003_T51QVV_20181224T041731.SAFE |         2 | 2019-05-20 07:04:16.454+08
       6 | S2A_MSIL1C_20181224T022111_N0207_R003_T51QVU_20181224T041731.SAFE |         2 | 2019-05-20 06:20:51.906+08
       6 | S2B_MSIL1C_20181219T022109_N0207_R003_T51QUU_20181219T051028.SAFE |         2 | 2019-05-20 05:30:10.757+08
       6 | S2B_MSIL1C_20181219T022109_N0207_R003_T51QVV_20181219T051028.SAFE |         2 | 2019-05-20 04:50:52.56+08
       6 | S2B_MSIL1C_20181219T022109_N0207_R003_T51QVU_20181219T051028.SAFE |         2 | 2019-05-20 04:10:37.648+08
       6 | S2B_MSIL1C_20181219T022109_N0207_R003_T51QUV_20181219T051028.SAFE |         2 | 2019-05-20 03:33:54.136+08
       6 | S2A_MSIL1C_20181214T022101_N0207_R003_T51QUU_20181214T055023.SAFE |         2 | 2019-05-20 02:42:32.837+08
       6 | S2A_MSIL1C_20181214T022101_N0207_R003_T51QVV_20181214T055023.SAFE |         2 | 2019-05-20 01:58:05.219+08
       6 | S2A_MSIL1C_20181214T022101_N0207_R003_T51QVU_20181214T055023.SAFE |         2 | 2019-05-20 01:09:51.174+08
       6 | S2A_MSIL1C_20181214T022101_N0207_R003_T51QUV_20181214T055023.SAFE |         2 | 2019-05-20 00:19:49.154+08
       6 | S2B_MSIL1C_20181209T022059_N0207_R003_T51QUV_20181209T040545.SAFE |         2 | 2019-05-19 20:01:36.817+08
       6 | S2B_MSIL1C_20181209T022059_N0207_R003_T51QVU_20181209T040545.SAFE |         2 | 2019-05-19 21:08:17.811+08
       6 | S2B_MSIL1C_20181209T022059_N0207_R003_T51QUU_20181209T040545.SAFE |         2 | 2019-05-19 23:23:54.614+08
       6 | S2B_MSIL1C_20181209T022059_N0207_R003_T51QVV_20181209T040545.SAFE |         2 | 2019-05-19 22:14:27.486+08
       6 | S2A_MSIL1C_20181204T022041_N0207_R003_T51QVU_20181204T041620.SAFE |         2 | 2019-05-19 16:59:54.919+08
       6 | S2A_MSIL1C_20181204T022041_N0207_R003_T51QUV_20181204T041620.SAFE |         2 | 2019-05-19 16:13:26.508+08
       6 | S2A_MSIL1C_20181204T022041_N0207_R003_T51QUU_20181204T041620.SAFE |         2 | 2019-05-19 19:03:46.027+08
       6 | S2A_MSIL1C_20181204T022041_N0207_R003_T51QVV_20181204T041620.SAFE |         2 | 2019-05-19 17:49:48.061+08
       6 | S2B_MSIL1C_20181129T022029_N0207_R003_T51QVU_20181129T040454.SAFE |         2 | 2019-05-19 13:14:40.93+08
       6 | S2B_MSIL1C_20181129T022029_N0207_R003_T51QUV_20181129T040454.SAFE |         2 | 2019-05-19 12:09:06.887+08
       6 | S2B_MSIL1C_20181129T022029_N0207_R003_T51QUU_20181129T040454.SAFE |         2 | 2019-05-19 15:19:15.921+08
       6 | S2B_MSIL1C_20181129T022029_N0207_R003_T51QVV_20181129T040454.SAFE |         2 | 2019-05-19 14:08:48.337+08
       6 | S2A_MSIL1C_20181124T022011_N0207_R003_T51QVU_20181124T042639.SAFE |         2 | 2019-05-19 09:22:08.286+08

How to check and make MAJA work? By the way, here’s the screenshot of my sen2agri-config:

Hello,

Have you also updated the GIPP files for MAJA? If not, you will have to update also the GIPP files for MAJA.
You can either use the update_gipp.sh script or you can do it manually by:

  • copy the gipp_maja folder from the installation package into the /mnt/archive/gipp_maja
  • update the config table in order to have into the key ‘demmaccs.gips-path’ the value ‘/mnt/archive/gipp_maja’
  • restart the demmaccs service
    Also, please note that starting with version 2.0 there is another useful table called “l1_tile_history”. Here are some status reasons and informations about the processing of each tile of the products. You could check here also the status reason to see if there are some useful information that could help you finding the problem.
    Please note that starting from 2.0 a retry mechanism was also implemented (a number of 3 retries are performed, with a timeout of 1 day) so basically the product will stay with status 2 until al 3 retries were performed (3 days) and after that the status will be set to 6 (processing failed).

Hope this helps.

Best regards,
Cosmin

Hello Cosmin,

I tried the manual method and here are the results:

   0 drwxr-xr-x. 7 sen2agri-service sen2agri-service     95 Nov 20 22:28 bukidnon_pantaron
   0 drwxr-xr-x. 3 sen2agri-service sen2agri-service     30 May 17 19:24 demmaccs_tmp
   0 drwxr-xr-x. 4 sen2agri-service sen2agri-service     38 Jan  2  2018 dwn_def
   0 drwxr-xr-x. 4 root             root                 51 May 17 18:10 gipp
   0 drwxr-xr-x. 4 brentf           brentf               51 Feb 21 15:27 gipp_maja
   0 drwx------. 4 sen2agri-service sen2agri-service     51 Jan  2  2018 gipp_OLD_2018_10_17T15_51_50
   0 drwxr-xr-x. 4 root             root                 39 Oct 17  2018 gipp_OLD_2018_11_15T11_08_43
   0 drwxr-xr-x. 4 root             root                 51 Nov 15  2018 gipp_OLD_2019_05_17T18_10_03
   key           |                   value                    | site_id 
-------------------------+--------------------------------------------+---------
 demmaccs.srtm-path      | /mnt/archive/srtm                          |        
 demmaccs.swbd-path      | /mnt/archive/swbd                          |        
 demmaccs.working-dir    | /mnt/archive/demmaccs_tmp/                 |        
 demmaccs.output-path    | /mnt/archive/maccs_def/{site}/{processor}/ |        
 demmaccs.maccs-launcher | /opt/maja/bin/maja                         |        
 demmaccs.gips-path      | /mnt/archive/gipp_maja                     |        
 demmaccs.remove-fre     | false                                      |        
 demmaccs.remove-sre     | true                                       |        
 demmaccs.compress-tiffs | true                                       |        
 demmaccs.cog-tiffs      | true                                       |     

Restarted the demmaccs service but here is the result:

May 20 17:37:29 nirvana-ESSC_MNL systemd[1]: Started Runs MACCS on L1C products.
May 20 17:37:29 nirvana-ESSC_MNL demmaccs_launcher.py[31197]: CompressTiffs = true, CogTiffs = true, RemoveSRE = true, RemoveFre = false
May 20 17:37:29 nirvana-ESSC_MNL demmaccs_launcher.py[31197]: CompressTiffs = True, CogTiffs = True, RemoveSRE = True, RemoveFre = False
May 20 17:37:29 nirvana-ESSC_MNL demmaccs_launcher.py[31197]: Starting thread dmworker_0
May 20 17:37:29 nirvana-ESSC_MNL demmaccs_launcher.py[31197]: Starting thread dmworker_1
May 20 17:37:29 nirvana-ESSC_MNL demmaccs_launcher.py[31197]: MainThread:139921279187984: Exception 42883 received when trying to execute sql queries: select
* from sp_clear_pending_l1_tiles();. Number of retries = 0
May 20 17:37:29 nirvana-ESSC_MNL demmaccs_launcher.py[31197]: MainThread:139921279187984: Exception 23503 received when trying to execute sql queries: select
* from sp_start_l1_tile_processing();. Number of retries = 0

What’s the reason why there was an exception received when trying to execute sql..?
Also, my l1_tile_history table has no data.

Regards,
Brent

Hello Brent,

Did you encountered any errors during the update of the system?
Is the version 2.0 displayed on the Web system interface or it is still displayed the version 1.8.3?
Could you also provide me also :

  • the structure of the l1_tile_history (sudo -u postgres psql sen2agri -c “\d l1_tile_history”)
  • the implementation of the function sp_clear_pending_l1_tiles (sudo -u postgres psql sen4cap -c “\sf sp_clear_pending_l1_tiles”).
    You could also try to execute the function sp_clear_pending_l1_tiles and see the actual error message using the command:

sudo -u postgres psql sen4cap -c “select * from sp_clear_pending_l1_tiles()”

Please let me know.

Best regards,
Cosmin

Hello Cosmin,

As far as I know I didn’t encounter any error from the update (Maybe I didn’t notice it). Yes the database is already 2.0.

Structure of l1_tile_history:

Column         |           Type           |       Modifiers        
-----------------------+--------------------------+------------------------
 satellite_id          | smallint                 | not null
 orbit_id              | integer                  | not null
 tile_id               | text                     | not null
 downloader_history_id | integer                  | not null
 status_id             | integer                  | not null
 status_timestamp      | timestamp with time zone | not null default now()
 retry_count           | integer                  | not null default 0
 failed_reason         | text                     | 
 cloud_coverage        | integer                  | 
 snow_coverage         | integer                  | 
Indexes:
"l1_tile_history_pkey" PRIMARY KEY, btree (downloader_history_id, tile_id)
Foreign-key constraints:
"l1_tile_history_downloader_history_id_fkey" FOREIGN KEY (downloader_history_id) REFERENCES downloader_history(id)
"l1_tile_history_satellite_id_fkey" FOREIGN KEY (satellite_id) REFERENCES satellite(id)
"l1_tile_history_status_id_fkey" FOREIGN KEY (status_id) REFERENCES l1_tile_status(id)
brentf@nirvana-ESSC_MNL:[~/Videos]: sudo -u postgres psql sen2agri -c "\sf sp_clear_pending_l1_tiles";
could not change directory to "/home/brentf/Videos": Permission denied
ERROR:  function "sp_clear_pending_l1_tiles" does not exist
brentf@nirvana-ESSC_MNL:[~/Videos]: sudo -u postgres psql sen2agri -c "select * from sp_clear_pending_l1_tiles()";
could not change directory to "/home/brentf/Videos": Permission denied
ERROR:  function sp_clear_pending_l1_tiles() does not exist
LINE 1: select * from sp_clear_pending_l1_tiles()
                      ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

I’m missing the sp_clear_pending_l1_tiles table.

Regards,
Brent

Could you try running the following command?

cat <INSTALL_PACKAGE_ROOT>/install_script/database/05-functions/sp_clear_pending_l1_tiles.sql | sudo su - postgres -c 'psql 'sen2agri''

And then check again the function execution and the demmaccs logs.
Could you check also the call to the function sp_start_l1_tile_processing? Is this function also missing from the database?

Please let me know,
Cosmin

Hello Cosmin,
Here are the results:

brentf@nirvana-ESSC_MNL:[~/Videos]: cat /home/brentf/Documents/SEN2AGRI_INSTALLER_FILES/Sen2AgriDistribution/install_script/database/05-functions/sp_clear_pending_l1_tiles.sql | sudo su - postgres -c 'psql 'sen2agri''
[sudo] password for brentf: 
CREATE FUNCTION
brentf@nirvana-ESSC_MNL:[~/Videos]: sudo -u postgres psql sen2agri -c "select * from sp_clear_pending_l1_tiles()";
[sudo] password for brentf: 
could not change directory to "/home/brentf/Videos": Permission denied
 sp_clear_pending_l1_tiles 
---------------------------
 
(1 row)

demmaccs log:
    -- Logs begin at Thu 2019-05-09 16:43:46 PST, end at Tue 2019-05-21 10:21:45 PST. --
    May 21 10:14:40 nirvana-ESSC_MNL systemd[1]: Stopping Runs MACCS on L1C products...
    May 21 10:14:40 nirvana-ESSC_MNL systemd[1]: Stopped Runs MACCS on L1C products.
    May 21 10:14:40 nirvana-ESSC_MNL systemd[1]: Started Runs MACCS on L1C products.
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: CompressTiffs = true, CogTiffs = true, RemoveSRE = true, RemoveFre = false
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: CompressTiffs = True, CogTiffs = True, RemoveSRE = True, RemoveFre = False
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: Starting thread dmworker_0
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: Starting thread dmworker_1
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: dmworker_1 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: Waiting for queue to join...
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: 2019-05-21 10:14:46.117192:[8436]:dmworker_1: No tile to process. Gracefully closing...
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: dmworker_0 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: 2019-05-21 10:14:46.117327:[8436]:dmworker_0: No tile to process. Gracefully closing...
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: 2019-05-21 10:14:46.117603:[8436]:dmworker_0: Exit thread
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: 2019-05-21 10:14:46.117695:[8436]:dmworker_1: Exit thread
    May 21 10:14:46 nirvana-ESSC_MNL demmaccs_launcher.py[8436]: All the workers finished their job. Exiting...
    May 21 10:15:06 nirvana-ESSC_MNL systemd[1]: Started Runs MACCS on L1C products.
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: CompressTiffs = true, CogTiffs = true, RemoveSRE = true, RemoveFre = false
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: CompressTiffs = True, CogTiffs = True, RemoveSRE = True, RemoveFre = False
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: Starting thread dmworker_0
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: Starting thread dmworker_1
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: Waiting for queue to join...
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: dmworker_0 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: 2019-05-21 10:15:11.788387:[8467]:dmworker_0: No tile to process. Gracefully closing...
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: dmworker_1 will consume: None | l2a | /mnt/archive/maccs_def/{site}/l2a/
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: 2019-05-21 10:15:11.788551:[8467]:dmworker_1: No tile to process. Gracefully closing...
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: 2019-05-21 10:15:11.788924:[8467]:dmworker_1: Exit thread
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: 2019-05-21 10:15:11.789033:[8467]:dmworker_0: Exit thread
    May 21 10:15:11 nirvana-ESSC_MNL demmaccs_launcher.py[8467]: All the workers finished their job. Exiting...
    May 21 10:16:06 nirvana-ESSC_MNL systemd[1]: Started Runs MACCS on L1C products.
    May 21 10:16:06 nirvana-ESSC_MNL demmaccs_launcher.py[8621]: CompressTiffs = true, CogTiffs = true, RemoveSRE = true, RemoveFre = false
    May 21 10:16:06 nirvana-ESSC_MNL demmaccs_launcher.py[8621]: CompressTiffs = True, CogTiffs = True, RemoveSRE = True, RemoveFre = False
    May 21 10:16:06 nirvana-ESSC_MNL demmaccs_launcher.py[8621]: Starting thread dmworker_0
    May 21 10:16:06 nirvana-ESSC_MNL demmaccs_launcher.py[8621]: Starting thread dmworker_1
    May 21 10:16:06 nirvana-ESSC_MNL demmaccs_launcher.py[8621]: MainThread:139904189279248: Exception 23503 received when trying to execute sql queries: select * from sp_start_l1_tile_processing();. Number of retries = 0
brentf@nirvana-ESSC_MNL:[~/Videos]: cat /home/brentf/Documents/SEN2AGRI_INSTALLER_FILES/Sen2AgriDistribution/install_script/database/05-functions/sp_start_l1_tile_processing.sql | sudo su - postgres -c 'psql 'sen2agri''
CREATE FUNCTION

Both of the functions are now in the database.

brentf@nirvana-ESSC_MNL:[~/Videos]: sudo -u postgres psql sen2agri -c "\sf sp_start_l1_tile_processing";
could not change directory to "/home/brentf/Videos": Permission denied
CREATE OR REPLACE FUNCTION public.sp_start_l1_tile_processing()
 RETURNS TABLE(site_id integer, satellite_id smallint, orbit_id integer, tile_id text, downloader_history_id integer, path text, prev_l2a_path text)
 LANGUAGE plpgsql
AS $function$
declare _satellite_id smallint;
declare _orbit_id int;
declare _tile_id text;
declare _downloader_history_id int;
declare _path text;
declare _prev_l2a_path text;
declare _site_id int;
declare _product_date timestamp;
begin
	if (select current_setting('transaction_isolation') not ilike 'serializable') then
		raise exception 'Please set the transaction isolation level to serializable.' using errcode = 'UE001';
	end if;

    select l1_tile_history.satellite_id,
           l1_tile_history.orbit_id,
           l1_tile_history.tile_id,
           l1_tile_history.downloader_history_id
    into _satellite_id,
         _orbit_id,
         _tile_id,
         _downloader_history_id
    from l1_tile_history
    where status_id = 2 -- failed
      and retry_count < 3
      and status_timestamp < now() - interval '1 day'
    order by status_timestamp
    limit 1;

    if found then
        select downloader_history.product_date,
               downloader_history.full_path,
               downloader_history.site_id
        into _product_date,
             _path,
             _site_id
        from downloader_history

brentf@nirvana-ESSC_MNL:[~/Videos]: sudo -u postgres psql sen2agri -c "\sf  sp_clear_pending_l1_tiles";
could not change directory to "/home/brentf/Videos": Permission denied
CREATE OR REPLACE FUNCTION public.sp_clear_pending_l1_tiles()
 RETURNS void
 LANGUAGE plpgsql
AS $function$
begin
    delete
    from l1_tile_history
    where status_id = 1; -- processing

    update downloader_history
    set status_id = 2 -- downloaded
    where status_id = 7 -- processing
      and not exists (
        select *
        from l1_tile_history
        where status_id = 1 -- processing
    );
end;
$function$

While sp_start_l1_tile_processing:

brentf@nirvana-ESSC_MNL:[~/Videos]: sudo -u postgres psql sen2agri -c "select * from sp_start_l1_tile_processing()";
could not change directory to "/home/brentf/Videos": Permission denied
ERROR:  Please set the transaction isolation level to serializable.

Regards,
Brent

Hello Brent,

For this function you should actually run :

sudo -u postgres psql sen2agri -c “select * from set transaction isolation level serializable; sp_start_l1_tile_processing();”
But this is not really important. The important thing is to have this function in your database.
I am not sure how this is missing (for the sp_clear_pending_l1_tiles it seems that is indeed missing from the update scripts but not sp_start_l1_tile_processing).
Anyway, please check if the demmaccs is functioning now but first you might want to reset the statuses of the failed products:

  • stop the sen2agri-demmaccs:

sudo systemctl stop sen2agri-demmaccs sen2agri-demmaccs.timer

  • reset the products that were marked as failed in the database:

sudo -u postgres psql sen2agri -c “delete from l1_tile_history”
sudo -u postgres psql sen2agri -c “update downloader_history set status_id = 2 where status_id = 6”

  • start the sen2agri-demmaccs:

sudo systemctl start sen2agri-demmaccs sen2agri-demmaccs.timer

Please let me know if the demmaccs is OK now.

Best regards,
Cosmin

1 Like

Hello Cosmin,

I’ve followed the abovementioned commands but I still have the same errors:

May 21 17:07:08 nirvana-ESSC_MNL systemd[1]: Stopping Runs MACCS on L1C products...
May 21 17:07:08 nirvana-ESSC_MNL systemd[1]: Stopped Runs MACCS on L1C products.
May 21 17:09:34 nirvana-ESSC_MNL systemd[1]: Started Runs MACCS on L1C products.
May 21 17:09:34 nirvana-ESSC_MNL demmaccs_launcher.py[21019]: CompressTiffs = true, CogTiffs = true, RemoveSRE = true, RemoveFre = false
May 21 17:09:34 nirvana-ESSC_MNL demmaccs_launcher.py[21019]: CompressTiffs = True, CogTiffs = True, RemoveSRE = True, RemoveFre = False
May 21 17:09:34 nirvana-ESSC_MNL demmaccs_launcher.py[21019]: Starting thread dmworker_0
May 21 17:09:34 nirvana-ESSC_MNL demmaccs_launcher.py[21019]: Starting thread dmworker_1
May 21 17:09:34 nirvana-ESSC_MNL demmaccs_launcher.py[21019]: MainThread:140611417656336: Exception 23503 received when trying to execute sql queries: select * from sp_start_l1_tile_processing();. Number of retries = 

Also, my l1_tile_history is empty. Anyway (just curious, -though i know this is not important), is this the right syntax: sudo -u postgres psql sen2agri -c "select * from set transaction isolation level serializable sp_start_l1_tile_processing();" ?

Hello Brent,
Indeed the command is not right, seems that copy/paste is not my friend (I did not noticed the error, sorry).
The command should be:

sudo -u postgres psql sen2agri -c “set transaction isolation level serializable ; select * from sp_start_l1_tile_processing();”

Could you try running please?

Best regards,
Cosmin

Hello Cosmin,

Here is the result of the said command:

brentf@nirvana-ESSC_MNL:[~]: sudo -u postgres psql sen2agri -c "set transaction isolation level serializable ; select * from sp_start_l1_tile_processing();"
could not change directory to "/home/brentf": Permission denied
ERROR:  insert or update on table "l1_tile_history" violates foreign key constraint "l1_tile_history_status_id_fkey"
DETAIL:  Key (status_id)=(1) is not present in table "l1_tile_status".
CONTEXT:  SQL statement "insert into l1_tile_history (
                satellite_id,
                orbit_id,
                tile_id,
                downloader_history_id,
                status_id
            ) values (
                _satellite_id,
                _orbit_id,
                _tile_id,
                _downloader_history_id,
                1 -- processing
            )"
PL/pgSQL function sp_start_l1_tile_processing() line 88 at SQL statement

Regards,
Brent

Hello Brent,

It seems that something went wrong to the update of the system in your case or there is something missing in the installation package.
Could you try to run also the following command and then check again the previous execution:

sudo -u postgres psql sen2agri -c “insert into l1_tile_status values (1, ‘processing’), (2, ‘failed’), (3, ‘done’);”

If the sp_start_l1_tile_processing execution is OK, then check also the MACCS execution.

Best regards,
Cosmin

Hello,

Demmaccs already started but it stopped after a few minutes. I’m currently having the same issue with this post.

By the way here are the maccs logs, similar to abovementioned post.

demmaccs_51QVV_log.txt (4.4 KB)
demmaccs_log.txt (1.3 KB)

Should I just follow the same answers?

Regards,
Brent

Hello,

Yes, normally running:

sudo chmod -R a+r /opt/maja

should solve the problem.

Please let me know.

Best regards,
Cosmin

Hello Cosmin,

I followed all the instructions from the said post. But I still have the same error. Do you think it would help to have a clean install instead, like putting the maja folder in the install script, instead of a manual maja install?

Also my sen2agri-service seemed to stop downloading sentinel images. I’ll update you again tomorrow.

Regards,

Brent

Hello Brent,

Normally, the mechanism for MAJA is exactly the same as for MACCS the only difference are the rights that are set during MAJA installation.
Before performing a clean install, you could try the following:

  • sudo su -l sen2agri-service
  • ls -l /opt/maja/bin/maja -> Is the command OK? If not, please check again the execution rights for the /opt/maja and its subfolders and files rights
  • ls -l /mnt/archive/gipp_maja/ -> Is the command OK?
  • I assume /mnt/archive/srtm, swbd, demmaccs_tmp, maccs_def … have the right rights as maccs functioned.
  • execute

/usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py --srtm /mnt/archive/srtm --swbd /mnt/archive/swbd --processes-number-dem 1 --processes-number-maccs 1 --gipp-dir /mnt/archive/gipp_maja --working-dir /mnt/archive/demmaccs_tmp/ --maccs-launcher /opt/maja/bin/maja --delete-temp True /mnt/archive/dwn_def/s2/default/isabela19/S2B_MSIL1C_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE /mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/ --tiles-to-process 51QVV

Is there a particular error there?

Please let me know.

Best regards,
Cosmin

P.S. Could you tell me what was the original version that was used for first installation of the system you use and was upgraded? Was it 1.8.3 or an older version? I am asking as I assume it was the version 1.8 or 1.7 as the problems you had with the missing functions were only possible in this case. I tried upgrading from 1.8.3 to 2.0 and I was not having these problems so I want to confirm my assumption. Thanks.

I think there was an error in my post before (I missed one flag to be set). The command for MAJA folder rights should be:

sudo chmod a+rx /opt/maja

Please let me know.

Cosmin

Hello Cosmin,

My first installation is 1.7, but I repeatedly had a clean install then. From that, I updated the system to 1.8.1, 1.8.3 and finally to version 2.

By the way here are the results of the command:

brentf@nirvana-ESSC_MNL:[~]: sudo su -l sen2agri-service
Last login: Thu May 23 10:11:47 PST 2019 on pts/0
su: warning: cannot change directory to /home/sen2agri-service: No such file or directory
-bash-4.2$ ls -l /opt/maja/bin/maja
ls: cannot access /opt/maja/bin/maja: Permission denied
-bash-4.2$ ls -l /mnt/archive/gipp_maja
total 28
drwxrwxrwx. 26 brentf brentf  8192 Feb 21 15:27 LANDSAT8
drwxrwxrwx. 57 brentf brentf 12288 Feb 21 15:28 SENTINEL2

While this is the error for the maja manual execution:

    2019-05-23 10:16:13.806026:[9090]:Starting MACCS/MAJA in L2INIT for /mnt/archive/dwn_def/s2/default/isabela19/S2B_MSIL1C_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE | TileID: 51QVV
    2019-05-23 10:16:13.806132:[9090]:MACCS_COMMAND: ['/opt/maja/bin/maja', '--input', '/mnt/archive/demmaccs_tmp/tmpUPzMfD/51QVV', '--TileId', '51QVV', '--output', '/mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/maccs_51QVV', '--mode', 'L2INIT', '--loglevel', 'DEBUG', '--enableTest', 'false', '--CheckXMLFilesWithSchema', 'false', '--conf', '/usr/share/sen2agri/sen2agri-demmaccs/UserConfiguration']
    2019-05-23 10:16:13.806250:[9090]:Running command: /opt/maja/bin/maja --input /mnt/archive/demmaccs_tmp/tmpUPzMfD/51QVV --TileId 51QVV --output /mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/maccs_51QVV --mode L2INIT --loglevel DEBUG --enableTest false --CheckXMLFilesWithSchema false --conf /usr/share/sen2agri/sen2agri-demmaccs/UserConfiguration
    Traceback (most recent call last):
      File "/usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py", line 509, in <module>
        out = maccs_launcher(demmaccs_contexts[0])
      File "/usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py", line 293, in maccs_launcher
        if run_command(cmd_array, demmaccs_context.output, tile_log_filename) != 0:
      File "/usr/lib/python2.7/site-packages/sen2agri_common_db.py", line 155, in run_command
        res = subprocess.call(cmd_array, shell=False)
      File "/usr/lib64/python2.7/subprocess.py", line 524, in call
        return Popen(*popenargs, **kwargs).wait()
      File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
        errread, errwrite)
      File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
        raise child_exception
    OSError: [Errno 13] Permission denied

Apparently, I figured out a workaround by sudo chmod -R 777 /opt/maja/bin/maja/ and it worked.

B2.tif> in directory </mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/maccs_51QVV/SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0> with ModeStreamDivisions=1 and BufferNumberOfLinesDivisions: 800  [vnsMuscateL2ImageFileWriter.txx:WritePublicImages:404]
2019-05-23T10:47:49.509810 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the SRE B2 image file done (time is 8.2 second).
2019-05-23T10:47:49.509967 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the SRE B3 image file...
2019-05-23T10:47:49.510008 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [D] Write the file <SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0_SRE_B3.tif> in directory </mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/maccs_51QVV/SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0> with ModeStreamDivisions=1 and BufferNumberOfLinesDivisions: 800  [vnsMuscateL2ImageFileWriter.txx:WritePublicImages:404]
2019-05-23T10:47:57.829620 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the SRE B3 image file done (time is 8.3 second).
2019-05-23T10:47:57.829741 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the SRE B4 image file...
2019-05-23T10:47:57.829768 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [D] Write the file <SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0_SRE_B4.tif> in directory </mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/maccs_51QVV/SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0> with ModeStreamDivisions=1 and BufferNumberOfLinesDivisions: 800  [vnsMuscateL2ImageFileWriter.txx:WritePublicImages:404]
2019-05-23T10:48:05.948087 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the SRE B4 image file done (time is 8.1 second).
2019-05-23T10:48:05.948209 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the SRE B8 image file...
2019-05-23T10:48:05.948237 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [D] Write the file <SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0_SRE_B8.tif> in directory </mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/maccs_51QVV/SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0> with ModeStreamDivisions=1 and BufferNumberOfLinesDivisions: 800  [vnsMuscateL2ImageFileWriter.txx:WritePublicImages:404]
2019-05-23T10:48:14.123835 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the SRE B8 image file done (time is 8.2 second).
2019-05-23T10:48:14.123970 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [D] vnsCachingMacro Proceed caching... with file name vns_caching_L2ImageFileWriterBaseFREImage0.tif (ModeStreamDivisions=1; NumberOfStreamDivisions=800).  [vnsMuscateL2ImageFileWriter.txx:WritePublicImages:414]
2019-05-23T10:48:36.180903 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [D]  => Caching with file name vns_caching_L2ImageFileWriterBaseFREImage0.tif run in 22 second.  [vnsMuscateL2ImageFileWriter.txx:WritePublicImages:414]
2019-05-23T10:48:36.185568 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the FRE B2 image file...
2019-05-23T10:48:36.185629 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [D] Write the file <SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0_FRE_B2.tif> in directory </mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/maccs_51QVV/SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0> with ModeStreamDivisions=1 and BufferNumberOfLinesDivisions: 800  [vnsMuscateL2ImageFileWriter.txx:WritePublicImages:437]
2019-05-23T10:48:44.448880 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the FRE B2 image file done (time is 8.3 second).
2019-05-23T10:48:44.449020 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the FRE B3 image file...
2019-05-23T10:48:44.449050 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [D] Write the file <SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0_FRE_B3.tif> in directory </mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/maccs_51QVV/SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0> with ModeStreamDivisions=1 and BufferNumberOfLinesDivisions: 800  [vnsMuscateL2ImageFileWriter.txx:WritePublicImages:437]
2019-05-23T10:48:52.631857 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the FRE B3 image file done (time is 8.2 second).
2019-05-23T10:48:52.631978 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [I] Writing the FRE B4 image file...
2019-05-23T10:48:52.632009 nirvana-ESSC_MNL maja-processing-3.2.2 3.2 [000000012252] [D] Write the file <SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0_FRE_B4.tif> in directory </mnt/archive/maccs_def/isabela19/l2a/S2B_MSIL2A_20190128T021949_N0207_R003_T51QVV_20190128T040527.SAFE/maccs_51QVV/SENTINEL2B_20190128-023247-444_L2A_T51QVV_C_V1-0> with ModeStreamDivisions=1 and BufferNumberOfLinesDivisions: 800  [vnsMuscateL2ImageFileWriter.txx:WritePublicImages:437]

Thank you for patience and support. Now my next problem is how to continue the downloading process as the sen2agri-service encountered error. I’ll post this in another topic.

Regards,
Brent

As said, the said command didn’t enable maja, I was able to use it using the chmod 777 command.

Regards,
Brent

Hello Brent,

It seems that there is a problem with the upgrade from the original installation older with 1.8.3. The problem is not present if the original installation is 1.8.3. Thanks for notifying us. If you have a second similar machine, please find attach a patch that normally should solve all the problems above (missing functions and the MAJA). patch_1_upgrade_183_20.zip (3.3 KB)

Even the second command that I sent (with “rx” - please note the “x” compared to the first command I gave) did not worked in your case? Normally on our case, it was not needed the 777 but only +rx was enough. But I think is OK also this way if it is working for you.
Concerning the downloads, could you tell me if it worked a priori with the current version of services from 2.0 or you had problems from the start with the downloads?
You can try the following:

  • go into /usr/share/sen2agri/sen2agri-services/config/
  • edit the file application.properties in order to set the value “TRACE” (it is info in this moment) for the following keys: “logging.level.org.esa.sen2agri” and “logging.level.ro.cs.tao”.
  • log the journalctl messages from sen2agri-service
  • restart sen2agri-services and wait several minutes - 20-30 minutes
    Please provide us the logs you saved.

Best regards,
Cosmin