Setting AWS EC2 Instance and Sen2agri

ernesto_luna · March 16, 2020, 10:47am

Dear Sen2agri community,

I want to know if someone could guide me to properly setup an AWS EC2 instance with CentOS to install and run Sen2agri for just one tile and in the future two tiles. Is there any guide or material for this? I have never used AWS so I have doubts such as:

Which Amazon Machine Image (AMI) to choose from since there are 222 CentOS AMI (checked on the 16th of march, 2020)?
-Instance Storage options (EBS only or SSD)
-Network performance
-Volume storage type (general purpose SSD, IOPS SSD,cold HDD,st1,standard)
-Which configuration is optimal to save resources and money.

Once the instance is ready, I also need help to install Sen2agri on this instance.

Thank you for your help,
Kind Regards,

alansgz · March 16, 2020, 4:53pm

I have already done this however it doesn’t run the whole web Sen2Agri environment, I only install it for manual usage as described in the documentation which means that it’s not automatic, you have to manually invoke the commands for every product you want to process (look in software user manual for “manual usage”), if this is what you are looking for this is what I did:

CentOS 7 (x86_64) - with Updates HVM from centos.org is the AMI I used
GP2 SSD Storage, for L2A processing a 50 GB drive will do, for an L3A you will need at least 130 GB to store the intermediate products
Network performance is quite good if downloading from PEPS mirror
I haven’t measured the difference of speed in storage types, I used GP2 SSD
To save money use Spot instances, you can reduce your costs by almost 70% by taking advantage of AWS surplus computing capacity, worst case scenario is that the instance gets terminated and you have to do it again.
Another suggestion, use the c5d.xlarge instance, it is optimized for computing and the ‘d’ class of c5’s have local NVMe SSD storage, it has 4 vCPUs and 8 GB memory which is the least you need to run L2A and L3A processors, if you try with the c5d.large which has only 4 GB of memory it will crash because of memory errors with the latest version of Sen2Agri 2 that uses MAJA 3.3, although 4 GB will be enough for the previous version of Sen2Agri with MACCS if you are using that.

I have a script to provision the minimal CentOS machine up to a working Sen2Agri2 installation in AWS, after installation you can use the system by manual commands as specified in the docs, there is also https://github.com/olivierhagolle/Start_maja that will help to manually process L2A products.

github.com

alansalinas/sen2agri-aws/blob/master/provision-sen2agri2-maja-ec2.sh

#!/bin/bash

### Installation script for Sen2Agri 2 with MAJA 3.3
### This software will provision the minimal CentOS 7 machine with the required
### dependencies and then install the Sen2Agri System, it requires you to provide
### the Sen2Agri installation package, the MAJA 3.3 installation package

yum -y install epel-release # Add extra repositories
yum -y update  # Update repositories
yum -y install perl dkms kernel-devel make bzip2 # headers need restart for vmware tools install
yum -y install gcc gcc-c++ python-devel
yum -y install python-pip
yum -y install unzip
yum -y install python-beautifulsoup4
yum -y install git
yum -y install postgresql-devel # required for psycopg2
yum -y install postgis
# Install Python 3.6
yum -y install https://centos7.iuscommunity.org/ius-release.rpm
yum -y install python36u

This file has been truncated. show original

A suggestion is to use AWS S3 as your output filesystem, check s3fs_fuse and mount your bucket in a directory, this allows to put all the L2As there and then the L3A processor can grab them from there. In my experience the cost is very accesible, making an L2A product costs a few cents and completes in less than an hour. Hope this helps.

pmalcorps · March 23, 2020, 1:49pm

Thanks @alansgz for the reply.

For @ernesto_luna, at UCLouvain, we are currently running the Sen2-Agri system on a VM from AWS (and we had 2 in the past) so we can give you some specs and advices for the configuration of the machine and the installation of the Sen2-Agri system on it. We are just compiling the information, and we will add it here as soon as it is ready (normally today ;)).

pmalcorps · March 31, 2020, 9:13am

Sorry for the answer delay, some slides will be specifically dedicated to that during the Thursday (2nd of April) webinar. Don’t hesitate to join in: http://www.esa-sen2agri.org/ninth-sen2-agri-webinar/. In any case, I will copy here the slides after the webinar.

Akash_Verma · July 2, 2020, 2:01pm

Hi I have installed Sen2agri on AWS CentOs 7 but the web interface is opening only login page and product table is empty after spending some time I found out that product table is empty in database. Can you suggest something?

cudroiu · July 3, 2020, 1:53pm

Hello,
Did you defined a site and a season?
Also, the AWS plugin is not installed by default in the system but you can easily install it by copying the file

<PACKAGE_ROOT>/Sen2AgriDistribution/sen2agri-services/datasource-additional-plugins/tao-datasources-aws-1.0.1.jar

into the following directory:

/usr/share/sen2agri/sen2agri-services/lib

and then restart the services using:

sudo systemctl restart sen2agri-services.

Once you did that, you will have in the Datasources tab the AWS datasource available that you’ll have to configure according to your local repository path by filling the local root and the other parameters.

Best regards,
Cosmin

Gajavelli_Sai_Venkat · March 11, 2021, 5:16am

Is this configuration mentioned for the installation of whole sen2agri software or it’s just for installing SNAP and then Sen@agri adaptors (For manual)? Actually i am also looking for installing and run Sen2agri for just one tile. So i need some help getting started with this. I wanted to know which one is the best “SNAP + Sen@agri adaptors” or “Whole Sen2Agri” software.?

cudroiu · March 22, 2021, 3:34pm

Hello,

If you need to automatically download products, pre-process them into L2A and then run the other processors, you should install the whole system (that includes Postgres, SLURM etc. installation too), not only the processors.

Best regards,
Cosmin