Setting AWS EC2 Instance and Sen2agri

Dear Sen2agri community,

I want to know if someone could guide me to properly setup an AWS EC2 instance with CentOS to install and run Sen2agri for just one tile and in the future two tiles. Is there any guide or material for this? I have never used AWS so I have doubts such as:

  • Which Amazon Machine Image (AMI) to choose from since there are 222 CentOS AMI (checked on the 16th of march, 2020)?
    -Instance Storage options (EBS only or SSD)
    -Network performance
    -Volume storage type (general purpose SSD, IOPS SSD,cold HDD,st1,standard)
    -Which configuration is optimal to save resources and money.

Once the instance is ready, I also need help to install Sen2agri on this instance.

Thank you for your help,
Kind Regards,

1 Like

I have already done this however it doesn’t run the whole web Sen2Agri environment, I only install it for manual usage as described in the documentation which means that it’s not automatic, you have to manually invoke the commands for every product you want to process (look in software user manual for “manual usage”), if this is what you are looking for this is what I did:

  • CentOS 7 (x86_64) - with Updates HVM from centos.org is the AMI I used
  • GP2 SSD Storage, for L2A processing a 50 GB drive will do, for an L3A you will need at least 130 GB to store the intermediate products
  • Network performance is quite good if downloading from PEPS mirror
  • I haven’t measured the difference of speed in storage types, I used GP2 SSD
  • To save money use Spot instances, you can reduce your costs by almost 70% by taking advantage of AWS surplus computing capacity, worst case scenario is that the instance gets terminated and you have to do it again.
  • Another suggestion, use the c5d.xlarge instance, it is optimized for computing and the ‘d’ class of c5’s have local NVMe SSD storage, it has 4 vCPUs and 8 GB memory which is the least you need to run L2A and L3A processors, if you try with the c5d.large which has only 4 GB of memory it will crash because of memory errors with the latest version of Sen2Agri 2 that uses MAJA 3.3, although 4 GB will be enough for the previous version of Sen2Agri with MACCS if you are using that.

I have a script to provision the minimal CentOS machine up to a working Sen2Agri2 installation in AWS, after installation you can use the system by manual commands as specified in the docs, there is also https://github.com/olivierhagolle/Start_maja that will help to manually process L2A products.

A suggestion is to use AWS S3 as your output filesystem, check s3fs_fuse and mount your bucket in a directory, this allows to put all the L2As there and then the L3A processor can grab them from there. In my experience the cost is very accesible, making an L2A product costs a few cents and completes in less than an hour. Hope this helps.

Thanks @alansgz for the reply.

For @ernesto_luna, at UCLouvain, we are currently running the Sen2-Agri system on a VM from AWS (and we had 2 in the past) so we can give you some specs and advices for the configuration of the machine and the installation of the Sen2-Agri system on it. We are just compiling the information, and we will add it here as soon as it is ready (normally today ;)).

1 Like

Sorry for the answer delay, some slides will be specifically dedicated to that during the Thursday (2nd of April) webinar. Don’t hesitate to join in: http://www.esa-sen2agri.org/ninth-sen2-agri-webinar/. In any case, I will copy here the slides after the webinar.

Hi I have installed Sen2agri on AWS CentOs 7 but the web interface is opening only login page and product table is empty after spending some time I found out that product table is empty in database. Can you suggest something?

Hello,
Did you defined a site and a season?
Also, the AWS plugin is not installed by default in the system but you can easily install it by copying the file

<PACKAGE_ROOT>/Sen2AgriDistribution/sen2agri-services/datasource-additional-plugins/tao-datasources-aws-1.0.1.jar

into the following directory:

/usr/share/sen2agri/sen2agri-services/lib

and then restart the services using:

sudo systemctl restart sen2agri-services.

Once you did that, you will have in the Datasources tab the AWS datasource available that you’ll have to configure according to your local repository path by filling the local root and the other parameters.

Best regards,
Cosmin

1 Like

Is this configuration mentioned for the installation of whole sen2agri software or it’s just for installing SNAP and then Sen@agri adaptors (For manual)? Actually i am also looking for installing and run Sen2agri for just one tile. So i need some help getting started with this. I wanted to know which one is the best “SNAP + Sen@agri adaptors” or “Whole Sen2Agri” software.?

Hello,

If you need to automatically download products, pre-process them into L2A and then run the other processors, you should install the whole system (that includes Postgres, SLURM etc. installation too), not only the processors.

Best regards,
Cosmin