Unlocking MACCS

Cloud Free Composite Processor
1

Hello,

Here follow three questions linked to improvements for the MACCS processor wrapper.

  1. The MACCS processor is run in parallel to the SLURM pool. Is there a reason to that ? Is it a lot of work to transform the wrapper to run it in a sbatch like the others ?

  2. Apparently, there is a “working directory” option in the MACCS exe. Is it possible to link it to a user-defined parameter in the DB ? We are observing a I/O bottleneck on our servers and want to use a SSD space. We also want to use a volume shared via NFS for the output storage, larger than the space available on a single server (S2 images are quickly huge), and don’t want to saturate that volume with disk write. In brief, the ideal is a local computation on SSD, then a move to the remote volume.

  3. We have found that MAJA can be put into a docker box. It is possible to wrap MACCS into one to make it available on other servers without CentOS7 ? (e.g. via a shared SLURM pool)

Thanks

2

Hi,

The MACCS processor is run in parallel to the SLURM pool. Is there a reason to that ? Is it a lot of work to transform the wrapper to run it in a sbatch like the others ?

No, it runs like that only because of “historical” reasons. It’s probably not hard to fix, the larger issue is that the system doesn’t process multiple L2As at once for a single site. This drastically slows down the processing and is harder to fix.

Apparently, there is a “working directory” option in the MACCS exe. Is it possible to link it to a user-defined parameter in the DB ?

I don’t think it will help that much, but the temporary files are saved to /mnt/archive/demmaccs_tmp/ (as set in demmaccs.working-dir in the config database table), while the outputs are under /mnt/archive/maccs_def/{site}/{processor}/ (demmaccs.output-path). You can either change those paths or make symlinks (which is usually easier). You might also find it helpful to tune the nfs mount options, although I don’t have any particular recommendations right now. For SMB we’re using vers=2.1,cache=loose.

It is possible to wrap MACCS into one to make it available on other servers without CentOS7 ?

It should be, with the caveats mentioned above.