How to check if the shapefile for in situ processing is working in custom jobs section


#1

Hello,

I was trying to upload my in situ shp file zip in the custom jobs section of the web interface.
However, it says that there was an error and the upload is not successful.
I’m sharing my in situ shapefile here: https://ufile.io/t1irz

Is there a way to check the validity/integrity of the shp file zipped before uploading it?


#2

Hi,

Depending on the actual error you get, it might be an actual issue with the shapefile. There is a script that checks it, you can run it like:

/var/www/html/scripts/check_shp.py /path/to/shapefile.shp

If it does report errors, there’s a “fix” mode that tries to handle them. You can use it by adding the -f argument to the command above.


#3

ok thanks much, will update you if the script worked


#4

Hello again,

I tried to upload again my fixed shapefile in the custom jobs of the web interface, but I encountered the same problem.

Here are the results of fixing my shapefile:

  1. I checked it:
    /var/www/html/scripts/check_shp.py /home/brentf/Downloads/PH_BUK_LC_PI_201712/PH_BUK_LC_PI_201712.shp
    960 feature(s) found
    Warning 1: Self-intersection at or near point 740359.92763765831 830476.7972673676 0
    Invalid feature: ID=21
    Warning 1: Self-intersection at or near point 717357.97096149693 881168.98342688382 0
    Invalid feature: ID=433
    Warning 1: Self-intersection at or near point 725221.06657073402 883105.55843447871 0
    Invalid feature: ID=448
    Warning 1: Self-intersection at or near point 686774.41494734993 863222.37843357422 0
    Invalid feature: ID=528
    ERROR 1: TopologyException: Input geom 0 is invalid: Self-intersection at or near point 740359.92763765831 830476.7972673676 0 at 740359.92763765831 830476.7972673676 0
    Traceback (most recent call last):
    File “/var/www/html/scripts/check_shp.py”, line 135, in
    sys.exit(main())
    File “/var/www/html/scripts/check_shp.py”, line 91, in main
    if g0.Overlaps(g1) and g0.Intersection(g1).Area() > 0:
    AttributeError: ‘NoneType’ object has no attribute ‘Area’

  2. Fixed it:

    /var/www/html/scripts/check_shp.py -f /home/brentf/Downloads/PH_BUK_LC_PI_201712/PH_BUK_LC_PI_201712.shp
    960 feature(s) found
    Warning 1: Self-intersection at or near point 740359.92763765831 830476.7972673676 0
    Invalid feature: ID=21
    Fixed feature: ID=21
    Warning 1: Self-intersection at or near point 717357.97096149693 881168.98342688382 0
    Invalid feature: ID=433
    Fixed feature: ID=433
    Warning 1: Self-intersection at or near point 725221.06657073402 883105.55843447871 0
    Invalid feature: ID=448
    Fixed feature: ID=448
    Warning 1: Self-intersection at or near point 686774.41494734993 863222.37843357422 0
    Invalid feature: ID=528
    Fixed feature: ID=528

  3. And checked it again:
    [brentf@nirvana-ESSC_MNL mnt]$ /var/www/html/scripts/check_shp.py /home/brentf/Downloads/PH_BUK_LC_PI_201712/PH_BUK_LC_PI_201712.shp
    960 feature(s) found

Does this mean that my is now valid?


#5
AttributeError: ‘NoneType’ object has no attribute ‘Area’

This was an oversight in the validation tool; it’s been fixed in the meanwhile. -f might still work, though.

[brentf@nirvana-ESSC_MNL mnt]$ /var/www/html/scripts/check_shp.py /home/brentf/Downloads/PH_BUK_LC_PI_201712/PH_BUK_LC_PI_201712.shp
960 feature(s) found

I’m also seeing 960 features in the fixed version. You can download my version from https://nas.dend.ro/public/PH_BUK_LC_PI_201712.zip.

Does this mean that my is now valid?

It should be.

I tried to upload again my fixed shapefile in the custom jobs of the web interface, but I encountered the same problem.

This works for me. Did you repack the fixed version as a zip archive and upload that? Can you try with my version?


#6

Hello,

I tried using your version, but still the same problem occurs. Yes, the repacked it again after fixing the shapefile. Maybe I could just delete the features that were fixed? I’ll update you with this after deleting the said features.


#7

Hello again,

Unfortunately, I accidentally closed the terminal tab that contained the fixed feature IDs of my shapefile.
I’m sharing here my fixed version of shapefile (https://ufile.io/eca3s). Again, if it not worked for me, perhaps there was a problem in my system’s web interface?


#8

Oh sorry, I forgot that I posted the names of the said feature IDs in this thread! :joy:

Anyway, I tried the shp file with the deleted file IDs, anyway the same result error showed. :disappointed_relieved:

I’m uploading the deleted feature IDs: https://ufile.io/odzip

If the said shp files worked for you and not with my system, I will try the manual processing through the CLI.
Thanks for you help. I really appreciated it.


#9

Yes, your shapefile works on my side:

image

Can you share a screenshot of the page and the error message?


#10

Hello again,

Actually I’m running Sen2Agri on two separate computers, one is a local one while the other one is controlled remotely.

My local machine has successfully uploaded the shop file, but the remote computer has not.
Does it have something to do with the minimum disk size required? My remote host has only 1.7 TB space.
Though I’m only processing 4 tiles for both Landsat and Sentinel, with the season of June 1 to Sept. 30 of 2017.

Thank you very much!


#11

Hi again,

I read that slurm is needed in custom jobs (in my case, L4A and L4B). The slurm in my remote machine is not active, therefore not running, perhaps this is the reason I can’t upload my shapefile?

Cheers.


#12

SLURM is needed to run the processor, but that shouldn’t affect the upload. Do you have two different full installs of the system?

Maybe there’s an issue with the permissions on your local system.


#13

No I only have one full install of Sen2agri system on each machine.
I think you already answered my question with a similar issue here: How to enable slurm services

I’ll update you about if it worked.

Much thanks.


#14

Hi @lnicola,

I’m experiencing the same issue with the in-situ shapefile. You mentioned the validation tool has been fixed?
Also, using the -f flag returns Could not open ...


#15

Hi,

Can you share your -f invocation? That error message suggests that the tool is unable to open or modify the shapefile.


#16

Here it is:

python /var/www/html/scripts/check_shp.py -f /mnt/upload/SouthSudan_NorthBahreGhazal_Warrap/sen2agri_1515397973/SSD_WARR_LC_VM_28112017/SSD_WARR_LC_VM_20171128.shp

I realize this might be because the upload directory contents belong to apache


#17

That’s possible. You can copy the files to your home directory, run it from there, and repack them.

Alternatively, you can download and replace the following files:

The first one will fix a crashing bug (AttributeError: ‘NoneType’ object has no attribute ‘Area’) and the second one will enable -f to be used by default.


#18

I downloaded and replaced said files, and the error persist.

Now when I run check_shp.py, I’m receiving this error message:

python /var/www/html/scripts/check_shp.py SSD_WARR_LC_VM_20171128.shp
1414 feature(s) found
ERROR 1: Error in fread() reading object of size 328 at offset 100 from .shp file
Traceback (most recent call last):
  File "/var/www/html/scripts/check_shp.py", line 141, in <module>
    sys.exit(main())
  File "/var/www/html/scripts/check_shp.py", line 50, in main
    if not geometry.IsValid():
AttributeError: 'NoneType' object has no attribute 'IsValid'

Thanks


#19

You have some features with no geometry. I’ll send you a new fix, but can you share the shapefile so I can test it? You can mail it to me at lnicola@c-s.ro.