In-situ Data XML


#1

Hello,

I am currently collating our training data to genereate In-situ Data to run the L4B processor for our site. I have a shapefile in the requried format and was wondering if there is a template or an example of the schema for the corresponding XML file that is mentioned in the Software User Manual.

Thank you

Graham


#2

Hi Graham, doesn’t look like you got a response on this yet. Did you find the answer yet on your own maybe? I would also like to know please.


#3

No, unfortunatley I don’t have a conclusive answer. I suspect the system isn’t looking at the contents of the xml; according to the user manual the xml largely contains user defined information relating to the capture of training data, but I don’t know.

So far I have been unable to successfully upload In-situ data under version 1.8.3 - trying to upload In-situ data results in the following error message “Missing shape file due to a problem with your selected file (Unable to access your selected file)”.


#4

Yes I have also been through the manual and suspect the same as you about the xml file. The problem I have is that I also get the same error as you when trying to upload my in-situ data “Missing shape file due to a problem with your selected file (Unable to access your selected file)”. This makes me wonder if the reason I get the error is that I didn’t upload the xml file with the shapefiles?

I checked the validity of the shapefile I tried to upload with this script: /var/www/html/scripts/check_shp.py /path/to/shapefile.shp

It did return some errors:

Overlapping features: ID=439 and ID=434
Warning 1: OGR_G_Area() called against non-surface geometry type.
Warning 1: OGR_G_Area() called against non-surface geometry type.
Warning 1: OGR_G_Area() called against non-surface geometry type.
Warning 1: OGR_G_Area() called against non-surface geometry type.
Warning 1: OGR_G_Area() called against non-surface geometry type.
Warning 1: OGR_G_Area() called against non-surface geometry type.
Warning 1: OGR_G_Area() called against non-surface geometry type.
Warning 1: OGR_G_Area() called against non-surface geometry type.

I then manually fixed the “overlapping features” in qgis and uploaded it again but I still get “Missing shape file due to a problem with your selected file (Unable to access your selected file)”

So maybe those “Warning 1: OGR_G_Area() called against non-surface geometry type.”, I get is the problem still or maybe it is something else i don’t know. Will update you if I have figured something out.


#5

I’m not sure that the error message and lack of xml are linked. I tried creating an xml for my site and adding it to my zipped shapefile archive but I got the same error message as before. Of course that could be because I made up the xml and the system was looking for something that wasn’t there.

The message reads like an issue with permissions to me, but having worked through the suggestions in other posts I think everything has the correct permissions and the problem persists. I can’t see any issues with the system that would prevent access to the shapefiles.


#6

Hello,

Could you provide the data and give more details about what you are trying to upload as in-situ data?
Please note that the in-situ data should be provided as ESRI shapefiles, zipped as an archive.
Also, I am not sure I understood the role of the xml file as in-situ data that you specified in your posts.

Best regards,
Cosmin


#7

Hi Cosmin,

Here is the in-situ data I am trying to upload through the “custom jobs” tab on the web interface.

But it gives me the following message when I try to submit the job: “Submit failed, Missing shape file due to a problem with your selected file (Unable to access your selected file)”

This in-situ data of mine is a esri shapefile consisting of polygons with an attribute table in the format specified in Chapter 4.5 of the Software user manual.

About the xml file, the manual says :
“The in-situ data should be gathered in a single shapefile, projected in the WGS 84/UTM grid system, containing both cropland and no-cropland information and in a corresponding XMLfile containing the metadata.”

So it specifies there that my in-situ data must include metadata in the form of an xml file. What I couldn’t find is how the format of that xml file should look. What I was wondering is if there isn’t by chance an example of a xml file of the metadata that I could have a look at and maybe copy and change the details to my site’s metadata.

Another thing that was not clear to me or that I could find an answer to in the manual, is how important is the xml metadata file, does the system need it to create the L4A and L4B products?

Thanks for your help and replies in advance,
Reinhardt


#8

“Submit failed, Missing shape file due to a problem with your selected file (Unable to access your selected file)”

Just as a quick test, can you please try a systemctl restart httpd?


#9

I also tried your shapefile on an installation that we have and everything went ok, so your insitu data seem ok.
Additionally to what lnicola added, could you also check that the user sen2agri-service has the right to write in the following 2 locations:

  • /mnt/upload
  • /mnt/archive/insitu (and in the subfolders corresponding to your site, if any).
    (assuming that you didn’t changed the default locations in the configuration, of course).

Best regards,
Cosmin


#10

I think you meant the apache user.


#11

Indeed :slight_smile:


#12

Ok tried it and it could not write into the insitu directory

Then I changed so that it could write

Still get the same message though when trying to upload shapefile “Submit failed, Missing shape file due to a problem with your selected file (Unable to access your selected file)”


#13

Hello, thank you for your replies.

I’m not sure what role the xml file plays either; it is mentioned in the Software User Manual so I thought it was worth pursuing to see if it affected the upload of the in-situ Data.

Please find attached my in-situ data. It is an Esri-shapefile recording 296 Land Covers in our test site and has been created in-line with the attribute table in the software manual.

I have checked that apache has read-write access to /mnt/upload and /mnt/archive/insitu and I have tried systemctl restart httpd but I still get the same “Missing shape file due to a problem…” error message.

Thank you for your help

Graham


#14

Oops, I forgot to actually attach my shapefile…
kelso.zip (241.4 KB)


#15

@cudroiu, @lnicola can you maybe see anything else here that might be the problem?


#16

Hello,
We are still investigating the problem. From what we noticed, the last insitu data that you uploaded seem to be OK (the shapefile was uploaded and validated OK on one of our platforms).
Could you try removing the /mnt/archive/insitu/kelso* and try uploading again the insitu data?

Best regards,
Cosmin


#17

Hi Cosmin,

I have tried removing and uploading the data again. It successfully uploads to the Sites tab but when I try to upload the insitu data under the Custom Jobs tab, to run the L4A or L4B processors, it still fails and gives the “Missing shape file…” error message.

Kind regards
Graham


#18

Hello,

It seems that you just hit a bug introduced in the latest version by some evolutions.
v_1_8_3_dashboard_config.zip (622 Bytes)
In order to solve the problem, please download the attached patch file and execute, from the folder where you downloaded the patch, the following command:

sudo patch /var/www/html/config.php < v_1_8_3_dashboard_config.patch

Please let me know if it solves your problem.
Sorry for the inconvenience.

Best regards,
Cosmin


Upload Reference Data in Custom Job Dialog
Some problem with in-situ dataset
Insitu Data L4B Crop Type Custom Processor
#19

Thank you Cosmin, that seems to have solved the problem. I have successfully uploaded my Insitu data to the custom processors.

Thanks again
Graham


#20

Thanks @cudroiu, it worked for me to :ok_hand: