Re: A couple of filtering problems with CSV

Hi Robert,

You've helped me indirectly solve one problem, and I think I've  
further refined the second!

Turns out I had a couple of comments at the top of the ignore.csv as  
follows

# Temporary Finds
# Found since 2007-02-22

This was producing a couple of extra waypoints in the resulting GPX  
file at 0,0 and 0,22 that I hadn't noticed before. As there wasn't  
much information, they didn't appear to be showing in GE which is why  
I had missed them previously. I removed the comments and the file  
processed correctly. So I'd mark the bounding box issue as resolved  
from my perspective. Bad assumption of mine to be able to place  
comments in the file.

For the second, I think I have refined it down to something choking  
on unicode characters. When viewing the source in terminal, the  
breaks appear to be occurring at <E2><80><99> - which appears to be a  
unicode apostrophe or quote. I can't yet confirm that it isn't  
happening with other unicode characters as well.

To give you some idea why I figured it has something to do with the  
csv file, have a look at the following commands I run in a script to  
combine and filter a number of my PQ's. All of the gpx-only commands  
work perfectly and give the expected output.

(for those interested, I have an older version on our website, I'll  
update it again once I've got the final wrinkles worked out)
<http://www.gps.org.nz/resources/paperless_geocaching/gpsbabel>

The only command that produces the truncated XML container content is  
the final one that invokes the ignore.csv.

gpsbabel -i gpx -f 388021.gpx -f 388021-wpts.gpx -o gpx -F caches- 
world-found.gpx
gpsbabel -i gpx -f 854701.gpx -o gpx -F caches-nz-placed.gpx
gpsbabel -i gpx -f 158140.gpx -f 158139.gpx -f 166136.gpx -f  
166137.gpx -f 410114.gpx -f 896898.gpx -f 158140-wpts.gpx -f 158139- 
wpts.gpx -f 166136-wpts.gpx -f 166137-wpts.gpx -f 410114-wpts.gpx -f  
896898-wpts.gpx -o gpx -F caches-nz.gpx
gpsbabel -i gpx -f 905319.gpx -f 905319-wpts.gpx -o gpx -F caches-nz- 
unavailable.gpx
gpsbabel -i gpx -f caches-nz.gpx -f caches-world-found.gpx -f caches- 
world-found.gpx -f caches-nz-placed.gpx -f caches-nz-placed.gpx -f  
caches-nz-unavailable.gpx -f caches-nz-unavailable.gpx -x  
duplicate,shortname,all -o gpx -F caches-nz-active.gpx
gpsbabel -i gpx -f caches-nz-active.gpx -x polygon,file=ni-poly.txt - 
o gpx -F caches-nz-ni-active.gpx
gpsbabel -i gpx -f caches-nz-active.gpx -x polygon,file=si-poly.txt - 
o gpx -F caches-nz-si-active.gpx
gpsbabel -i gpx -f caches-nz-si-active.gpx -x  
radius,distance=85K,lat=-43.478,lon=172.702 -o gpx -F caches-nz-si- 
chc-active.gpx
gpsbabel -i gpx -f caches-nz-si-active.gpx -x  
radius,distance=35K,lat=-41.25,lon=173.1 -o gpx -F caches-nz-si-nsn- 
active.gpx
gpsbabel -i gpx -f caches-nz-ni-active.gpx -x  
radius,distance=25K,lat=-41.2,lon=174.8 -o gpx -F caches-nz-ni-wlg- 
active.gpx
gpsbabel -i gpx -f caches-nz-ni-active.gpx -x  
radius,distance=20K,lat=-36.85,lon=174.8 -o gpx -F caches-nz-ni-akl- 
active.gpx
gpsbabel -i gpx -f caches-nz-si-active.gpx -x arc,file=chc- 
nsn.txt,distance=4K -o gpx -F caches-nz-si-chcnsn-active.gpx
gpsbabel -i gpx -f caches-nz-active.gpx -x polygon,file=chc-dud.txt - 
o gpx -F caches-nz-si-chcdud-active.gpx
gpsbabel -i gpx -f caches-nz-si-chc-active.gpx -f caches-nz-si-nsn- 
active.gpx -f caches-nz-ni-wlg-active.gpx -f caches-nz-ni-akl- 
active.gpx -f caches-nz-si-chcnsn-active.gpx -x duplicate,shortname - 
o gpx -F caches-nz-popular-active.gpx
gpsbabel -i gpx -f caches-nz-si-chc-active.gpx -i csv -f ignore.csv - 
x duplicate,shortname,all -o gpx -F caches-nz-si-chc-active-ge.gpx

I have also reproduced the error using the attached test case files.

Command used was...

gpsbabel -i gpx -f source.gpx -i csv -f ignore.csv -x  
duplicate,shortname,all -o gpx -F results1.gpx

This produced a truncated description for GC100XE in results1.gpx.

I also copied GCNE66 to ignore.gpx and ran a similar filter

gpsbabel -i gpx -f source.gpx -f ignore.gpx -x  
duplicate,shortname,all -o gpx -F results2.gpx

This resulted in GC100XE having a complete and correct description -  
i.e. it didn't choke on the unicode characters.

So, it would appear that truncating at unicode characters is only  
occurring when the csv file is used.

Cheers Gavin

results2.gpx
Description: Binary data

ignore.gpx
Description: Binary data

results1.gpx
Description: Binary data

ignore.csv
Description: Binary data

source.gpx
Description: Binary data

On 14/03/2007, at 15:41, Robert Lipe wrote:

> Gavin, thanx for the detailed report.
>
> > encompasses much of the world when extended to the caches down in
> > little ole New Zealand.
>
> That may part of the problem - you're in the opposite hemispheres from
> the bulk of our users (and developers!) so there may be some weirdness
> there.
>
> > maxlat of 0 being produced. Could this have anything to do with us
> > being located near the +/-180 boundary?
>
> That's certainly not impossible. It's not obvious to me how a bounding
> box that spans a hemi boundary should work but I can poll some  
> other GPX
> writers and see how they represent it.
>
> > Is there some way of having the bounds recalculated so that they
> > always properly express the co-ordinates to be found in the resulting
> > file, rather than some temporary state during processing? This is
> > only a problem when filtering the GPX XML with CSV.
>
> By inspection, the GPX writer *should* be computing bounding boxes  
> based
> on the filtered set and not the source set, but a test case (a few
> waypoints and the precise commands issued) would be most helpful.
>
> > I assume this is occurring as the CSV parser is choking on some
> > special characters. Interestingly, it has only occurred when being
>
> The CSV parser isn't involved in reading GPX.
>
> > Let me know if I can provide more info on either of these issues to
> > help.
>
> I'd really like to see the source and the commands it takes to trigger
> this.  If you can trim it down to just a cache or two that's great.
>
> RJL

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV_______________________________________________
Gpsbabel-misc mailing list http://www.gpsbabel.org
Gpsbabel-misc@xxxxxxxxxxxxxxxxxxxxx
To unsubscribe, change list options, or see archives, visit:
https://lists.sourceforge.net/lists/listinfo/gpsbabel-misc