Pages

Friday 11 July 2014

Updating cache data

GSAK is my database of geocaches. The question is, how do I keep it updated?

1. I have seven PQs that all run each day. They cover "Southern England", "London" and the other five regions that I'm likely to visit, my total "caching area". There's a limit of 1000 hits in each query, so I ask it to find all those listed within the last month.

That gives me an up-to-date database of caches, and the first 30 days of logs. A good start. And GSAK has an option to grab all the PQs from the web site for me without me having to download each one and feed it in.

2. GSAK also has an option to update cache listings. But there's an allowance of 6000 full listings (including logs) or 10,000 partial listings (wthout logs). I have my caching area divided up into a grid of 16 squares. Each day, I can ask GSAK to get an update of one of those squares. That means that my log list is updated even after the first month.

3. There's a problem. People archive caches, and that isn't reflected in either of the updates above. But GSAK has an option "Status update"; that updates the "current/disabled/archived" flag. I need to do this every few days, so that I dont go hunting for a cache that is no longer there. For a while, they limited the number of caches you could check to 10,000, but that limitation seems to have gone now.

4. I have a second account on the web site. That means that I can do 6,000 more updates each day, if I want to.

So with all the above, I can keep my GSAK database pretty much up-to-date. But then I have to get information onto my PDA, TomTom satnav and iPhone. I want it on the iPhone as a backup in case my PDA stops working while I'm in the middle of nowhere (this happened a couple of days ago when the rain got me).

So I run a query on my GSAK database, making a GPX file of all the caches within 130 kilometers of my home that I don't own and haven't found. That gives me a huge file, but then I process it. I have a program I've written called gpxtocsv that:

1) Creates POI files for the TomTom satnav, showing trads, multis, puzzles, solved puzzes and "other" caches.
2) Creates cut-down GPX files of the iPhone, for importation into Memory Map. There's four of these: NW, NE, SW and SE.
3) Creates gpx file for Geosphere; Geosphere is my preferred iPhone app for caching. There's 16 files, in a 4 by 4 grid. If I'm not in a place where I can get 3G signal, I can use this offline database.
4) Creates CSV files for Memory Map. One big one, which is about 46,000 caches; I  an load that onto my main desktop computer. But Memory Map on the PDA can't handle such a big file, so I make four files: NW, NE, SW and SE.
5)  Creates MMO files, 16 of them. They get stored on my PDA CF card, and that means that if I have the wrong Memory Map file, I can switch to any of the 16. That's been useful once, when I loaded the NW quadrant onto Memory Map, but on the way home I passed through the NE quadrant. I saw the cache on TomTom, but I had to load the relevant MMO file on the PDA to find the cache.
6) Creates html files, so that when I click on a cache on my PDA, it displays the cache information, including the last 25 logs. I find that reading the logs sometimes gives me that bit of extra help that lets me find a stubborn cache. But I can't store 46,000 files; the PDA uses the FAT file system, and if you put that many files into a single directory, it runs like cold treacle. So, I lump into a single file, all the caches that have the same starting three letters (two letters for four-character cache codes). And very often, when I'm going round a circuit, that means that adjacent caches are on the same file, so displaying the info is instant, because it's already loaded. Doing it this way means only 4400 files, and the FAT system can cope with it.

So.

I update GSAK, I run a query to output all unfound caches within 130 km, I run gpxtocsv to create all the various files I want. Then I update the html files on the cf card (and on my backup card), and load the Memory Map info for the quadrant that I'll be visiting. I copy the POI files onto my TomTom, and copy the Geosphere and Memory Map files onto my iPhone.


I have no idea how other people cope with this.


5 comments:

  1. I am having the same problem currently. I'm thinking of creating a daily PQ with all the new caches to help me spot potential circuits. I tend to plan my routes quite meticulously as renumber so I usually spot archived caches that way. How I used to do it was to search for everything that's Last GPX update was a couple back and then do a search and replace to archive a bulk load.

    ReplyDelete
  2. Try my method, I think it works well.

    ReplyDelete
  3. I cache in smaller areas but I still find that a home-brew chunk of Perl is handy to munge the GPX files and my local cache notes (e.g. puzzle solutions) into GPX files, POI files, and KMZ files.

    ReplyDelete
  4. I tried to visit those two blogs.

    "The Blogger Profile you requested cannot be displayed. Many Blogger users have not yet elected to publicly share their Profile."

    ReplyDelete
  5. Things have changed with API ver2. You can now download 16,000 full waypoints and 10,000 light.

    I can get a 100km radius from my house with 25 PQs in 2.5 days. Use Project-GCs PQ splitter to do the hard work:

    https://project-gc.com/Tools/PQSplit?multi_country%5B%5D=United+Kingdom&hidefound=on&location=N+50%C2%B0+51.576+W+001%C2%B0+11.263&max_distance=100&submit=Filter

    As you know GC.com does not give you all the fields so use the APIv2 to get your Personal Notes and Corrected Coordinates with GSAK's Get Geocaches->GCXXXX code option (This covers the same area as the PQs but only those caches that contain Personal Notes. All my caches that have corrected coordinates on the cache page also have a personal note. Which ensures they all get downloaded):

    location:[50.xxxxxx,-1.yyyyyy]+radius:100km+fby:not(username)+hn:

    Then plan your day by flagging caches within a polygon of your search area. Filter on the userflags. Full refresh with 30 logs. Export to your device/s.

    I've created a macro that also creates vcards, all sorted in my required order, that loads directly into my cars satnav via USB.

    As you mentioned, also run a status check on caches that did not get updated to weed out any archived ones.

    So I am always up-to-date and have plenty of download allowance left for stepping outside my area and other things.

    ReplyDelete