My new book on Amazon – raster/vector data manipulation using GDAL/OGR

Read More
Geospatial Power Tools by Tyler Mitchell now on Amazon
Geospatial Power Tools by Tyler Mitchell now on Amazon
Geospatial Power Tools by Tyler Mitchell now on Amazon

Ten years ago I wrote a book for O’Reilly called Web Mapping Illustrated – using open source GIS tools. It was mostly about how to use MapServer and PostGIS to publish maps on the web and was the first of its kind in the marketplace.

This year I’ve completed my second book, for Locate Press, which focused on even more low level geospatial data manipulation using the GDAL/OGR command line tools. This was a work-in-progress for a couple of years, but has just now been released on Amazon as Geospatial Power Tools.

If you’re looking for a resource to understand how to convert imagery, vector data or to build elevation shaded maps or contours, and more, then this book is for you. It includes complete GDAL and OGR documentation. A third of the book presents new material geared to help you learn how to do specific kinds of processing tasks – from downloading from web services, to quickly converting imagery into an online map. A PDF version is also available and Kindle will likely come over the next 6 months.

I’m always interested in feedback on the book and to learn more about how to improve the next edition.

Create Tile Map Structure – gdal2tiles command

Read More
OpenLayers mapping application showing natural earth dataset
Default OpenLayers application produced by the gdal2tiles command and a Natural Earth background dataset.

Tiles in a Tile Map Server (TMS) context are basically raster map data that’s broken into tiny pre-rendered tiles for maximum web client loading efficiency. GDAL, with Python, can chop up your input raster into the folder/file name and numbering structures that TMS compliant clients expect.

OpenLayers mapping application showing natural earth dataset
Default OpenLayers application produced by the gdal2tiles command and a Natural Earth background dataset as input.

This is an excerpt from the book: Geospatial Power Tools – Open Source GDAL/OGR Command Line Tools by me, Tyler Mitchell.  The book is a comprehensive manual as well as a guide to typical data processing workflows, such as the following short sample…

The bonus with this utility is that it also creates a basic web mapping application that you can start using right away.

The script is designed to use georeferenced rasters, however, any raster should also work with the right options. The (georeferenced) Natural Earth raster dataset is used in the first examples, with a non-georeferenced raster at the end.

There are many options to tweak the output and setup of the map services; see the complete gdal2tiles chapter for more information.

Minimal TMS Generation

At the bare minimum an input file is needed:

The output created is the same name as the input file, and include an array of sub-folders and sample web pages:

Open the openlayers.html file in a web browser to see the results.

The default map loads a Google Maps layer, it will complain that you do not have an appropriate API key setup in the file, ignore it and switch to the OpenStreetMap layer in the right hand layer listing.

 

The resulting map should show your nicely coloured world map image from the Natural Earth dataset. The TMS Overlay option will show in the layer listing, so you can toggle it on/off to see that it truly is loading. Figure 5.2 (above) shows the result of our gdal2tiles command.


Geospatial Power Tools is 350+ pages long – 100 of those pages cover these kinds of workflow topic examples.  Each copy includes a complete (edited!) set of the GDAL/OGR command line documentation as well as the following topics/examples:

Workflow Table of Contents

  1. Report Raster Information – gdalinfo 23
  2. Web Services – Retrieving Rasters (WMS) 29
  3. Report Vector Information – ogrinfo 35
  4. Web Services – Retrieving Vectors (WFS) 45
  5. Translate Rasters – gdal_translate 49
  6. Translate Vectors – ogr2ogr 63
  7. Transform Rasters – gdalwarp 71
  8. Create Raster Overviews – gdaladdo 75
  9. Create Tile Map Structure – gdal2tiles 79
  10. MapServer Raster Tileindex – gdaltindex 85
  11. MapServer Vector Tileindex – ogrtindex 89
  12. Virtual Raster Format – gdalbuildvrt 93
  13. Virtual Vector Format – ogr2vrt 97
  14. Raster Mosaics – gdal_merge 107

Create a Union VRT from a folder of Vector files

Read More
Unioned OGR VRT layers - source layers beneath final resulting merged layer
The following is an excerpt from the book: Geospatial Power Tools – Open Source GDAL/OGR Command Line Tools by me, Tyler Mitchell.  The book is a comprehensive manual as well as a guide to typical data processing workflows, such as the following short sample…

The real power of VRT files comes into play when you want create virtual representations of features as well.  In this case, you can virtually tile together many individual layers as one.  At the present time you cannot do this with a single command line but it only takes adding two simple lines to the VRT XML file to make it start working.

Here we want to create a virtual vector layer from all the files containing lines in the ne/10m_cultural folder.

First, to keep it simple, create a folder and copy in only the files we are interested in:

Then we can create our VRT file using ogr2vrt as shown in the previous example:

If added to QGIS at this point, it will merely present a list of four layers to select to load. This is not what we want.

So next we edit the resulting all_lines.vrt file and add a line that tells GDAL/OGR that the contents are to be presented as a unioned layer with a given name (i.e. “UnionedLines”).

The added line is the second one below, along with the closing line second from the end:

Now loading it into QGIS automatically loads it as a single layer but, behind the scenes, it is a virtual representation of all four source layers.

On the map in Figure 5.8 the unionedLines layer is drawn on top using red lines, whereas all the source files (that I manually loaded) are shown with a light shading. This shows that the new virtual layer covers all the source layer features.

Unioned OGR VRT layers - source layers beneath final resulting merged layer
Unioned OGR VRT layers – source layers beneath final resulting merged layer

 


Geospatial Power Tools is 350 pages long – 100 of those pages cover these kinds of workflow topic examples.  Each copy includes a complete (edited!) set of the GDAL/OGR command line documentation as well as the following topics/examples:

Workflow Table of Contents

  1. Report Raster Information – gdalinfo 23
  2. Web Services – Retrieving Rasters (WMS) 29
  3. Report Vector Information – ogrinfo 35
  4. Web Services – Retrieving Vectors (WFS) 45
  5. Translate Rasters – gdal_translate 49
  6. Translate Vectors – ogr2ogr 63
  7. Transform Rasters – gdalwarp 71
  8. Create Raster Overviews – gdaladdo 75
  9. Create Tile Map Structure – gdal2tiles 79
  10. MapServer Raster Tileindex – gdaltindex 85
  11. MapServer Vector Tileindex – ogrtindex 89
  12. Virtual Raster Format – gdalbuildvrt 93
  13. Virtual Vector Format – ogr2vrt 97
  14. Raster Mosaics – gdal_merge 107

Query Vector Data Using a WHERE Clause – ogrinfo

Read More
Geospatial Power Tools book cover
The following is an excerpt from the book: Geospatial Power Tools – Open Source GDAL/OGR Command Line Tools by Tyler Mitchell.  The book is a comprehensive manual as well as a guide to typical data processing workflows, such as the following short sample…

Use SQL Query Syntax with ogrinfo

Use a SQL-style -where clause option to return only the features that meet the expression. In this case, only return the populated places features that meet the criteria of having NAME = ’Shanghai’:

Building on the above, you can also query across all available layers, using the -al option and removing the specific layer name. Keep the same -where syntax and it will try to use it on each layer. In cases where a layer does not have the specific attribute, it will tell you, but will continue to process the other layers:

NOTE: More recent versions of ogrinfo appear to not support this and will likely give FAILURE messages instead.


Geospatial Power Tools is 350 pages long – 100 of those pages cover these kinds of workflow topic examples.  Each copy includes a complete (edited!) set of the GDAL/OGR command line documentation as well as the following topics/examples:

Workflow Table of Contents

  1. Report Raster Information – gdalinfo 23
  2. Web Services – Retrieving Rasters (WMS) 29
  3. Report Vector Information – ogrinfo 35
  4. Web Services – Retrieving Vectors (WFS) 45
  5. Translate Rasters – gdal_translate 49
  6. Translate Vectors – ogr2ogr 63
  7. Transform Rasters – gdalwarp 71
  8. Create Raster Overviews – gdaladdo 75
  9. Create Tile Map Structure – gdal2tiles 79
  10. MapServer Raster Tileindex – gdaltindex 85
  11. MapServer Vector Tileindex – ogrtindex 89
  12. Virtual Raster Format – gdalbuildvrt 93
  13. Virtual Vector Format – ogr2vrt 97
  14. Raster Mosaics – gdal_merge 107

GPS Controls OSSIMPlanet

Read More
Two wires to hook an eTrex data cable up to the Arduino

With OSSIMPlanet’s nifty camera control and listener functionality, as demonstrated in my last post, you’ve got so many neat opportunities. A couple nights ago I got a basic GPS NMEA parser working. Here’s a pic of the ultra-professional connection method I use to hook it to my arduino board 🙂

Two wires to hook an eTrex data cable up to the Arduino
Two wires to hook an eTrex data cable up to the Arduino

Oops, just realised that the picture shows the wires in the wrong spot (it was a late night photo). Pin 2 from the serial cable goes into the Power GND on left side of board. Pin 5 from serial cable goes into the Digital Pin 0 aka RX pin on lower right.

Note that I had a lot of confusion regarding the pin-out options for the Etrex. I looked at several diagrams and couldn’t get the right pins to work. It appears that you may actually need to swap the ground/TX pins to make it work – that’s what I had to do. This is something to do with how the serial connection works.. something I won’t pretend to understand – but swapping the wires worked, that’s all I know!

Of course you could always use your nifty new bluetooth GPS receiver, or plug your receiver directly into the PC and this would still work. But for me, I will have additional devices going into the arduino, where their signals will get mixed together before being sent to the PC.

Arduino setup

The arduino doesn’t do much here, except filter the strings coming from the GPS. It grabs only the $GPGGA string that has the location info I want. I specifically used the GPGGA because the Python NME parser code example I had used that one and I didn’t want to have to learn how to handle the $GPRMC strings.

Here’s the basic code I run on the Arduino:

 

I had originally hoped to do the XML prep in the arduino, but skipped it for now due to my poor understanding of variable types in the processing language. So for now I’ve still got a lot of cruft leftover in the above, that I didn’t need from the original tutorial code. But as I add more sensors I’ll want to do more mixing in the board itself.

So the arduino board just sends a raw NMEA $GPGGA string to the serial port, where Python takes over.

Python Serial Reader and OSSIM Controller

I then use a Python script that checks for strings coming from the arduino board. It does a bit of filtering, but not much error checking at the moment. This is the first time I’ve used Python to connect to the serial port, so it was fun to learn and so simple!

I found this GPGGA parser code and incorporated it into my script. I won’t paste it here as it is quite long. But here is the rest of my script – reading from Serial, parsing results, then reformatting and sending to OSSIMPlanet… comments, cruft and crummy coding.. all yours for free!

Be sure to have OSSIMPlanet running and set its Preferences to listen on port 5000 first.

That’s about it. In OSSIMPlanet the updates are practically instantaneous, but I wait 1 sec for new GPS data to come in. When I start using the nunchuks I’ll want to do more updates faster to emulate the movement smoother. But that’s for another day…