Make Stellar/Star Data Maps In GIS

Read More
hygdata_quick_qgis_starmap

I have a latent interest in stargazing but haven’t done much about it for a long time until the past weekend when I had a bit of time and wondered if I could create a star map using QGIS. I found a couple tutorials and stackexchange questions that referenced David Nash’s HYG Database from www.Astronexus.com. Some of the tutorials showed equations for computing latitude and longitude from the star position values – ascension, declination specifically. However, I found that there is already an X and Y column in the latest version of the dataset which makes it easy to map, here’s how.

Download the Star Location Data

From the HYG Database page, grab the HYG version 3.0 dataset:

  • HYG 3.0: Database containing all stars in Hipparcos, Yale Bright Star, and Gliese catalogs (almost 120,000 stars, 14 MB)

The resulting 34MB CSV file contains about 20 columns and 119,615 rows.

Import CSV File into QGIS

  1. Launch QGIS (I’m using 2.10.1-Pisa)
  2. Select Layer -> Add Layer -> Add Delimited Text Layer
  3. Browse to the file location, select File Format radio button as CSV and the rest should take care of itself

Form importing the HYG Database star map CSV file into QGIS

Tweak the Map

If you use a typical WGS84 projection you’ll get a ton of points oriented in a circle.  I took a few minutes to zoom into a meaningful scale (for me it was around 1:5M), change the background colour and then use the LUM attribute to change the relative size of the stars.  I also changed the colours to be shades of white.  Here is what I was able to produce.  It’s not all that meaningful to me but I know that when I need to dig into the data further it will be available!

hygdata_quick_qgis_starmap

I’ve attached my project file here so you can get started quickly.

I’d love to hear what you would use this kind of data for and if you have any tips for making it more useful or better rendered.

Trick serving a hillshade image from GeoServer

Read More
When GeoServer skews the image it creates a region around the image that has value=0.  This image already had nodata=255, so it can be very confusing.
When GeoServer skews the image it creates a region around the image that has value=0. This image already had nodata=255, so it can be very confusing.

I had the perfect hillshade raster ready to throw on top of my web map, I had previewed it in QGIS and the colour stretching and values looked fine.
Hillshade in QGIS

I then loaded it into GeoServer and it looked great in the OpenLayers Layer Preview window as well.

However, once I added it as a WMS layer in a Leaflet map, I got really confused.  Even though I had my NODATA=255 parameters seemingly working well there appeared a small strip around the outside of the image that was obnoxious.

When GeoServer skews the image it creates a region around the image that has value=0.  This image already had nodata=255, so it can be very confusing.
When GeoServer skews the image it creates a region around the image that has value=0. This image already had nodata=255, so it can be very confusing.

I figured I could change it with an SLD but hadn’t done it before so it took a while to figure out.  For a couple hours I tried various permutations of the default “raster” Style, but with no luck.  The image, otherwise, looked great so the Style obviously wasn’t a real problem in and of itself.

The trick came when I realised that I need to set my color maps for the gray band explicitly.  Up to this point no particular band had been isolated for color mapping.  I found the right place to put the elements in the SLD and all worked out well.

My takeaway: there must be a better way to develop SLD for rasters.  The options in the GeoServer docs are amazing and exciting, but trying to fit them into a specific context has always been a challenge for me.  That’s why I had to write this down, but I’m sure I’ll have to remember it all again a year from now!

Note there is no skewed "collar" around the image as value=0 is ignored.
Note there is no skewed “collar” around the image as value=0 is ignored.

 
My SLD:

 

 

Day 1 – Building a 3D Game using Geo Data (Hobby project)

Read More
Day 1 - Unity 3D real time rendering of geodata

We are a team of three guys (so far) with a design idea for a simple yet challenging racing/strategy game based on real-world geospatial data coupled with a 3D gaming engine. Tonight we started building it together in Unity after downloading a bunch of public geospatial datasets. Here’s a summary of what we’ve done and some basic screenshots. The goal is to make progress every couple days to show we is possible with a little work and a combination of creative, technical and group management skillsets.

First Images

Day 1 - Unity 3D real time rendering of geodata

Starting with Geography

The landscape of our 3D game is built using real world geographic information – aka geospatial data. We wanted this so that we could build games around real locales and market the game to local users. One side benefit is the players will learn a little about Canadian geography!

We’ve got a great collection of geo data for all of Canada available through the government Geogratis website. After we’ve decided what map tile numbers we needed, we grab the CDED data which is a TIFF elevation image file with geographic coordinates included.

We also found a BC provincial shaded relief map layer that was pretty nice as a starter. As both these data files are geolocated we can load the data into GIS (geographic information system) software from QGIS.org and combine it with any other data we have for the region. In our case we have a “populated places” point file from naturalearthdata.org that we show as stars on top of the shaded relief map.

Then we export the elevation data and the relief data (including the stars) into two files for Unity to ingest. The result is one PNG that is going to be our texture and the other will be the heightmap for the terrain.

day1_geodata

Loading into Unity

We created a terrain object with some pretty huge dimensions and height. There are mountains in the region we are working with, so it’s pretty cool to see. We apply the heightmap using a commonly used Unity script (I’ll have to put the link here when I remember where I got it!).

Then we add the shade relief map as a texture to the terrain et voila! We threw in a plane of basic water and raised it to the level where it filled the main river valleys. As our game is going to first start as a river racing game, we want to start with water from the very beginning.

We added a car, parented it to the camera and were racing down our waterways within 2 hours of starting. We spent a lot of time adjusting sizes of terrains and texture so we could try to match real-world scale. There are some further ideas we are going to try to optimise this as well as nail down a workflow for easily ingesting new geodata for other regions (as we had to manually export and adjust things in the GIS).


Reposted from our Indiedb blog, hence writing for a slightly non-geo audience: http://www.indiedb.com/members/1tylermitchell/blogs/edit/1mitchellco-day-1-cross-country-river-run-race-game

 

PDF Map Hack – Remove Embedded Image

Read More
pdforthomap

Need to remove an embedded image from a PDF file?  You can easily chop out parts of it as needed with the PDFtk command line tool and a little bit of text editing.  Here’s how… Continue reading “PDF Map Hack – Remove Embedded Image”

Generate terrain with flowing water from DEM in Unity

Read More
terrain-waterflow-gis-unity

Just a quick video follow up based on a reader asking how I did what I did in my previous post with Unity (http://unity3d.com) game engine.

Using Surface Wave asset and built-in Unity Terrain generator, plus a script for taking DEMs and creating Terrains easily.  I’m really new at this by the way, but have a brilliant teacher showing me this stuff in my spare time 🙂

 

High speed water flow simulation on a DEM

Read More
waterflow-gis-unity

I’m helping some friends who are working on a project to visualise a whole whack of GIS data in Unity (Unity3D.com) game engine.  It looks like we’ll end up working on a GIS -> Unity workflow for generating terrains from DEMs and texture maps from orthophotos.  To top it off they’ve already got a landcover classification app running that takes landcover raster classes and creates 3D objects (grass, trees, water) in the model.  (Don’t worry, I won’t tease you by mentioning their voxel based subsurface soil model interaction).  It’s still early but really encouraging so far.

Next up is to simulate water flow in the environment and it was slim pickings for options for doing this.  Then they found the Unity asset called Surface Waves (US$80) – it does the water flow work we wanted but much more.  I just posted a really short test video to see how it worked – with both an auto generated water source and a manual placement water source, like a paint brush, that allows you to see how things will flow. It is amazingly performant on a notebook.

Be sure to check out Surface Waves’ demo video – it frees you from trying to emulate the look of water movement through shader trickery to actually simulating water flow over and around objects.  Things that used to take a sophisticated GIS quite a while to compute actually, the last time I tried anyway 🙂

More to come as we play around with it, but I put it out there in case other spatially oriented folks might be interested as well in the GIS -> Unity workflow challenges being worked through.  If so, I’ll do more video highlighting the work that’s underway.

Partitioned Data & Why It Matters

Read More
partitioned_bread

Would love to hear how you partition your spatial data..

As data volumes grow, so does your need to understand how to partition your data.  Until you understand this distributed storage concept, you will be unable to choose the best approach for the job.  This post gives an introductory explanation of partitioning and you will see why it is integral to the Hadoop Distributed File System (HDFS) increasingly used in modern big data architectures….

Read the full article on Make Data Useful >>

Parsing GPS with Arduino – retro-post

I dug up one of my oldest blog posts from about 7 years ago.  In the post, I show how I had connected my Garmin eTrex GPS receiver to an Arduino board and used it to control a camera in a desktop application.

After pumping the data into the Arduino, I parsed the raw GPS data coming from the eTrex and streamed it out to a Python app on the desktop, via a serial port.  I was using the output to position a virtual camera in the OSSIMPlanet platform by formatting XML in Python and sending it to a listener built into the OSSIMPlanet app.  (OSSIMPlanet is a sort of pre-Google Earth, Google Earth on steroids product.)

I haven’t used it for years, but thought some of the methods from this old post may still apply if you have an Arduino + GPS or Arduino -> Python streaming requirement.   Enjoy the flashback – I know it’s inspired me to pick up a new Arduino to continue where I left off.


 

GPS Controls OSSIMPlanet - With OSSIMPlanet’s nifty camera control and listener functionality, as demonstrated in my last post, you’ve got so many neat…

Two wires to hook an eTrex data cable up to the Arduino
Two wires to hook an eTrex data cable up to the Arduino

Read Post

Home Energy Monitor Series – Internet of Things

Read More
IoT Energy Series Header Image showing home energy gateway and monitor services

Recently I started an Internet of Things series on my experiences installing, using and analysing data from a smart electrical meter.  This included a BC Hydro smart meter, Eagle monitoring gateway from rainforest automation, and a cloud-based analytics service from Bidgely.

I’ve collated all the posts on the topic for you below.  More will be added as I write them.  Enjoy!


Continue reading “Home Energy Monitor Series – Internet of Things”

IoT Day 4: Bidgely Cloud Energy Monitor Dashboard

Read More
Bidgely Energy Monitor Dashboard - Usage

After a week of collecting smart meter readings, I’m now ready to show results in a cloud-based energy monitor system – Bidgely – complete with graphs showing readings, cost and machine learning results breaking down my usage by appliance.


Bidgely Energy Monitor Dashboard - UsageThis is part 4 of a series of posts about the Internet of Things applied to Home Energy Monitoring.  I have a Smart Meter from BC Hydro, an Eagle energy monitor and various cloud apps helping me understand it all.

 

See my post from Day 1 – getting started or Day 2 – connecting to cloud services – Day 3 – viewing data


3 Value Added Parts to Bidgely

In this post I’ll show you the three parts of Bidgely that I’ve found most helpful:

  1. Usage dashboard
  2. Cost dashboard
  3. Appliance breakdown (best for last!)

Usage Dashboard

Bidgely Energy Monitor Dashboard - Usage
Bidgely Energy Monitor – Usage Dashboard

Continue reading “IoT Day 4: Bidgely Cloud Energy Monitor Dashboard”