Spatialguru change on Twitter/Google Plus accounts

As a result of moving slightly away from “spatial” as a core focal area in my day-to-day work at Actian.com – (I do way more with Hadoop than spatial these days),  I started a new Twitter account with a less domain specific name.

My original Twitter account was spatialguru – I still use it, but less often than before . Now I’m using 1tylermitchell instead.

When I started calling myself spatialguru it was a bit of an inside joke around our home, I didn’t think it would still around this long.  🙂 Anyway, follow my new account if you want to see more about what I’m reading, etc.

Similarly, I have tried to migrate my previous Google plus account – tmitchell.osgeo – to a new one here.  Add me to your circles and I’ll probably add you to mine if you aren’t already.

Now, what to do about this blog name.. hmm.. more to come.

— Tyler

Leveraging Analytics for Personal Health

Read More
nikeplus-weekly

I spent the last 6 months undergoing some dramatic health changes (ping me for details), primarily diet, and now I’m getting around to refactoring my fitness.  Naturally, I want to try some apps that both collect lots of sensor data but that also present it back to me in a meaningful (and hopefully motivating) way.

While I’m not sure that I’d classify all the sensor data from my iPhone as a “big data” stream, it has some common attributes – in particular, that I want to keep it all and it will grow endlessly and never stop (to quote Actian’s CTO).  So as I’m moving around, walking, running and more, I want to capture that.  Then I want to use whatever tools I want to process, analyse and visualise it.

If you’re already into fitness than you’ve already hear about the myriad of devices waiting on store shelves for you to pick up – from pedometers to bathroom scales – and strap to your persons.  Internet of Things anyone?  It’s not your fridge measuring energy and toaster monitoring temperature – it’s your wrist band monitoring your every footstep and your phone correlating it with your location, time of day and weather.

What a great time to be alive for a data junkie working in analytics and visualisation!

A Few Apps

So let’s talk devices, apps and services…

MapMyFitness app on iPhone 5
MapMyFitness app on iPhone 5

One of the most popular ones I hear about is MapMyFitness.  You can read lots about it elsewhere.  I used it to track some walks and runs, but mainly to share with a friend who does the same (keeping an eye on the competition!).  I’ve used it on and off for a few years to keep track of walks and gym visits.  I only used it on my iPhone and found GPS mapping was really good.  I particularly like the elevation gain stats and the ability to select from past routes.

 

Moves app on iPhone 5.
Moves app on iPhone 5.

I briefly tried another app called Moves and it had a really interesting interface and visualiser.  All computed as automatically as possible, guesstimating where you are at (Home vs Gym) and what you are doing.  It’s neat, I might try it again someday now that I’m actually doing more than just driving around.

I see there are lots of other online services and apps that can read from or populate both these apps.  Do you see how the line between data, application and services continues to blur.  Just as the “open data” revolution has rocked governments we’ll see “open monitoring” apps rock our personal lives.  Or at least thats what I’m experiencing.

When (re)starting my fitness programme I had to decide what to do and what to stick with.  My main goal was to track general movement, but also to get good stats and motivation while walking or running.  Since I’d already used some of these other services, I figured it wouldn’t hurt to go with something else.  Plus, getting a gift for a birthday always helps 🙂

Nike+ Fuelband

So I went with the Nike+ Fuelband and have found it very useful.

Nike+ Fuelband activity tracker
Nike+ Fuelband activity tracker

In a nutshell, I’ve found that it:

  • Is unobtrusive
  • Fits comfortable (adjustable too)
  • Uses barely any power
  • Syncs constantly even during a workout
  • Doubles as a watch
  • Comes with some interesting apps – on phone, in browser and feedback on the band itself

I wasn’t familiar with NIKEFUEL – a single unit measurement that sums up all your activity.  It’s handy because you can compare your FUEL numbers with someone else on equal footing.  It roughly converts to calories but is more of a sum-total picture.

Nike+ Fuelband App on iPhone 5
Nike+ Fuelband App on iPhone 5

The Phone App

On the phone I fire up the Fuelband app and it syncs with the band and updates all my stats.  The basic stats screen shows how close I am to my FUEL points for the day, while changing the colour of items to also bring the point home that I’m aiming for “green”.  Other screens show how I’m doing by the week, month and year.

It’s all about goals.  I’ve found it recommended a good starting goal (which is still pushing me beyond what I was doing before this).  And it also recommended adding additional goals when I saw that I was starting to run (i.e. Try running 3x per week).

Nike+ Fuelband app on iPhone 5 - activity stats and motivations
Nike+ Fuelband app on iPhone 5 – activity stats and motivations

They also have various motivations throughout the app – “awards” and comparisons with others in my age bracket during the same time of day.  I’m really impressed with how they’ve leverage crowdsourced stats, while also keeping them anonymised yet useful for me.

I’ll show you how to can access that raw data in another post – but for now, enjoy the screenshots of their apps, knowing you can also build your own!

For most of the first week I only used the phone app and the band directly.  By clicking the single button on the band it displays time and daily stats: fuel, steps, hours won (hours in which you were moderately active for 5 minutes) and calories.  It has a great little coloured indicator that also changes as you reach toward “green”.  This kind of feedback completes the full circle of – monitor, analyse and act – what so much of big data analytics aims to do.. but I digress…

Nike+ Fuelband - goal status indicators
Nike+ Fuelband – goal status indicators

Web Apps

But the device and the mobile app are only part of the package.  By creating a Nike+ account you get access to all your Fuelband app data (and the Nike+ Running App data if you are using it. More on that another time I think.)

nikeplus-dashThe site includes a daily dashboard view, which is excellent – providing a good mix of high level goal achievement with meaningful visuals and links to more summarised weekly or monthly info logs.  This is a very useful free service and could be worth the price of a device in and of itself.

 

nikeplus-weeklyUp Next

Aside from using the Fuelband app, you can also use the Nike+ Running app, along with the Nike+ site, which uses your iPhone to collect everything.  More on that in another post.  I’ll also give an intro to using Nike+ developer API to get access to my data!

 

Check out FOSS4G event in Portland

Though it’s been years since I attended a FOSS4G event, it’s getting temptingly closer to home (geographically speaking). I know several good folks down in Portland and Oregon, so can rest assured that recommending you to check it out is a safe bet.  I’m sure it will be an all-you-can-eat smorg of geospatial software goodness!

Training: extending QGIS, intro too

If I needed QGIS beginner or advanced training, I’d try to take it from the founder of QGIS. Gary Sherman is advertising his 2014 courses on GeoApt.net:
Training from GeoApt LLC

We are currently organizing courses for the upcoming year:

If you are interested in either of these offerings, please register and we will send you details as to venue and dates once they are firmed up.

2 Dashboard examples – Devops and more

Read More
panopticon.com

I came across a couple interesting packages recently for both system and app monitoring scenarios.

AppFirst

Their “Beautiful Dashboards” slogan says it all – they have a free version of their DevOps Dashboard system for monitoring up to 3 servers.  Among lots of cool stuff, I especially like how they show historic ranges in the background of charts, and flag anomalies visibly.

appfirst.com

DataWatch / Panopticon

I met some of the DataWatch team at the recent Data Viz Summit and was impressed with their tutorial on using their dashboard building tools.  It all looked good and was easy to build and use – also the ability to drill-down was quite powerful as was their ability to attach to streams and perform playback.

panopticon.com

Tons of demos at http://panopticon.com/demos

QGIS Training Manual and Locate Press changes

Today’s an exciting day on several accounts…

First, it’s the day that Locate Press’s second title is now starting to become available through Amazon!  There are several titles in the works but this one made it past the post first.

This relates also to my second item…  I’ve stepped back from Locate Press over the past month to hand the leadership over to Gary Sherman.  This is really good news for all our readers and future authors, because he will have more time than I did to keep things moving forward.

Personally, my day job is demanding enough for me at the moment and I knew it wasn’t going to get any easier, so finding a replacement to take over the company was a no-brainer.  Gary’s been involved with Locate Press from the very beginning and knows the toolchain, editorial processes and publishing paradigm just as well as I ever did.  In fact, he single handedly put the QGIS Training Manual into production, which was inspiring to watch from the sidelines.

I’m still around and helping where needed, but am focused on my continuing work with Actian Corp which is growing and innovating in some interesting ways – it’s a great company to being working with and I’ll share more as things progress there.

By the way, with this transition, it’s a great time to pitch your new geospatial book ideas to Gary – drop him a note!

Thanks for all the support I received when launching Locate Press!!

Ecstatically Hyperlocal Communication Matrix – aka Coffee House

Jack into the ‘net -make an impact and realise your full potential!
Taken from future advert for the first Matrix implant computing system.

By metacortex on redbubble.com

Don’t sign me up…

I really enjoy technology that allows me to do work from a remote home office, but outside of work unless the technology enables face-to-face time and with people geographically near to me then my time may be best spent elsewhere.  Having children helps keep it real for me.  My filter operates on the premise that unless something I’m doing online is going to benefit my work/career, my family, my community, or teach me something useful within those spheres – then it’s not high priority.

Stay with me, this isn’t just a rant about disconnectness or a deadbeat Internet obsessed parent!  This potentially applies to you, your business network and how you come up with new ideas.

Social apps leave me cold.  I guess if they’re called social apps I can’t fault them for focusing on solving social “problems” such as “How to find your friends on Friday night!” or answering important life questions like “should I eat here?”  Now we’re able to find “deals” locally and much much more with all this hyperlocal app mumbo jumbo.  But how did we ever survive without these precious tools?  I dare say we had to talk with friends, family and neighbours to get recommendations or phone them to ask what they’re up to next week.

All the work-related benefits of Internet/social apps/LBS technology aside, consider how information technology can easily push us toward isolation from others around us.  On the surface it doesn’t seem to be a bad thing – with 10 minutes noticed I can hang out online with more geo-geeks than I can ever muster up within a 400km radius from my home office.  Seriously.

But when it comes to connecting with my neighbour, our city council or even local businesses, it’s a complete and utter fail.  Granted, I don’t live in the largest municipality, but the impact is similar in large centres too.  You can occasionally find some new local shop or service you didn’t know was just around the corner, but more often than not you find one that is across town or out of town instead.  As we travel to these non-local sites we continue the isolation from our “real” local communities and local business networks.

If the technology is helping isolate us, how do we bring it back around?  Egads!  It gets kinda messy to be liberated from this Matrix machine sometimes, but there are some time-tested models for rehabilitation that you might consider.  Here are two…

In a previous post (Open Gov/Data Needs Collaborative Common Spaces) – I proposed that we need to get data providers into the same rooms as those who are users or, better yet, those who best add value to the data.  Rubbing shoulders, face-to-face, sharing problems and solutions across domains and disciplines is key.  We don’t always know what the outcome will be, but the more isolated these stakeholders and providers are, the more insular the outcome.  Guaranteed.  I was pointing to the co-working model as one way to help make that happen – though enabling public servants to hang out in more accessible locations may be the largest barrier to that idea!

Steven Johnson touches on a similar vein in his talk on where good ideas come from.  Hint: it’s not from reviewing the results of a multi-year stakeholder survey.  Nor is it from mining mailing lists or user groups dedicated to a particular topic.  His conclusion is similar to mine, using the coffee house as an example of the kind of unstructured communication that leads to breakthroughs.  Now we are way beyond metaphors here, we’re talking real, meaningful and messy ways of bringing people together to serendipitously meet, chat and share.  That could be around the boardroom table once a week when you bring in disparate team members, but even better if you can be rubbing shoulders with others in a non-exclusive environment.

What does that look like?

I have some vague ideas – co-working space, coffee shop, impromptu customer visits, hackfests, heck even the water cooler! – but I think we’re still early in defining what these look like and how to best engage at a more meaningful level.  Ironically, I do believe that much of the answer lies in (pre-Internet era) history, so asking relatives that are not as online connected may reveal some interesting results.

However, I do believe that the first step in truly coming to grips with the reality of techno-fueled isolation is to admit there is a need to “bring ourselves back together” in a meaningful way.  It’s early days, but mark my words we’ll have a whole industry devoted to helping people get “unplugged” and we’ll see friends and colleagues disappear from online while they reclaim their local relationships.  I don’t think it needs to be that dramatic but is worthy of some consideration to find an effective balance and to check that our pulses are not timed with our router packet traffic.

We’ll know there has been progress when hyperlocal starts to refer to relationships between people and not our relationship with applied technology.  Likewise, we could benefit by reclaiming the definition of communication as a conversation between two people.  And the time is now, not after Skynet becomes sentient or the Matrix materialises… without our knowing.

Open Gov/Data Needs Collaborative Common Spaces

When innovation is sought after, for example in the government open data initiatives, it becomes even more important to have effective connection with people in the trenches.

I propose the use of collaborative meeting spaces, a sort of Demilitarized Zone for government and constituents to connect – physical places where they can meet together, and not just so they can meet face-to-face, but to implement an innovative (new?) way of making spontaneous collaboration happen

Read my full article on GoGeomatics.ca.

Desktop + Processing Server = GeoShangri-La

Those who know me are likely sick of my retelling of stories about my observations on the impact of migrating away from command line GIS (circa ArcInfo Workstation) to the typical desktop GIS with graphical user interfaces. More often than not, however, my desire for server-side scripting is now done within spatial databases (Ingres and Postgres/PostGIS databases being the two I recommend the most).  The interface between the GIS client and the server is still ripe for innovation however.

While some desktop applications can leverage server-side processes, instead of pulling it all down to the client, I have seen it less in-the-wild than I have the traditional client-side approach.  I’m hopeful that with products like ArcGIS Server, GeoServer or even GIS Cloud, PyWPS, deegree, 52North, Zoo, etc. much of that work will finally be put where it belongs.

Enter the Web Processing Service (WPS) to help glue it all together.  A couple years back I barely knew this existed, but now that it is easy to enable this on GeoServer, it’s starting to draw my attention more and more.  However, I can’t help but think that lack of client support is hampering the adoption of a WPS-based workflow.  When I saw there was a WPS plugin for QGIS then I really started paying attention!

Here’s why…

Imagine having layers loaded, say, through a WFS.  So you’re just drawing some features, selecting a few and then sending off a request to the server to process a buffer or to do a clip function.  Behind the scenes, QGIS would send just the select shape of interest, tell the server what process to run (along with some other parameters) and return the result to your map view.  Only a few local temporary files and all is done in one spot and kept clean.

Too good to be true?  You be the judge and let me know 🙂

p.s. The ultimate end game for me here is to have my Ingres spatial db serving up customer data via WFS, with options to process the data with GeoServer WPS.  I’ve tested against other platforms with more success, but I’ve only gotten halfway there with GeoServer.  If you’ve done QGIS with WPS plugin against a GeoServer WPS instance, I’d like to hear from you (twitter: spatialguru).  I can get it to create a grid, for example, but I can’t get the client to send data to the server and get a result.

Convert lines/polys to point data – ogr_explode.py

Read More
poly2point

How do I convert my linestring shapefile into points?

It’s a common enough question on the OSGeo mailing lists and stackexchange.com. Even with the power of GDAL/OGR command line utilities, there isn’t a single command you can run to do a conversion from lines/polygons into points. Enter ogr_explode.py.

I’ve co-opted the term “explode” to refer to this need and put together a simple Python script to show how it can be done. You can access it on GitHub here or see just the main guts of it listed below.

It’s just a start, without any real error checking, and can be greatly improved with more options – e.g. to take elevation from the parent lines/polys attributes, to output to another shapefile, etc. Because it outputs in a CSV (x,y,z) format, you can give it a .csv extension and start to access it with OGR utilities right away (VRT may be required). For example, you could then push it through gdal_grid, but that’s another topic…

Since it is a common enough need, it seems, I encourage you to fork a copy and share back.

[Note, QGIS has a nice option under its Vector -> Geometry Tools -> Extract Nodes tool to do the same thing with a GUI. Props to NathanW for having the most easily accessible example to refer to 🙂 ]


...
for feat in lay:
geom = feat.GetGeometryRef()
for ring in geom:
points = ring.GetPointCount()
for p in xrange(points):
lon, lat, z = ring.GetPoint(p)
outstr = "%s,%s,%s\n" % (lon,lat,z)
outfile.write(outstr)
...