Friday, May 7, 2010

Assembling Virginia's Geospatial Data with Google Earth Enterprise

Over the course of the past few weeks I've spent some time scouring the internet and university websites in the Commonwealth of Virginia to try to collect as much publicly available geospatial data as I could. To my delight, there is a lot more data online now than back in the 2000-2005 timeframe while I was a student at Virginia Tech's Geography department, working with Virginia GIS data quite regularly.

However, it could be a lot simpler. Not only did I have to spend a great deal of time searching for the data, often it was warehoused in sites that were not friendly to bulk downloading the entire state at once. Some scripting helped here, but it was still somewhat cumbersome.

I also found lots of dead links, even on sites like Virginia.gov. This includes the occasional missing file, or worse, sometimes entire data-sets have disappeared.

Also I have to say that my overall experience working the nearly 500GB of data that I did find was disappointing. Data in the Commonwealth of Virginia is primarily found in one of five projections.
  • UTM Zone 17
  • UTM Zone 18
  • Virginia State Plane North
  • Virginia State Plane South
  • A custom Department of Transportation Lambert Conical Statewide projection
  • or, Unprojected WGS84
Now, knowing this helped a great deal because SOOOOO MUCH of the data comes without accompanying projection files, or any metadata that describes what projection any given piece of data may be in. Frustratingly, much of the data was also commingled so that you might literally have all 5 projections for a single data-set seemingly randomly distributed throughout a single data-set. This likely occurred because the data was never really assembled all together in a seamless GIS before, and instead the tiles were used one or two off at a time in desktop GIS software. 

I wanted to be able to visualize all this data in one environment though, so I've done a lot of work to fix the problems, and where possible, track down the source data from the original providers. I'm very happy to find a good steward of this data for the Commonwealth now that I've corrected it so that others can find clean data to use in their project. 

I have only just scraped the tip of the iceberg of the total GIS data for the Commonwealth - there is so much more out there hidden in all of municipalities web-based GIS systems that would be fantastic to incorporate. 

I also hope that one day the work I've done on the Google Earth Enterprise system will be made available to the citizens of the Commonwealth. After all, it is your tax dollars that are paying for all this data - you should demand a way to use it easily! 

If you're interested in seeing the Google Earth Enterprise for the Commonwealth of Virginia's progress thus far, and for more discussion about existing GIS data-sources in the Commonwealth, please view the video below. 

If you have any questions please leave a comment, or e-mail my colleague at Google, Mary Jean Clark.