Geeking out on a GIS
A couple of months ago Paul Wilson asked a question on KnowledgeHub about the best way to organise data in a GIS. Well, that got the GIS folk out of their data loading torpor and provided some interesting discussion. It highlighted the many different approaches but all seemed to agree that it could always be done better. And it changes over time too.
Doesn't every GIS officer like a question like this? And the challenging opportunity to rearrange things in a way they see fit?
My response to the original question seemed to strike a chord amongst other Local Authority GIS people so maybe we are doing something right...
Morning all
I am looking at changing our GIS file structure as it is all getting in a bit of a mess! I was just wondering how others go about storing their GIS data. I am thinking of doing it using each service area as the main structure and then various departments as sub folders
Any thoughts would be appreciated
Thanks
Paul
Morning Paul
Doesn't every GIS officer like a question like this? And the challenging opportunity to rearrange things in a way they see fit?
We've finally managed to turn off our Oracle and ArcSDE services after one or two lingering applications where finally migrated to our PostgreSQL/PostGIS SDW. This has just about completed our migration to an open source GIS base and the consolidation of six file servers into The One Share. We pull a lot of business data into the PostGIS database on a daily basis using FME and batch files and this is made available to all desktop GIS users via QGIS and ArcGIS. OS rasters, historic rasters and aerial photography datasets are available through WMS (both OSMA and internal Geoserver) and can be consumed by most business applications requiring mapping (IDOX excluded (for now)). The One File Share is available to all data creators and editors for their spatial datasets but they are encouraged to store their data in the PostGIS database in a working schema and this is published to a live schema on a defined schedule. Datasets in the working database are organised by business function in different schema - gazetteer, roads, planning, transport, education, etc. - and in the live database by source - council data and thirdparty data. We're looking at ways of improving this and may move to a single database rather than a split work/live environment.
For all the other data consumers we publish a subset of the datasets to our intranet/internet mapping application (Location Centre) where they can view, print, share, and query data in a permission controlled spaced.
We've tried to make the data management process as efficient as possible so that we have more time to do the interesting stuff. Most of our processes run on scheduled tasks and others are set to be run by batch file when required. Making the data and information as accessible as possible to end users is where we put a fair amount of time. We also spend a lot of time working on gaining access to the different silos of data across the council to join it up and look at ways of using it and adding value. There is a lot of expensive unused information waiting to be mined...
We had all our raster datasets (OS, Landmark, Aerial, obliques, DEMs and LiDAR) on the file system already as there were a few applications that couldn't read them from Oracle and ArcSDE. So it was relatively easy to consolidate the different datasets onto one new file share and configure an internal Geoserver cluster (x2) (on a dedicated server) to build the catalogue and serve then as WMS. We use a set of batch files that call GDAL processes to process the OS COU and overwrite the old files with new files in situ. So now we only maintain a set of rasters for Geoserver and a smaller subset on another server for IDOX PA and OLDP. Everything else gets them via WMS. If there is a special requirement for say, terrain analysis, then we have a set of VRT files that users can load and access the raw data directly.
Fin.