I recently got a dataset made up of 2364 images files representing 1km x 1km tiles of 5m resolution Digital Terrain Model (DTM).  There are also another 2364 files of 1km x 1km 2m resolution Digital Surface Model (DSM).  I want to serve these with Geoserver which prefers fewer, larger files over more smaller files.

GDAL is my tool of choice for batch processing images and in this case I want to use gdalwarp or gdal_merge to create my larger tiles.  The files are named logically based on the National Grid - NO4321.tif - and I can use this to create 10km x 10km tiles from the 1km x 1km tiles.  If I take the first (4) and third (2) numbers after the grid square reference (NO) I get the 10km tile reference - NO42.

So, how do I automate this process to create my tiles?

First up I need some text files, one for each 10km tile, with the list of image tiles making up the larger tile.  The following batch command has a nested loop that iterates through two sets of numbers (0 to 9) and substitutes those in the DIR command to find all image filenames that match the string (the ? represents a single character in the string):

FOR /L %%G IN (0,1,9) DO (  FOR /L %%H IN (0,1,9) DO (DIR /b /s NO_tiflzw\NO%%G?%%H?.tif > NO%%G%%H.txt  ))

I end up with a directory of text files like NO00.txt, NO01.txt, NO02.txt, etc.  Each text file has a list of the 1km tiles that make up the 10km tile.

gdal_merge and gdalwarp can take a text file list as an input so now we can loop through the list of text files and generate a 10km tile for each one:

FOR /F %%I IN (*.txt) DO gdal_merge -o NO_tifmerged%~nI.tif -of   GTiff -co TFW=YES -co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=LZW -ot Float32 -v --optfile %%I

This creates an internally tiled, LZW compressed, floating point GeoTIFF with accompanying world file.  If you want more control over the output raster try gdalwarp instead:

FOR /F %%I IN (*.txt) DO gdalwarp -s_srs EPSG:27700 -t_srs EPSG:27700 -ot Float32 -r average --config GDAL_CACHEMAX 500 -wm 500 -of GTiff -co TFW=YES -co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=LZW --optfile %%I NO_tifwarped%~nI.tif

After this I can generate some stats for each image:

FOR /F %%G IN ('dir /b *.tif') DO gdalinfo %%G -stats

And some overviews with gdaladdo for performance:

FOR /F %%H IN ('dir /b *.tif') DO gdaladdo -r average %%H 2 4 8 16

*Remember, if you run this from the command line replace the "%%" with "%"

Now I have 35 tiles to serve instead of 2364 which should make Geoserver faster.

If you're running on Linux here are some simple Bash shell scripts which do the same as the Command shell loops above:

for i in {0..9}
 do
   for j in {0..9}
   do
     ls -Q NO/NO$i?$j?.asc > NO$i$j.txt
   done
 done
exit 0

for f in .txt;
 do
   gdalwarp -s_srs EPSG:27700 -t_srs EPSG:27700 -ot Float32 -r average
--config GDAL_CACHEMAX 500 -wm 500 -of GTiff -co TFW=YES -co TILED=YES
-co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=LZW
--optfile $f NO_10k/"${f%.}".tif
 done
exit 0

How can we visualise our driving distance dataset another way? Can we create something like this which is using Voronoi polygons:

Palvealueanalyysi avoimella lähdekoodia #Hämeenlinna #PostGIS #pgRouting #sote #OSGeofi pic.twitter.com/ssyCD0vOxN

— Pekka Sarkola (@posiki) May 23, 2017

Well, yes we can. And PostGIS gives us the tools to put this together very quickly. First, create a view of the distinct nodes in your output table created by my batch function in a previous post.

CREATE OR REPLACE VIEW vw_sdd_nodes AS
  SELECT DISTINCT ON (node) *
  FROM school_driving_distance
  ORDER BY node, agg_cost ASC;

(from this StackExchange post).

Then using a new function in PostGIS 2.3.x, ST_VoronoiPolygons, create a voronoi polygon layer based on the road nodes in the view above. See this StackExchange post for more info.

CREATE OR REPLACE VIEW vw_sdd_voronoi AS
  WITH voronoi AS
  (SELECT geometry FROM vw_sdd_nodes)
SELECT
 (ST_Dump(ST_SetSRID(ST_CollectionExtract(ST_VoronoiPolygons(ST_Collect(geometry)),3),27700))).geom
AS the_geom
FROM voronoi;

Then, link the nodes back to the voronoi polygons so that each polygon gets a node ID and a cost value in a new table. This takes some time - in my case I had 24,000 nodes intersecting with 24,000 polygons in about 3 minutes.

CREATE TABLE edn_sdd_voronoi AS
  SELECT row_number() OVER () AS gid, n.node, n.agg_cost, v.the_geom
  FROM vw_school_driving_distance_nodes n, vw_sdd_voronoi v
  WHERE ST_Intersects(n.geometry, v.the_geom);

CREATE INDEX edn_sdd_voronoi_geometry_ixs
  ON corporate.edn_sdd_voronoi
  USING gist(the_geom);

I also created a set of Voronoi polygons using the QGIS > Geometry Tools > Voronoi Polygons and that took several hours to create a new shapefile layer. Using PostGIS you can be done in less than 5 minutes depending on how many points you have. As always this could probably be done more efficiently but 4 hours to 5 minutes is good enough for me.

Style the data in QGIS using a graduated renderer and the aggregated cost column and you get something like this.

If you have any suggestions for improvements, get in touch.

I have a dataset of pupils and the schools they attend. For each pupil I have a source XY coordinate and a target XY coordinate. I have a table of schools where the school name matches the name in the pupil table. I also have a full featured road network complete with road routing information and turn restrictions. All this sits in a PostgreSQL database with PostGIS and pgRouting installed. I need to use all this to calculate the shortest and quickest routes to school for each pupil. The results will then be used to inform some decision-making process about school transport and school location.

Two different solutions will be applied - routing from nearest node to nearest node and routing from nearest edge position to nearest edge position.

First, the pupil location needs to be associated with the nearest node on the road network.

-- Find the nearest network node for each pupil
CREATE TEMP TABLE temp_pupil_nn AS 
SELECT DISTINCT ON (a.pupilid) a.pupilid, b.id AS nearest_node
FROM edn_pupilschoolroute a
  LEFT JOIN hw_roadlink_vertices_pgr b ON ST_DWithin(a.geometry, b.the_geom, 1200)
ORDER BY a.pupilid, ST_Distance(a.geometry, b.the_geom);

This query creates a temporary table with a row for each pupil and the nearest node ID within a specified distance. You can find all the pupils that don't have a node within the specified distance. Adjust the distance in the ST_DWithin function until every pupil has a node assigned.

-- Find pupils without a node
SELECT a.pupilid
FROM edn_pupilschoolroute a
  LEFT JOIN hw_roadlink_vertices_pgr b ON ST_DWithin(a.geometry, b.the_geom, 1200)
WHERE b.id IS NULL;

Once the temporary table is populated update the pupil table with the node id. You could probably do it in a wunner but I like separate steps.

-- Update pupil data table with NN information
UPDATE edn_pupilschoolroute
  SET pupil_nn = 
  (SELECT nearest_node 
  FROM temp_pupil_nn b
  WHERE b.pupilid = edn_pupilschoolroute.pupilid);

Repeat the steps above for each school.

Using this approach works well where there is a dense road network and the road node is a suitable proxy for pupil location but most of my network is in rural areas where nodes are fewer and further apart. This can lead to differences between the actual and modelled distances. To get around this we can route from the point on the nearest edge closest to the pupil to the point on the nearest edge closest to the school.

So, find the closest edge and position of the point on the edge for each pupil by creating a view.

-- Find the nearest network edge to each pupil
CREATE OR REPLACE VIEW view_source_edges AS
SELECT DISTINCT ON (a.pupilid) a.pupilid, b.ogc_fid AS nearest_edge, ST_Distance(a.geometry, b.centrelinegeometry) AS distance
FROM edn_pupilschoolroute a, hw_roadlink b
  WHERE ST_DWithin(a.geometry, b.centrelinegeometry, 1200)
ORDER BY a.pupilid, ST_Distance(a.geometry, b.centrelinegeometry);

Do the same for schools with a view of the target edges.

-- Find the nearest network edge to each school
CREATE OR REPLACE VIEW view_target_edges AS
SELECT DISTINCT ON (a.schoolname) a.schoolname, b.ogc_fid AS nearest_edge, ST_Distance(a.geometry, b.centrelinegeometry) AS distance
FROM view_edn_dataschool a, hw_roadlink b
  WHERE ST_DWithin(a.geometry, b.centrelinegeometry, 1200)
ORDER BY a.schoolname, ST_Distance(a.geometry, b.centrelinegeometry);

Make sure the pupil table has the required fields for the edge calculations

ALTER TABLE edn_pupilschoolroute
  ADD COLUMN source_eid integer,  --source edge id
  ADD COLUMN source_pos double precision,  --source edge position
  ADD COLUMN target_eid integer,  --target edge id
  ADD COLUMN target_pos double precision,  --target edge position
  ADD COLUMN p_ndist double precision,  --node to node distance
  ADD COLUMN p_ntime double precision,  --node to node time
  ADD COLUMN p_edist double precision,  --edge to edge distance
  ADD COLUMN p_etime double precision;  --edge to edge time

The last four columns added are those that are populated by the function. Update the first four columns with the edge ids and positions.

UPDATE edn_pupilschoolroute a
  SET source_eid = b.source_eid,
      source_pos = b.source_pos,
      target_eid = c.target_eid,
      target_pos = c.target_pos
  FROM view_source_edges b, view_target_edges c
  WHERE a.pupilid = b.pupilid
  AND a.schoolname = c.schoolname;

The function to process all this information into routes goes like this:

	-- Function: corporate.pupil_journeys_routes()

	-- DROP FUNCTION corporate.pupil_journeys_routes();

	CREATE OR REPLACE FUNCTION corporate.pupil_journeys_routes()
	  RETURNS character varying AS
	$BODY$
	DECLARE

	  cur_pupil refcursor;
	  v_pid integer;
	  v_seid integer;
	  v_spos float8;
	  v_teid integer;
	  v_tpos float8;
	  v_geom geometry;
	  v_sql varchar(1000);

	BEGIN
	  RAISE NOTICE 'Processing pupil journeys...';
	  OPEN cur_pupil FOR EXECUTE format('SELECT pupilid, source_eid, source_pos, target_eid, target_pos FROM edn_pupilschoolroute WHERE pupil_nn IS NOT NULL');
	  LOOP
	  FETCH cur_pupil INTO v_pid, v_seid, v_spos, v_teid, v_tpos;
	  EXIT WHEN NOT FOUND;
	  SELECT ST_Collect(r.centrelinegeometry) AS geometry FROM pgr_trsp('
		SELECT ogc_fid AS id,
		  source::integer,
		  target::integer,
		  cost_len::double precision AS cost,
		  rcost_len::double precision AS reverse_cost
		FROM hw_roadlink'::text,
		v_seid, --pupil source edge id
		v_spos, --source edge position
			v_teid, --school target edge id
			v_tpos, --target edge position
		true,
		true,
		'select to_cost, teid as target_id, feid||coalesce('',''||via,'''') as via_path from hw_nt_restrictions'::text) AS d
	  INNER JOIN hw_roadlink r ON d.id2 = r.ogc_fid INTO v_geom;
	  
	  -- insert route cost into pupil data table
	  EXECUTE format('INSERT INTO %s(pupilid,geometry) VALUES ($1,$2)','corporate.edn_pupilroutes')
		USING v_pid, v_geom;
		
	  END LOOP;
	  RETURN 'Oooh, get in!';
	  CLOSE cur_pupil;
	END;
	$BODY$
	  LANGUAGE plpgsql VOLATILE
	  COST 100;

For 16,000 pupils and 57 schools this took about 2 hours to return a result.

A function to batch process driving distance results for a set of input points, in this case, schools.

The function creates a table to hold the results (dropping it if it exists) and then selects each school from the your_schools table and calculates the driving distance numbers. For each school it writes the results to a temp table (temp_dd, dropping it first if it exists) and writes those results to the output table (school_driving_distance). In the end you have a table with a lot of rows (more or less depending on your cost) with which you can do neat things with QGIS Time Manager.

The schools data needs to be pre-processed to give each school an id field, school_nn, which contains the node id of the nearest network node.

Note: You'll need to VACUUM ANALYZE your output table to update the statistics.

-- Function: public.school_driving_distance()
-- DROP FUNCTION public.school_driving_distance();

CREATE OR REPLACE FUNCTION public.school_driving_distance()
  RETURNS character varying AS
$BODY$
DECLARE

  cur_school refcursor;
  v_snn integer;  --variable for the school nearest node
  v_node integer;
  v_seq integer;  --variable for the sequence id
  v_cost double precision;  --variable for the cost results
  v_geom geometry;  --variable for the output geometry
  v_sql varchar(1000);  --variable to hold some SQL
  
BEGIN
-- Drop the results table first
	RAISE NOTICE 'Dropping table...';
	v_sql:='DROP TABLE IF EXISTS public.school_driving_distance;';
	EXECUTE v_sql;
	
-- Create a new results table next
	RAISE NOTICE 'Creating table...';
	v_sql:='CREATE TABLE IF NOT EXISTS public.school_driving_distance 
	(
	  gid serial NOT NULL,
	  seq integer,
      node integer,
	  school_nn integer,
	  agg_cost double precision, 
	  geometry geometry(Point,27700), 
	  CONSTRAINT school_dd_ipk PRIMARY KEY (gid)
	);

    CREATE INDEX school_driving_distance_geometry_ixs
    ON public.school_driving_distance
    USING gist(geometry);';

	EXECUTE v_sql;
	
-- Process the schools    
	RAISE NOTICE 'Processing school...';
	OPEN cur_school FOR EXECUTE format('SELECT school_nn FROM your_schools WHERE school_nn IS NOT NULL');
	LOOP

-- Drop the temp table
	v_sql:='DROP TABLE IF EXISTS temp_dd;';
	EXECUTE v_sql;

-- Create a temp table to hold results for each school
	v_sql:='CREATE TEMPORARY TABLE temp_dd(
	  seq integer,
      node integer,
	  school_nn integer,
	  agg_cost double precision, 
	  geometry geometry(Point,27700)
	  );';
	EXECUTE v_sql;
	
	FETCH cur_school INTO v_snn;
	EXIT WHEN NOT FOUND;
	
-- Do 30 minute driving distance calculation for each school
	INSERT INTO temp_dd(seq, node, school_nn, agg_cost, geometry)
	SELECT foo.seq AS v_seq, foo.node AS v_node, v_snn AS school_nn, foo.agg_cost AS v_cost, n.the_geom AS v_geom FROM pgr_drivingDistance(
	  'SELECT ogc_fid AS id, source, target, cost, reverse_cost 
	  FROM your_network',
	  v_snn,          -- school name
	  1800) AS foo    -- change this cost to change the output
	  LEFT JOIN your_network_vertices_pgr n ON n.id = foo.node;
	  
-- Insert results into driving distance table
	EXECUTE format('INSERT INTO public.school_driving_distance(seq,node,school_nn,agg_cost,geometry) SELECT * FROM temp_dd;');
	END LOOP;
	RETURN 'Completed the driving distance calculations';
	CLOSE cur_school;
	
END;
$BODY$
  LANGUAGE plpgsql VOLATILE
  COST 100;
COMMENT ON FUNCTION public.school_driving_distance() IS '30 minute driving distance function for each school';

Once all the results are written to the output table you can then query the table to get the overall minimum distance to each school:

SELECT DISTINCT ON (node) *
FROM school_driving_distance
ORDER BY node, agg_cost ASC;

Here's a result from the function above processed with QGIS Time Manager and stitched together with Avidemux and VLC.

And another one.

See this stackexchange post for another way of doing it.

Checklist

• QGIS with the Time Manager plugin installed.
• a PostgreSQL database with PostGIS and pgRouting extensions installed.
• access to said database through psql or pgAdmin3/4

A quick and dirty workflow to create some animations using a road network and some driving distance calculations. I won't go through the process of buiding your routing topology - there are examples here and here to get you up and running with open data. You should have two tables in the database - your_network and your_network_vertices_pgr - once you've completed the load and build process.

pgRouting comes with a number of built in routing functions one of which is the Driving Distance function. This function extracts all the nodes (and edges) from the network that are within a specified distance of a specified start point.

For the source features you can either select a node on the network that is closest to your area of interest and use the node id in your query or you can use an existing dataset and assign each feature a node id from the nearest node on the network. The query below creates a view of the point data in the school table (adjust as required) with the nearest network node id. You can then just pick the school and the node id you want to use.

CREATE OR REPLACE VIEW view_school_nodes AS
SELECT DISTINCT ON (a.schoolname) a.schoolname, b.id AS node, ST_Distance(a.geometry, b.the_geom) AS distance
FROM schools a, your_network_vertices_pgr b
  WHERE ST_DWithin(a.geometry, b.the_geom, 120)
ORDER BY a.schoolname, ST_Distance(a.geometry, b.the_geom);

The results of the nearest node query:

schoolname ; node ; distance
Aberlemno Primary School ; 58810 ; 27.8567765543682
Airlie Primary School ; 60834 ; 80.8949936646267
Andover Primary School ; 49398 ; 45.9162068337352
Arbroath Academy ; 55679 ; 42.0475920832573
Arbroath High School ; 56431 ; 26.1166270792929

My network is set up with cost fields - length and time - and I can use these to model journeys across the network. In the example below I am using time (in seconds) to show me how much of the network I can cover in 450 seconds (7 minutes 30 seconds). This number can be set to what ever you like.

SELECT seq, node, edge, cost, agg_cost FROM pgr_drivingDistance(
'SELECT ogc_fid AS id, source, target, cost, reverse_cost 
  FROM your_network',
  58810,  -- this is the start node
  450     -- this is the maximum cost (seconds in this case)
  ); 

The results are presented below.

seq  ;  node  ;  edge  ;  cost  ;  agg_cost
1 ; 58874 ; 67982 ; 4.68003566289286 ; 46.337640876984
2 ; 58727 ; 67931 ; 7.45750698693065 ; 7.45750698693065
3 ; 58728 ; 67932 ; 1.86503491582722 ; 9.32254190275786
4 ; 58793 ; 67946 ; 4.79924189436584 ; 23.1936531505147
5 ; 58794 ; 67939 ; 8.04220277283646 ; 18.3944112561489
6 ; 58808 ; 67928 ; 5.65869396139883 ; 14.9812358641567
7 ; 58809 ; 67930 ; 3.89677598786481 ; 13.2193178906227
8 ; 58810 ; -1 ; 0 ; 0
9 ; 58811 ; 67936 ; 0.800410785992549 ; 10.3522084833124
10 ; 58812 ; 67945 ; 2.09429071038919 ; 9.55179769731984
11 ; 58821 ; 67947 ; 24.9065391490445 ; 43.3009504051934
12 ; 58864 ; 67985 ; 28.4382873234685 ; 41.6576052140912

The field we are interested in is the agg_cost which can be sorted smallest to largest and we'll use this in QGIS to render them in order. But before we get there we need some geometry to work with. We need to join the results of the driving distance calculation to the network geometry. Then we'll create a view to load the data into QGIS. These views can be created using pgAdmin or through the DB Manager in QGIS.

To get the edge or line geometry do this:

SELECT seq, node, edge, cost, agg_cost, l.geometry FROM pgr_drivingDistance(
'SELECT ogc_fid AS id, source, target, cost, reverse_cost 
  FROM your_network',
  58810,  -- this is the start node
  450     -- this is the maximum cost (seconds in this case)
  ) AS foo
  LEFT JOIN your_network l ON l.ogc_fid = foo.edge; 

To get the node geometry do this:

SELECT seq, node, edge, cost, agg_cost, l.the_geom FROM pgr_drivingDistance(
'SELECT ogc_fid AS id, source, target, cost, reverse_cost 
  FROM your_network',
  58810,  -- this is the start node
  450     -- this is the maximum cost (seconds in this case)
  ) AS foo
  LEFT JOIN your_network_vertices_pgr l ON l.id = foo.node; 

Turn these results into a view by adding the create view statement (make up your own name):

CREATE OR REPLACE VIEW view_dd_lines_58810_450 AS
SELECT seq, node, edge, cost, agg_cost, l.geometry FROM pgr_drivingDistance(
'SELECT ogc_fid AS id, source, target, cost, reverse_cost 
  FROM your_network',
  58810,  -- this is the start node
  450     -- this is the maximum cost (seconds in this case)
  ) AS foo
  LEFT JOIN your_network l ON l.ogc_fid = foo.edge; 

To get the node geometry do this:

CREATE OR REPLACE VIEW view_dd_nodes_58810_450 AS
SELECT seq, node, edge, cost, agg_cost, l.the_geom FROM pgr_drivingDistance(
'SELECT ogc_fid AS id, source, target, cost, reverse_cost 
  FROM your_network',
  58810,  -- this is the start node
  450     -- this is the maximum cost (seconds in this case)
  ) AS foo
  LEFT JOIN your_network_vertices_pgr l ON l.id = foo.node; 

OK, we've got some data with aggregated costs and geometry. Add the views to QGIS and let's style it up. We can style the points and lines using a colour ramp and the aggregated cost field so that colours change with distance. We'll use data-defined colour expressions for this. Open the layer properties and switch to the Style tab. Select the simple line or point symbol and click the data-defined override button to edit.

Copy this expression in:

ramp_color( 'PiYG',  scale_linear( "agg_cost", 0, 450, 0, 1))

This uses the PiGn colour ramp and scales the colour in a linear way based on the values in our data (min: 0, max: 450) to values between 0 and 1. So as costs increase colours will change from purple through white to green. Make sure both the point and line layers are styled like this.

Note: you can use any colour ramp name and even a custom one you designed yourself. Check the Style Manager under Settings to get the list of available colour ramps.

I set the project background colour to black for that classy touch. You can also set the feature blend in the point layer to multiply. Or not - see how you're feeling.

Right, fire up QGIS Time Manager plugin. Hear that? The sound of a quality QGIS plugin loading... Turn the plugin on using the on/off button and click Settings.

Add the point layer. Choose it from the drop down. Choose the cost field as your start time. Set the end time to No end time - accumulate features. Click OK. Repeat for the line layer. In the animation options I set the Show Frame For option to 40 ms for a frame rate of 25. Click OK.

Back in the Time Manager window adjust the time frame size to the value of [max cost / 25]. This will ensure your animation plays out in a second at 25fps. Again adjust as required. Click the play button and see how the animation runs.

Before we export the video, adjust the canvas size to the size of the output video. You can go big by hiding all the panels or smaller by expanding the panels to get the desired dimensions. The images that are exported are the same size as the canvas. Zoom to the extent of you layer to fill the screen. You can rotate the canvas too.

To export the animation click the Export Video button. On Windows the animation is exported as a PNG image sequence. On Mac OSX and Linux you can export directly to video. Choose an output directory and click OK. The Time Manager starts exporting your animation one frame at a time. You can watch the files appearing in the directory or go and get a biscuit. When you get back you will have a directory full of PNG and PNGW (world) files. Delete the PNGW files.

I use Avidemux to create my video output (Windows Movie Maker works well too) and it needs JPG files. Convert the PNG files to JPG. I use the batch converter in Irfanview for that. Once you have some JPG files fire up Avidemux and open the first image in the sequence.

Important: make sure you don't change the file name otherwise it work work. It needs the Name### format.

Avidemux automatically recognises the image sequence and creates the video. Save it to AVI format. This makes for a large file as it is basically all the JPG images bundled into a container and then told to display in order. If you want to upload your animations to the Twitter then you need to convert to MP4. I convert the AVI to OGG or MP4 with VLC.

And a couple of Blue Peter outputs...

You might want to use the TauDEM Toolbox in QGIS. The QGIS docs give you all the detail on the available algorithms but you need to install and configure the toolbox yourself. You might get error messages about mpiexec not being being found on your system.

QGIS installs most of the required TauDEM dependencies and so you only need to install the Microsoft MPI executable (on Windows) or Open MPI (on Linux / Mac OSX). Get the latest MPI installer (v8 at time of writing) here. Install in the default location.

Register and download the TauDEM archive from here and then extract to some location - e.g. C:\Program Files\TauDEM537exeWin64

Switch to QGIS and open the Processing toolbox option in QGIS (Ctrl + Alt + C)

• Expand the Providers section
• Expand the TauDEM (hydrologic analysis) section
• Check the Activate box - x
• Add the MPICH2/OpenMPI bin directory - C:\Program Files\Microsoft MPI\Bin
• Add the TauDEM command line tool folder - C:\Program Files\TauDEM537exeWin64
• Add the TauDEM multifile command line tool folder - C:\Program Files\TauDEM537exeWin64
• Click OK

Run a TauDEM algorithm.

Another attempt at Blender landscapes using the Ordnance Survey Terrain50 DTM and Meridian2 datasets.

The Meridian2 features are draped over the DTM which is vertically exaggerated by 150%.

Water features and a sea layer are used to create an image texture. This is used in a UVmap to create the glossy looking sea and rivers in the final render.

Some work needs to be done on the image texture resolution and some smoothing of the DTM needs to happen to get rid of the contouring effect. But I still like them.

Inspired by an image created by Owen Powell I thought I would have a look at Blender again and see what I could get it to do with elevation and other mapping datasets.

Here are some first attempts getting to grips with lighting and UV maps and materials.

Auchmithie sunrise (above)

Reekie Linn - largest fall (24m) in the east of Scotland (above)

Newtyle path network (above)

Line-of-sight study (above)

These guys do amazing things creating maps - John M. Nelson, Stephen "Mapsmith", Craig Taylor, Daniel Huffman - and their maps, visualisations and images are inspiring. A tweet by Terry Stigers pointed me at the work of Leland Brown and his texture shading algorithms.

I do like a nice shaded relief map. Wouldn't it be nice to be able to use those in your desktop GIS of choice and make your hill-shaded backdrops even nicer?

Well, you can. The source code and some pre-built binaries for Windows and Mac OS X are available. As I don't have a Windows or Mac OS X machine and use Linux (Mint) I need the source code. Head over to the texture shading repository on box.com and look in the Software folder. Download the Source_Code folder in its entirety and save the zip file somewhere. Extract it. Read the User Guide.

The user guide gives clear instructions on how to compile the code into something usable. To compile the source code into the texture and texture_image executables you need to have a GCC compiler installed on your Linux box. Install it if you don't:

sudo aptitude install build-essential

This will install the necessary bits - gcc, g++ and make.

I found the compiler commands in the User Guide failed with "undefined references to sin/cos/pow/sqrt" and other errors.

Try the commands in the User Guide - they may work for you. I had to append a -lm to each command. This worked on my Linux Mint. Run the following commands in the directory where you extracted the source code

gcc -O2 -funroll-loops -DNOMAIN -c *.c -lm
gcc –O2 –funroll-loops *.o texture.c -o texture -lm
gcc –O2 –funroll-loops *.o texture_image.c -o texture_image -lm

The first command creates a bunch of files with .o at the end. The second and third commands use these new files to build the texture and texture_image files that we'll use to generate our textured shading. Once the commands have completed successfully you'll have two new files: texture and texture_image. Run each command without any arguments to print out the basic help information for each one.

$ ./texture
Terrain texture shading program - version 1.3.1, built Oct  6 2016

USAGE:    texture detail elev_file texture_file [-options ...]
Examples: texture 0.5 rainier_elev.flt rainier_tex.flt
      texture 1/2 rainier_elev rainier_tex
      texture 2/3 rainier_elev rainier_tex -mercator -32.5 45

Normal range for detail is 0.0 to 2.0.
Typical values are 1/2 and 2/3.
(Either decimal or fraction is accepted.)

Requires both .flt and .hdr files as input  (e.g., rainier_elev.flt and rainier_elev.hdr).
Writes   both .flt and .hdr files as output (e.g., rainier_tex.flt  and rainier_tex.hdr).
Also reads & writes optional .prj file if present (e.g., elev.prj to tex.prj).
Input and output filenames must not be the same.
NOTE: Output files will be overwritten if they already exist.

Available option:
-mercator lat1 lat2    input is in normal Mercator projection (not UTM)
Values lat1 and lat2 must be in decimal degrees.

Right, you've got some working commands now. Give it a go. Here are my steps in quickfire mode: I downloaded some Terrain 50, a 50m DEM available as open data from Ordnance Survey. I picked the NN tile as it has Ben Nevis and some other fantastic mountains and used QGIS to built a VRT from all the ASCII grid files*.

I created a hill-shaded image in QGIS using the NN tile.

From QGIS I exported this as both a RAW and a RENDERED GeoTiff (EPSG:27700). I then used gdal_translate to convert to GridFloat format:

gdal_translate -of EHdr -ot Float32 nn_t50.tif nn_t50.flt

I found the texture commands didn't like the files created from the QGIS rendered GeoTiff as they were multi-band so use the RAW GeoTiff.

Using the ./texture command I created the first texture shaded image as a .flt file. Then using the ./texture_image command I exported the results as a texture shaded GeoTiff.

Load the texture shade into QGIS, add your hill-shade and coloured DEM. Play around with blend modes and transparency until you get something that works for you.

Hill shade and texture shaded multiply blended

Add some tinting by blending in the 50% transparent DEM

No transparency just multiply blended

Did I mention reading the User Guide? It has a heap of tips on how to get the best out of your DEM. I created a matrix of detail parameters and contrast levels to see what effect they had. Another post maybe.

texture detail | texture contrast
0.25 | -2 -1 0 1 2 3 4 5
0.33 | -2 -1 0 1 2 3 4 5
0.50 | -2 -1 0 1 2 3 4 5
0.66 | -2 -1 0 1 2 3 4 5
0.75 | -2 -1 0 1 2 3 4 5

* Creating a VRT from the ASCII grid files and converting to GeoTiff introduces grid artifacts that can be seen in some of my images. See here and here for ways around it. I found that converting the ASCII grid files to GeoTiff with gdal_translate and then warping them into a single mosaic with gdalwarp (set "-r average" instead of default "near" and "--config GDAL_CACHEMAX 7000" to speed things up) worked. Hillshades created with SAGA, GRASS and QGIS had none of the stepping/terracing artifacts present. You might also get different results trying the default Horn and alternative ZevernbergenThorne algorithms.

After seeing the post by John Nelson on hex binning using ArcGIS I thought I better try it in QGIS as the effect is pretty neat.

Open the Processing toolbox and find the QGIS geoalgorithm > Vector Creation Tools > Create Grid

Select your grid type: hexagon

I created grid using 300000,400000,700000,800000 and cell size 5000, 5000. Note the order of the coordinates for the extents! Output as hex_grid_1.shp

The new layer should be added to your QGIS project. Nice.

Save this hex grid layer as a new layer - hex_grid_2.shp Now you'll have two layers exactly the same.

QGIS creates a hex grid with flat tops and bottoms and pointy sides. I'll rotate this later on to get the pointy bits top and bottom.

Add the QGIS Affine Transformation plugin to QGIS if not already installed.

Select all features in the layer hex_grid_2. Start an edit session.

Open the Affine Transformation plugin, select the hex_grid_2 layer and choose to transform the whole layer.

In the transformation matrix set the parameters like this:

X' = 1x + 0y + -1443.37567297
Y' = 0x + 1y + 2500.0

How did I get these numbers? Well, leave the four cells on the left at their default values of 1 and 0. We only want to change the two cells on the right to shift the layer in the XY plane. My hex grid had a cell size of 5000, 5000 and so my shift in the X direction needs to be half the distance between two parallel sides: 5000/2 = 2500. My shift in the Y direction is 1/4 of the distance between points on a hex cell. This shift of -1443 and 2500 will move the cells and align the vertices of the cells perfectly to create a pseudo 3D effect.

The attribute table of the hex grid layer has values for top, bottom, left and right for each cell. So, if you have a different cell size to mine, just divide the difference between top and bottom by two and the difference between left and right by four to get your parameter values.

Your layer will shift. Check it is in the correct place, save edits and stop editing.

Now we need to intersect these two layers to create a final layer of hex bins with internal partitions.

Do your point in polygon count, style 'em up and see what you get!

Rotating to get the pointy bits up

Now, if you want to rotate your hex grid layer so the pointy bits are top and bottom you need to fiddle with the Affine Transformation plugin and work out how to rotate it 90 degrees around the layer centre point.

After some searching I found the solution on hydrogeotools.

This is what I used:

x' = cos(θ)*x -sin(θ)*y + (x0 - cos(θ) * x0 + sin(θ) * y0)
y' = -sin(θ)*x + cos(θ)*y + (y0 - sin(θ) * x0 - cos(θ) * y0)

Caveat: Theta (θ) needs to be in radians and not degrees and that had me scratching my head for a bit. To rotate 90 degrees, convert to radians with pi * 90 / 180 and you'll get 1.570796. And x0 and y0 are the coordinates of the point you want to rotate around.

Thus, the parameters to be fed into the Affine plugin are:

x' = a, b, c = cos(θ), -sin(θ), (x0 - cos(θ) * x0 + sin(θ) * y0)
y' = d, e, f = -sin(θ), cos(θ), (y0 - sin(θ) * x0 - cos(θ) * y0)

which in my case resulted in

x' = 0, -1, 1100000
y' = 1, 0, 400000

Bung those in the transformer and the layer should rotate 90 degrees around 350000, 750000.

I also did it in PostGIS with the ST_Rotate function (a lot easier but then I did have a handy PostGIS DB nearby):

CREATE OR REPLACE VIEW hexgrid1_rotate AS
SELECT id, ST_Rotate(geometry, -pi()/2, 350000, 750000) FROM hexgrid1;

I loaded my shapefile layer into the database and created a rotated view of it. The SQL above takes the geometry, rotates it 90 degrees (converting to radians) clockwise around a point (350000,750000). I added this into QGIS, saved is as a shapefile, saved it again and went through the affining process (same as above but remember your X and Y shifts are now swapped because you've rotated your layer) to shift the layer into the correct position.

As ever, there may well be a better way of doing this (maybe I should learn some Python...) and thank you to the giants with big shoulders...

Other useful resources

• http://gis.stackexchange.com/questions/13433/moving-vectors-to-specified-coordinates-in-qgis
• http://postgis.net/docs/ST_Rotate.html
• http://gis.stackexchange.com/questions/21696/rotating-a-vector-layer-in-qgis-with-qgsaffine-or-other-method?rq=1
• http://gis.stackexchange.com/questions/30559/how-to-compute-parameters-for-qgis-affine-transformation?noredirect=1&lq=1

This is my talk on pgRouting from FOSS4G UK 2016.

My workshop notes and data are available here. The workshop was designed around using the OSGeo Live DVD so that everyone could work in a controlled environment. If you've got QGIS and PostgreSQL (with PostGIS and pgRouting) installed you should be able to follow along quite easily making the small changes to database connections where required.