data

Merging CSV files

Ross McDonald

May 27, 2016 • 1 min read

Let's say you want to merge a whole directory of CSV files into one large file to make data management easier. For example, the Ordnance Survey Code-Point Open dataset comes as multiple nested CSV files (121 of them) in one large zip archive. In addition to merging the files into one, you might also only want certain fields from the individual files like POSTCODE, EASTING and NORTHING.

Using the Code-Point Open example the following command line will, on Windows, do what you want it to do:

for /f %a in ('dir /b *.csv') do for /f "tokens=1,11,12 delims=," %b in ('dir /b *.csv') do echo %b %c %d >> alluk.csv

To translate this into English: For all the CSV files in the directory select the records in the first (postcode), eleventh (easting) and twelfth (northing) columns from each file and merge them into one file called alluk.csv.

This file can then be loaded into QGIS and displayed very quickly.

OS Code-Point Open

Sign up for more like this.