Introduction

The purpose of this webpage is to document my testing of Automatic Orienteering Map generation tools for map material typically available in Norway.

Summary: Make your own Pullautin map (Norway edition)

Update! New version of Pullautin is now available with my suggested changes (and probably also some improvements) included. That means FKB-data should render fine with road borders and everything when you use the new Pullautin-version. Thanks, Jarkko!

In this summary I describe the use of Pullautin to make an orienteering map from LIDAR + FKB data for Norway. For details about importing to OCAD and all of my experiences (including many failures and challenges), see below the summary. Note that using vector data and using other tools for e.g. the vegetation part may give a better final result, as also described below the summary.

For simplicity I assume that your working directory is C:\data, but you may of course put the files anywhere you want.

Download files & data

  1. Make an agreement with the municipality in order to get LIDAR and FKB data for your area of interest (more info about how to get the data you need here). To follow this HowTo, you want LIDAR data in las or laz-format and FKB/AR5-data in Shape-format (If you got the FKB-data in SOSI-format, you have to work more).
  2. Zip all the shape-files you get into a zip-file "fkb-shape.zip", and put this in the folder "C:\data" (note that you can speed up the later process by not including all shape-files in the zip-file).
  3. Put all the las/laz-files you got from the municipality in a subfolder "C:\data\tilesorg".
  4. Download Karttapullautin from here, and unpack the zip-file to your work directory.
  5. Download an old version of lastools from here,unpack it wherever you want, and copy las2txt.exe, lasview.exe and lastile.txt from the bin-folder to C:\data. (Note! The newest version of lastools has a las2txt which does not work correctly with Pullautin. If there is problems with the download, search for another version of lastools here).
  6. Download the file tools.zip and unpack into "C:\data\". This file contains fkb.txt and a few bat-files which simplifies the process for you.

Retile data (optional)

  1. Run EITHER "retile1000.bat" or "retile400.bat" (from tools.zip). This takes the original las/laz-data you got, and splits it into tiles of either 1km x 1km (retile1000.bat) or 400 x 400 meters (retile400.bat) into the directory "tiles". I have been using 400x400 for my tests, but both should work.
  2. Note: If you do not want to retile the data, you need to rename your directory "tilesorg" to "tiles".

Make a Karttapullautin map for a single tile

It is strongly recommended to test with a single tile before you make the complete map.
  1. Copy one of your tiles from the "c:\data\tiles" folder to "C:\data" and rename it to "testtile.laz"
  2. Do some changes in pullauta.ini
    • Change the line "vectorconf=" to "vectorconf=fkb.txt"
    • Set the correct angle for the north lines in the line "northlinesangle=7" (find the correct angle here)
    • Consider changing "thinfactor = 1" to e.g. "thinfactor = 0.33" if you have very good LIDAR data - this will speed up the process later on
    • (you can also download my pullauta.ini file and it in C:\data, but northline angle may be wrong. Remember to check that the file has name pullauta.ini and not pullauta.ini.txt!)
  3. Run the batch-file "run-single.bat" from "tools.zip" (actually runs "pullauta.exe testtile.laz"). You will now get an image "pullautus.png" which hopefully looks OK. If you were unlucky with your choice of tile, you might not have much sensible there. In that case, choose another tile.
  4. Run the batch-file "shape_single.bat" from "tools.zip" (actually runs "pullauta.exe fkb-shape.zip"). You will now get a new image "pullautus.png" where the FKB-data should also be on. Again, choose another tile if this one is not representative for what you want to test.
Examples for my test tile (I see now that I used the wrong northline angle in this test - at this point you should try to get rid of all such problems before continuing with the batch job for the complete map):
After run-single.bat

After run-shape.bat

Note that in the current official Pullautin-version, you will not get road/lake boundaries. I hope this will be fixed by Jarkko at some time in the not so distant future (read more about this problem here).

Batch job for complete map

When you are happy with your single tile (and maybe even tried a few to make sure everything works as it should), you are ready to process all tiles. This is very easy,
  1. Make sure you have the following set in your pullauta.ini file (in addition to what you did above):
    • batch=1
    • lazfolder=./tiles
    • savetempfiles=1
    • processes=1 (set this to a higher number if you have several processors on your computer; I typically used 3)
    • (you can also download my pullauta.ini file and it in C:\data - I actually use the same file when running in batch-mode and in single tile mode.
  2. Start "pullauta.exe"
  3. Wait some hours or days - depending on the size of your map. Make sure you have enough disk space (As savetempfiles=1 you will save all the dxf-files which can later be imported in OCAD. Set savetempfiles=0 if you are sure you do not want to work with vector files in OCAD)
When you have finished this process, your folder "C:\data\out\" should be full of processed files. You can look at them one at a time until you merge them together (see below).

Merge the tiles together

The final step is to merge the tiles together to a single map. This is also easy, Here is my complete map (click for full 600 dpi output 5 Mb file from Pullautin, without any cropping - warning slow download):

Note that this map is made with a modified version of Pullautin which handles the road/lake boundaries. With the current version of Pullautin (as of September 7th 2014), you would not get road/lake boundaries.

Viewing your result in 3D

I did some work on rendering this in 3d - still some work to be done there, but following the instructions you should be able to do the same with your map tile.

Background

For test purposes, I got LIDAR + FKB-data (buildings, roads, rivers, lakes etc.) for an area north of Bergen in Norway. The LIDAR data is on LAS-format and the FKB data is on a Norway specific SOSI format. The plan is to use this data as a test case for auto-generating orienteering maps for training use with Karttapullautin - there are many fantastic unmapped areas for orienteering in Norway (and even close to Bergen). I also want to see how easy it is to use Open Street Map data for cases where FKB data is not available. And I want to test Terje Mathisen's workflow for making orienteering maps from LIDAR/FKB data. His approach is a bit different than Karttapullautin as the goal is not to make a finished map out of the box.

I have previously done a little bit of testing with LIDAR data and Karttapullautin from another area in Norway (Sotra / Algrøy) with quite good results. This was only LIDAR data though (no FKB data), and I only used standard settings in Karttapullautin. Now I want to get one step further.

Get started

If you want to try to repeat my work, you should start by downloading Karttapullautin. You also need to download lastools and copy las2txt.exe from the bin-folder to your Karttapullautin-folder.

Here are two tiles from my dataset to test with:

Initial testing

First tests were done with Karttapullautin and the same method as I used for Algrøy/Sotra some months ago - just after receiving the data. This resulted in an "out of memory problem" when trying to generate the map in OCAD-format due to the LAS file being too large. I managed to make a JPG of the complete area without memory problems though (see below).

New start August 23rd 2014: Research on useful resources

Now I am restarting from scratch again after getting a tip from Jarkko (maker of Karttapullautin) on how to split the LAS-file into tiles using lastime (part of lastools). I also found some other useful resources:

Tiling LAS-file

I start experimenting with the lastile tool (download). I run the following command (sent to me by Jarkko) in the directory where I have all my LAS-files. This generates a new subdirectory "tiles" with new, smaller files of one square kilometer each: 
lastile -i *.las -olaz -o tiles/tile -tile_size 1000
Seems to work as expected - at least I get a new directory with several smaller laz-files (compressed las-files). To see if it really worked I used "lasview" from lastools and saw the following:

For Terje's tool I am supposed to use smaller tiles with overlap (see here). I do a test of this as well using the following command (250 meter width/height of tiles, 35 meter overlap): 

lastile -i *.las -tile_size 250 -buffer 35 -o tiles_raw\tile -olaz
Trying this instead gives me 236 small laz-files instead of 24 larger laz-files as i had earlier - and with overlap.

Now I've at least got two different sets oflas-files to play with, and none should be too large. I'll figure out later if I want to have overlap or not.

I then found a better way to check the location of the LAS-files, OL-Laser (I remembered doing it before, but I needed a new version of OL Laser with my new LAS-files).

Still did not want to show my new tiled laz-files, though. "Keep file and path names out of space-characters" is the message, but I've got none of those, so who knows what the problem is.

Karttapullautin

Now over to Pullautin. First I revisit my "pullauta.ini" file from last time.

I first do the simplest thing - process a single "tile" with default settings: 

pullauta.exe tile_289000_6716000.laz
Some minutes later I get a PNG-file like this as expected:

I also get a separate vegetation file and some DXF-files. I see that there are too many stones/cliffs here (more about that further down), so I have to do something with the settings. But at least it works as expected.

Next I want to let Karttapullautin crunch through the complete area (I know I should first get nice stone/cliff/vegetation settings, but I first want to check out how I can make a big orienteering map in one go). To do this I need to run Pullautin in batch-mode. I make a few changes to the "pullauta.ini" file (my file) compared to the original:

Now doubleclicking "pullauta.exe" makes Pullautin start crunching. 5(!) hours later the crunching is finished, and I've got it all there as expected. Now ready to import to OCAD - or at least that is what I thought. The problem now was that I had separate dxf-file and png-files for each tile. My first tests importing to OCAD went well for a tile at a time, but I struggled to get them all imported at once. After using half an hour searching the Internet for a way to merge DXF-files, I finally went back reading the instructions for Karttapullautin and found that there is a built-in tool for merging both DXF-files and PNG-files. Thus before proceeding with OCAD, the following commands had to be executed:

For the dxf-files: 

pullauta dxfmerge
For the png-files: 
pullauta pngmerge 1
pullauta pngmergedepr 1
pullauta pngmergevege 1
And here is a message from Jarkko about the merge-command: "Note, you easily run out of memory if you try merging together too large area with too high resolution.". In that case use e.g. "pullauta pngmerge 2" which reduces size to 50%. This of course makes the resolution worse, so there is a tradeoff.

Anyway, this gave me the following final map (reduced size here):

This could already be printed and used for orienteering, but of course we want the roads and stuff as well - and I'd rather want it all in OCAD format to be able to manipulate it.

In addition to a corresponding merged vegetation files and dxf-files. Next I tried Terje's tool before trying to put everything together in OCAD. For import to OCAD see further down.

Terje Mathisen's tool

In parallel I also chose to try out Terje Mathisen's automatic tool (includes download link). This is not to make a final map for use for training, but rather in order to make a tool for a mapper. Still it is interesting to see if it gives me something useful which I don't get from Karttapullautin.

After downloading Terje's zip-file from his page, you find some basic info in the file "lidar-batch.bat". What I had to do to get it to work was to make sure I had all lastools files in my path (I put them in C:\data\orientering as Terje suggested, and type set PATH=%PATH%;C:\data\orientering in my DOS-window). As I already have ActivePerl installed, I did not need to install Perl to use Terje's tools.

I then removed all pause-commands in the script (to let it run while going out for a training), and then I wrote the following command (my original LIDAR-files are in d:\download\Jan\lidar\): 

lidar-batch.bat d:\download\Jan\lidar\*.las
From this I got a contour DXF-file (terje_contour.dxf) and GIF-files for vegetation, cliff and slope. I hoped to also get DXF-files for vegetation and cliffs, but that somehow did not come out. I then removed some of the "REM" in front of lines in Terje's script and restarted the last part again. A restart did finally generate dxf-files for vegetation, but I got one dxf-file for each tile, 238 files. I found no way to merge them all into one dxf-file. I also tried to import into OCAD, but in OCAD10 I had to import them one-by-one... I finally tried to import in OpenOrienteeringMapper, but no luck there either. I also tried to merge the DXF-files with Karttapullautin, but no luck there either.

So for now I have contour DXF-files and vegetation and cliffs as separate GIF-files from Terje's tool. I hope to get more, but I'll have to e-mail Terje and ask how to get that. For now I have something to import in OCAD and compare to/use along with Karttapullautin data.

Update 1: Terje answered in an e-mail that (1) In OCAD11 you can import many DXF-files at the same time, so this was not an issue for him = no solution for me coming there. Might have to program merging of DXF-files myself? (2) There is no hope for Terje's side to get the cliffs in any vector format. So I'll have to use the ones from Pullautin I guess.

Update 2: I made a small hack in perl to merge the DXF-files (mergedxf.pl, place it in the folder where all his other tools are and doubld click when all else is finished). Works only for Terje's files I guess. Using his "veg2dxf.crt" CRT-file I can now import the vegetation into OCAD as vectordata. The small issue I faced was that white forest is mapped to "Sandy ground" 211. I had to play a bit with colors and new symbols to get it to look OK - not sure if this is the perfect way. Anyway, I get the vegetation as vectors now (but a large 37 Mb dxf-file for my dataset)

FKB-data

I got the FKB-data as a number of different SOSI-files ( 1m.sos, 1256_VANN.sos, 1256_TERRENG_5M.sos, 1256_TERRENG_1M.sos, 1256_LEDNINGELTELE.sos, 1256_VEG.sos, 1256_BYGNANLEGG.sos, 1256_BYGNING.sos, 1256_EIENDOM.sos, 1256_EIENDOM_T1.sos, 1256_TEKST1000.sos). These now have to be converted either to DXF-format or to SHP-format to be imported in OCAD and/or to be used in Karttapullatautin.

Note! In OCAD11 you can import SOSI-format directly, but I have not got access to OCAD11.

I have read about two different ways to handle the FKB-data on SOSI-format, and I wanted to try both:

  1. Terje's script sosi2dxf.pl (download via his webpage in the same zip-file as the other tools)
  2. SOSICON: SOSI to SHP converter

Sosi2dxf.pl

I started with sosi2dxf.pl, and this was actually very easy. I just wrote the following in a command window for each SOSI-file, 
sosi2dxf.pl 1256_VANN.sos 1256_VANN
This gives a dxf-file 1256_VANN.dxf and a description-file 1256_VANN.crt in the same directory. I found that these can be easily imported in OCAD. The water bodies looks like this:

The two issues I see,

  1. Based on my first trials with import to OCAD, it looks like roads are not area objects, only the boundarys of the road are included. I see the same issue with buildings. With water bodies however it looks like they are imported as an area, and thus I guess it is the original SOSI-files which are the issue and not the import.

    Update: (1) I took a second look at the SOSI-files, and roads are actually both area-objects for the center of the road + line objects for the sides of the road. Thus it could be possible to import them as area objects. Will have to try that later on. (2) I got an e-mail from Terje with new version of sosi2dxf which also handles AR5-data (which I have not got for any area yet, but which I hope to get for testing)

  2. I have to import the DXF-files one by one. It could have been nice to merge the ones which I need into one to do the import in one go. Based on a quick look at the file format this looks quite straightforward to fix. It is also easy to make a batch-process so that I don't have to write sosi2dxf for each file separately (might already be possible, I did not check). I also need DXF-file merging for the contours etc., so this is is a priority to look at. Update: Terje answered by e-mail: "Når det gjelder SOSI data så kan du kopiere sammen så mange .SOS filer du vil, og sende summen inn i sos2dxf.pl". I.e., it should be possible to process/import all in one go, sounds good for the workflow.

SOSICON

With the success with sosi2dxf.pl for direct import to OCAD it is probably not even worth testing SOSICON for this purpose, but I decided to still give it a quick try (especially since I probably need this if I want to get it into Karttapullautin). Downloading SOSICON gave me the error message "The program can't start because MSVCP120.dll is missing from your computer". Installing a Visual C++ Redistributable Packages for Visual Studio 2013 from here did the trick (Thanks, Google). Now I run the following command, 
sosicon -2shp 1256_VEG.sos
This gave me 92(!) files, one shp-file with accompanying prj/dbx/shx files for each object type in the SOSI-file. I tried importing one of these into OCAD, and while it works OK, I can not see any advantages by using this method compared to Terje's tool for OCAD import. So I'll put this on ice and use Terje's tool instead. But I'll come back to this later for testing with Karttapullautin.

Open Street Map data

For the other area I want to work on, I do unfortunately not have any FKB-data (yet). Therefore I wanted to check if it is possible to import Open Street Map data to at least have the roads in there (although FKB data clearly is a big advantage).

What I tried first was to export data from OSM via the export functionality in OSM-format (link) - following O-Zeugs advice - and then importing this to Open Orienteering Mapper. That did give me some lines in Open Orienteering Mapper, but I would probably need some kind of CRT-file or similar to translate it properly.

I then tried to follow instructions from this Attackpoint discussion - which Jagge also points to in the readme-file of Pullautin. The link pointed to there (http://www.osm974.re/osm2gis/) is however dead. There was also an alternative link where I tried to download data. I am still waiting for the download though. I did try to download in OpenStreetMap format there as well, so I might get the same problems in Open Orienteering Mapper.

The plan now is to install Gdal and try the ogr2ogr-command from the Attackpoint discussion in order to import. That will have to wait until finishing with the other stuff, though.

Getting it all together in OCAD

So now I had most of the bits and pieces, and had tried importing each part separately in OCAD just to see that it worked. Finally putting it all together into a map. Exciting!

August 24th: Temporary conclusion

For now this looks quite good. If I have LIDAR + FKB-data available, I can make maps which looks like they are good enough for training. I have to find out about the issue with the cliffs, though. Also the form lines from Karttapullatin is an item which I need to look into. I'd also want to be able to import the vegetation in vector-format from Terje's tool. Maybe it would even be possible to use Terje's tool to convert Pullautin's PNG-files with vegetation data to vector-format?

I also see that for the terrain we have here in Bergen, it would be very nice to also have marshes included, as most yellow areas are probably marshes. I think that marshes might be in "AR5-data" available from Statens Kartverk. I should try to get hold of some AR5 data for further testing.

Finally I need to look at Open Street Map data for the case where I do not have FKB data. And it would be nice to see if it is possible to get the FKB-data directly into Karttapullation - that might very well be possible (there is such functionality for Swedish/Finnish base data).

More testing with Open Street Map data

I started with downloading the FWTools windows binary which was linked to in the Attackpoing discussion (note, only the mirror download site worked).

Now to get started with the import, I first reread this Attackpoint message and accompanying discussion, which lead me to some more instructions here.

My (tiny) steps to try to get this to work:

After thinking a bit, it might be better to try to import DXF-files to OCAD. So I try to use ogr2ogr to generate DXF-files, 

ogr2ogr -skipfailures -f "DXF" lines.dxf osm_line.shp -t_srs EPSG:32632 
ogr2ogr -skipfailures -f "DXF" polygons.dxf osm_polygon.shp -t_srs EPSG:32632
This gives me a lot of "ERROR 1: DXF layer does not support arbitrary field creation" errors, and a few "ERROR 1: No known way to write feature with geometry 'None", but I got DXF-files for both lines and shapes which I can import in OCAD. The good thing with DXF-files is that I know how to use CRT-files for them.

The bad thing for now is that these DXF-files do not align with my other map data when importing. This is another indication that there is something wrong with the projection I use (i.e. the EPSG:32632).

So next up is taking another look at the projections.

FKB-data directly into Pullautin

With the OSM data import stopped for now due to the projection issue, I'll go back to trying to get FKB-data directly into Pullautin. First I repeat what I did with SOSICON yesterday, but for all the three relevant SOSI-files (roads, water bodies, buildings). I've now got nearly 200 files after running SOSICON in "pairs" of shp, shx, dbf and prj-files. To check the projection I import a few of them in OCAD and see that that works well.

Next I choose just the roads (1256_VEG_Veg_FLATE.shp) and pack the four files belonging to these in a zip-file (i.e. shx, dbf and prj-files with same name, the file is available here).

Now I have to make a new vector definition file. I see that for OSM-data lines are typically of type 

road-path|503|highway=motorway&bridge!=yes
parking|529|amenity=parking
farm|401|landuse=farm
For Swedish FASTIGHETSKARTAN you have got 
building|526|DETALJTYP=HUS
watercourse|306|DETALJTYP=VATTDR.M
watercourse|306|DETALJTYP=VTUB.M
Thus it looks like the first entry in each line is the description, the next the OCAD symbol number and the third the SQL-query to find the object in question. I thus first tried with the following, 
road|529|OBJTYPE=Veg
The reason for trying just this value was that importing the SHP-file in OCAD gave me the following:

I am on quite unsteady ground now, though - and probably need some help to get on. I know try to rerender in Pullautin with roads with the command: 

pullauta.exe vegflate.zip
My new "pullautus.png" is exactly the same as I had before unfortunately, without any roads on it. Possible errors can be my fbk.txt or the projection info in the shape file. Or that Pullautin does not like this kind of shape files.

Cliffs in O-Laser

After rereading what O-Zeugs writes, I decided to try to get better cliffs by using O-Laser.

The first problem occured when trying to open my laz-files. I get the error message "Keep file and path names out of space-characters" however I work with my laz-files. The only way I managed to open my 1 km x 1 km test tile was to decompress it (from laz to las) with laszip.exe (from lastools). Then I got a (very large) file which I managed to open in O-Laser. This will be enough for my test purposes at least.

So what I did:

Projections

Thought some more about the projections and tried EPSG:25832 in the ogr2ogr command instead as I then get something which is very similar to what I find in the shp-files I get from the SOSI-data. Still no luck when trying to import the resulting file in OCAD, though. I tried with Pullautin and OSM-data as well with this projection, but it still did not work.

August 25th: Temporary conclusion

So today's work did not give many results (so far). No luck in getting any vector format into Pullautin (neither FKB data or OSM data). And no luck in importingOSM data into OCAD. Time to ask around to get some help I guess - both these problems should be possible to solve...

And for the fun of it: A course on the test map.

August 25th 2014: E-mails from Jarkko and Terje

I sent e-mails to both Terje an Jarkko - getting me a lot of useful information. I also got hold of some AR5-data for the area. I have just briefly tested their tips, and am already many steps ahead. I'll just have to do it all step-by-step.

Here is the result of the quick testing. Starting to look good, I think (click to see larger)!

August 31st 2014: Back to systematic work

I have got a few steps later the last week based on help by e-mail from Jarkko (new version of Pullauta with some small changes), Terje (some tips + a new script earlier in the week) and the AR5-data (as both SOSI-format and Shape format). But busy days, so I did only quick tests without proper documenting of the steps. I'll now continue from where I was last weekend, documenting it step by step.

Fix of problem with "black dots" in Pullautin + cliff optimization

Based on feedback from Jarkko, I finally found out about the problem with all the "black dots" (you see them e.g. here). It turned out that I had triggered a "building mode" in Pullautin by uncommenting the line 
buildingsclass=6
in the Pullautin ini-file (not on purpose). With a # in front of this line, it was all OK! That helped a lot!
Before:

 After:

This is very good news. The black dots on one of the cliff-layers made it impossible for me to use this cliff layer in may mapping tests in OCAD, and thus results would not be realistic. Also, the output from Pullauta looked "horrible", and I had no fix. So thanks, Jarkko!

On the same note, Jarkko also said I should check the settings for cliff generation. The threshold should maybe be put up a bit in this type of terrain. This is of course not easy to know without either comparing with an existing orienteering map or going out to check in the forest, but at least I could do some testing with the settings.

The cliff settings in the pullauta.ini file are handled in the following lines: 

## cliff maker min height values for each cliff type. vertical drop per 1 meter horisontal distance
##  cliff1 = these cliffs will be erased if steepness is bigger than steepness value below
##  cliff2 = impassable cliff
cliff1 = 1.2
cliff2 = 2.5
Thus by increasing these values, the cliffs must be steeper/higher to be included in the final orienteering map. I just briefly did some tests on a small tile:
More cliffs (cliff1=1.0, cliff2=2.0)

 Standard settings (cliff1=1.2, cliff2=2.5):
Less cliffs (cliff1=1.5, cliff2=3)

 Even less cliffs( cliff1=1.75, cliff2=3.5)
I have got to experiment more with this setting when I have an actual old orienteering map to compare with - or when I have been in the terrain with a HeadCam. At least it is easy to do the change.

Open Street Map data

I got a tip from Jarkko about why I had trouble georeferencing the OSM map data (in both OCAD and Pullautin). It turned out that the problem was that I ticked the wrong coordinate system when downloading data from OpenStreetMap, so the data was in a different coordinate system than I thought. Note that the OSM-data is often not very accurate/useful, but for my initial tests where I had only LIDAR-data and no FKB-data, it would have been great to have the roads there to show people where the starting point was!

Step by step exporting from OSM (Note! You can only do this once per hour, make sure you choose the correct area the first time!):

This is the zip-file I got for my test area. I will use that in the following.

We now need to convert them to the coordinate system we have used for our map, which is "UTM zone 32N" (I guess all maps in Norway will be in this coordinate system, so the same command can be used for all?). Step by step:

OSM data into OCAD

Depending on which version of OCAD you have, there are different ways to do this,

OSM data into Pullautin

Next up is getting these data into Pullautin. This was actually not difficult now when we have the data in the right projection! Results also get better "out of the box" than I managed to do quickly in OCAD, as Pullautin has support for OSM-data. Steps are easy:

Vector data into Pullautin: FKB data, initial tests

Update: I now also got hold of FKB-data directly as Shape-files. Turns out that Pullautin takes these shape-files directly without any issues. That is, if you get the FKB-data as shape-files you can skip all this trouble!

Earlier I had big trouble importing the FKB-data into Pullautin, and it turned out that the problem was that the "fields" in the shape-file (which I had converted from SOSI) had some blanks in them, while Pullautin took away the blanks. Thus is was actually impossible to get Pullautin to work with these shape-files. I tried to modify both SOSI and shape-files, but Pullautin didn't really like the modified files. The solution: Jarkko made a new (temporary) version of Pullautin for me which fixed that issue. He might very well publish a new official version of Pullautin later on which includes this fix, at least I hope so. I've also some other suggested changes as you will see below.

With the new version of Pullauta.exe, I am back to where I was earlier on:

Conclusion so far: I have finally managed to get FKB-data into Pullautin, i.e. everything is fine with projection etc. The next step is to translate the shape-files into stuff which looks nice in Pullautin. Before going there, a look at AR5 data below and importing them into OCAD. Then I'll continue with Pullautin and trying to optimize the results in Pullautin.

AR5-data

The initial FKB-data I got did not contain the socalled AR5-data (see details here - a brief description:

"AR5 is a national (for Norway) land capability classification system and map dataset that describes land resources, with emphasis on capability for agriculture and natural plant production. The dataset is primarily intended for land use planning, public management, agriculture and forestry. The primary level of classification is surface type (arealtype) based on criteria for vegetation, cultivation and drainage. Second level attributes are forest site quality class (skogbonitet), forest cover type (treslag) and soil conditions (grunnforhold).")

These are actually some of the most useful FKB-data for orienteering maps as far as I can see, at least in areas where they are updated to the best standard (See here for type and availability of FKB-data for Norway; you can also see availability of LIDAR data via the same link; my test area has B-standard which is the second best standard. Only very limited areas in city centers seem to have A standard). See the types of FKB-data here.

The map symbols I found useful from AR5-data in my test case were

What I didn't find in AR5 which I took from other FKB-layers was: There might be other objects/layers I may have missed; I have only tested for a limited area.

The structure in the AR5-data is somewhat different from the other FKB-data (there are two fields which need to be taken into account in the database in the shape-files). Note also that the data are mostly area symbols, and thus Terje's script does not play that well with them (but he sent me a new script which are very good friends with the AR5 data). Except for that, there are no big differences.

AR5-data - import into OCAD

I got an alternative version of "sosi2dxf.pl" from Terje called "sosi2dxf4.pl" (he hasn't posted it on his website yet, but I guess he'll do if somebody asks him to). This can work with area data - except for that it works exactly like the old script. Note that you need both scripts to get the work done as of now, but if demand rises I am sure Terje might consider to put them both into one...

Step by step:

Now this is what I see in OCAD:

Starting to look good! As you can see, we have only got the inner part of the road/lake; missing road/lake boundaries. I had hoped that running the old "sosi2dxf.pl" would give me those boundaries, but that part was actually missing in "sosi2dxf.pl". Luckily this is written in a programming language which makes it easy to make changes, so I modified Terje's script to give me the boundaries. So with my new script (I hope Terje will accept my suggested changes), I do:

AR5-data + all the other stuff: Import into OCAD

Now I want to combine the AR5 data with the other data we already have there from earlier. The goal is to get a fully automatic process here in due time (which I think is feasible), for now I'll do some manual work while finding out what is the best approach and gives the best accuracy.

Continuing with the map I started above with contours + AR5-data, I do the following:

And here is the final result:

You can see that it is a bit tricky to see the difference between houses and cliffs with all these cliffs, so I tried to make the houses gray:

Also, there is the formline-problem in OCAD currently (which you can read about further down). So here is what it looks like completely without formlines. Later on I have to find a way to include only the required formlines.

Finally a comparison with everything rendered in Pullautin. I will get back to all the Pullautin-details - this is just because I had it ready now... Note that Pullautin has the "correct" formlines. Note also some differences with the yellow between Pullautin and my OCAD rendering - due to some differences in my import-setup. Anyway, both results look quite nice and usable for at least training - looking forward to get out and try some of these maps!

And here is the complete map - all tiles together:

See also complete map here with all formlines and without any formlines at all.




Vector data into Pullautin: FKB data, making it nice-looking

Update: I now also got hold of FKB-data directly as Shape-files. Turns out that Pullautin takes these shape-files directly without any issues. That is, if you get the FKB-data as shape-files you can skip all this trouble!

One of the main goals when starting this small project was to get the Norwegian FKB-data into Pullautin to automatically generate orienteering maps good enough for o-technical training and evaluation of areas for making a "real" orienteering map.

Above I showed how to import one of the FKB-layers into Pullautin (Buildings). Now I want to include all the data in the FKB-files, including the AR5-data. For this I needed to:

  1. Fix an issue with Norwegian special characters in the SHP-files
  2. Make some new symbols in the Pullautin source code to render thin enough lines
  3. Make a conversion table fkb.txt which converts the symbols in the FKB files to the forrect symbols in Pullautin

First problem: Norwegian characters æøå in filenames in shape files

The first problem I struggled with was that some of my shape-files were named with Norwegian special characters. Pullautin didn't like this, and I have found two different workarounds,



Second problem: Thin lines for road borders etc.

The second problem I had was that Pullautin did not render thin enough lines for road borders etc. Jarkko sent me a new version with thinner lines, but I was still not happy and implemented a symbol with even thinner lines.

For reference, here is the code I put in just before the code for settlement: 

  if ( $ISOMcode eq '529.1' || $ISOMcode eq '301.1' ) {
     $ok = 1;

     foreach $ehto (@ehdot) {
         ( $operator, $ehtoname, $ehtovalue ) =
           split( /\|/, $ehto, 3 );
         if ( $operator eq '=' ) {
             if ( trim( $dbfrec{$ehtoname} ) ne $ehtovalue )
             {
                 $ok = 0;
             }
         }
         else {
             if ( trim( $dbfrec{$ehtoname} ) eq $ehtovalue )
             {
                 $ok = 0;
             }
         }
     }
     if ( $ok == 1 ) {

         $imgblacktop->setThickness(2);
         $thickness = 2;
         $area      = 0;
         $vari      = $black;
         $border    = 0;
         $image     = 'black';
     }
 }
I made the OCAD code for this symbol 529.1 (road boundary) and 301.1 (water boundary). Alternatively I would go for Jarkko's new symbol 414 which was a thicker line. This is however also not in the official Pullautin version as of September 7th 2014.



Which SHP-files to include?

Following the instructions for import of FKB/AR5 data into OCAD above, content from the following SOSI-files are to be imported: 1256_BYGNING.sos, 1256_VEG.sos, 1256_VANN.sos, 1256_AR5.sos.

When converting these SOSI-files to SHP-files with SOSICON, I get more than 250 different files (to each shp-file there belongs a prj, shx and dbf file, so the number of shp-files is around 65), one for each layer. Not all these files are actually needed, but you will not loose anything (except that it takes more time and memory) by just zipping all together and giving them all to Pullautin. If you want to save some computation time (not much compared to the overall time), you could take only these files:

Note also that the shp-files I got are for a large area. Pullautin would work a lot faster if I managed to get shp-files for a smaller area. I have not looked into how one can do that.

Conversion table for FKB-data

The job now was to make a conversion-file fkb.txt which took as much of the FKB-data as possible and brought it into the Pullautin map. After some trials, I made the following fkb.txt file: 
Building|526|OBJTYPE=Bygning
Other building|526|OBJTYPE=AnnenBygning
Road inside (from AR5)|529|OBJTYPE=ArealressursFlate&ARTYPE=12
Road edge (from AR5)|529.1|OBJTYPE=ArealressursGrense&ARAVGRTYPE=7200
Road inside (from road layer; not included, we take from AR5)|0|OBJTYPE=Veg
Road boundary (from road layer; not included, we take from AR5)|0|OBJTYPE=Vegdekkekant
Bicycle road (not included)|0|OBJTYPE=GangSykkelveg
Parking area|529|OBJTYPE=Parkeringsomrade
Parking area boundary|529.1|OBJTYPE=ParkeringsomradeAvgrensning
Path|507|OBJTYPE=Sti
Large path|505|OBJTYPE=Traktorveg
River|306|OBJTYPE=ElvBekk
River edge|301.1|OBJTYPE=ElvBekkKant
Water channel|306|OBJTYPE=KanalGroft
Lake 1, water layer|301|OBJTYPE=Vann
Lake 2, water layer|301|OBJTYPE=Innsjo
Lake boundary|301.1|OBJTYPE=Innsjokant
Ocean|301|OBJTYPE=ArealressursFlate&ARTYPE=82
Lake 1 (AR5) - note we include lakes from both layers|301|OBJTYPE=ArealressursFlate&ARTYPE=81
Lake 2 (AR5)|301|OBJTYPE=ArealressursFlate&ARTYPE=80
Lake boundary (from AR5)|301.1|OBJTYPE=ArealressursGrense&ARAVGRTYPE=3000
Glacier (ignored for now)|0|OBJTYPE=ArealressursFlate&ARTYPE=70
Marsh|310|OBJTYPE=ArealressursFlate&ARTYPE=60
Open land with scattered trees (ignored, use Pullautin vegetation)|0|OBJTYPE=ArealressursFlate&ARTYPE=50
Various forest types (ignored, use Pullautin vegetation)|0|OBJTYPE=ArealressursFlate&ARTYPE=30
"overflatedyrka" - we use 401|401|OBJTYPE=ArealressursFlate&ARTYPE=23
"overflatedyrka" - should be 403?|401|OBJTYPE=ArealressursFlate&ARTYPE=22
"fulldyrkamyr" - should be 415?|401|OBJTYPE=ArealressursFlate&ARTYPE=21&ARGRUNNF=45
"fulldyrkajord" - should be 415?|401|OBJTYPE=ArealressursFlate&ARTYPE=21&ARGRUNNF=44
Settlement area|527|OBJTYPE=ArealressursFlate&ARTYPE=11
Remember to reference the "fkb.txt" file in your pullautin.ini-file: 
vectorconf=fkb.txt
From my comments in the fkb.txt file, you can see that I for now chose to use AR5-data for roads; it would be possible to use the road layer instead, although I did not yet manage to render the road edges from the road-layer in Pullautin.

Note that this is with my new 529.1/301.1 symbols. If you use the original version of Pullautin (which is the one available for download) you will not get any road/water boundaries with this fkb.txt file; you'll have to choose another symbol here. I will relay the information to Jarkko, and he might just implement these symbols (or alternative symbols so that the FKB-file can be changed accordingly).

Running Pullautin with rendering of roads etc. from FKB-data

With the new fkb.txt-file ready, all the shape-files in a zip-file "fkb_shape.zip" and the "vectorconf=fkb.txt" line in the ini-file, we are ready to go: 
pullauta.exe fkb_shape.zip
This now gives the following result for a single tile:

And the complete map (click for larger - and see the direct 600 dpi output 5 Mb file from Pullautin here, without any cropping):

For comparison - here is my start with just running Pullautin "out of the box" - including some self-inflicted problems and without FKB-data. I learned a lot since starting with this project two weeks ago!

Fine tuning: Buildings/cliffs/roads

A note about vegetation

In OCAD I have used Terje's vegegation, as I think it looks more like the local one. However, there are big possibilites to improve the vegetation output in Pullautin. Here is a description Jarkko wrote on the Routegadget Facebook page about how he fixed vegetation, 
Method used for green mapping: First I made tiny clip if the laser data, same as one of the teaser clips. I set Pullautin to have tens of green shades and make it do green mapping quite detailed way. I iterated "pointvolumefactor" until overlapping scanning lines could not be seen any longer as green stripes. Then couple of more iterations to set lightest green, just ta get the amount of white colour about right. I tried not to guestimate value for second and third tone, instead I left there plenty of tones - human eye is quite good at seeing where the colour it turns darker. Then I let it process the whole area and used other three teaser clips to judge/verify green mapping.
I would like to do some real testing of the vegetation mapping, but preferrably I should use an area where there is an existing orienteering map. Or at least an area where I can go out and check the terrain. So I might postpone this until I get more relevant data available.

However, as a starter I want to look into the "pointvolumefactor". Here is what Jarkko writes (in an e-mail) about this:

Most likely green is affected by overlapping scanning lines/point density. There is parameter to balance that, "pointvolumefactor". When one makes vegetation mapping with Pullautin, the first thing to do is balance this issue right. It depends on the way data is processed, is all raw points there or are some filtered out and so on. The default value is compromise for Finnish data of varying type and age. Your data looks better and most likely you should set that to zero or something like 0.1

I did do just a quick comparison with my current settings and a setting with "pointvolumefactor=0"

Original pointvolumefactor=0.5

Settings with pointvolumefactor=0

Not the biggest differences (note however that I used thinning for both).

I also tried with settings suggested by Jarkko where I just render a lot of shades of greens. The idea is that it might be difficult to get a general setting for all areas. With many shades, you can at least see the differences. Here is what Jarkko writes:

The question is how much this should be generalizes for training vs mapping and in what shade(density) should one place the boundary between white and green and between tow three greens. I's say it's difficult task, the value working in one place and nicely illustrating green in one place may not work that well elsewhere. To get it better/right there should be algorithm for detecting edges where shade turns fast considerably darker. Before that is available the best solution may be using lots of tones and let human eye see the edge like it is done here. And maybe editing the green as raster image in photo editor (~brightness, contrast, sharpen, median filters, etc.).

Here is the result with many shades of green:

Original

Settings with many shades of green

And here are the settings: 
pointvolumefactor=0 
greenshades=0.0|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1.0|1.1|1.2|1.3|1.4|1.5|1.6|1.7|1.8|1.9|2.0|2.1|2.2|2.3|2.4|2.5|2.6|2.7|2.8|2.9|3.0
lightgreentone=230
greendetectsize=1
groundboxsize=3
medianboxsize=4
medianboxsize2=1
My conclusion is that there is still a lot to learn about the green-settings, but I need to have LIDAR-data for an area with an old (good) orienteering map in our area to really start learning it. Anyway, I am sure that the green information we get out of Pullautin is a lot better than no green information at all.

Finally I still think that the way Terje does it in his tool is interesting, there I get the good visibility-low runnability green (green stripes) which we have so much of here in our region. We'll just have to see if it really is put in the right place....

Anyway, green mapping is postponed for now!




Availability of LIDAR and FKB-data in Norway

LIDAR and FKB-data are available in many parts of Norway. See here for availability of LIDAR and FKB-data for Norway (search for your region/area). It is, however, not freely available as in e.g. Finland (and in Sweden for students?). However, orienteering clubs will usually be able to get data without paying as long as they deliver back accurate "path data" from the area. Here is the corresponding text from orientering.no: 

  GEOVEKST sentralt har godkjent at lokale orienteringsklubber kan få tilgang til gratis geodata for å utarbeide orienteringskart. Dette inkluderer også LIDAR-data (laserdata). Kommunen skal som gjenytelse få data som er nyregistrert og synfart av kartprodusenten. Det kan være data som kulturminner, stier, løyper, veier og ferdselsårer.
And here is a suggestion for agreement. This is only valid for municipalities which are part of the "GEOVEKST" agreement. All municipalities are part of Geovekst except (unfortunately) Oslo, Bærum, Stavanger, Bergen and Trondheim.

You want the data on the following format:

Note that you should get both LIDAR- and FKB-data in the same projection; this should typically not be a problem.

Fun in 3D

I did some work on rendering this in 3D - looks spectacular, I think! Try it out yourself here.

A note about formlines in Pullautin

The way Pullautin works in formline mode (as far as I understand), first all formlines (i.e. 2.5 meter curves) are calculated. This is written to the contour dxf-file. Then an algorithm determines which formlines should be kept and which are not needed (most formlines are removed). This reduced set of formlines is used along with all the main contours when making the pullautin png-file.

Unfortunately, the dxf-file is not updated with the reduced set of formlines, and thus it is currently impossible to load just the normal contours and the reduced set of formlines into OCAD as a vector file. Instead you have to either remove all formlines or use all formlines. Alternatively it is possible to use a png-image instead of vectors in OCAD for the contours, but this does not feel like a good long-term solution. Thus currently this is a quite significant issue when using Pullautin to generate contours to OCAD.

Comparison with aerial photos in OCAD

As I currently did not have the possibility to go into the terrain to check out the map, I wanted to compare the generated map with aerial photos. There are a number of web services with aerial photos of the area, the best being norgeibilder.no. Unfortunately you can not download georeferenced images for import into OCAD from norgeibilder (at least I did not manage), and comparing without overlaying in OCAD was not easy. What I did in the end was to install SAS.Planeta and follow these instructions. My bat-file for the conversion process in the 4th/5th step: 
:: setfw.bat
gdal_translate -a_srs "holsnoygmapsn_17.prj" -co COMPRESS=LZW "holsnoygmapsn_17.png" "holsnoygmaps1n_17.tif" 
gdalwarp -s_srs EPSG:3785 -t_srs EPSG:25832 "holsnoygmaps1n_17.tif" "holsnoygmapsn_17.tif"
This worked well, and I could open the Google aerial image in OCAD as a background layer and compare with my Pullautin and/or OCAD map. Note that by doing this, you "may be violating Google's license if you do this with Google's maps or images, and/or if you distribute the generated product". For my quick test I think this should be within "fair use", but if you use this for real mapping, I think you'd be on the wrong side of the law.

Remaining issues