My dad came to inspect our progress on The Xmas Grifter (see Kustoms) and bought some of his old bike manuals/leaflets etc.  I had given up on removing the handle bars / forks but when flicking through the leaflets i noticed an exploded diagram that looked very familiar (albeit with racer handlebars).  It showed how to remove the handle bars and forks to check the bearings.  If you look on the diagram point 4 was the part i had not done.  Now i expect i am telling everyone what they already know but once i had tapped down the centre bolt (two lots of 5mm) the handlebars were out in less than 5 mins with no effort at all!  If you are struggling to read let me know - not sure what quality will be like when i add a link.