He probably pummelled me a bit and told me a thing or two about life - I've always been pretty naive - but I don't remember him hurting me in any way.
From that time though we parted ways, because a Catholic school started in town and he was part of its first intake, while I stayed at the state primary school.
61 years later I was in Melbourne editing a magazine called Earthmover and Civil Contractor and my editorial and picture in the magazine prompted that boy, Allan Hoy, to send me an email to see if I was indeed the same David Palmer.
Shortly afterwards in April 2011, we met for coffee near his office in Collins Street and for starters we naturally reminisced about our early lives in the small Western District town.
Allan said suffering a displaced hip, playing football for Mortlake and subsequently missing months of school, had changed his life. Because he had missed so much school and was then 15, the school principal thought he might be better off in the workplace.
Fortuitously a job came up as supernumerary station assistant at the Mortlake railway station - one train a week but more than enough work for two people - and Allan got the job.
Nearly 60 years later he is still working in the railways but now in South Africa, Taiwan, South Korea, China and India as well as Australia.
Allan says that while he was not a good student at school, once he joined the Victorian Railways, he embraced the system which gave him the education he needed to do his job and live life to the full. In 1965 he married a Mortlake girl and they had two daughters who each in turn had two children each.
At the peak of his VR career in the 1980s, Allan was appointed to run the Melbourne Met, a job which entailed looking after 600 train drivers, 500 guards and all the Melbourne region's stations and signalling systems.
Allan retired from the Victorian Railways in 1991, on the eve of a massive shake up and has been consulting around the world on railway operations since then.
The spur railway line from Terang to Mortlake we enjoyed reminiscing about closed in 1978 and was later torn up.
Allan and I have stayed in contact since meeting for coffee in Collins Street in 2011.
I did not then and have not since though, reminded him that he once "beat me up".