The two-time Olympic gold medallist finished the 21.1-kilometre route in 60min 18sec according to unofficial results, beating the previous record of 60min 53sec established by Moroccan runner Mohamed El Hachimi in 2009.
This was Gebrselassie's first long-distance run since announcing his short-lived retirement in November following a knee injury, and he had wanted to show that he was back, he told the Austria Press Agency after the race.
Gebrselassie, who turns 38 on Monday, had been set the solo challenge of catching up with the best runners who started two minutes ahead of him, and did just that after barely 34 minutes.
He will next run a 10-kilometre event in Birmingham in May before deciding which marathons to run in the autumn.
"Berlin is a possibility," he said Sunday.
The smiling Ethiopian, who has two Olympic golds in the men's 10,000 metres and has won four world championships, beat the world marathon record in Berlin in 2008, finishing in 2hr 03min 59sec.
He announced his retirement however, after dropping out of the New York marathon in November and had to pull out of the Tokyo marathon earlier this year due to knee injuries.
The full marathon on Sunday was won by Kenyan John Kiprotich, 21, who beat two of his compatriots to finish in 2hr 08min 29sec, becoming Vienna's youngest winner ever.