South Sudan's president Salva Kiir cast the first vote in the region just after 8 a.m. (0500 GMT) in the southern capital of Juba, a Reuters witness said.
Voters slept outside overnight and huge queues built up outside polling stations before dawn in Juba where banners described the ballot as a "Last March to Freedom" after decades of civil war and perceived repression by north Sudan.
"Yes, of course, I will vote for separation. We need our independence. We need to be free from the Arabs," said Justin Victor, reverend at All Saints Episcopal Cathedral, as his congregation sang inside.
The referendum was promised in a 2005 peace deal that ended decades of civil war, fuelled by oil and ethnicity, between the mostly Muslim north and the south, where most people follow Christianity and traditional beliefs.
While many say both sides reaching referendum day without a relapse into war was an achievement in itself, some burning issues have yet to be resolved, such as where the new border will run and how Sudan's oil revenues will be shared.
U.S. President Barack Obama on Saturday said a peaceful, orderly referendum could help put Sudan back on a path toward normal relations with the United States after years of sanctions but warned a chaotic vote will mean more isolation.
In the north, the prospect of losing a quarter of the country's land mass -- and the source of most of its oil -- has been greeted with resignation and some resentment.
"It is a feeling of sadness and anger at the same time ... It is also a feeling of disappointment in the political leadership of the south who have guided the southerners towards secession," said Ibrahim Ghandour, a senior member of the north's ruling National Congress Party (NCP).
In the build-up to the vote, Juba and Khartoum already looked liked the capitals of two different countries.
In Juba, trucks blaring out music and slogans roared past buildings covered in pro-separation posters. Voters waiting outside one polling station burst into a communal rendition of the hymn "This is the day that the Lord has made".
Kiir earlier had repeated promises that there would be no return to war with the north.
"We are left only with a few hours to make the most ... important decision of our lifetime. I urge you to make the decision in a peaceful manner," he said in a speech.
In Khartoum, traffic was light and there were no banners acknowledging the historic referendum.
The vote's organising commission told Reuters it had defied gloomy forecasts of delays to deliver all voting materials on time for Sunday's deadline.
The logistical achievements have not been matched by political progress. Southerners went to the polls without knowing the exact position of their border with the north or how much of Sudan's debt they will have to shoulder after a split.
The two sides have been locked in negotiations for months over how they might share out oil revenues -- the lifeblood of both their economies -- and settle other issues after secession. There is no public sign of progress.
The south also will have to face up to its own internal ethnic rivalries and resolve a bitter dispute with the north over the ownership of the central Abyei region.
Still, north and south proceeded to the referendum while drawing a line under more than half a century of fighting.
"The risk is always there. There is always lots of tinder about and there are a lot of unresolved issues, including Abyei," said Derek Plumbly, chairman of the Assessment and Evaluation Commission that monitors the north-south peace deal.
"But neither side really wants to go back to war. I believe they will find their way through."