With almost a quarter of votes tallied, the ANC was leading but with a score of 43 per cent -- well below the 57 per cent it won in 2019 -- followed by the liberal Democratic Alliance (DA) at 25 per cent, according to authorities.
The leftist Economic Freedom Fighters (EFF) was in third place with nine per cent of the vote, trailed by former South African president Jacob Zuma's uMkhonto weSizwe (MK) with eight.
The final results are not expected to be known before the weekend.
"The broad church of the ANC has taken a substantial knock. This is a shock to the system for the ANC and ultimately will also be a shock to the system for the average South African, who has only known ANC rule since 1994," said political analyst Daniel Silke.
"It redraws the political boundaries of South Africa and creates a degree of uncertainty".
If President Cyril Ramaphosa's party is confirmed as dropping below 50 per cent, it would force him to seek coalition partners to be re-elected to form a new government.
That would be a historic evolution in the country's democratic journey, which was underlined by newspaper headlines on Thursday.
"SA on the cusp of a shift in politics," read the front page of daily BusinessDay. "The people have spoken," headlined The Citizen.
New chapter
The ANC has dominated South Africa's democracy with an unbroken run of five presidents from the party.
The party remains respected for its leading role in overthrowing white minority rule, and supporters credit its progressive social welfare and black economic empowerment policies with helping millions of black families out of poverty.
But over three decades of almost unchallenged rule, its leadership has been implicated in a series of large-scale corruption scandals, while the continent's most industrialised economy has languished and crime and unemployment figures have hit record highs.
"The people in power are hopefully going to come down and we will have a new political party," Shaun Manyoni, a 21-year-old student, said as he enjoyed a morning beer outside his university in Johannesburg on Thursday.
Ramaphosa's opponents from both the left and the right came to the polls on Wednesday hoping either to replace the ANC with an opposition alliance or force the party to negotiate a coalition agreement.
"Zuma ran this country perfectly ... so let's put him back and let South Africa run again," Don Naidoo, a middle-aged small business owner from the province's largest city of Durban, told AFP.
Voting was marked by hours-long queues in many districts, which in some cases forced polls to remain open well beyond closing time.
The Independent Electoral Commission (IEC) said a last-minute rush in urban voting and high turnout were to blame for the late finish.
Almost all votes had been counted by 3:00 pm on Thursday, but most had yet to be validated, IEC's head Sy Mamabolo told a press conference.
While the process was "proceeding well", officials said they expected it would take longer than the usual 24 hours to reach an 80 per cent tally, due to delays caused by the new three-ballot system.
What bedfellows?
If the ANC gets close to 50 percent it could shore up a majority by allying itself with some of the four dozen smaller and regional parties contesting the election.
But this appeared increasingly unlikely.
A projection by the respected Council for Scientific and Industrial Research (CSIR), showed it was in line to win less than 42 per cent, a share that could force it to partner with a bigger rival.
Yet, experts were split on who the ANC would prefer as bedfellows and on whether the poor performance threatened Ramaphosa's leadership.
"His power is gone within the ANC," said analyst Sandile Swana, predicting that the party would patch up ties with one or both of the radical left groups led by former ANC figures: firebrand Julius Malema's EFF or Zuma's MK.
In a major upset, the latter was leading with 43 per cent of preferences in Zuma's home province of KwaZulu-Natal, a key electoral battleground.
Siphamandla Zondi, a politics professor from the University of Johannesburg said MK was a natural partner for the ANC.
"They have similar policies and similar tendencies," he said.
But analyst and author Susan Booysen said the rift between Ramaphosa and Zuma, who has long been bitter about the way he was forced out of office in 2018, was "too far reaching" to mend.
The ANC might prefer the centre-right DA, which pledged to "rescue South Africa" through better governance, free market reforms and privatisations, to the leftist EFF, which is perceived as "too erratic" and "unpredictable" in its demands, she added.
Swana said he expects pressure from civil society to push for a convention to publicly discuss the makeup of a coalition.
"We don't want politicians to talk among themselves," he said.
Short link: