Fighting on both sides of the boundary has strained ties between the two capitals - Khartoum and Juba and complicated talks over issues such as oil and debt left unresolved since the south seceded in July under a 2005 peace deal that ended decades of civil war.
For its part, the insurgent group, the Sudan People's Liberation Army North (SPLA-N), denied it had troops stationed in the area named by Sudan's armed forces, located in the loosely demarcated border region with newly independent South Sudan.
Both sides have accused the other of supporting rebel groups on either side of the border. "The Sudanese army attacked and took control of an SPLA base at 4:30 this afternoon in the Baheirat al-Abiyad Jau region," Sudan's armed forces spokesman Al-Sawarmi Khalid said by phone on Saturday.
"This is considered the headquarters of the SPLA in South Kordofan," he said, adding that "a number" of rebels were killed in the fighting.
Qamar Dalman, a spokesman for the SPLA-N in South Kordofan, denied the group had soldiers in the area. "Our troops aren't present in that area because it falls in South Sudan according to the ... 1956 border," he said, referring to an internal boundary drawn shortly before Sudan became independent.
A spokesman for South Sudan's military was not immediately available to comment.
The SPLA-N fighters in South Kordofan and Blue Nile served as divisions of the southern rebel army during Sudan's civil war, but were left on the northern side of the border when the south seceded.
Clashes broke out between the insurgents and government troops in June, with both sides blaming the other for provoking the violence.
Khartoum accuses Juba of continuing to back the SPLA-N rebels, which both South Sudan and the rebels deny.
South Sudan voted overwhelmingly to secede in a January referendum, culmination of the 2005 peace deal that ended what was one of Africa's longest and deadliest wars. Some 2 million people died in the conflict.