"We tried to reach agreement on a way forward to resolve all the outstanding issues ... We approached this trip in a constructive spirit. Unfortunately we could not get agreement," Herman Nackaerts told reporters at Vienna airport on Wednesday.
"We could not get access (to the Parchin military site), we could not formalise the way forward. We will now report to the (IAEA) director general and later to the board of governors.
"Then we will have to see what are the next steps."
The International Atomic Energy Agency said in a statement in the early hours of Wednesday that "intensive efforts" to reach a deal on a document covering the way forward had failed.
The team requested access both during this visit and during a first trip in late January to Parchin, near Tehran, where it believes explosives testing was carried out, but Iran "did not grant permission," the IAEA said. "It is disappointing that Iran did not accept our request to visit Parchin during the first or second meetings," IAEA Director General Yukiya Amano said in the statement.
Iran's envoy to the IAEA, Ali Asghar Soltanieh, was quoted by the Iranian news agency ISNA as saying the talks "would continue" but neither he nor the IAEA said whether another visit was planned.
And following the visit, Iran's supreme leader Ayatollah Ali Khamenei insisted that the regime in Tehran was not seeking nuclear weapons.
"We are not after an atomic weapon. We want to break the supremacy (of the world powers) that relies on nuclear weapons. God willing, the nation will reach this goal," he told a meeting with Iranian nuclear scientists, according to an official government statement.
The high-ranking IAEA team led by Nackaerts hoped to clarify issues raised in a watershed November report from the agency that substantially raised suspicions that Iran had done work developing nuclear weapons.
The trip was also seen as an important precursor to a possible resumption of talks between Iran and the P5+1 powers, the United States, China, Russia, France, Britain and Germany, which broke down in Turkey 13 months ago.
Since the report's publication, the United States and the European Union have ramped up sanctions on Iran's oil sector, and speculation has grown that Iran's arch rival Israel might launch air strikes.
Iran has threatened to close the Strait of Hormuz, has announced the halt of oil sales to Britain and France -- sending the price of crude soaring -- and has defiantly trumpeted advances in its nuclear programme.
Iran, whose economy has been hit hard by the recent ramping up of sanctions, has consistently denied wanting nuclear weapons, insisting its programme, including the enrichment of uranium, is for peaceful purposes.
The UN Security Council has imposed four rounds of sanctions on Iran because of its repeated failure to declare nuclear sites and materials to the IAEA.