The prisoner exchange deal comes less than a month after Saudi Arabia, which has had uneasy relations with Iraq's Shiite Muslim-led government, named an ambassador to Baghdad for the first time since Saddam Hussein invaded Kuwait in 1990.
It also comes ahead of an Arab summit in Baghdad on 29 March which has been delayed twice by regional turmoil and acrimony between Baghdad and some Sunni Muslim Gulf Arab states over a crackdown by Bahrain's Sunni rulers on Shiite protesters.
"The agreement emanates from the strong relations between the two brotherly people and in the interest of strengthening friendship and cooperation between them," the Saudi Justice Ministry said in a statement.
It was issued after the accord was signed in Riyadh by Justice Minister Muhammad Al-Eissa and his Iraqi counterpart Hassan Al-Shimari. Iraqi officials were not immediately available for comment.
It was not immediately clear how many people would benefit from the accord. But rights activists say hundreds of Saudis held in Iraq, mostly for involvement in an insurgency against US-led troops, as well as Iraqis held in Saudi Arabia for criminal offences, are expected to be affected.
Thousands of Saudis travelled to Iraq to fight alongside Islamist insurgents after the 2003 US invasion of Iraq. Official rights group statistics show that less than 100 Saudis are held in Iraq, but the number is believed to be much higher and could reach hundreds.
Under the accord, prisoners from each country will serve out the remainder of their sentences in their home countries without being eligible for pardons. Saudi media said the deal does not cover inmates facing the death penalty.