"According to preliminary investigations, Al Makki is a Yemeni national and has been working directly under Al-Qaeda leaders along Pak-Afghan borders," the military's media wing said.
"The arrest of Al Makki is a major development in unravelling the Al-Qaeda network operating in the region," it said.
The arrest follows a US covert operation in the garrison city of Abbottabad near Islamabad on May 2 that killed Al-Qaeda leader Osama bin Laden, and comes a day after US senator John Kerry visited Pakistan to smooth a damaging row caused by the raid.
Pakistan's civilian and military leaders were left angry and embarrassed after the unilateral US assault that killed the Al-Qaeda chief, who had been living, possibly for years, two hours drive from the capital.
The raid rocked the country's powerful security establishment, with its intelligence services and military widely accused of incompetence or complicity over the presence of bin Laden in a suburban house in Abbottabad.
Al Makki is apparently not on the list of internationally most wanted Al-Qaeda operatives but the military's statement mentioned his network was operating in the region.
His arrest came a day after a Saudi Arabian diplomat was killed in a hail of bullets on his way to the Saudi consulate in Karachi.
Interior Minister Rehman Malik blamed Al-Qaeda for the attack which was the second on Saudi interests in Pakistan's biggest city in less than a week, media reports said.
The Pakistani Taliban denied carrying out the assassination and authorities said they were investigating possible links to sectarian groups or the death of Saudi-born Osama bin Laden by US Navy SEALs.
The killing in an upmarket area of Karachi followed a grenade attack on the mission's building last week.
Saudi Arabia expelled bin Laden in 1991 and later revoked his nationality.
The government in Riyadh, which is allied to the authorities in Islamabad, last week welcomed his killing as a boost to international anti-terror efforts.
Pakistan is holding in protective custody three of bin Laden's widows, who come from Saudi Arabia and Yemen, and 13 of their children.