The International Organization for Migration flew 172 Nigerian migrants including six women home from Libya on Friday, the organisation said.
An AFP cameraman said a child was among the migrants who left Tripoli's Mitiga airport at 4:30 am (0230 GMT) dressed in tracksuits and sports shoes.
"Almost all the migrants travelling on this charter were detained as they were trying to cross to Europe," IOM said in a statement.
"One hundred and forty-two had spent months in immigration detention centres," it said, adding the flight was organised with the Tripoli authorities and the Nigerian embassy.
Before embarking, they were given back personal belongings including mobile phones that were confiscated when they were detained.
On Thursday, they had travelled to the airport in buses from the Salaheddin and Abu Slim detention centres in the Libyan capital.
At the end of February, IOM organised a similar flight for 117 migrants from Burkina Faso who chose to return home, the organisation said.
Libya has long been a stepping stone for migrants and refugees seeking a better life across the Mediterranean in Europe.
The perilous sea journey from the North African country to Italy has taken the lives of 97 migrants and refugees this year alone, IOM said.
Libya has been in turmoil since the 2011 uprising that toppled longtime dictator Moamer Kadhafi and has had two rival administrations since the summer of 2014 when an alliance of mostly Islamist militias overran the capital.
People smugglers have taken advantage of the power vacuum to step up their lucrative business, while the Islamic State jihadist group has extended its influence in the oil-rich country.