South Africa's le Clos, third at the final turn, relentlessly reeled in Phelps and plunged past him at the finish to win in 1min 52.96sec.
Phelps, who had led at every turn in a quest to become the first man to win the same Olympic swimming event at three successive Games, was just five-hundredths of a second back in 1:53.01.
Japan's Takeshi Matsuda earned bronze in 1:53.21.
Le Clos thrust his body out of the water in delight upon seeing the scoreboard. Then he perched on a lane rope as he tried to absorb his achievement.
Phelps has dominated the 200m fly for a decade. He owns the four fastest times in history and his world record of 1min 51.51sec is more than one second faster than the second-best performer in history.
When he saw the result, he flipped his cap away in disappointment.
Phelps' silver medal took him level with Soviet gymnast Larisa Latynina as the most decorated Olympic athlete of all-time with 18 medals.
He was to return later Tuesday as anchor of the US team in the men's 4x100m freestyle relay final.
Phelps' 18 Olympic medals include a record 14 gold -- eight of them from his spectacular Beijing Games campaign.
He won six gold and two bronze in the 2004 Olympics in Athens, and has won two silvers so far in London -- in Tuesday's fly and in the 4x100m freestyle relay.
(For more sports news and updates, follow Ahram Online Sports on Twitter at @AO Sports and on Facebook at AhramOnlineSports)
Short link: