Australia now go to Sydney on Wednesday to face the winners of the final last-eight tie between England and Colombia.
Cortnee Vine scored the winning penalty to end a remarkable shoot-out that saw both teams take 10 spot-kicks, the quarter-final having ended 0-0 after 120 nerve-shredding minutes.
Vine held her nerve to send the crowd into raptures and keep the Matildas' dream of winning the World Cup on home soil alive.
Australia goalkeeper Mackenzie Arnold had missed a chance to win the shoot-out when her kick hit the post, but then saved twice from Kenza Dali after the VAR spotted she had both feet off her line the first time.
Vicki Becho missed France's 10th penalty, and it was left to Vine to take Australia through by beating France's substitute goalkeeper Solene Durand, who had been sent on especially for the shoot-out.
"I'm so freaking proud about this team. The bravery that they showed tonight, unbelievable," said Australia coach Tony Gustavsson.
"I think we'll have maybe tonight to celebrate it and then tomorrow we'll review it and move on to the next game," added Arnold, the player of the match.
It was an agonising way for France to go out after a tense encounter watched by a sell-out crowd of 49,461.
Les Bleues had been hoping to get to the semi-finals for just the second time, following their defeat in the last four in 2011, but instead go home.
"We had a whole stadium and a nation against us. We produced an exceptional performance, but that's football. It was destiny," French coach Herve Renard told broadcaster France 2.
"Good luck to Australia. I think we deserved more but that's how it is."
Australia's victory was achieved despite Sam Kerr again being left on the bench at kick-off, with the talismanic Matildas captain, now fit after a calf injury, coming on early in the second half and going on to convert her penalty in the shoot-out.
- Kerr comes off bench -
Gustavsson stuck with the same team that started against Denmark in the last 16, while France brought the fit-again Maelle Lakrar back into their defence.
Lakrar really should have given France an early lead to silence the hostile crowd, but the 23-year-old somehow succeeded in diverting an Eugenie Le Sommer shot over the bar with the goal gaping.
That was a let-off for the hosts, who were then grateful to Arnold for tipping a Le Sommer effort behind and for stopping a stinging Lakrar attempt following a corner.
It looked like the occasion was getting to Australia, but they began to threaten in the final minutes of the first half.
They were desperately unlucky not to go ahead in the 41st minute when French defensive hesitancy allowed Emily van Egmond to tee up Mary Fowler for what seemed like a certain goal, but Elisa De Almeida raced in to produce a miraculous block.
French goalkeeper Pauline Peyraud-Magnin then had to come out to clear in front of Fowler, as it somehow remained goalless at the interval.
Kerr emerged 10 minutes into the second half, replacing Van Egmond to the delight of the crowd, and her introduction provided such a lift that Australia very nearly went ahead moments later.
Hayley Raso tried her luck with a rasping drive from outside the box, but Peyraud-Magnin saved and the Juventus goalkeeper topped that with a brilliant block to deny Fowler from point-blank range on the hour mark.
That appeared to pierce some of Australia's momentum, and the tension increased as the clock ticked down, making extra time almost an inevitability.
France thought they had the breakthrough 10 minutes into the extra period when Ellie Carpenter turned the ball into her own net, but Australia were rescued when the Chilean referee blew for a foul.
Arnold saved superbly from Becho, before France replaced Peyraud-Magnin with Durand as penalties loomed.
Durand is their penalty-saving specialist and denied both Steph Catley and Clare Hunt in the shoot-out, but it was not enough for France.
(For more sports news and updates, follow Ahram Online Sports on Twitter at @AO_Sports and on Facebook at AhramOnlineSports.)