Napoli moved up to second and trail leaders AC Milan by just three points after their 3-0 derby victory over Inter Milan, third at five points, on Saturday.
Goals from Stefano Mauri and Andre Dias gave Lazio a two-goal lead before Andrea Dossena and Cavani, with his first, levelled matters.
A Salvatore Aronica own goal put the visitors back in front but Cavani, from the penalty spot, equalised and then hit a splendid winner for Walter Mazzarri's team.
The game wasn't without controversy, though, as Cristian Brocchi had a perfectly good goal for Lazio disallowed and the visitors also finished with 10-men as Giuseppe Biava was very harshly dismissed in giving away the penalty.
After some ineffective probbing, Napoli created the first clear chance on 22 minutes with Ezequiel Lavezzi playing Marek Hamsik down the inside right channel but goalkeeper Fernando Muslera was equal to his near post shot.
But on 29 minutes the visitors took a shock lead as Mauri luckily bundled past Michele Pazienza, beat Paolo Cannavaro and then with the outside of his left boot somehow slipped the ball past Morgan De Sanctis.
Moments later Brocchi played Mauri in behind the dithering Napoli defence but this time he hooked the ball past the far post with his right.
Napoli came back at Lazio but Dossena headed over at the back post from Christian Maggio's cross.
Lazio seemed to have the game in hand and Napoli's title hopes appeared to be going up in smoke on 57 minutes when the visitors went two ahead.
Javier Garrido swung over an inswinging free-kick and Dias got behind the ball-watching Napoli defence to poke home.
That, however, began a crazy 11-minute spell that saw the match swing back and forth.
On the hour a Lavezzi free-kick flicked off Giuseppe Sculli's head and Dossena arrived at the back post to head high into the roof of the net.
A minute later and another Lavezzi free-kick from the other side this time ballooned off Dias's head to the back post where Maggio nodded it back into the six-yard box for Cavani to head home and complete a rapid turnaround.
Just a couple of minutes after that Lazio had a goal controversially ruled out as Brocchi's pile-driver beat De Sanctis, bounced down over the line and came out, only for the referee to wave play-on.
Yet a minute after that Lazio were back in front as De Sanctis parried a Mauro Zarate strike only for Aronica to inexplicably prod the rebound past his own goalkeeper.
Having already given one poor decision Napoli's way, the referee then handed the hosts an even bigger gift.
He awarded Napoli a penalty after Cavani tumbled theatrically in the box under contact from Biava just as the Uruguayan was about to pull the trigger with what was nonetheless a difficult chance.
If the penalty award was soft, the red card shown to Biava was nothing short of outrageous.
Cavani stroked home the penalty and Napoli had eight minutes to find a winner with a numerical advantage.
That came two minutes from time with a brilliant lobbed finished over Muslera after Giuseppe Mascara's header over the defence sent Cavani clear.
That was too much for Lazio's former Napoli coach Edy Reja who received his marching orders for, wrongly, protesting that Cavani was offside.