With star striker Erling Haaland absent due to injury, Pep Guardiola's side trailed to Elijah Adebayo's goal late in the first half at Kenilworth Road.
But, after squandering a host of chances, City finally clicked into gear with two goals in three minutes after the break.
Silva equalised and Grealish bagged the winner to give City a much-needed first victory in five league matches.
After their lethargic 1-0 defeat at Aston Villa on Wednesday, which followed draws with Chelsea, Liverpool and Tottenham, City were in danger of falling out of the title race.
Liverpool's win at Crystal Palace on Saturday had moved the leaders seven points clear of City.
But fourth-placed City responded with a gritty display to close the gap on Liverpool to four points.
City still have work to do to regain control of their bid for an unprecedented fourth successive top-flight title and Haaland's fitness scare could not have come at a worse time.
Haaland, who had started all of City's league matches this season, was sidelined with a foot injury that could impact his availability for the treble winners' hectic Christmas schedule.
City boss Pep Guardiola said Haaland, who has 19 goals this season, noticed the injury after Wednesday's defeat at Aston Villa.
Guardiola will be sweating on his fitness with City travelling to the FIFA Club World Cup in Saudi Arabia before returning for the busy festive programme.
Despite their lowly position, Luton have been a tough proposition at home on their return to the top-flight after a 31-year absence.
They drew 1-1 with Liverpool, who needed a late equaliser to avoid defeat, and came within seconds of a point against Arsenal, before losing 4-3 on Tuesday.
On their first visit since 1998, City were the latest star-studded side to find an inhospitable welcome amid the terraced houses that surround tiny Kenilworth Road.
While they dominated the first half, Guardiola's men were hamstrung by their profligate finishing.
Phil Foden accelerated onto Silva's pass for a stinging drive that Thomas Kaminski saved at full stretch before Silva sliced the rebound wastefully wide.
Kaminski rescued Luton again as he stuck out a hand to repel Rodri's strike, then plunged to his right to stop Foden's shot.
City kept coming and Alvarez poked narrowly wide from Grealish's pass, while Foden was denied by Kaminski's brave save at his feet.
By the time Mateo Kovacic's blast flashed over, Guardiola was shifting uneasily on the touchline.
The Spaniard's fears were realised with seconds left in the first half when Luton grabbed the lead.
Josko Gvardiol allowed Andros Townsend too much time on the right flank and the Luton winger's cross reached the far post, where Adebayo climbed above Ruben Dias and Kyle Walker to head home.
Guardiola left the pitch in animated discussion with the officials over the goal being scored beyond the two minutes allotted stoppage-time.
When Rodri side-footed woefully wide from Walker's cross in the second half, the hoots of derision from the Luton fans exacerbated City's misery.
But Dias thumped a free-kick against the bar to foreshadow City's impending revival and they drew level in the 62nd minute.
Rodri was the catalyst with a rampaging run into the Luton area and when the ball broke for Silva, he curled a fine finish into the far corner.
Three minutes later, City's comeback was complete as appeals for handball against Julian Alvarez were ignored before the Argentine's low cross reached Grealish for a clinical close-range finish.
Guardiola celebrated by sarcastically blowing kisses towards the Luton coaching staff, while inwardly breathing a sigh of relief at City's escape.
(For more sports news and updates, follow Ahram Online Sports on Twitter at @AO_Sports and on Facebook at AhramOnlineSports.)