Referee Vladislav Bezborodov, center, shows a red card to Steven Whittaker, 2nd right, of Rangers after he threw a ball at the head of a Malmo FF player (AP)
The Ibrox side, trailing 1-0 from the first leg, were reduced to ten men in the 18th minute when Steven Whittaker saw red for an off-the-ball incident.
However, the ten men levelled the game on aggregate in the 24th minute through Nikica Jelavic's superb volley.
Madjid Bougherra was ordered off in the 66th minute for elbowing Dardan Rexhepi in the head before Malmo's Ricardinho joined him for an early bath for a second bookable offence.
Rangers, with only one win in their past 22 European matches, were only 10 minutes away from extra time when Jiloan Hamad got a vital equaliser with a 20 yard volley to put the Swedes into the next round.
Ally McCoist's side will now join rivals Celtic in the final qualifying round for the Europa League group stages.
Despite receiving clearance to play from UEFA a hamstring injury ruled out Rangers' new signing Dorin Goian and it was Sasa Papac that partnered Bougherra in the centre of defence while Ricardinho returned for the home side.
A moment of madness saw Rangers reduced to ten men when Whittaker was shown a straight red.
The Scotland international, whose mistake led to Malmo's goal at Ibrox last week, reacted angrily to a challenge from Jimmy Durmaz and threw the ball off the Malmo player's back earning his marching orders from referee
However, the set back seemed to spur Rangers into action and Jelavic levelled on aggregate. Maurice Edu swung a ball to the back post from the right wing and the Croatian met the ball sweetly to send a volley across Dusan Melicharek and into the far corner of the net.
The first-half ended on a nasty note when Lee McCulloch picked up a booking for a nasty tackle on Durmaz, who kicked out at the Rangers midfielder from the ground but escaped any punishment much to the anger of the Scottish side's management team.
Malmo began to pile the pressure on after the break as they tried to take advantage of their extra man as Rangers started to lose their discipline.
Steven Naismith picked up a yellow for a trip on Durmaz before the Scottish champions' task became even tougher as they lost another player to a moment of stupidity.
Algerian defender Bougherra was shown a straight red when he appeared to elbow Rexhepi on the back of the head when the two players went up for a high ball.
There was more red card drama seven minutes later when Malmo's Ricardinho, who was booked in the first half, received a second yellow for a challenge on Jelavic.
Jelavic then missed a golden chance when he sliced his shot wide with the goal gaping after being played clean through by Naismith.
McGregor made a great save from Pontus Jansson's header but he could do nothing to stop Hamad scoring the equaliser in the 80th minute.
Daniel Larsson chested the ball into the Swedish international's path and he took a touch before lashing a half-volley into the top right-hand corner.
Naismith volleyed over with almost the last kick of the ball as Rangers' misery was completed.