Without a Championship win since 1977, London were galvanised by an early goal from Paul Geraghty and led 1-04 to 0-04 at the break in Ruislip.
Mayo, who had six Senior Football Championship debutants on show, missed chance after chance - they hit 19 wides in the end - but late points from Trevor Mortimer and Kevin McLoughlin forced extra-time at 1-09 to 0-12.
In the added periods Andy Moran added four vital points to Mayo's tally, cancelling out a second London goal.
This was not the sort of game James Horan had in mind for his first Championship outing as Mayo manager. The result was enough to earn his side a shot at Galway in Castlebar on Sunday, 26 June, but the performance was of the horror show variety.
Poor decision making, wayward shooting and a general lack of intensity blighted Mayo's play against a London team that finished eighth in Division 4 of this year's Allianz Football League.
Much credit is due to Paul Coggins and his committed London team - they gave it their all and deservedly led for most of the opening 70 minutes, until Mayo began to get their act together late on.
Goal-scoring centre-forward Geraghty, full-back Michael Maloney and centre-back Tony Gaughan were the pick of the London players.
The Exiles grabbed an early lead with points from free-taker Eoin O'Neill and captain Sean McVeigh. Their panel contained four Mayo men, Noel Tuohy, Sean Kelly, John Scanlon and Tony Gaughan, who are either working or studying in the English capital.
Mayo skipper Alan Dillon replied with a point from play in the eighth minute and a foot block by Maloney on Jason Doherty produced a penalty for the visitors, four minutes later. However, Aidan Campbell struck the woodwork with his penalty and London cleared the danger.
By the midpoint of the half, the hosts had moved into a 1-03 to 0-01 lead, maintaining a high work-rate and using the breeze behind them to good advantage.
Geraghty's 18th-minute goal gave them a huge confidence boost, allied to a point from the hard-working Mark Gottsche.
Mayo had a good deal of possession before half-time, Dillon taking his personal haul to 0-03 amid a series of wides.
Alan Freeman and Campbell were particularly guilty, while Trevor Howley also fluffed a goal-scoring opportunity.
O'Neill kicked his second successful free late in the half and with Doherty replying with a fisted point in injury-time, London's lead were down to three points at the interval.
The majority of the 4,000-strong crowd expected Mayo to kick on in the second half but their wastefulness continued and an upset looked on the cards as corner back Barry Comer raided forward to two well-taken points.
Mayo's deployment of a sweeper system backfired as London used the extra man, Comer, to good effect. Scores from substitute Kevin McLoughlin, Moran and Dillon shows signs of a recovery from the westerners.
With 10 minutes to go, Mayo had London's lead whittled down to a single point, Moran and James Kilcullen finding their range in front of goal.
The London defence was coming under increasing pressure and another pointed free from Moran had the sides level at 1-07 to 0-10.
Back came the home side with Geraghty and substitute Killian Phair registering quick-fire points.
Substitute Trevor Mortimer responded for Mayo and then McLoughlin landed the levelling score, with time still left for the winner.
But again Mayo's shooting boots let them down - Moran and Doherty both missed frees in injury-time, so extra-time was required.
With London showing signs of fatigue, it looked like it was set up for Mayo to coast clear.
The momentum was certainly with the visitors when Moran landed three successive points, one of them with his fist. Cillian O'Connor added another for a 0-16 to 1-09 scoreline, but a goal towards the end of the first period of extra-time gave the Exiles some hope.
Sean Kelly lobbed a free in on top of the Mayo defence and fellow substitute Cathal O'Sullivan rose to knock the ball into the net.
Any lasting hope for London was extinguished though as Freeman, Robert Hennelly ('45') and Moran, with his sixth of the day, tagged on points in the second period to give Mayo a victory that they will quickly want to forget.