Ludogorets, who won 1-0 at seasoned European campaigners Partizan Belgrade on Tuesday thanks to a late penalty, and Shakhtyor have not got past the second qualifying round before.
Defending a 1-0 lead from the first leg, Basel were ahead after five minutes through Fabian Schaer before Mohamed Salah and Marcelo Diaz added further goals with half an hour played.
But an own goal by centre-back Schaer revived Maccabi and Eran Zahavi kept the dreams of the 13,000 home fans alive with a goal eight minutes before the end of a breathless first half.
The build-up to the match had been dominated by reports of anti-Israel remarks attributed to Salah. He and another Egyptian, substitute Mohamed Elneny, were booed whenever they touched the ball at the compact Bloomfield stadium.
The Swiss, who reached the Europa League semi-finals last season, were rocked again eight minutes into the second half when Maharan Radi brought the Israeli side level with a shot from 30 metres that beat Basel goalkeeper Yann Sommer.
But Basel held their nerve to reach this month's two-legged playoffs to decide who will join the likes of last season's winners Bayern Munich, Barcelona and Manchester United in the group stage of European club soccer's elite competition.
Basel coach Murat Yakin said: "We could not have asked for a better start with three unanswered goals. But then Maccabi had nothing to lose. We were expecting a hot game, both in terms of the crowd and the climate."
Lyon, Champions League semi-finalists in 2009-10, had won the first leg against Grasshoppers 1-0 in France and just about deserved to seal the same result in Zurich thanks to a late goal from Clement Grenier.
Fenerbahce, who drew the first leg 1-1 in Salzburg, won 3-1 on Tuesday but went behind after four minutes with a Jonathan Soriano goal before Raul Meireles hit a quick reply for the home side.
Senegal's Moussa Sow scored a second for the Turkish side, who had grabbed a 1-1 draw in the first leg only after a 95th-minute penalty, and Cameroon striker Pierre Webo sealed the win for last season's Europa League semi-finalists.
Steaua Bucharest all but guaranteed their playoff place after just six minutes of the second leg against Dinamo Tbilisi when Iasmin Latovlevici scored to make it 3-0 on aggregate.
Dinamo equalised early in the second half but the 1986 European champions eased through with plenty to spare.
Maribor reached the playoffs for the second season in succession, on away goals, after a 0-0 home draw with APOEL Nicosia, which followed a 1-1 draw in the first leg.
Albania's Skenderbeu, 3-0 down after the first leg against Shakhtyor, shocked the Kazakhs by taking a 2-0 lead after just 20 minutes of the return leg through goals from Zelijko Tomic and Blendi Shkembi, from the penalty spot.
The Albanian midfielder added a second penalty on the half-hour to bring Skenderbeu level on aggregate but a goal by Sergei Khizhnichenko steadied the visitors' nerves before the break.
Skenderbeu's Pero Pejic was sent off early in the second period and Shakhtyor took advantage when Makzat Bayzhanov hit their second of the night to take them through 5-3 on aggregate.
Partizan Belgrade were missing left-back Vladimir Volkov and midfielder Milos Jojic after red cards and failed to make their home advantage count against Ludogorets, who had come from a goal down in the first leg to win 2-1.
The Bulgarians rubbed salt in the Serbian fans' wounds when Hristo Zlatinski scored from the spot in the 88th minute after a foul by Nikola Gulan, who was sent off.
The draw for the playoffs is on Friday. Teams will be divided into a champions group and a non-champions group which already includes Arsenal, AC Milan and Schalke.
Five of the 10 teams in each group will be seeded for the two legs on August 20-12 and August 27-28.
(Additional reporting by Ori Lewis in Tel Aviv; Editing by Martyn Herman)
