The 47-year-old Herrlich, who had taken over in 2017, left despite the team's 3-1 win on Saturday against Hertha Berlin that lifted them to ninth place.
Bosz had briefly coached Borussia Dortmund last year after spells at several Dutch clubs, including Ajax Amsterdam, and a stint in Israel in 2016.
He will take over on Jan. 4 when the team meet up after the winter break ahead of the season restart on Jan. 18, and has signed a contract to 2020.
"Heiko Herrlich gave our team important impulses last year after a very difficult season prior to that and led us back to the international scene," Leverkusen sports director Rudi Voller said in a statement.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"Until recently we were determined to stick with him as head coach and turn the corner.
"But unfortunately we cannot deny the fact that there is currently a stagnation in the team's development. We are in an unsatisfactory position at the halfway mark and we considered a change of coach as necessary."
The 55-year-old Bosz led Ajax to the 2017 Europa League final, which they lost to Manchester United, before joining Dortmund. But a bad run until December 2017 led to his departure that month.
(Reporting by Karolos Grohmann; editing by Clare Fallon)
