The 56-year-old Swiss coach's contract runs until Dec. 31 and the OeFB confirmed in a statement that it would not be renewed.
The decision was widely expected after his side won two of their first eight games in their 2018 World Cup qualifying group, leaving them all but mathematically eliminated.
"Due to the changed sporting situation, the (OeFB) sports director has been commissioned to provide a sound analysis of how the change can be achieved," said the statement. There was no immediate comment from Koller.
Koller, a former coach of German clubs VfL Bochum and FC Cologne, has taken charge of 52 games, with 23 wins, 13 draws and 16 defeats. Under his leadership, Austria failed to qualify for the 2014 World Cup but their performances improved considerably and they took Sweden to their final game in the race for second spot in the group.
The high spot of Koller's reign was when they qualified for Euro 2016 by winning their group, dropping only two points in the process and crowning the campaign with a 4-1 win in Sweden.
It was the only time that Austria have reached the finals through the qualifying competition, although they took part as co-hosts in 2008.
Their performance in France, however, was disappointing as they picked up one point from three games and were eliminated after losing their final match to Iceland.
That poor form continued with a home defeat against Ireland early in the World Cup qualifiers which left them with an uphill battle.
(Reporting by Brian Homewood; Editing by Toby Davis)
