Organisers of the 2016 Games included the Amazonian city as a potential venue earlier in the day.
It joins two stadiums in Rio and others in the possible host cities of Belo Horizonte, Brasilia, Salvador and Sao Paulo. All of them built or renovated stadiums for football's 2014 World Cup.
FIFA, however, told Reuters it preferred to have venues close to Rio, not 1,770 miles away like Manaus is.
"We do not consider Manaus as a suitable first option for a hosting venue," world football's governing body said in an emailed statement.
"Given the Olympic football tournament is only two weeks long, FIFA's preference has also always been on short distances between the other football venues and the Olympic city Rio de Janeiro to provide the participating teams with the best possible Olympic experience."
FIFA and the Brazilian Football Confederation will decide which cities to use by March, Olympic committee president Carlos Nuzman told reporters in Rio.
Sixteen men's teams and 12 women's teams will take part in the competition. Olympic gold is the only major title Brazil have not won.
(Reporting by Rodrigo Viga Gaier and Pedro Fonseca in Rio de Janeiro; writing by Andrew Downie in Sao Paulo; editing by Justin Palmer)
