Fears about the Zika virus have been cited by many as the reason for opting out of Rio, while other golfers have pulled out due to scheduling conflicts.
Montgomerie, who will play in next week's British open for the first time in six years, expressed his disappointment at those who had opted out of Rio.
"Am I worried it might affect golf's future in the Olympics? Very much so. First time we're back in the Olympics since 1904 and we don't show up," Montgomerie told British media.
"The IOC will have a good look at this and think, 'hang on a minute, what's happened here? We've suddenly had to find around 300 new rooms, and we've had to build a golf course.
"That goes into the multi, multi-millions -- and then they don't show up?'
"If I was in charge of the IOC, of course I'd have a second look at it."
British golfer Chris Wood, who is in contention to qualify for the Olympics, revealed he is considering withdrawing from the Games due to health fears from the Zika virus.
The mosquito-borne disease can cause crippling birth defects in infants and has been linked to the neurological disorder Guillain-Barre in adults.
"I'm getting married the week after and we want to start a family," Wood was quoted by the Telegraph. "I thought Rory (McIlroy) made the case perfectly when outlining his Zika fears."
The Rio Games will be held from Aug. 5-21.
(Reporting by Nivedita Shankar in Bengaluru; Editing by Peter Rutherford)
