Watford scored again in the 84th minute and Silva believes a late comeback could have been completed had United been reduced to 10 men.
"We scored the first goal from a penalty and for me in that moment it was a second yellow card for Marcos Rojo, with 11 against 10 it would be really difficult for them," Silva said after the match.
"We made it difficult for them because we scored the second goal. Imagine if it was 11 v 10 at that moment. We started to create more problems for them, but then one counter attack from them and we should do different in that moment."
A superb solo effort by Jesse Lingard sealed victory for the visitors and helped second-place United stay in touch with league leaders Manchester City, while Watford remain eighth in the standings.
United next face a tough trip to Arsenal on Saturday and are likely to be without midfielder Nemanja Matic, who was substituted in the 54th minute due to a muscle injury.
"Matic is injured, I don't know how badly," United manager Mourinho said after the game. "For Matic to tell me he wants to be replaced, he does not do that if it is nothing important. It is muscular."
Watford next host a slumping Tottenham Hotspur, who have dropped out of the top four after a three-game winless streak, on Saturday.
(Reporting by Aditi Prakash in Bengaluru)
