Even tough your signs may not seem "perfectly compatible" at first.
As it happened with me, just a few years ago, I could read a guy who is stone cold.
It could be that your Virgo guy picks up his cellphone to call or text you, but then being poor at managing his emotions, the poor soul will just set it down.
So he could very well be into you, but due to one reason or the other he may not show it openly or he may pull back because he gets into an "analyzing" mode that Virgos are known for...
A Virgo man has a very bright side and a kind heart, like you... They may not reveal their true emotions or reveal entirely different emotions when they have their doubts with a woman or a certain relationship situation.