While I'm not a VC, and I don't know enough to comment on how hard the job itself is, I do know that it takes a significant amount of capital. Not everyone and their mom has enough capital, and therefore, they're not in a position to become a VC.

It is plausible that the barrier to entry is high, but the job is easy.