A problem I've noticed with every comment/article voting system, including HN, is that a single downvote shouldn't have any effect. Here on HN one downvote turns the comment dark which is to encourage "pile-on" voting. Any vote up or down should not be visible to other users until X votes have been cast.
I don't think it's a really big issue, but I agree with you wholeheartedly.
The color of the post should only change when it hits a negative score. A post downvoted by only one person is a reflection of that person rather than the post. Also, since they start at 1 point anyhow, it would be a very elegant system.