This is a great suggestion. Perhaps a good compromise would be to tie downvoting ability to a "number of days visited" threshold, rather than a karma threshold. Those who visit HN often but comment rarely may be particularly well qualified to judge whether a comment is constructive.

Edit: This might work best for voting ability in general, not just downvoting ability.