In that case it should definitely be up to the individual community.

Then create a community for your niche interest that isn’t popular in the more general communities. I mostly browse my subscribed, but some communities have some content I like and maybe a few I don’t. If i’m the only one, then it won’t make much difference. But if more people on the community down vote some content and up vote others it may just tell people a split in the community is necessary.