If it was so cut and dry it wouldn’t be a “dilemma” don’t you think?

The prisoners dilema is a worst case scenario as you can have an infinite number of prisoners but just one can create the worst possible scenario for everyone else, the commons generally has the ability to sustain a couple of selfish people without breaking down. Everyone has their own threshold for how much abuse of shared resources they can tolerate before they too will begin to abuse those resources out of self preservation, society stops working properly when that process hits a tipping point that makes it self sustaining so it encompassas everyone.