"whoever is not against us is for us" —Mark 9:40Because the rationalizing animal can rationalize anything, many Christians rationalize this, but if you think the Bible is accurate, the only sensible explanation is Jesus couldn't make up his mind. No matter what your take, Luke 9:50 has him saying, "whoever is not against you is for you", so if he vacillated, he favored the idea that only those who oppose you are your opponents.
"whoever is not with me is against me" —Matthew 12:30
Most students of the mind would agree the outlooks in Mark and Luke are healthy ways to think of others, and the version in Matthew is paranoid—which is the version cults love. Matthew's take may be an accurate reflection of the writer and perhaps of his immediate community, while Mark and Luke were less afraid of the people around them, so I'm happy to think they all wrote what they believed Jesus said.
But I'll go with Mark and Luke.
ETA: Brother Will is a bad scholar: Luke has two versions of the saying. The second is at Luke 11:23: "Whoever is not with me is against me."