Again, you really should read his book. I don't have it handy and it's been a while but I'll try to answer why your intuition (and mine before I read the book) may not be correct.

First, in the first small communities it's not believable that people would barter goods with other people they know. The reason is that it's difficult to set a fair value between goods. Also, you want to be productive in your community. I see this in my own life and family. I've never bartered anything with my family or friends. Instead, I'll just give something they may want to them and later they'll give me something.

Further, I rarely buy anything with money from family or friends. Again, for the same reason. Graeber doesn't see why it would different from the first human communities, if not more so. You don't do that kind of "business" with friends and family. He goes deep into the anthropology of family compared to strangers.

If you buy that argument, barter never existing in any meaningful way. Though, again, it DID exist, but it as one-offs and not as a substitute for money. That is, no one made things to barter on spec and store (like money).

Barter doesn't make sense between large makers because then they're back where they started from. Instead of 100 shoes to barter with various other makers they have 100 jugs. Therefore, money will come up quickly between communities (strangers). It might also come up between family, in communities, but more as a representation of favors, not goods.

Like you, I've always believe barter than money. After reading Graeber's arguments I changed my mind. I thought people used to barter more from my condescension of ancient people. I believed in the progress or development of human nature. Maybe I'll change my mind again. For now, I don't ever barter, or want to, so don't see why anyone would have 5,000 years ago. As a kid we used baseball cards as a kind of money. I don't see why that wouldn't have also been true back then. I hope you have time to read the book at some point!