My experience was the same as yours in some ways.
I think the only problem was a result of my own unrealistic expectations. I, like you, found sharing space difficult, especially when it was with those who didn't seem to vibrate at the same rate as I did.
I really enjoyed the aulburges where silence and prayerfulness was a requirement (there were a few, such as Emaeus).
I was on the road to Santiago as a pilgrim, to contemplate, to pray, to work out some issues in my own life, to think, to pray for others who had supported my trip, and to enjoy God's nature as I walked.
Much of my frustration was in the aulburges, which is why this trip, I will camp outdoors more.
It manifested as the challenge of often-irreverent chatter, loud and late night cellphone talking, blaring radios, people turning lights on, human excrement and fouled paper just thrown on the ground along with other litter, rustling bags and glaring lights without concern for those sleeping, rude talking in the chapels when people (locals included) were trying to pray/worship. On occasion, people were demanding, which was comical as well as shocking. I mean, really, when you're paying 3 euros, what type of coddling do you expect?
For me, all of these behaviors are displays of how we're becoming less empathetic, less caring, less loving of our fellow humans. I often just had to shake my head and walk away, disappointed in some of my fellow pilgrims for their seemingly blatant unkind and rude behavior.
This was the hardest thing for me... and luckily was not my ENTIRE experience. But you asked for the hardest, and I agree, that was it.
I guess I was expecting more of a silent vigil type of experience and my Camino certainly was not that.