Absolutely NO worries about safety!!! You may not encounter many other pilgrims but plenty of pleasant people. It would be best to try to speak French, generally in less touristy parts of France little english is spoken. I think the French are less inclined to speak a foreign language than even us English!!! Even in trourist offices we someimes found that little english was spoken. Keys for the pilgrim gites are usually to be found in the Tourist office or the Town Hall (Mairie) so that should be your first port of call at the end of the day.
The GR 654 tends to go round three sides of a square, up a steep hillside, across a forest and two unnecessary valleys even when there is a quiet lane which wil take you somewhere directly. However there are yellow arrows to follow. The 'definitive' guide is the Chassain guide of which I believe a new edition is due out. This has maps and details about every town and village you will pass through, with lots of infor about where to stay. This route is more direct. hwever, on the ground, you will find that there are other possibilities suggested which may be nicer. For example we took the advice offered in La Tripterie, a caravan/albergue on a farm between Issoudon and Chateauroux (I think) for a route through a pleasant forest which avoided the town of Chareauroux (or however you spell it!!). You call visit the office of Les Amis of the Vezelay route in Vezelay (obvious from the mannequin pilgrim standing outside the entrance - the office is upstairs) for more advice about routes - again, I would expect to speak French.
best wishes
Bridget
But really do not worry about safety; you are walking through rural France.