You are correct, the climb out of Conques would make for quite a demanding start!
I would suggest resuming your chemin at Decazeville or Figeac- unlike Conques, both towns have a train station, and may be easier to reach depending on your point of departure.