not such a silly idea, I do like the sound of it, but I would not do it, in the camino you want to forge about your every day chores (like charging your mobile).
at the same time, it is true it would be ideal, walking to Santiago, normally the sun is on your back/rucksack, you any solar panel would have an excellent exposure to sun, so, technically, it does make sense ...
for convenience, it would be really easier to just charge every few days in the albergues as you move along, even better if you have your mobile off all day but one hour in the evening or so, just forget about mundane things...
and enjoy, buen camino!