I'll take the other side of the coin - fully enjoy SJPP and then start at your leisure once you've had breakfast and seen what you want to see around town. The center of SJPP is pretty compact, and you'll be done with your self-guided tour in an hour or so. The walk to Orisson is a steep 8 km, but it will still only take you a couple of hours of actual walking time. Plus, there are more food choices in SJPP, so I'd eat lunch there.
But do throw all this aside if the weather forecast is for inclement weather or unclear skies in the afternoon; the views are that stunning.
In other words, just do what feels right when the time comes...!
Nancy