when you walk down the pass, you arrive at the Fuente de la Reina (as said at the map) there you don't follow the arrows, but take the road to the right (a little backwards.)
But doing so you don't have the views at Segovia, so maybe make the turn to Valsain/ La Granja later on the route.