It depends on which route you come in on. The trail markers give two choices, shortly before Ponferrada. If you go to the left, via the road, you will end up in the old historic center and may feel a bit lost. Nonetheless, it's the shorter way to go. If you go to the right, you walk through a suburb of Ponferrada (Campo?) and once you arrive in the city, just after a small bridge, you have a choice. At that juncture, you can go to the right to get to the albergue, or to left which does take you directly past the castle.
Regardless of which way one goes, it's good to ask for help to find the castle, because it's well worth a look!