I would suggest you think of it as a layering system. I usually do T shirt, then long sleeve shirt, next fleece jacket then finally a rain jacket, if needed. I agree that you should be able to do without the merino sweater, as long as your fleece is not a thin, flimsy fleece jacket.