Ryanair used to be quite brutal about it, they would patrol the boarding queues with a cardboard box and if your carry on didn't fit, they'd send you off to the check in desk. Nowadays, they are a bit more relaxed. Now that everyone has got the message that you can't lug a suitcase into the cabin there's no point in badgering their customers. The secret is in their business model, which depends on a fast turn-over. Checked in luggage slows down the amount of time they need between landing and take-off. Therefore their staff are discouraged from checking in too much hold luggage because it works out cheaper for budget airlines to let a few slightly oversize bags into the cabin, hence the leniency.