Just can't see how it can be done without the actual plastic card.
I could tell you my PN but what good will that be if you don't have the actual card. With a credit card you can give somebody the PN and they can go to town, but how can somebody empty you ATM account without the actual physical plastic card?