It's more fair because the consumer pays a 2-3% premium for the convenience of a credit card vs. counting out cash, and people who don't want that convenience aren't obliged to pay for it with universal price inflation. Since this is a choice that the consumer makes, it's reasonable to make him pay for it.
In practice, I think most traditional retailers like it when consumers use cards because it makes their lines go faster and wouldn't really increase their prices for most card carriers if they were allowed to do so (AMEX might be a different story). The cashier doesn't have to wait for the consumer to fumble around for the correct combination of bills and coins, the customer doesn't have to wait for the cashier to count out change, and that means the line moves faster and the retailer can process more transactions and make more money.
In practice, I think most traditional retailers like it when consumers use cards because it makes their lines go faster and wouldn't really increase their prices for most card carriers if they were allowed to do so (AMEX might be a different story). The cashier doesn't have to wait for the consumer to fumble around for the correct combination of bills and coins, the customer doesn't have to wait for the cashier to count out change, and that means the line moves faster and the retailer can process more transactions and make more money.