Depends on your car and what the charger is plugged in to.

phone chargers plugged in

If the phone charger doesn't have any crazy lighting and you're not leaving a phone connected to it and your battery is in decent health - Then probably no issue.

On my cars I always have my chargers connected and have never had an issue with them - but my cars all turn off the 12V sockets when they're not running.

As @kaizen pointed out, in some older cars, the feed to the cigarette lighter is always hot, while in others, it's only hot with the ignition on.  If your lighter is always hot, a charger plugged in with a phone, will definitely pull down the battery, no matter how new it is or the CCAs, just like a parasitic draw.

You mean just the charger?

No. If you think about it, many cars already have charging built in (either a standalone USB socket or as part of the entertainment/nav unit), and they're fine. Electronics almost always use a tiny bit of standby power, but we're talking about microwatts... it's insignificant.

Caveats:

If you buy cheap Chinese no-name crap from Amazon, fleaBay, AliExpress and the like... ( you know .... the $3 ones) then all bets are off.

Or are you talking about leaving a device plugged in?  Small smartphones shouldn't be an issue. I have one plugged in 24/7 right now. Bigger power hungry things like tablets and laptops ... that could start to become an issue. I have had issues with dashcams leaving me stranded.

If your phone is plugged into it, then yeah, a little.  If not, then no (assuming there's no short it in or something).

In older cars I err on the side of taking chargers out of the cigarette socket.  I could be wasting seconds of my life doing so, but it gives me piece of mind. 

In newer cars, they have built in USB, and a better battery, so I usually just leave them plugged in.