Summary: Inevitably the additional smart switches will use some power, but it's probably going to be negligible compared to the other consumers in your home. I certainly can't imagine it being more than $50/year in electricity even if you run dozens of smart devices.
As stated in How much power do smart sockets consume themselves? you can probably expect a smart socket (or similar device) to consume around 1–2 W. Wi-Fi devices will be on the upper end, ZigBee on the lower end (perhaps even 0.5 W). That figure of course includes all of the losses inherent in the electronics, but Bence Kaulics' answer there goes into more detail if you want to work through the calculations yourself.
For comparison, incandescent light bulbs easily use 100 W while running (so over 50 times the consumption of a smart socket). A newer LED bulb will still use about 10 W while running, so several times your smart switch's usage.
Say you installed 20 smart devices each consuming 2 W of power—that's 40 W. Generally electricity is billed per kWh (a ballpark figure is $0.12/kWh in the US—and about £0.12 in the UK, too). You'd use up 1 kWh every 25 hours, so you'd be looking at about $0.11 per day due to your smart devices, which is $40/year. That is of course some money, but if you used fewer or more efficient devices, that could be much less. 10 devices at 0.5 W, for example, cuts that cost by a factor of 8—just $5/year.
I would only worry about the power consumption of smart devices if you also care about how much your other electronics use, e.g. set top boxes, routers and so forth. Most smart devices use less than those and are designed more with power consumption in mind. If you're worried, consider checking the power usage of the devices you want to buy and add it up yourself. Likely you'll find the idle consumption of these devices to be quite low, fortunately.