I always say if you can get an electrician to move the chandelier spot so it sits perfectly centered over your dining table, go for it. But if it's impossible, like in a condo with concrete ceilings, or if it is cost prohibitive, there are ways to make the swag look right. You have to make it look intentional. If you try to pretend it isn't swagged, it will look worse. So embrace the swag. Do a contrasting colour or make it look industrial.
Here are a few examples of what you can do if you are faced with this dilemma.
And another option is to simply add more. No swagging needed, but symmetry is in order.