I installed PulseAudio on an old, small, fan-less x86 box that is close to the hi-fi system. On my home server I have a mpd installed that uses this audio sink box.
Now it always tries to open up a X11 connection as client. If I'm logged in such that the original display is available it will fail to play music just printing "X connection to localhost:11.0 broken (explicit kill or server shutdown)." or similar (DISPLAY variable may change) on the screen. Not that there is no local X installed on the box running the media player.
If I log out I can play music and get "E: client-conf-x11.c: XOpenDisplay() failed" in the log file. It seems that the client always tries to open an X11 connection, which is obviously not the right thing to do. At least there should be a configuration parameter in client.conf to turn this off. Also it should not abort at that point if it cannot open the X connection, but it should just try to play.
Doing an "unset DISPLAY" in the startup script just before mpd is started is a workaround.