Headphones don't work right.

Question

I have a Yamha PSR 500 keybord. When i try to use headphones with the mini plug and an adaptor the volume is very diminished. I have to turn the volume up all the way to barely hear it when using the piano voice. The headphones work fine when plugged into another device(mp3 player, my computer). I'm wondering if the adaptor is the problem? Without headphones the volume level is fine and everything else on this old keyboard seems to work!