We are like fish out of water. Just as a fish cannot be happy unless he is in the water, we cannot be happy apart from the spiritual world.
The more we are happy in Krishna Consciousness the more we become purified.
Actual joy is a product of the spiritual nature. It spontaneously manifests when the living entity begins to love God.
Without the grace of the Lord, no one can become happy and prosperous.
Srimad Bhagavatam 3.1.26,
The happiness in serving is the happiness that material nature cannot impede because we can always serve in some way or the other.