We are like fish out of water. Just as a fish cannot be happy unless he is in the water, we cannot be happy apart from the spiritual world.
The happiness in serving is the happiness that material nature cannot impede because we can always serve in some way or the other.
When we understand our relationship with God then we all will be happy. It is an essential factor for establishing peace in the world.
Without the grace of the Lord, no one can become happy and prosperous.
Srimad Bhagavatam 3.1.26,
That procedure which in the beginning may be just like poison but at the end is just like nectar and which awakens one to self-realization is said to be happiness in the mode of goodness.