Why do we need a Humanist Society?

When I browse the web or talk to people socially I often encounter those who say things to this effect:
"Yes, of course religion is a sham and a means by which the priests, mullahs, the pentecostal preachers and others like them mislead and dominate people who lack the intellectual skills or training to resist them. However I am busy with other things in my life and don't want to waste time on all that nonsense so why worry about it?"

Here is my response to that view.
I don't worry about the fact that some other people believe in a supreme being who created this universe and that a man executed 2000 or so years ago came back to life.
I don't worry if they believe holy water can cure cancer even if it can't restore an amputated limb. That's quite OK. Its OK if they pray to this god and to various dead people declared to be saints.
As long as they confine themselves to these beliefs and prayers they are harmless enough.
The problem arises when they inflict on the rest of us their views about family planning, sexual behavior, use of condoms for disease prevention and a list of other things designed to interfere with normal human life and its improvement.
If it wasn't for the fact that they invoke their supposed god to authorize themselves to interfere with the law making process, the schools, the taxation system etc, they need not be of concern to us.
It is because they do all of these things that Humanists and others who value normal human morality, secular law and education need to oppose them.
We can't afford the luxury of leaving the field to them.
If we ignore the need to enter a contest of values we risk further erosion of basic freedoms.