What does "their jokes were quiet and they smiled rather than laughed" tell about the mood of the story?

When the townspeople gather together for the lottery, they are subdued and quiet. They engage in small talk because they would rather not discuss the lottery itself. They go about the ritual mechanically because it has become a habit. But there is also a prevailing sense of foreboding. They uneasily go through the ritual. The mood is dismal and the people smile rather than laugh because they all have a shared sense of dread.

Thus, their discussions and jokes are restrained by these shared senses of fear and apprehension. To combat this dread, they make jokes to lighten the mood. But they can't bring themselves to laugh because the dread is simply too much. They might also be restraining any sense of laughter and happiness out of respect for the unfortunate person who will become the victim that day.

The mood might also be described as solemn. It has the solemnity of something like a church service or even a funeral. The people engage in restrained small talk and light humor in order to make the gruesome ritual more bearable.