The non-fallacy answer is that there is nothing compelling you to work out an accommodation with other people other than personal preference, regardless of what that preference is informed from. We don't HAVE to do anything. You can be tolerant, intolerant, or any level inbetween. It's entirely up to you. But there is no "have to" involved in it.

Once you recognize that, you can understand why others don't feel they have to work out an accommodation. Once you eliminate "have to" as a compelling premise, other's actions become understandable. There's no guideline, no force, no law of nature or physics that requires it.