As it relates to survival, memory is a particular type of perception; it is not an accurate imprint of an event. In this sense, it is the process by which the organism creates a gestalt (functional unit) of the experience. This gestalt can be a faithful representation of an actual event or it can just as easily be a rendering consisting of unrelated data from several different events -- in other words, a mosaic.-Peter Levine, from Waking the Tiger (Healing Trauma)
My parents' bedroom is a location I cannot describe in a single sitting. While at certain speaking or writing takes I can catch the plants in the window straight ahead and my mother's closet, it is as if remembering the whole thing at once is simply too much. This is not surprising, considering that both of my parents died there, and I had some pretty traumatic experiences around both their lives and deaths in that room. Each time I go to write about one of their deaths, or even just a regular, non-traumatic scene taking place in that fifteen by fifteen square space, I seem to miss a few things that I then only catch later.
This is also not unusual.
Whether or not we carry trauma associated with a place or person, the fact is it will always take more than one take to capture the details necessary to evoke what is needed in our tale. Many people believe you can only get it in one draft, that somehow it isn't authentic if you don't write all of it at once. But that is simply not how the writing process goes, and certainly not how very significant scenes are built.
It's also not, as Levine points out, how memory works.
How we remember what happened changes over time, as our present changes. Our filter affects the story, which doesn't mean it wasn't true before - just as true as it is now. In life, in general, I have gotten less focused on being "right" than staying "true" - to the moment, to the memory as it exists now. There is no locus of exact memory - not inside our minds and not, despite your brother's opinion to the contrary, even in family annals. No one knows for sure what happened.
But you can take advantage of the fact that memory - and scene - are built over time. Gain from feedback that insists on more from the mother's perspective, and splice together your drafts. Stop expecting that the masterpiece of your conflict scene will appear in whole cloth, and instead lean into the mastery of mosaic.
What if your memories change drastically?
A student recently asked me which version of a significant memory in her past should she use - the one she recalled originally, the one she realized was "more right" later, or the one a friend of hers from that era insists is right? I asked her - which one feels most right, right now, as you are writing? She's worried that later, once it is all done and handed out, she'll think of yet another take.
She might. She'll have to embrace what was the mosaic of what she knew then, took to be true at that moment, imperfect as it always will be. She chose to work with her current memory, weaving in some historical accuracies from her friend's account. Not to satisfy her friend, but because that is what felt most true to her.