Born in 1924 in Belgrade, Yugoslavia, to an assimilated Sephardic family, Fischer did not experience antisemitism in her youth. When the Germans were fighting to capture Belgrade, she, her parents, and sister fled to a border town to escape the bombing. Her parents wanted to return to their hometown after the fighting ended because of a sense of unity with the community, but they did not want their two daughters to return.

Aryan papers were secured for Fischer and a non-Jewish friend of the family drove her down to the Dalmatian coast of Yugoslavia to join her sister, who was already there. The sisters lived for the next few years with several hundred other Jewish families on Curzola, Yugoslavia, an island that had been a lovely resort area. They remained there until September 1943, when they were taken by boat, under attack, to Bari, Italy, where they were greeted by American soldiers.

Fischer met her husband in Italy after the war. Several years later they met again in New York and were married. Fischer estimates that 30 members of her family were killed in the Holocaust.