If my soul memory was 200k and I lost 20k souls would my soul memory stay at 200k or show as 180k?
The Souls Memory is basically the sum of every soul you've possessed, whether you used it to buy something, to level up, to repair items or simply lost it.
Note: The souls of fallen bosses or the souls you find in corpses (eg. Soul of a Hero) do not count towards your Soul Memory until you actually use them.
Second Note: Answering specifically your question. By the time you lost those souls, they were already part of your Soul Memory, in other words, every single soul you receive adds to your current souls (the ones you have to spend) and your soul memory, what happens to those souls after that doesn't matter, they will stay as part of your soul memory.
As mentioned by @3ventic in the below comment, it is worth noting that the retrieved souls DO NOT add to your soul memory, as those are souls you already acquired.