Most likely, the document get's unloaded (if not pre-loaded of course), but the cache generated for the document by users will remain - that is the memory you are seeing. Cache will never flush until the lower working set limit is hit. When the document is loaded into memory again, the generated cache will either be flushed or merged with the document again.
This is expected behavior.