The OS does keep stuff in the cache for a long time. I'm not even sure it clears it out until the memory is needed elsewhere.
I guess as long as you have some free memory listed then it shouldn't be an issue.
You can run the following command as root which will clear the cache.
sync; echo 3 > /proc/sys/vm/drop_caches
You shouldn't use this command when the server is busy as it stops everything else while the command runs and can take a few seconds to complete if the cache is really full.