Once the file has been added to a conversation it will stay there until it's deleted.
To remove a file from the File Drawer, right-click on the file in the File Drawer and choose Delete from the drop-down menu. This will erase the file from this conversation’s File Drawer. Be aware that this will remove the file for everyone, not just yourself.
Similar to Pinboards all files are visible to the specific conversation members. Anyone can download the files and delete them if they should wish to do so.
If you’d like to limit the amount of people that have access to your files we suggest you create a specific chat with only those people to share the file.