If you're writing down your passwords for personal use, and you keep them collected in a notebook or something you're fine. If a bad guy has broken in to your home where they would have physical access to your notebook, then you have much bigger security problems than having your passwords compromised. Keeping a physical notebook of passwords would be much safer than keeping them in a file on your computer.
Password reuse is a much bigger security problem than writing your passwords down.