It's never too late. "Back then", when I started using Signal (called TextSecure), only one other single friend used it. Nowadays, almost all my personal contacts use it. Every additional Signal user adds a contact in someone other's address book as a potential Signal contact. It just takes time. Good luck!