Delete those you dont play with often? Or actually, you can do it the other way too: Delete those that you know are always on, since you know they will always be there to join lobbies. ^^'
True, I just feel like an asshole deleting people of my list. Most I play with at least once every few weeks.