It often happens that, when you attempt to remove a COM Add-In from an Office application, it shows a message saying that 'This add-in is installed for all users on this computer and can only be connected or disconnected by an administrator.'.
If you face this case, it is because the said add-in was installed using admin permission. To disconnect that add-in, you need to follow the steps as mentioned in this post.