You can only remove packages which are not required by another package.
NOTE: This only removes the reference to the package in the manifest. The package itself, including its contents, remains intact, even for local packages.
When you remove a package from a Project, any Editor or run-time functionality that it implemented is no longer available in that Project.
To remove an installed package:
Open the Package Manager window.
Click the Remove button.
When the progress bar finishes, the package disappears from the list.
If you want to restore a removed package, follow the instructions for installing a package.