Users are shared between all Trac project and svn repos. If you don’t want them to have access you just remove their permissions. Deleting them would remove them from everything.
By default everyone who can authenticate has these basic Trac permissions:

To create a limited set of users with permissions copy the authenticated permissions to a new group

Then remove any (all?) permissions from the authenticated group. (fyi “authenticated” is a special group name like “anonymous”)
On the permission list click the “authenticated” row header to select all permissions then click the button “Remove selected items”
Then for each user add them to the new permissions group. Anyone you don’t add won’t have access to that trac project.
