As you try new service, one is bound to make a lot of mess. I did the same with my Team Foundation Services account. After initial testing was done I decided to make a clean start and delete a project.
It took me quite a while to give up on finding delete. It seems that somebody forgot to take his pills during design phase and thus delete button does not exist.
Fortunately, this is just an Team Foundation Server so good old command line will do nicely:
> CD "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE"
> TFSDeleteProject.exe /collection:https://jmedved.visualstudio.com/DefaultCollection/ Test
Warning: Deleting a team project is an irrecoverable operation. All version control, work item tracking and Team Foundation build data will be destroyed from the system. The only way to recover this data is by restoring a stored backup of the databases. Are you sure you want to delete the team project and all of its data (Y/N)? Y
Deleting from Build ...
Deleting from Version Control ...
Deleting from Work Item Tracking ...
Deleting from TestManagement ...
Deleting from ProcessManagement ...
Deleting from LabManagement ...
Deleting from ProjectServer ...
Warning. Did not find Report Server service.
Warning. Did not find SharePoint site service.
Deleting from Team Foundation Core ...
Of course, if you want to create new project with same name, do not forget to visit
Source Control ->
Workspaces in order to remove last traces.