Yes they can connect. In terms of what they can execute
where, that depends.
If a login isn't assigned a database, they will access
databases under the guest account. If guest is in the
database, they have access to the database and whatever
public permissions have been set as well as whatever has
been explicitly granted to the guest user in that database.
Guest always exists in master and tempdb - you can't delete
the account from those databases.

-Sue