Hi Yash,

With Windows Auth on the endpoint, it won't work to use Local System as the
service account because Local System has no privileges outside of its own
machine (no network identity). Using the same domain user as a service
account on both instances would work. Using different domain users or the
Network Service account as the service account would work if each domain
user/other machine's network service had a login on the other instance and
CONNECT privilege on each other's endpoint.

This article http://msdn.microsoft.com/en-us/library/ms166077.aspx shows how
to allow ENDPOINT security as certificates. Note that the example shows
setting up one side of the authentication; if you are setting up both sides,
you need to do this on both sides. Note that you must create a login and a
user in the master database because users own certificates and logins have
CONNECT privilege.

In the example at http://msdn.microsoft.com/en-us/library/bb839483.aspx,
they create certificates for server broker conversation security (this is
distinct from endpoint security). This is used for authorization in the
database in the target's side. Note that, on the target, SEND permission on
the target service is granted to the user (in the target database)
associated with the certificate (or the target database). This certificate
was imported from a cerificate file for a certificate created in the
initiator's database.

Remote databases don't know about users created in other
databases/instances, so when the initiator sends a message it's encrypted by
the certificate owned by the user specified in the remote service
binding/user that owns the service. On the target side, the user that owns
the same certificate (and only that user) has SEND permission on the target
service. Other messages/conversations will be rejected.

Cheers,
Bob Beauchemin
SQLskills