Authentication to Mesos master does not seem to work

[zhitaoli]

Hi,

I'm trying to write a testing framework with mesos-go, but the authentication to my testing mesos master does not seem to work.

Description:
I ran a mesos master in a local Vagrant VM with ip 10.141.141.20 with port forwarded to my Mac and use a fake principal/secret pair to authenticate.

The official python testing framework built from current Mesos release master is able to connect.

When I tried to run the persistent_volume example, I got authentication errors like:

$ /tmp/mac-persistent-scheduler-test  -address="10.141.141.1" -alsologtostderr=true -master="zk://10.141.141.20/mesos" -mesos_authentication_principal=fake_principal1 -mesos_authentication_secret_file=/tmp/fake_secret1
I1218 15:54:23.705075   65718 main.go:337] Initializing the Example Scheduler...
I1218 15:54:23.709051   65718 scheduler.go:323] Initializing mesos scheduler driver
I1218 15:54:23.709212   65718 scheduler.go:792] Starting the scheduler driver...
I1218 15:54:23.709291   65718 http_transporter.go:407] listening on 10.141.141.1 port 50847
I1218 15:54:23.709309   65718 scheduler.go:809] Mesos scheduler driver started with PID=scheduler(1)@10.141.141.1:50847
I1218 15:54:23.709415   65718 scheduler.go:999] Scheduler driver running.  Waiting to be stopped.
2015/12/18 15:54:23 Connected to 10.141.141.20:2181
2015/12/18 15:54:23 Authenticated: id=95057675430527014, timeout=40000
I1218 15:54:23.714933   65718 scheduler.go:374] New master [email protected]:5050 detected
I1218 15:54:23.714976   65718 scheduler.go:838] authenticating with master [email protected]:5050
I1218 15:54:23.715235   65718 http_transporter.go:407] listening on 10.141.141.1 port 50849
I1218 15:54:23.722448   65718 authenticatee.go:260] Received SASL authentication mechanisms: [CRAM-MD5]
I1218 15:54:23.724992   65718 authenticatee.go:305] Received SASL authentication step
I1218 15:54:28.719369   65718 messenger.go:269] stopping messenger..
I1218 15:54:28.719418   65718 http_transporter.go:476] stopping HTTP transport
E1218 15:54:28.719433   65718 scheduler.go:421] Scheduler failed to authenticate: context deadline exceeded
I1218 15:54:28.720558   65718 scheduler.go:464] failed to authenticate with master [email protected]:5050: master changed
I1218 15:54:28.720602   65718 scheduler.go:838] authenticating with master [email protected]:5050
I1218 15:54:28.720809   65718 http_transporter.go:407] listening on 10.141.141.1 port 50852
I1218 15:54:28.724000   65718 authenticatee.go:260] Received SASL authentication mechanisms: [CRAM-MD5]
I1218 15:54:28.725372   65718 authenticatee.go:305] Received SASL authentication step

Interesting, mesos master log within the vagrant seems to suggest that master authenticates the example successfully and repeatedly:

I1218 23:40:56.703946 16091 master.cpp:4694] Authenticating scheduler(1)@10.141.141.1:50841
I1218 23:40:56.705118 16091 authenticator.cpp:92] Creating new server SASL connection
I1218 23:40:56.712682 16092 authenticator.cpp:197] Received SASL authentication start
I1218 23:40:56.712858 16092 authenticator.cpp:319] Authentication requires more steps
I1218 23:40:56.714017 16095 authenticator.cpp:225] Received SASL authentication step
I1218 23:40:56.714100 16095 authenticator.cpp:311] Authentication success
I1218 23:40:56.714175 16095 master.cpp:4724] Successfully authenticated principal 'fake_principal1' at scheduler(1)@10.141.141.1:50841
I1218 23:41:12.740057 16096 master.cpp:4694] Authenticating scheduler(1)@10.141.141.1:50847
I1218 23:41:12.740550 16096 authenticator.cpp:92] Creating new server SASL connection
I1218 23:41:12.746835 16094 authenticator.cpp:197] Received SASL authentication start
I1218 23:41:12.747391 16094 authenticator.cpp:319] Authentication requires more steps
I1218 23:41:12.749446 16092 authenticator.cpp:225] Received SASL authentication step
I1218 23:41:12.749568 16092 authenticator.cpp:311] Authentication success

The context deadline exceeded log line seems to suggest some message is not properly sent back from master to the scheduler?

Any help is greatly appreciate. Please feel free to request my vagrant files if that could facilitate investigation.

Many thanks!

[jdef]

what version of mesos are you testing against?