Please scroll to the bottom for the latest update on 18/3/2016.
Some people have run into difficulties connecting to SMB shares served by OS X servers running Open Directory. I saw this phenomenon myself with a clients’ machine that was able to connect to the SMB share using a Local User account on the server, but when attempting the same connection with an OD Local Network User account the logs returned errors akin to not logging in to the correct domain.
After searching various online communities I found that a small group of people had run into this problem but no-one had sussed out a solution yet. I banged my head against the wall for a while with this on the machine that was causing the problem without making any headway and then set up a new fresh VM of Windows 10 to test in my home lab setup. To my surprise, the machine logged in to the SMB shares on my OS X Server machine without issue.
Comparing the fresh VM to the machine that failed to authenticate, I noticed some differences in Security Policy settings. Many OS X admins who had Windows 7/8 machines that exhibited issues authenticating on earlier versions of OS X Server (10.7 mainly) used a set of “tweaks” to the Windows security policy to overcome these problems, and the client machine I was working with I suspect had these adjustments at an earlier point in it’s life.
The main “tweaks” were setting:
Network Security: LAN Manager Authentication Level to Send LM & NTLM -use NTLMv2 session if negotiated.
Network Security: Minimum session security for NTLM SSP Based (including secure RPC) Clients to unchecked (No Minimum).
I was re-pointed back to these settings via this site.
When examining the Local Security Policy on the fresh VM, I noticed that the first item, Network Security: LAN Manager Authentication Level, was set to Not Defined.
Returning to the problem machine, setting this item to Not Defined was not an available option but changing it to Send NTMLv2 response only did the trick. The machine now authenticates to the SMB Share with OD Local Network User accounts without issue.
Press the Windows key and “R” together on the keyboard and enter secpol.msc in the run dialogue.
Navigate to Security Settings / Local Policies / Security Options.
Find Network Security: LAN Manager Authentication Level, and set it to Send NTLMv2 response only.
Finally, ensure that Network Security: Minimum session security for NTLM SSP Based (including secure RPC) Clients is set to Require 128-bit encryption.
These settings got my problem machine singing the right tune, hopefully it will help yours as well!
UPDATE – 18/3/2016
A few people have contacted me to say that this process is no longer working, and I have seen myself on the Home version of Win10 that you cannot access the secpol.msc applet to edit security policies. Obviously, there have been many patches to both Windows 10 and OSX Server since this post was written back when 10 was first released. Having just worked with an end user on this issue I thought I’d post an update with what worked for us on this Win10 Home machine.
Watching the log from the server I saw this on failed authentication:
Mar 18 11:39:10 server.company.com digest-service: digest-request: user=MicrosoftAccount\\username
Mar 18 11:39:10 server.company.com digest-service: digest-request: kdc failed with 36150275 proto=unknown
Mar 18 11:39:10 server.company.com digest-service: digest-request: guest failed with -1561745590 proto=ntlmv2
So the Windows 10 client was forcing the OSX server to find the user’s account in an unknown domain called MicrosoftAccount. Let’s say that the OSX server’s Open Directory domain is “OD-DOMAIN”. I was able to get the user to authenticate correctly after clicking Use Another Account and entering “OD-DOMAIN\username” in the username field and then supplying the user’s password.
Mar 18 11:51:34 server.company.com digest-service: digest-request od: ok user=OD-DOMAIN\\username proto=ntlmv2 flags: ENC_128, NEG_VERSION, NEG_TARGET_INFO, NEG_NTLM, NEG_SIGN, NEG_TARGET, NEG_UNICODE
I hope this helps somebody!