Service Account Issues with SEPM 12.1.6 on SBS/DC Machine

Quick, quick note on a long and tired day…

In trying to (re)install Symantec’s Endpoint Protection Manager on a client’s SBS server (obviously containing a DC and other roles) I received this message, shamelessly copied from Symantec’s KB article:

SEPM-Error1

The message reads “Symantec Endpoint Protection Manager services require user rights in Windows domain security policies. The management console cannot run until you assign user rights to the services in the specified policies:”, in my case the GPO in question was the Default Domain Controllers Policy not giving NT Service\semwebsrv and NT Service\semsrv service accounts the SeServiceLogonRight permission as obviously would not be contained in that GPO.

The catch? I couldn’t add them – the GPO won’t let me add the account name because it doesn’t resolve, instead giving me the error The following accounts could not be validated: NT Service\semsrv:

SEPM-Error2

Some quick research showed me the sc showsid <servicename> command. Thinking I’d be slick, I did a sc showsid semwebsrv and sc showsid semsrv, copied the SIDs and pasted them into the GPO, did a gpupdate /force and clicked the “Try Again” button in the Symantec window – problem solved.

SEPM-Error4

Backscatter SPAM

A quick tidbit of information on Backscatter SPAM, as I fix a customer’s servers …

Backscatter SPAM is a way for Mr. Spammer to email Mrs. Victim indirectly. Mr. Spammer sends an email spoofed from Mrs. Victim’s account to a non-existent account on your domain and server. If things are set up (in)correctly, your email server will accept that email, find out it cannot deliver it and then send the email back to what it thinks the sender’s address is – Mrs. Victim – effectively delivering the message while trying to be helpful.

Here’s what a mail queue will look like full of Backscatter SPAM:

Backscatter

You’ll have a bunch of Undeliverable: messages sitting in your queue, not delivered because the remote end is refusing it (identifying it as SPAM) with a 400 4.4.7 Message Delayed error. You might even find yourself on a blacklist.

Fixing this is (fortunately) pretty easy on an Exchange server – here’s my four-step fix.

Turn off NDR Reports for External Domains

This is simple. Open the Exchange Admin Console and go to Organization Configuration -> Hub Transport. Once there, a tab for Remote Domains will be visible – click that, and configure each domain as shown:

Backscatter2

(For each remote domain, uncheck Allow non-delivery reports in the Message Format tab)

Turn On Recipient Filtering

Many people disable the Exchange Anti-SPAM options to reduce trouble, but the Recipient Filtering service can only help things – reject emails destined for non-existent mailboxes before you accept the message. In Organization Configuration -> Hub Transport, select Anti-spam. Right-click Recipient Filtering to enable it, then edit it as shown below:

Backscatter1

(In the Blocked Recipients tab, make sure Block messages sent to recipients that do not exist in the directory is checked.)

Clear out your Mail Queue

Open the mail queue appropriate for your Exchange (Exchange 2007-2013 have it in the Exchange Console) and delete any messages (obviously without NDR) that are stuck. Simple enough.

Check for blacklisting

Using a service like MXToolbox or the like, check your IP for any blacklist entries and perform the necessary steps to remove yourself – but only after the above steps, or you’ll end up right back on it – and sometimes, ending up back on it enough times can result in a permanent ban!

 

Cisco ASA IPSEC VPN Tunnel (with NAT!)

I’ve been asked about this a number of times on the job. You have to create a VPN tunnel between two ASA devices – using the command line. There’s also a twist – the IP address of your destination device(s) is already in use by someone else they’ve peered with, so you’ll need to NAT your device’s IP address(es) when speaking to the remote end.

Take this diagram:
CiscoVPNwithNAT1

Let’s make a few assumptions, then onto the configuration:

  • We’re routing between two hosts on the remote end and one on the local end
  • You’ll be using NAT to present your local host (192.168.1.20) as 192.168.40.97
  • Your remote peer is 205.206.207.50
  • Phase 1: ISAKMP, AES-256 Encryption, SHA1 hashing, Group 5, Lifetime=86400
  • Phase 2: ESP Encapsulation, AES256 Encryption, SHA hashing

My config (with explanations):

Group our remote addresses together:
object-group network REMOTE_VPN_IPs
 network-object 10.20.30.1 255.255.255.255
 network-object 10.20.30.2 255.255.255.255

This ACL is used to NAT traffic below:
access-list REMOTE_VPN_REWRITE extended permit ip host 192.168.1.20 object-group REMOTE_VPN_IPs

This ACL matches traffic for our VPN tunnel - notice the rewritten address:
access-list outside_5_cryptomap extended permit ip host 192.168.40.97 object-group REMOTE_VPN_IPs

This Static NAT statement re-writes traffic to the new address:
static (inside,outside) 192.168.40.97 access-list REMOTE_VPN_REWRITE

We need a transform set for our next statements:
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac

We're creating a unique crypto map (in our router, this is the fifth VPN tunnel):
crypto map outside_map 5 match address outside_5_cryptomap
crypto map outside_map 5 set pfs group5
crypto map outside_map 5 set peer 205.206.207.50
crypto map outside_map 5 set transform-set ESP-AES-256-SHA

We need to create an ISAKMP policy allowing Phase 1 traffic:
crypto isakmp policy 110
 authentication pre-share
 encryption aes-256
 hash sha
 group 5
 lifetime 86400

And our tunnel-group:
tunnel-group 205.206.207.50 type ipsec-l2l
tunnel-group 205.206.207.50 ipsec-attributes
 pre-shared-key Pr3$h@ReDK33

Hope this helps someone!

Flashing a Cisco 1130AG WAP to Autonomous Mode

The Cisco 1130AG can ship as a lightweight WAP, designed to connect to a Wireless LAN Controller for its configuration and setup. However, sometimes you just want it to act like a basic WAP – this is how you reflash the device as such.

  1. Probably the most annoying/difficult step – download the firmware from cisco.com for your specific model.
  2. Download and install a TFTP Server. I use 3CDaemon, but it appears discontinued. You might try this TFTP Server.
  3. You’ll want to connect a laptop or desktop to the same L2 Collision Domain (same VLAN, etc) as the WAP(s). On that NIC, configure an IP address of 10.0.0.2 (or .3->.30), /24.
  4. Start your TFTP Server. Place the firmware you downloaded in whatever root folder is used by the TFTP software and rename it c1130-k9w7-tar.default (different models will have different file names).
  5. Unplug the device, and then plug it back in while holding the Mode button down. Wait 20 seconds until the R light turns solid red. At this point:
    1. The WAP will assume an IP Address of 10.0.0.1;
    2. It will attempt to download the above filename using TFTP from 10.0.0.2
    3. If it does not connect or see that file, it will repeat for 10.0.0.3 through 10.0.0.30.
    4. Once the file is downloaded, it will flash and reboot to factory defaults.
  6. Check your DHCP Server for a lease matching the device’s MAC address (mine began with 68:EF:BD) and telnet to the device’s IP.
  7. Log in with Username = Cisco, Password = Cisco.
  8. Copy your favorite config file over.

Here’s what the Cisco 1130AG looks like:
Cisco_1130AP

Speaking of which, a typical configuration might look like this:

!
! Last configuration change at 18:15:28 UTC Thu May 14 2015 by Cisco
! NVRAM config last updated at 18:15:31 UTC Thu May 14 2015 by Cisco
!
version 12.4
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname wap-hostname-here
!
logging rate-limit console 9
enable secret 5 <Redacted>
!
no aaa new-model
ip domain name your-domain.local
ip name-server 192.168.1.10
ip name-server 192.168.1.11
!
!
dot11 syslog
!
dot11 ssid YOUR-SSID-HERE
 authentication open 
 authentication key-management wpa version 2
 guest-mode
 wpa-psk ascii 7 <Redacted>
!
!
!
username Cisco privilege 15 password 7 <Redacted>
username second-admin privilege 15 password 7 <Redacted>
!
!
bridge irb
!
!
interface Dot11Radio0
 no ip address
 no ip route-cache
 !
 encryption mode ciphers aes-ccm tkip 
 !
 ssid YOUR-SSID-HERE
 !
 station-role root
 bridge-group 1
 bridge-group 1 subscriber-loop-control
 bridge-group 1 block-unknown-source
 no bridge-group 1 source-learning
 no bridge-group 1 unicast-flooding
 bridge-group 1 spanning-disabled
!
interface FastEthernet0
 no ip address
 no ip route-cache
 duplex auto
 speed auto
 bridge-group 1
 no bridge-group 1 source-learning
 bridge-group 1 spanning-disabled
!
interface BVI1 (Management Address)
 ip address 192.168.1.12 255.255.255.0
 no ip route-cache
!
ip default-gateway 192.168.1.1
ip http server
no ip http secure-server
ip http help-path http://www.cisco.com/warp/public/779/smbiz/prodconfig/help/eag
logging 192.168.1.254 (Syslog Server)
bridge 1 route ip
!
!
!
line con 0
line vty 0 4
 login local
!
sntp server 192.168.1.10
sntp server 192.168.1.11
end

 

Simultaneous VPN Users on Cisco ASA

A quick-and-dirty post for Cisco ASA Users who have people logging in from different devices – say, a doctor’s office with doctors who might like their tablets, laptops and home computers connected all at once…

Normally, they are are limited to one simultaneous session per username at a time, but you can change this (by the user or by the tunnel policy).

By Username:

test-FW(config)# username example-name password test-password
test-FW(config)# username example-name attributes
test-FW(config-username)# vpn-simultaneous-logins 4
test-FW(config-username)# exit

By Policy:

(Find Policy)
test-FW# show run tunnel-group
[Deleted]
tunnel-group SSLAccess type remote-access
tunnel-group SSLAccess general-attributes
 default-group-policy SSLVPN
tunnel-group SSLAccess webvpn-attributes
 group-alias RemoteUsers enable

test-FW# config t
test-FW(config)# group-policy SSLVPN attributes
test-FW(config-group-policy)# vpn-simultaneous-logins 3

Don’t forget to write mem or copy running-config startup-config!

Interface Statistics on a Sonicwall

I absolutely love the Sonicwall products as you might have guessed. One of my peeves is hearing that you can’t show interface statistics – you can see that and so much more! You just have to know where to look.

Just open up SSH Management access on any interface, then connect to it with an SSH client like PuTTY – look at all of the fun stuff you can show in this little IOS-like environment:

TZ 210> show
access-rules device nat syslog
address-group firmware netstat system
address-object gms network tech-support
alerts ha pp-stats timeout
all hw-stats processes tracelog
arp interface route tsr
ars language security-services vpn
baud log service wd-stats
buf-memzone log-categories service-groups web-management
build-info log-filters session zone
continuous mem-pools sonicpoint zones
cp-stats memory ssh
cpu memzone sslvpn
debug messages status

Can’t show interface statistics? I bet you’re wrong:

TZ 210> show interface statistics X1
Interface statistics for X1
 InDiscards : 0
 InNUcast : 2188
 InUcast : 2336686
 InOctets : 2673340832
 InErrs : 0
 OutDiscards : 24
 OutNUcast : 139
 OutUcast : 1608500
 OutOctets : 338528455
 OutErrs : 0
 InUnkProto : 0
 InMcast : 824
 InBcast : 1364
 OutMcast : 0
 OutBcast : 139

There’s show system to get basic system info, show device to get a quick and dirty summary of the configuration, show messages to see those little status messages when you log into the device, show log to see a list of administrative events, show vpn policy to dump the current VPN tunnel configurations, …

Trend WFBS Mail Security Breaks Exchange Transport On Uninstall

I have a customer for whom we just uninstalled Trend WFBS with the messaging security component. I sent and received test emails just fine, all appeared well. Until a reboot, when these events popped up in the event log:

MSExchangeTransport #16023:
Microsoft Exchange couldn’t start transport agents. The Microsoft Exchange Transport service will be stopped. Exception details: Failed to create type ‘TrendMicro.SMEX.hookE12TransportAgent.hookE12SmtpReceiveAgentFactory’ from assembly ‘C:\Program Files\Trend Micro\Messaging Security Agent\hookE12TransportAgent.dll’ due to error ‘Invalid agent assembly path.’. : Microsoft.Exchange.Data.ExchangeConfigurationException: Failed to create type ‘TrendMicro.SMEX.hookE12TransportAgent.hookE12SmtpReceiveAgentFactory’ from assembly ‘C:\Program Files\Trend Micro\Messaging Security Agent\hookE12TransportAgent.dll’ due to error ‘Invalid agent assembly path.’. —> System.ArgumentException: Invalid agent assembly path.

MSExchange Extensibility #1052:
The creation of an agent factory for the agent ‘ScanMail SMTP Receive Agent’ failed with error ‘Failed to create type ‘TrendMicro.SMEX.hookE12TransportAgent.hookE12SmtpReceiveAgentFactory’ from assembly ‘C:\Program Files\Trend Micro\Messaging Security Agent\hookE12TransportAgent.dll’ due to error ‘Invalid agent assembly path.’.’. Verify that the corresponding transport agent assembly and dependencies with the correct version are installed.

The cmdlet to check for transport agents is Get-TransportAgent – let’s see what it says:

[PS] C:\Windows\system32>Get-TransportAgent
Identity Enabled Priority
-------- ------- --------
ScanMail SMTP Receive Agent True 1
ScanMail Routing Agent True 2
Transport Rule Agent True 3
Journaling Agent True 4
AD RMS Prelicensing Agent False 5
Connection Filtering Agent True 6
Content Filter Agent True 7
Sender Id Agent True 8
Sender Filter Agent True 9
Recipient Filter Agent True 10
Protocol Analysis Agent True 11

And the ScanMail components?

[PS] C:\Windows\system32>Get-TransportAgent ScanMail* | fl
Identity : ScanMail SMTP Receive Agent
Enabled : True
Priority : 1
TransportAgentFactory : TrendMicro.SMEX.hookE12TransportAgent.hookE12SmtpReceiveAgentFactory
AssemblyPath : C:\Program Files\Trend Micro\Messaging Security Agent\hookE12TransportAgent.dll
Identity : ScanMail Routing Agent
Enabled : True
Priority : 2
TransportAgentFactory : TrendMicro.SMEX.hookE12TransportAgent.hookE12RoutingAgentFactory
AssemblyPath : C:\Program Files\Trend Micro\Messaging Security Agent\hookE12TransportAgent.dll

Let’s uninstall those bad boys and see if that fixes it:

[PS] C:\Windows\system32>Get-TransportAgent ScanMail* | Uninstall-TransportAgent

A service restart confirms – resolved without further errors!

Why You Should Always Use Root Hint Lookups (and avoid DNS Forwarding)

I’ve had this on my mind a while, but never fully published this until now. I’ve been called out a number of times to various customers who have internet connection issues… and found bad DNS setups the culprit, oftentimes DNS Forwarding. I’ve also argued over more than a few beers about using them.

I also want to mention: I know IT people like to pontificate. I’m talking about companies with 10-2,000 users in a typical Active Directory environment. I’m not talking about companies with 2,500,000 users and external-facing infrastructures or service providers who offer DNS as a service. Those companies are likely to use advanced technologies way outside the scope of your typical Microsoft DNS Server setup and the people who implement those technologies aren’t likely to be reading my blog to understand far simpler setups.

Why You Should Not Use DNS Forwarders:

  • DNS Servers Change
    … and they change well after you’ve forgotten about your customer. If you’re even aware of the changes, will you remember every customer you’ve set forwarding up for using that IP?
  • DNS Servers Go Down
    Here in Rochester NY, we’ve had a few circumstances where our area’s primary DNS servers have been down. Granted, it’s 15-20 minutes at most, but that’s enough to anger people. Meanwhile, your internal DNS Server needs to be up using Root Hints or Forwarding anyway, so why have the added dependency?
  • You Marry Your ISP
    If I had a dollar for each time someone has changed ISPs and then “went offline” I’d … well, I’d be worth about $15. Regardless of how you value your money, you will have a server you forgot about lose DNS when you change your ISP and your forwarders no longer work.
  • WAN Failover Breaks
    A half-dozen times or more, I’ve assisted engineers with WAN Failover setups that didn’t work when the primary connection went down. Why? Because the DNS Forwarding was configured using the Primary ISP’s DNS server and when they failed over, those DNS Servers stopped answering queries. Worse, people then just slap Google’s 8.8.8.8 in there to “fix” it.
  • Problems Are Twice As Complex
    I had a school district who couldn’t load specific websites. After some packet-level analysis, I figured out why: the Windows DNS Server and their ISP weren’t playing nicely when using DNSSEC, which the specific sites had configured. The fix was either disable DNSSEC on the Windows DNS Server or use Root Hints. Luckily, they’d recently set up WAN Failover so I could give them my lecture on why it was a bad idea.
  • It’s More Secure
    ISP DNS Servers are going to be the first targets of hackers trying to redirect your banking traffic, vendor logins or other DNS queries to their malicious servers. Your DNS Server is secured nicely behind a firewall and hackers are much more likely to go after the machines in your ISP’s DNS pool.
  • You’ll Never Get Answers from Servers That Aren’t There
    I had a customer continuously receive certificate errors whenever they opened Outlook. After investigation, I discovered that the autodiscover.domain.com record was receiving replies – when it shouldn’t have been. Why? Because the ISP’s DNS server was redirecting traffic from the non-existent domain to their helpful pain-in-the-ass search website, which was responding on HTTPS with an invalid certificate. Sure, there was some fault in the Exchange setup but this side-effect shouldn’t have happened at all. It’s also happened when someone loads http://domain.com and the record doesn’t exist, and in other situations. Using Root Hints means you’re going to get a real, honest DNS lookup.
  • You Get Live Data
    When you query against the Root DNS Servers, you get the most recent information possible. I’ve had nameserver records change for domains that required 72 hours for an end-user to properly resolve because their ISP has (likely inappropriate) levels of caching and returned the wrong values when forwarded.
  • It’s Idiot Proof
    Seriously, direct lookups have been working flawlessly since … well, the mid-1980s. There’s no compelling reason to go the more complicated route with forwarders. This is pretty simple and very reliable technology, there’s no reason to rely on your ISP to get it right when you can.
  • You Probably Set It Up Wrong
    I’d like to ask Microsoft why, in their wisdom, they left the default timeout for DNS Forwards at 3 seconds. Most DNS resolvers time out after 2 seconds, so the second or third DNS server you’re forwarding to is never going to really service the end user on a lookup unless you drop this to 1 second. Also, have you looked at caching and other considerations? While not really complex, there are some considerations I never see accounted for when forwarding is set up. Again, let the DNS Server do its job.

DNS Forwarding Myths:

  • It Reduces Traffic
    This is silly – there’s very little difference in traffic between lookups. Recursive or not, the results are cached for the next lookup – the majority of DNS queries come from the cache after an initial loading period.
  • It Reduces Server Load
    Unless you’re familiar with your DNS Server’s source code, this argument is silly and uninformed. Who is to say that a recursive lookup through an external DNS server is any more or less computationally intensive on a machine than a non-recursive lookup? And unless you’re processing tens of thousands of DNS queries a second, it won’t matter.
  • It Prevents DNS Exposure
    … just wrong, wrong, wrong. Your DNS Server is going to resolve any query for a name it doesn’t know about – whether forwarding or using root hints. If your DNS server is attempting to resolve internal hostnames through your WAN port you have it configured wrong.
  • It’s More Secure
    Some people claim that by going to your ISP’s DNS Server, you’re reducing exposure. Firstly, DNS is a pretty simple (and pretty darned secure) protocol, particularly when it’s behind a firewall and not answering public queries. If a potential hacker has the ability to modify or inject malicious packets into lookups done through a DNS root server, why could they not do the same with your ISP’s DNS server addresses?
  • It Prevents Cache Poisoning
    Since Server 2003, Microsoft’s DNS server has had protections against this by default. But what do you trust more to provide a reliable DNS query/response – your ISP’s DNS servers, or the internet’s heavily fortified root servers? And if the root servers are compromised, would your ISP’s results not also be in question?

I’ll state again – there are plenty of times when you want to use DNS Forwarding. If I had a campus with dozens of buildings or many floors of users, I might have my various internal servers forward external lookups to a DNS setup somewhere else to minimize the WAN traffic of 40 DNS servers (where caching at another level would really benefit things), but those purpose-built servers would be querying the root servers for answers…

The Woes of a Base Cisco ASA License

Though past the end-of-life announcement, the Cisco ASA 5505 is still a common router to see. I feel as though just yesterday I was installing them regularly. Today I’m recommending their replacements. Gosh, I feel old…

Anyway, the ASA5505 came with a base license that was – essentially – a total turd. Sure, it has the awesome ASA feature set, but it came with a limitation of 10 users and 1 LAN (and a DMZ, but you weren’t allowed to route traffic between them openly). But it was cheap and has the Cisco name on it and as such, many users bought them. Later on, they’ll add a printer, a credit-card machine and a little file server and boom – they’re over a long-forgotten limitation of ten hosts.

If you have a number of machines on a LAN that experience intermittent connectivity with the outside world (and one PC you test from never seems to go down), that’s because the Cisco has hit the license limitation for network hosts and is preventing other machines from getting online.

Enable console or buffered logging at (I believe) a debug level and check for messages like this one I stole from another blog:

11:29:05 450001 24.106.9.206 80 Deny traffic for protocol 6 src outside:216.81.128.197/23580 dst inside:24.106.9.206/80, licensed host limit of 10 exceeded

You can also issue a show local-host command to see the host limit and current host count:

someones-asa-5505# show local-host
Detected interface 'outside' as the Internet interface. Host limit applies to all other interfaces.
Current host count: 12, towards licensed host limit of: 50
[...]

There’s also the show activation-key command to see what you’re licensed for:

someones-asa-5505# show activation-key
Serial Number: ABC1234ABCD
Running Activation Key: [Redacted]

Licensed features for this platform:
Maximum Physical Interfaces : 8
VLANs : 20, DMZ Unrestricted
Inside Hosts : 50
Failover : Active/Standby
VPN-DES : Enabled
VPN-3DES-AES : Enabled
SSL VPN Peers : 2
Total VPN Peers : 25
Dual ISPs : Enabled
VLAN Trunk Ports : 8
Shared License : Disabled
AnyConnect for Mobile : Disabled
AnyConnect for Cisco VPN Phone : Disabled
AnyConnect Essentials : Disabled
Advanced Endpoint Assessment : Disabled
UC Phone Proxy Sessions : 2
Total UC Proxy Sessions : 2
Botnet Traffic Filter : Disabled

This platform has an ASA 5505 Security Plus license.

Server 2012R2 Install Issues on Hyper-V using Dynamic Memory

Posting this hoping I can help someone else out. I had a customer who wanted a Server 2012R2 VM built for a new accounting application… so I downloaded the ISO, created a VHD and built the VM using Dynamic Memory with a 512MB . Upon booting, I get as far as clicking “Install Now” when I received a similar error message:

 

Server2012R2ErrorWithDynamicMemory

For Google, it reads: “Windows installation encountered an unexpected error. Verify that the installation sources are accessible, and restart the installation.”

The error in the screenshot is easily searchable and resolvable, but initially I had a different error. The screenshot reads “Error code: 0xE0000100″, but when I first ran into this problem it read “Error code: 0xC0000005“. As I write this post, I wonder why the error code changes after one successful startup of the installation program, but alas …

If you try to install Server 2012R2 as a Hyper-V Guest and you get the “Error code: 0xC0000005″ message using Dynamic Memory – check your Startup RAM value. Despite this TechNet Article stating that 512MB is the minimum needed for Server 2012R2, there must be a RAM Disk or additional software that is utilized during the install process. I was successful with a Startup Memory value of 1024M. However, that was after I spent over an hour checking to make sure the HyperV services had access to the ISO files, re-downloading the ISO twice, checking MD5 hashes of the file against a known-good and making sure the VHD files were not corrupt…