View 5 Personas with Forefront Endpoint Protection

Posted on by
Reply

I have been trying to find a resolution to this for several weeks, scroll down for a full description of the performance problem that FEP causes with View 5 persona management. The issue found its way to the the View product manager. A GPO setting that specifically addresses this problem was accidentally left out of the ViewPM.adm file included with View 5.0 build 481677. The policy setting is called “Excluded Processes”. The description of the setting is: “Excluded processes are processes whose i/o is ignored by Persona.  Certain anti-virus applications might need to be added to prevent performance problems.  If an anti-virus application does not have a feature to disable offline file retrieval during its on-demand scans, this setting will prevent it from unnecessarily retrieving files.  However, changes to files/settings in the users’ profiles made by excluded processes are still replicated.” Using this setting eliminates the need to create any exclusions inside of FEP, so C:\Users will still be fully protected.

The solution for FEP is to use this GPO setting to exclude the FEP process MsMpEng.exe.  Here is the replacement ViewPM.adm file. There is also a viewPM_adm_patch that I created with WinMerge, you can apply the diff to the ViewPM.adm file that installs with View 5.0.

Here are 2 screenshots of the missing setting.

Special thanks to Kevin Goodman at VMware for providing this solution!

Update 1/12/2012
I opened a ticket with Microsoft support and they advise that there is no way to prevent the download and scanning of offline files. They have escalated the ticket internally to look for any way this could be supported, but the engineer is setting expectations that there will be no resolution. It appears that you can not use personas with FEP unless you disable scans of C:\Users. That’s not a risk that I’m going to recommend taking, so it looks like it’s roaming profiles for my View clients who use FEP.

Update 1/10/2012
The support engineer assigned to my case came back with this:

I was able to replicate the behavior you saw while working with persona management and FEP server. The reason I was able to identify is on logon FEP is downloading all data to the local machine to be scanned and thereby causing the delay.

Our expectation is that virus scanners will ignore scanning offline files, but not a case with MS FEP. I was unable to find any such option with FEP client to ignore scanning offline files. Hence this can only be worked around by excluding the folder to be scanned on individual VMs. This does not however completely compromises security as the file share where the persona eventually will be stored is being protected with FEP anyway.

Persona management works by selecting/clearing offline file attribute for the contents of entire %userprofile% folder.

I opened a ticket with Microsoft support with the above information. Setting up an exclusion on C:\Users prevents realtime scanning on that folder and I don’t think I want to let a potential virus sit out there unchallenged until the profile syncs up to the file server.

1/1/2012
I am deploying a new View 5 setup and the client has Forefront Endpoint Protection. After enabling persona management for my Windows 7 image, I found that the logon process hung at “Welcome” for 3+ minutes.

I found this VMware communities thread suggesting that MS Forefront Endpoint Protection was the culprit. I enabled debug logging and tried logging on with FEP both enabled and disabled. The debug logs are identical other than the amount of time spent between log entries. With FEP disabled, files in C:\Users are touched quickly, less than 0.1 seconds between log entries. With FEP enabled, files in C:\Users have a 3-5 second delay between touches. This leads to the 3+ minute logon delay.

The VMware community post suggested excluding C:\Users from scanning – I tested that configuration and it does resolve the logon slowness problem – logon completes within 5 seconds and profiles are synced successfully. Obviously, this comes at the expense of the security of the environment – I wouldn’t go to production excluding the entirety of C:\Users from antivirus scans.

I’ll be opening up a VMware support ticket, but given the deadline I’m working under it looks like roaming profiles are in my near future. Disappointing because I was impressed with the speed of the View persona management when FEP isn’t in the way.

Google Apps in the Enterprise – Not for the Faint of Heart

Posted on by
Reply

I helped with a massive Google Apps deployment to over 30,000 users at my last job with Fortune 250 RR Donnelley. We were moving away from Lotus Notes, which I was happy to be rid of. However, in retrospect, Notes is a relative joy to manage compared to Google Apps. Although being “in the cloud” is trendy and provides your management with a cool golf course buzzword, supporting Google Apps in the enterprise is nightmarish and should not be taken lightly.

Pluses

  • Email and calendar syncs with multiple devices –  Native apps for Android and Blackberry, and IMAP support for the iFolks.  No need for Blackberry Enterprise Server. Coming from an environment where only upper level management got Blackberries, having my calendar and e-mail on my phone was a huge plus.
  • Multiple editors – Google Docs lets you have multiple users in the same document editing simultaneously. This proved highly useful for many collaborative efforts.
  • Education – Provides a low cost, easy way to provide email accounts and collaboration tools for a constantly changing set of users.

Minuses

  • User Deletion – When you delete a user, it’s 5 days before you can put that user back in. It was an accident? Too bad. That user is without their e-mail account for 5 days.
  • Document Deletion – When a user is deleted, so are all their documents. Seriously. Every document they own is *poof* – gone.
  • Management Delegation - There are no degrees of admin rights – you’re either an admin or you’re not. Although you can do some interesting thing with delegation using Google Organizations, many corporations have a centralized helpdesk. If you want any helpdesk worker to be able to help any employee, the helpdesk worker has to be a full blown administrator over the entire Google Apps domain. This is not a particularly palatable choice when you’re talking about hundreds of outsourced workers.
  • Bandwidth – Your bandwidth costs are going up. WAY up. Using Lotus Notes, a file attachment sent within the company never left the WAN. With GMail, the sender has to upload the attachment to the Internet, and then each user who receives it has to download it over the Internet. Every time people check their email, it’s Internet bandwidth and not WAN bandwidth. That’s thousands of users generating Internet traffic that didn’t exist with Notes
  • Constant Development – Google Apps is continuously changing, and changes are pushed to Production with no warning. Things break overnight and it irritates the hell out of your users, and drives up support costs for IT.
  • No Logs – Logging is nonexistent. There’s no event viewer, no activity log, nothing. You have no idea who is doing what and no way to look it up
  • Extracting Information - You can’t even get a simple list of who your domain’s administrators within the Google GUI. If you want to extract the information, you’ll be writing a script. They have libraries in Java, JavaScript, .NET, PHP, Python, and Objective-C. If you’re a system admin, you’re probably not a programmer, so you’d better brush up on your Python. You’re not likely to make it past the learning curve to be able to use anything else.
  • Emergency support – You can’t get support without your support PIN, and you can’t get that without administrative access to your Google domain. The PIN changes periodically, so you can’t rely on having written it down.  If you’re locked out of the domain, you’re stuck with filling out a form and waiting for e-mail support to get back to you.
  • The apps – Using the apps feels a lot like using Office 1.0. They’re rough, good enough for the very basics but not really ready for primtime. Your accounting department would stage a revolt if you forced them to use Google Spreadsheets instead of Excel
  • Enterprise Integration
    • Passwords – You can use the Google Apps Directory Sync (GADS) tool to automate user provisioning up to Google Apps by syncing your existing LDAP infrastructure with Google. However, passwords are quite a challenge. If you have a way to extract the user’s password out of your LDAP directory, you can push that up to Google – but who wants Google to have a copy of the user’s corporate password? So you’re left with maintaining a separate Google password. Even if you implement Single Sign-On and authenticate users against your own LDAP server, that only applies to access via web browser.  The separate Google password is still used for mobile device access.
    • Single Sign-On – If you decide to go the SSO route, you have to have a secure way to store a user’s e-mail address within your directory structure. With SSO, no user credentials are sent to Google – instead, Google uses a certificate to ask your company’s SSO server to authenticate you. Your SSO server must send the user’s e-mail address back to Google. This directory attribute must be created and secured, otherwise anybody with access to that attribute can impersonate any user. Let me repeat that – *ANY* user in the organization with access to the “email” property in your directory system can use SSO to read *ANY* user’s e-mail. For many enterprises, this means an unacceptable number of non-privileged IT workers have the ability to read anybody’s e-mails. Google does not offer any kind of AD schema update like an Exchange installation would to create a properly secured set of properties for e-mail address, so you are forced to create the properties manually and come up with a set of ACLs to achieve the proper security. This can be extraordinarily tricky in a large enterprise.

While Google will try to sell you Google Apps as a simple solution, neither the implementation nor the ongoing maintenance are simple. The Google resources assigned to the implementation project were simply not up to the task of navigating the complex political and technical landmines of a Fortune 250 corporation. The resources weren’t necessarily at fault – it was Google’s fault for several reasons. The resources assigned were simply too junior with insufficient experience at the enterprise level. They also had no idea how to address many of the enterprise level problems we encountered because they had never had to deal with a corporation the size of RRD. We felt like we were flying by the seat of our pants with little to no direction from Google.

Google Apps a great choice for a small company and possibly for education, but it’s just not ready for the enterprise.

Free vSphere 5 ICM class giveaway

Posted on by
Reply

Just received this from Chicago VMUG leader Chris Wahl:
——
Want to get VMware certified in vSphere 5? The Chicago VMUG can help!

Have you been looking at the VMware Certified Professional (VCP) in vSphere 5 and wondering “how the heck do I save up $3000 to attend the required course”? It’s crossed a lot of minds and has been a topic I’ve heard often from colleagues and members of this group. In this day of tightened budgets and spend freezes, it can be nearly impossible to get your employer to justify sending you to Install, Configure, Manage on vSphere 5 in order to get your VCP5.

The Chicago VMUG is looking to help!

Thanks to a really solid sponsor showing at our upcoming VMUG Conference (October 31st!) we are putting aside funds to make sure one grand prize winner gets their Install, Configure, Manage on vSphere 5 class paid for, in full, on the date and location of their choosing. We only ask that you be working towards getting your VCP5 and not be eligible for the exam already (this excludes VCP4s who can take the exam without a course requirement until February 2012 and those who have already taken a qualifying vSphere 5 course).

What do you have to do to enter? Simple, register for the Chicago VMUG conference (link is attached to this post) and then stop by the VMUG booth at the day of the conference. We’ll scan your badge and pick one lucky winner to get trained! The VMUG staff will contact you to set up the details after the conference, so that you can pick the right time and place.

Already a VCP4 or VCP5? Please spread the news to a friend, co-worker, or your twitter followers! For more breaking news I also invite you all to follow the @ChicagoVMUG twitter account.

Best of luck to those who enter!

Event Registration (myvmug.org)
—–

My VCAP-DCA exam experience

Posted on by
2

9/29/2011
I passed! I am VCAP-DCA #421.

I initially found the test intimidating because I didn’t think that anybody outside of a consultant could possibly gain enough experience to pass this test. I wanted to update this to let people know that you don’t have to be a consultant. I was an admin at a Fortune 500 for my first attempt, and I’d only been doing consulting for a month when I took it the second time. Build a lab, put in your study hours and you can achieve the cert!

9/13/2011
I took the VCAP-DCA exam for the second time today because I failed my first attempt by 13 points. I walked out of my first attempt feeling like I had been run over by a truck. Today, I only felt like I had been pummelled by baseball bats. It is without a doubt the most difficult test I have ever taken. I am reasonably certain that I passed – I know I got more questions right than in my first attempt.

The version 4 exam remains the only version of the DCA available and it’s not clear when version 5 will become available. The exam is 225 minutes long and consists of 100% lab questions. You sit down at a remote desktop and you have a vCenter, 1 ESX host, 1 ESXi host, a vMA appliance, and the PDF documentation. No multiple choice, no guessing – you either have the knowledge to make the requested configuration changes or you don’t.

Many others have posted good exam reviews, including David Davis and Eric Sloof.  Some of my bits of advice are:

  • I found Sean Crookston’s study guides to be immensely useful.
  • Most reviews say this, but I will say it again – there is a lot of configuration to do and very little time. I ran out of time on my first attempt. There is no interface to “review later” like you can on the VCP exams, so you have to track your questions manually. On my second try, I used the Pearson-provided dry erase board to keep track of which questions I needed to revisit. Before I started the test, I wrote the digits 1 through 40 on my board. During the test, I marked an X through the ones I was confident on, and circled the ones I needed to come back to.
  • Certain tasks run for a while. Don’t sit and wait! Mark down the question number and move on to the next question.
  • Limit yourself to no more than 3 minutes per question the first time through. That will get you to the end of the test with almost 2 hours left to go back and tackle the tougher questions.  On my first attempt, I burned over 20 minutes on an early question; bad enough to waste time, but it also made me frustrated and threw me off for a while afterward.
  • You have to spend hours in a lab or you won’t be able to pass the test. I ran my entire lab on my laptop inside of Workstation – a domain controller, vCenter, ESX and ESXi host, vMA, and Openfiler for iSCSI.  It was slow as hell when it booted up, but it worked well enough. I’d rather have some kind of dedicated whitebox, but it’s not in my budget right now.
  • The lab engine during the test makes it possible for the edge of the vSphere client to be off the screen. This could end up hiding something important, or make you think you are losing your mind because you can’t find the place to make a configuration change. If you are feeling lost, step back and check your vSphere client.
  • Spend a little time learning what information is the official PDFs. The documentation is all sitting in a folder, i.e. a bunch of files like ”vsp_40_config_max.pdf” sitting in a folder. If you can’t remember how to do something, it helps to know where to look it up quickly. You don’t have enough time to open them all and search.
  • It’s 225 minutes. There is no bathroom break.
  • The Advanced Fast Track course was great, though quite expensive, preparation for the exam. Fortunately the training was employer-paid.

Hopefully I get a passing grade – nothing to do now but wait.

My VCP5 beta exam experience

Posted on by
1

I received an invitation on July 8th, 2011 to participate in the VCP5 beta exam. The availability period was from July 12th through July 24th, and registration opened on the 12th. I had been participating in the VCP5 beta program for a few weeks, I had installed and configured the product and had about 10 hours of hands-on experience when I received the invitation.

There weren’t all that many participating Vue locations, but that’s how it works with betas I guess. The exam consisted of 180 questions over 225 minutes – only 48 seconds per question. Seemed like nowhere near enough when I read about it, but it turned out to be plenty of time.

The exam had the familiar mark and review later interface. Beta exams have a comments button for every question – VMware expects the beta participant to give feedback on each question. Results did not arrive until today, 9/7/2011. I’m told that 7 weeks is quite a bit faster than it has been for past beta exams. As this was my first, I don’t have anything to compare it to.

I decided to follow my typical certification exam strategy – go through the exam as fast as I can. If I’m not sure of an answer within a few seconds of reading the choices, mark it for review and move on. Making comments on the questions slowed me down a bit, but I stuck with the strategy. By the time I made it to the end, I had about 50% that I thought were correct. I then went back and reviewed my questions. I walked out of there thinking that I had barely passed, and I chalked that up to a lack of preparation time. I would typically spend 6-8 weeks getting ready, 50+ hours of study time. For this I only put in about 15 hours.

When I took the beta, I found it to be extremely difficult. I think my perception of the difficulty of the exam was related to the lack of study time, coupled with the fact that I saw 2.5x more questions than the standard VCP5 candidate. Looking back, I do think it’s a bit harder than the VCP4 – it does require some hands-on knowledge. I don’t think you can pass the exam without actually playing with ESXi.

In the end, I passed!

VMware Tools Install Failure

Posted on by
Reply

I’ve installed VMware tools hundreds of times and never encountered this one before… the VMware tools installation fails on XP with “Setup failed to install the XXXX driver automatically.” The VMXNet, Mouse, and Video drivers all fail.

It’s because HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce had been deleted. Manually create the key and the problem is fixed.

VMware KB Article 1006382

APC PDU configuration trick

Posted on by
Reply

I was trying to configure an APC7811 PDU, which has a network port for remote access. It’s a metered PDU so you can log in remotely and look at power consumption.

I’d never had to configure one before so I just popped in the CD that came with it and tried to configure an IP address by using the discovery tool. Regardless of which IP address I tried, I kept getting “The IP is already in use on the network.”

This configuration method is in the documentation, but it’s buried at the end… I think I’ll use this every time now as it’s fast and doesn’t require the vendor’s CD.

The MAC address is on the back of the PDU and on the Quality Assurance slip included in the package.

  1. Use ARP to define an IP address for the PDU,
    and use the PDU’s MAC address in the ARP
    command. For example, to define an IP
    address of 156.205.14.141 for the PDU that has
    a MAC address of 00 c0 b7 63 9f 67, use one
    of the following commands:

    Windows command format:
    arp -s 156.205.14.141 00-c0-b7-63-9f-67

    LINUX command format:
    arp -s 156.205.14.141 00:c0:b7:63:9f:67

  2. Use Ping with a size of 113 bytes to assign the
    IP address defined by the ARP command. For
    example:

    Windows command format:
    ping 156.205.14.141 -l 113

    LINUX command format:
    ping 156.205.14.141 -s 113

  3. Use Telnet to access the PDU at its newly
    assigned IP address. For example:

    telnet 156.205.14.141

  4. Use apc for both user name and password.

At this point the IP address is configured for the interface, you can browse to the IP via HTTP. Don’t forget to delete your static ARP entry.

VMware View – Persistent / Non-Persistent

Posted on by
Reply

We are struggling with how to implement VMware View. If you use a persistent pool, you waste resources by having resources provisioned for users who aren’t doing any work at the time. You also run into problems maintaining the user profile – you can use a dedicated user disk, but how do you ensure a good backup? It’s also an administrative annoyance having to disconnect and reconnect user data drives. On the plus side, users can be given admin rights to install applications

With a non-persistent pool, desktops are spun up as needed based on demand, and they can be destroyed automatically when the user logs off. You are forced into using roaming profiles with folder redirection, which creates its own set of administrative difficulties. Users aren’t able to install applications as nothing is persistent, so the administrator has to install all apps inside the pool template. Alternatively, you can use application virtualization and try to ThinApp an application. However, my experience with ThinApp has been less than impressive. When it works, it works quite well. When it doesn’t work, it’s a baffling ritual of hocus pocus trying to get the app to work. In the end, it just doesn’t work for the majority of our software because the vendor won’t support ThinApp. When you call a 3rd party vendor for support for a problem and try explaning that it was installed via ThinApp, the call will often end with a statement that ThinApp isn’t supported.

Refusal to support ThinApp does sound a lot like when vendors refused to support apps installed in a VM – they eventually came around. But until that point, we’re stuck with what the vendor is willing to support.

Windows 2008 primary IP – DNS registration

Posted on by
7

Update 2011-07-16
We filed a request to have the 2008 R2 solution listed below backported to Windows 2008 SP2, but the request came back denied yesterday. The escalation engineer’s comment: “Considering that Windows Vista and 2008 SP1 hit End of Life this month, the resources were not available for a fix.” That will close the book on this issue – if you need the SkipAsSource functionality in Windows 2008, you have to use 2008 R2.

Update 2011-04-15
Microsoft support advises that there are no plans to bring the ‘netsh int ipv4 show ipaddresses level=verbose’ command to Windows 2008. It will only be available for 2008 R2.

Update 2011-04-10
I can confirm that KB2386184 does work on Windows 2008 R2, the netsh int ip command with the SkipAsSource flag works, and ‘netsh int ipv4 show ipaddresses level=verbose’ does display the flags on all IPs on the box. The issue I ran into was because of how the Hotfix downloader works. We don’t download patches directly from servers, so I was downloading from my workstation which is Windows 7 32-bit. I didn’t notice at the time, but it pushed a 32-bit version of the hotfix down because that’s what it detected on my workstation. When I tried installing the 64-bit version, it (not suprprisingly) worked.

I have our MS rep looking into why the same show ipaddress command doesn’t exist for 2008 SP2, KB975808.

Update 2011-04-06
My change request was closed. MS says they addressed the 2 biggest concerns – setting the SkipAsSource flag and being able to view the flag in the hotfix available at KB2386184. They opened a new ticket to address why I am unable to successfully install the hotfix – when I attempt the installation on 2008 R2, I get an error saying that the hotfix doesn’t apply to my system.

Update 2011-04-02
KB2386184 references a way to at least dump the existing SkipAsSource flag on Windows 7 and Windows 2008 R2, but I was unable to install the hotfix, it said the hotfix didnt apply to my 2008 R2 server. I have a meeting next week with a product manager, hopefully we can come to some resolution.

Update 2010-02-08
Heard back from Microsoft and option “a” below is being considered for backporting into Windows 2008. All of the features listed below are part of Windows 8.

Update 2009-12-17
We have filled out a design change request for Microsoft – we’re not holding our breath on implementation though.

a) There doesn’t appear to be a command to list out the IPs on the box and determine which ones are flagged SkipAsSource. When we have a server that’s reporting intermittent failures to connect to an SMTP server, it is not possible to figure out which IP is the offending IP.
b) The SkipAsSource flag should be in the GUI instead of a command line netsh.
c) You can only set the flag on an IP add, not on an existing IP.
d) We should be able to set the default behavior so new IPs that are added are automatically flagged SkipAsSource.

Update 2009-11-16
Microsoft advised the instructions in their hotfix were not 100% accurate, the syntax of the netsh command was incorrect. It has since been updated. I did get the hotfix to work with one caveat: the name of the interface can not start with a digit – the command fails unless the interface name starts with a letter i.e. interface 10.0.0.x won’t work, but if you name it i10.0.0.x it will work.

It’s not a great solution because 1) You can only use the netsh command to add an IP with the SkipAsSource flag, there’s no way to set it after it’s on the box, you have to delete and add it again 2) You have no command to display which IPs have the SkipAsSourceFlag set, 3) There’s no GUI option, 4) There’s no option for SkipAsSource to be the default setting for all IPs added to the box. But it’s the best we have for now.

2009-11-02
We’ve run into issues with Windows 2008. The first is that every IP on the box gets auto-registered in DNS. This causes issues with some of our monitoring software being unable to handle multiple IPs returned in the DNS query. The second is that there is no concept of “primary” IP on the box anymore. In Windows 2003, the first IP on the box is the primary IP and it’s the default IP for all outbound communication. In what appears to be a random fashion, 2008 picks different IPs for outbound traffic. This is causing problems as we’re having to add firewall rules for every new IP that we add to a box – we never know which IP is going to be used to hit a SQL box or SMTP server.

I found this MS article that explains how an IP is selected. This does seem to explain the behavior; however, it isn’t consistent. I also found a hotfix KB975808 that claims to fix the DNS registration and primary IP issue, but was unable to get it to work. The fix is a “SkipAsSource” flag addition to the netsh command, adding an IP with the flag enabled will force windows to skip the IP for DNS registration and outbound IP selection.

I opened a ticket with Microsoft and we’ll see what happens.