PowerCLI - Datastore Report

I recently was working in an environment where they are still running vSphere 6.0. While doing some storage expansions I discovered that the Flash Client and the C# Client were not reporting the same size on the Datastore. This can be very frustrating as you are not sure which one you can trust. So as a tie breaker I decided to turn to my good friend PowerCLI.

It turns out that the PowerCLI and the C# client were reporting the same thing. So I decided to create a PowerCLI Script to assist me in my work. You can find the script there on my Github:

https://github.com/kenbshinn/PowerCLI-Scripts/blob/master/VM_Datastore_Report.ps1


I hope you found this post helpful and will share it with your friends.

How to unlock and reset SSO password in vSphere 6.x (2146224)

So I have a bit of a embarrassing confession to make. I forgot to record the Administrator password for my VCSA Appliance. Total disclosure, I was freaking out and I really thought I was going to have to start from scratch. I did some research I was surprised to find out that you can actually reset the Administrator account on a VCSA appliance as long as you have the root password for the appliance and you have access to the VCSA Console. Below are a list of the links to the KB Articles from VMware.

Resetting SSO Administrator Password
https://kb.vmware.com/s/article/2034608
Resetting SSO Administrator - VCSA 6.x

Below is the PUTTY session as an example.

shinnk@Computer:~$ ssh root@devvcsa01.xxxx.xxxxx
ssh: Could not resolve hostname devvcsa01.xxx.xxxxx: Name or service not known
shinnk@Computer:~$ ssh root@172.26.44.18
The authenticity of host '172.26.44.18 (172.26.44.18)' can't be established.
ECDSA key fingerprint is SHA256:7E4K1HVpg2ExWz+vEkkRdJ0M5jUYftb3HZw6OSDKFEICSOEPWWKYERe4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.26.44.18' (ECDSA) to the list of known hosts.

VMware vCenter Server Appliance 6.5.0.21000

Type: vCenter Server with an embedded Platform Services Controller

Password:
Connected to service

    * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"

Command> shell.set --enabled true
Command> shell
Shell access is granted to root
root@devvcsa01 [ ~ ]# /usr/lib/vmware-vmdir/bin/vdcadmintool


==================
Please select:
0. exit
1. Test LDAP connectivity
2. Force start replication cycle
3. Reset account password
4. Set log level and mask
5. Set vmdir state
6. Get vmdir state
7. Get vmdir log level and mask
==================

3
  Please enter account UPN : Administrator@vsphere.local
New password is -
/a+p|8M?vRl`%"p4*+oZ


==================
Please select:
0. exit
1. Test LDAP connectivity
2. Force start replication cycle
3. Reset account password
4. Set log level and mask
5. Set vmdir state
6. Get vmdir state
7. Get vmdir log level and mask
==================

Once you go through all these steps you are now able to log into VCSA with that temporary password that you are given and you are also able to reset it as well.

I hope you find this post helpful, and if you do please share it out to your friends.

Power off entire Virtual Environment using a PowerShell Script and PowerCLI

So I have a lab that I manage where we have had several scheduled power outages in the last few months. So it is my job to make sure that we power off the lab so none of our equipment have issues when we bring it back up.

I would estimate that our lab is comprised of about 90% VMware ESXi Hosts, and after going through the exercise of powering off the whole lab 1 or 2 times, it became it bit of a pain.

So I created a PowerShell Script using PowerCLI to not only power off all the Virtual Machine, but also the Virtual Hosts, and the VCSA appliance itself.

I have posted the script to my Github which you an check out here:

https://github.com/kenbshinn/Poweroff_VirtualEnvironment

Feel free to check it out and let me know what you think.

I hope you found this post helpful, and if so, please share it with your friends.

All vCenters not showing up after adding a new one to an SSO Domain

As I have mentioned before, the company I work for had a disaster event that took place almost a year ago and because of that we have had  some of our infrastructure duct taped together.

Today I am proud to say that I had the opportunity to rip off another piece of that duct tape and actually move our Virtual Infrastructure forward.

You see, Pre-Disaster we had a single vCenter appliance which managed 3 sites. (Yes I know... yuck)

But, because of the disaster we had to move all of our services from the 1 site to the other 2. In the middle of the DR event I had to create 2 VCSA appliances to be able to manage the 2 sites, and due to a lack of sufficient network connectivity at the time, they were just islands. I even set them up a separate SSO Domains.

Fast forward to today, and I have now consolidated these 2 SSO Domains down to 1 and I must say it is pretty slick.

I did however run into a bit of an anomaly, which is the purpose of my post today. You see on the VCSA appliance that was added to the existing SSO domain, I discovered that I could see the first VCSA Appliance in the vSphere Web Client as well as it's inventory which was awesome!

However on the vSphere Web Client of the Original VCSA Appliance, I can only see the 1 VCSA Appliance.

I consulted Dr. Google but found nothing at first, until I came across this post on the IBM Cloud for VMware Solutions site.

It turns out that you need to restart the vSphere Web Client in order for the new vCenter server to appear.

Just encase the link dies I will post the resolution here:

This is a known VMware 6.5 issue.

To resolve the problem, you must restart the vSphere Web Client:

Using the root account, connect over ssh to the vCenter VM (virtual machine) of the previously ordered instance.
Type shell to enter the bash shell.
Enter service-control --stop vsphere-client to stop the client.
Enter service-control --start vsphere-client to restart the client.
After the vSphere Web Client of the previously ordered instance is restarted, confirm that the vCenter Server system for the newly added secondary instance is visible in the vSphere Web Client.

NOTE: Rebooting the VCSA Appliance will also resolve your issue.

I hope you found this helpful, and if so please let me know and share with your friends.

Unable to log into a new VCSA Appliance added to an SSO Domain

So today I ran into an interesting issue. I was adding a new VCSA appliance to an preexisting SSO domain for a post that will come out in the coming weeks and I discovered a few things.

First of all, I learned that there are not many blogs covering this so I figured I would make a post just encase I get hit with a brick and forget.

So I ran through the install of the new VCSA appliance and selected all the settings as you would normally do. However when the installation completed it said everything was fine and I should be able to log in at the new URL.

I opened the URL and tried to log in.... and It failed. This being the first time I have tried this I was not sure what to expect, and here is what I mean:

The Platform Services Controller (or PSC) is responsible for authentication to vSphere. That being said I figured I might be able to log in with my Domain credential.... I was wrong.

So I then tried logging in with the Administrator Credentials for the SSO Domain.... that too did not work.

I then logged into the vSphere Web Client for the First VCSA to see if I could see what is going on over there. It turns out, from an SSO Perspective, everything was running great. I went to see if I could reset the Administrator password and discovered that I must not have had enough coffee this morning. You see, because vSphere is based on some form of Linux Kernel, I forgot to take into account that THE USER NAME IS CASE SENSITIVE!!!!!

So I went back to the login screen and logged in with the correct credentials and IT WORKED!!

Now to figure out why AD Authentication was not working.....

Well it turns out that that lack of coffee I mentioned earlier came back to bite me yet again <face palm>

Although the SSO component was working and showed my AD domain as a valid and default authentication source, it doesn't mean squat if the server is not a member of the Active Directory Domain. A quick add to AD and a reboot later and we were in business.

Lesson Learned:

1. Don't skip out on Coffee
2. Linux based credentials are Case Sensitive
3. Drink more Coffee
4. Make sure that host is added to AD before you try to authenticate.

I hope you found this helpful and if I ever get around to it I will add screen shots for a more TL:DR experience.

Tools Discussion - VMWare Converter Standalone

Today I wanted to talk about VMware Converter Standalone which has been a constant in my bag of tricks for a long time. VMware Converter is a tool that allows you to convert a Physical Machine to a Virtual Machine which can run on a VMware ESXi host or VMware Workstation / Player. You can also use VMware Converter to resize Virtual machines or even convert them from Thick Provisioned virtual disks to thin.

Before VMware Converter was a free Standalone product, it was an add-on to vCenter (version 3-4.1 if my memory was correct) that was licensed by VMware. It enabled you (much like it does now) to convert a physical machine and turn it into a Virtual Machine. Pretty Sweet huh?!

Somewhere between vSphere 4 and 5 they decided to remove it from vCenter as an add on and make it a a separate product and best of all made it free!

During the installation process you are now asked to decide if you are doing a standalone installation or a Client Server install. I will probably do an install guide for this later so I can discuss further as both have their merits.

If you are interested in getting a copy of VMware Converter Standalone you can follow this link. It will also be available on the Tools page once this post goes live.

I hope you found this post helpful and please check back later for more articles.

How to convert VMDK from Thick to Thin Provision

As I have stated in a previous post, I have been trying to help out on the VMware Community Forum. One of the threads that I was able to help out on was how to convert a VMDK from thick to thin.

I have had the opportunity to do this several times in the past, but I have never needed to document. I was able to do a step by step procedure from memory for the thread, but I figured doing an actual guide might be helpful to someone else. 

The first thing that you need to realize is that the VM you want to change from Thick Provision to Thin MUST be powered off before you can convert it. 

Launch the VMware vCenter Converter Standalone Client software.

Click on the Convert Machine button.

That will launch the Converter wizard

By Default  the Source machine will be set to Powered on VM. You will need to select the Powered off radio button which will give you to option to select VMware Virtual Machine. 

Enter the Server Name or IP address, Username, and Password for the Source system and click Next

This will cause the Wizard to do a check against the Source machine to make sure that the credentials are able to access the vCenter server. 

Once you are logged in, will see the your vCenter server and ESXi hosts. Click on the vCenter server, or ESXi Host to see all the VMs that are available. 

Select the Powered off VM you wish to convert to thin provisioned and click Next. 

Next you will need to enter the Server Name, User name and Password for the destination host. Before you ask, yes the source and destination host can be the same server, however the Virtual Machine name will need to be different. 

Click Next to Continue

As I mentioned above, you will need to give the Virtual Machine a unique name, otherwise vCenter gets pretty mad. 

Select the Data Center you would like to put the VM on and click Next. 

Select the Host, Datastore, Virtual Hardware Version, and click Next.

NOTE: You should probably pick a different datastore that the one you used before, just to make sure that you don't run out of space. 

This will take you to the Options screen which is where you are allowed to change the Virtual Disk from Thick to Thin. 

Click Edit as seen in the screen below.

This will bring up the disk configuration for this Virtual Machine.

Click on the drop down under Type and change it from Thick to Thin and then click Next.

You will then be presented with the Summary screen which you can review, and then click Finish to start the conversion. 

 As you can see the Job gets submitted and will run until it completes successfully

Once it completes you will see in vCenter that there is a new Virtual Machine and if you check the settings of that Virtual Machine the VMDK will be Thin Provisioned. 

Thank you for stopping by, and I hope you found this post helpful and please check back later for more updates. 

Adding an IDE Storage Adapter for an OVF that is supposed to run in vSphere? What were you thinking???

A few weeks ago I had the misfortune of having to deal with a .OVF that was sent by a support team to one of our engineers so we can run a Virtual Machine in our lab environment.

That seems pretty cut and dry, except when I go to import the .OVF and I get this message when the import gets to the Validating Step.

I had never seen this message before, but after seeing the part of the message that says DiskControllerReference, I automatically assumed that there was an issue with the .VMX file on the VM. Well because this is not a VM, but an .OVF I decided to open the .OVF file in Notepad ++ and found this.

I do not know why the person who created this .OVF selected a IDE controller for the HDD., but here we are.

I informed my Engineer of the issue, and he said that they told him to use VMware Converter to change it into an acceptable VM image. 

Giving them the benefit of the doubt, I downloaded the lastest copy of VMware Converter Standalone (which took me a day, because VMware's Download site was not working) and tried to convert it.

Guess what, It didn't work!!

This is also not the first time I have run into this issue with this particular vendor (who shall remain nameless). Their .OVA files will give you the same error message.  So needless to say anytime I run into this vendor, I immediately tell my Engineers to get me an .ISO and I will build them a VM from Scratch. 

I hope you find this post helpful, and if you would like to know more information about this, please feel free to DM me on one of my social media accounts. 

Thanks for reading, and check back soon for more articles. 

How to install ESXi 5.5

Continuing the saga of building out my VMware home lab I will be covering the process off installing ESXi 5.5 as a Virtual machine in my environment. As stated in my previous posts, I am a VMUG Advantage Member so I do have access to newer versions of VMware software. 

That might make you wonder why I am going with ESXi 5.5, well you are going to have to wait and see what my plan is for that. 

Lets get started with the installation of ESXi 5.5

Once the ESXi .ISO has been booted up on the machine you will be presented with this screen.

Press Enter to boot, or let the countdown continue and it will boot automatically. 

The ESXi installer will load into memory. It may take some time for this to complete depending on how much Memory you have on your machine, so your mileage may vary. 

After all of this you will be presented with the ESXi 5.5 Welcome Screen. Press Enter to continue. 

Press F11 to Accept the End User License Agreement to continue on.

Once you press F11 the install will start scanning the System it is installing on to see what Storage is available. 

On the next screen you will be presented with a list of Datastores that are available to you to install ESXi on. Select the Datastore that you want to use and press Enter. 

You will then be asked to select your Keyboard layout. Press Enter to continue.

Now you will have to set the Root Password for the OS. Once you confirm the password press Enter to continue. 

The ESXi Installer will then can your system again .

You will then be asked to Confirm your decision to install ESXi on the datastore you selected. Press F11 to continue. 

Now the installation will begin, again your mileage may vary as to how long it will take to complete. 

Once the installation has been completed you can press Enter to reboot the machine to complete in installation. 

After the reboot has been completed and the system boots up you will be presented with the following screen. 

This concludes the ESXi install guide. I hope you found this helpful an please check back for more updates to come.

VMware Home Lab 1.0

GREAT DAY IN THE MORNING!!!!!

Alright I had to find a different way to start this post, so I decided to borrow Pierre Roberts Catch phrase. I don't think it worked so he can have it back. 

I wanted to take the time to talk about my home lab that I am currently using. As I mentioned in a previous post I have a Razer Blade 2018 which I will be using to run my lab for now. 

Stock the laptop comes with:

• 8th Gen Intel Core i7 - 8750H Processor 6 Cores/12 threads up to 4.1 GHz
• 1 TB M.2 SSD
• 16 GB of RAM. 

I have been wanting to upgrade it to 32 GB of RAM since I purchased it, however $400 for 32 GB of RAM was just too much for me to swallow at the time. 

I had been watching the price of RAM (particularly for this laptop) for the last few months and saw that it was steadily declining. I finally decided to pull the trigger when it was down to 49% off on Amazon. 

The RAM Arrived a few days ago, and I installed it in the laptop, and I have to say it is awesome. I have never seen a laptop run this good. 

I installed VMWare Workstation Professional 15 (Thank you VMUG Advantage) and I will have a install guide out for that soon. 

I am also beginning to Spec out my VMware Home Lab 2.0 which will also be a Nested Solution. I will probably leverage my Kit.com account to show my parts list once I get it running. 

I hope you find this post helpful and stay tuned for updates.

RVTools - A VMware Admin's best friend

Today I would like to talk about one of my favorite tools that I use both internally and with my clients.

RVTools - https://www.robware.net/rvtools/

RVTools is a free utility that you can use to discover what is going on with your VMware Environment. All you have to do it install the client on your machine and in a few minutes you are able to log into your vSphere Environment (using your normal vSphere Credentials) and it will give you a read out of your entire environment.

From the login Screen the look and feel is very reminiscent of the old C# vSphere Client used back in 5.x and 6.0 days.

Once you log in you are presented with the vInfo tab which is a summary of everything going on with the vSphere/ESXi host you Connected to.

Along the top you will see a series to tabs that will drill down into the various components of your environment such as the ones seen below

I am only going to cover a few of these features, but I encourage you to try it out for your self.

 vCD will show you what Virtual Machine has a CD Drive installed on it, if it is connected, and what .ISO is currently connected to it. So ,.for instance, if you have a Virtual Machine that will not vMotion, you could use this tool to see if it is attached to a .ISO that is on Local Storage.

vSnapshot will show you all the Virtual Machines that have snapshots associated with it. It will report back to you the name of the Virtual Machine, the name of the Snapshot, the Date/Time it was taken, and the File Name of the snapshot.

And Lastly we will touch on the vHealth tab. This tab will give you advice on things that you may not realize is going on in your environment such as inconsistent naming, Host logs not being stored on persistent storage, VMware tools being out of date, and Zombies (Files that exist on Datastores, but are not in inventory).

On a final note that I forgot to mention, If you go up into the file menu you are able to export everything you see in the tool to a .CSV which can become very handy for documenting your environment. 

I hope you find this post useful and if you do download this product, please consider donating to it's creator. Without tools like this it would make all of our jobs much harder. 

How to Change VM from BIOS to UEFI

I just had a question on the VMWare Community about how to change a VM from BIOS to UEFI.

I found that if you go into Edit Settings > Click on the VM Options Tab and you should see this

I hope this is helpful for anyone who needs this information in the future. 

It's Official!!!

I just received my digital books for VMware vSphere: Design Workshop [V6.5]! I am looking forward to taking this class next week

How to add Target Servers to vMA

I have had this one post sitting around for a while and I figured I should get it written before it gets lost.

Earlier I was working on getting vMA stood up in my lab environment and I figured that I should do a post about how to configure vMA to connect to a target. Adding a Target Server in vMA is a faster way for you to be able to issue commands. Rather than having to specify which server you want to run commands against (especially if you are running several commands on the same box) you should set the server as a target. In this instance I will run through the steps of setting a target for an AD Authenticated host or vCenter. 

From the vMA console you will type the following command out:

vifp addserver <vcenter server name.domain.local> --authpolicy adauth --username <AD DOMAIN\UserName>

and press Enter and you will be prompted for the Password of the AD account that you referenced.

To verify that the server is connected you can use the following command:
vifp listservers --long

This will tell you the server name as well as the method of authentication. Now anytime you want to run a command against that host you will be able to do so without having to re-authenticate every time.

To set your Target server you can use the following command prior to running your actual command or script.

vifptarget -s <servername>

VMWare Virtual Machine will not Start in vCenter

So I was asked by a client to do up "hardware upgrades" on one of their virtual machines. This VM is not production and is not running anything at the moment, but will soon become an SQL Server running an application. So I went and powered off the VM, logged into the C# vSphere client (they are running vSphere 6.0), and doubled the memory as requested by the client.

After making the change I went to power on the VM and I get presented with this pop up error message from VMWare

The message itself seems pretty generic but I have actually seen this one before. This error has to do with  an issue with the management agents on  your ESXi host.

I followed the instructions found on this KB article but it did not resolve my issue. I also found this KB article which might yield better results but as this whole environment is production, I will need to wait to test that.

WORK AROUND:

In order to get this VM powered back online, I needed to log directly into the virtual host and power on the machine.

The VM powered on with no issues.

I hope you find this helpful and I will update this when I have a solution.

How to add vMA to Windows AD Domain

While working in my lab at work I wanted to see about adding my vSphere Management Assistant (vMA) to our lab Active Directory so it would be easier for me to connect to resources. After going through the process I decided to make a guide on the steps I took to get it working.

Just an FYI I was working with vSphere Management Assistant version 5.5.0.4 at the time of taking these screenshots so I am sure things may have changed in later versions... or not.

First you will need to get to the console of the machine.

Select the option to login

and log in using the vi-admin account

Once you are logged in you are presented with a command prompt. You will need to use the following command to add your appliance to a Windows domain.

Sudo domainjoin-cli join <domain.name> <Domain User Name>

You will then be prompted for the vi-admin password

Once your Appliance is added to the domain you will see a screen like this.

Once the reboot has been completed you will can verify that you are on the domain by running this command:

sudo domainjoin-cli query

and you will be presented with a response something line this

Now your appliance is on the domain and you will now have an easier time connecting to resources in your VMware environment.  

Scenario 004 - Issue with connecting to NETAPP OnCommand System Manager

It has been a while since I did one of these posts do I figured I would try to get back to it. 

Today I was working at a client site and I had to make some changes to one of their VMs to increase storage. While working on their management host I attempted to connect to their NetApp OnCommand System Manager when I was presented with this:

Having seen this so many times I figured I would share the solution. The error message above is telling you that the file SystemManager.ks is corrupt. This file can be renamed and is located here:

C:\Users\<username>\NetApp\SystemManager\

Once you rename the file all you have to do it close you IE window and try again.

Exchange 2016 - Issue with changing URL for MAPI connections

As you know when you installed Exchange all of the Virtual Directories use the FQDN of the server as the URLs for the Exchange Virtual Directories. If your Exchange server is being used for OWA or CAS you will most likely have a SSL Cerfiticate to prevent you from getting those pesky SSL errors.

During an install that I recently did I ran into an issue where all the outlook clients were getting SSL pop up messages that looked like this:

From the message you are able to see that the name of the certificate does not match the server name. So I logged into the Exchange Admin Center and looked under Servers > Virtual Directories and checked each of the Virtual Directories listed. As this was my first 2016 installation I discovered that there was a virtual directory for mapi. So from the web interface I changed the internal url to match the name of the certificate "https://Mail.Domain.com/mapi" and restarted IIS. After that I go and try and configure an outlook client and good news I no longer get the SSL Error, but now I am unable to authenticate to the mailbox!!!

After playing around with it for a while I discovered that changing the name back did not fix the issue so I was forced to do a rebuild on the virtual directory and that took me back to square 1.

After hours of searching online for an answer I found this powershell command that worked


Set-mapivirtualdirectory -identity "[SERVERNAME]\mapi (default Web Site)" -internalurl  https://Mail.DOMAIN.com/mapi

After doing this command I restarted IIS again and the clients are now able to connect without issue.

UPDATE on this issue:

I would appear that do to a misconfiguration in the client's internal DNS their Autodiscover.domain.local was a A record and not a CNAME which it appears that outlook autodiscover does not like. So I changed the record to a CNAME by deleting the A record and recreating it. Then I went to a local machine and did a IPCONFIG /FLUSHDNS and then restarted outlook and that seems to have fixed the issue.

How to edit a VM hardware after it has been created

So after you go and make your Virtual Machine you realize you forgot to mount the .ISO for Windows Server 2012 and you left the Floppy Drive in the config which is not really hurting anyone but it makes things more aesthetically pleasing to not see it there.  So what do you do....

Here are the detailed steps to change the hardware configuration of a virtual machine from the VMware vSphere Client.

With the virtual machine selected in the navigation pane you should see the "Getting Started" tab in the center panel. On that tab towards the bottom you will see the "Basic Tasks" section as seen in the screenshot below. 

Click on "Edit Virtual Machine Settings" and the Virtual Machine Properties Window will open up for you.

From here you can see all of the virtual hardware that is assigned to the virtual machine in question. You can also make changes to the virtual hardware as you see fit.

Please note: you will not be able to modify all of the settings if the VM is powered on. Some things like Memory and CPU will require the VM to be off before VMware will let you modify it.

To remove the Floppy drive select the device by clicking on it and click the remove button.

You will now see that the device has a line though it, meaning that this is a pending change for removal and will not take effect until you click okay at the bottom. 

To add the boot ISO you will need to click on the CD/DVD Drive to highlight it.

To add an ISO stored on a Datastore like a LUN or the Local Datastore on the virtual host click Datastore ISO file radio button.

Once you click on the radio button the browse button then becomes available to you. Click on the Browse button to open the Browse Data store window. 

Navigate to where you ISO is stored and click Open.

The file path for the ISO is now populated for you. I suggest that you click connect at power on up at the top of the window. This will make sure that the CD drive presents the ISO to the VM at boot up and allows you to boot into the VM. 

Click okay to close the window and to commit the changes. Once the tasks have been committed you will then be able to power the VM back on.