Archives For sysadmin

Team+OpenStack+@+VMwareIn a previous article I showed the process to patch an existing VIO 1.0 installation, which as you were able to see it is a clean and easy process. VMware announced VMware Integrated OpenStack 2.0 during VMware US and it finally became GA a few weeks ago.

This new version of VIO has all OpenStack code updated up to the latest Kilo release and comes packaged with many interesting features like Load-Balancing-as-a-Service (LBaaS) or auto-scaling capabilities based on Heat and Ceilometer.

With a new VIO version hot of the press it is time now to upgrade your VIO 1.0.x environment to 2.0 and take advantage of all those new great goodies. The upgrade process is pretty straightforward and consist of three main stages.

  • Upgrade VIO Management Server
  • Deploy a new VIO 2.0 environment
  • Perform the data migration

Keep in mind that you will need to have enough hardware resources in your management cluster to be able to host two full fledged VIO installations at the same time during the migration process. Just for the sake of transparency, the lab environment where I test the upgrade is based on vSphere 5.5 Update 2, NSX for vSphere 6.1.4 and VIO 1.0.2.

Step 1 – Upgrade VIO Management Server

From VMware website download the .deb upgrade package and upload it to VIO Management Server.

Download VIO upgrade package

Stage the upgrade package.

viouser@vio-oms:~$ sudo viopatch add -l vio-1.0-upgrade_2.0.0.3037964_all.deb
[sudo] password for viouser:
vio-1.0-upgrade_2.0.0.3037964_all.deb patch has been added.
viouser@vio-oms:~$ viopatch list
Name            Version       Type   Installed
--------------- ------------- ------ -----------
vio-1.0-upgrade infra  No
vio-patch-2 infra  Yes


Upgrade the management server with viopatch command.

viouser@vio-oms:~$ sudo viopatch install -p vio-1.0-upgrade -v
Installing patch vio-1.0-upgrade version
Installation complete for patch vio-1.0-upgrade version

Go to the vSphere Web Client, logout and log back in to verify that the new version is correct.


Step 2 – Deploy a new VIO 2.0 environment

With VIO Management Server upgraded is now time to deploy a fresh 2.0 environment. In the VIO plugin go to Manage section and a new Upgrades tab will be there. Before


Before starting with the deployment check in the Networks tab that there are enough free IP address for the new deployment, if there aren’t then add a new IP range.


Click on the Upgrade Screen Shot 2015-10-20 at 13.49.34 icon. Select if you want to participate in the Customer experience improvement program, my recommendation here is to say yes to help our engineering team to improve VIO upgrade experience even more ;-), and enter the name for the new deployment.


Enter the IP addresses for the public and private load balanced IP addresses, keep in mind that these IP addresses must belong to the API subnet of the existing VIO 1.0 environment in case of the public and to the management network segment in the case of the private one.


In the last screen review the configured values and click Finish. The new environment will be deployed and you will be able to monitor it from the Upgrades tab.


Step 3 – Migrate the data

With the new environment up and ready we can start the data migration. From the Upgrades tab right-click in the your existing VIO 1.0 installation and select Migrate Data.


The migration wizard will ask for confirmation, click OK. During the data migration all OpenStack service will be unavailable.


When the migration process is finished the status of the new VIO 2.0 environment will appear as Migrated and the previous VIO 1.0 will appear as Stopped.


Open a browser and connect the VIO 2.0 public IP to access OpenStack Horizon interface, login and verify that all your workloads, networks, image, etc have been properly migrated. Logout from Horizon and go back to the Web Client. Now that the data has been migrated we need to migrate the original Public Virtual IP to the new environment.

Right-click on VIO 1.0 deployment and from the menu select Switch To New Deployment.


A new pop-up will appear asking for confirmation since again the OpenStack service will be unavailable during the IP reconfiguration.

After the reconfiguration the new VIO 2.0 deployment will be in Running status and the Public Virtual IP will be the same as the former 1.0 deployment.


The upgrading procedure is finished. We can now access now Horizon using the existing FQDN, verify that everything is still working and enjoy your new OpenStack Kilo environment.


In the same way as patching, with VIO upgrading your OpenStack cloud does not have to be a painful experience, VIO provides the best OpenStack experience in a vSphere environment. Kudos to my colleagues of the Team OpenStack @ VMware.

Happy Stacking!


Fedora 22 was released a few months ago and amongst many new features it came with a replacement for yum as package manager called dnf, or DaNdiFied YUM, oh yes yum is still around but it is now considered legacy software. Also DNF will become in the near future the default package manager for RHEL and CentOS so it is for the best that you get familiarized with it sooner than later.

DNF Commands

The first thing you need to understand about dnf is that many commands are basically still the same but there are differences. Package management commands can be executed with almost the same syntax previously used with yum.

Search for a package,

[jrey@fed22-srv ~]$ sudo dnf search htop
Last metadata expiration check performed 1:25:54 ago on Mon Oct 5 23:47:45 2015.
=================================== N/S Matched: htop ====================================
htop.x86_64 : Interactive process viewer
php-lightopenid.noarch : PHP OpenID library
[jrey@fed22-srv ~]$

Install a package.

[jrey@fed22-srv ~]$ sudo dnf install htop

Remove a package.

[jrey@fed22-srv ~]$ sudo dnf remove htop

Get information about a package

[jrey@fed22-srv ~]$ sudo dnf info htop
Last metadata expiration check performed 1:47:13 ago on Mon Oct 5 23:47:45 2015.
Available Packages
Name : htop
Arch : x86_64
Epoch : 0
Version : 1.0.3
Release : 4.fc22
Size : 91 k
Repo : fedora
Summary : Interactive process viewer
License : GPL+
Description : htop is an interactive text-mode process viewer for Linux, similar to
 : top(1).

[jrey@fed22-srv ~]$

Group and repository management commands are still the same as well.

[jrey@fed22-srv ~]$ sudo dnf repolist

Querying the available repositories for a specific command.

[jrey@fed22-srv ~]$ sudo dnf repoquery --whatprovides htop
Last metadata expiration check performed 1:54:52 ago on Mon Oct 5 23:47:45 2015.
[jrey@fed22-srv ~]$

dnf comes with some powerful capabilities like history query.

[jrey@fed22-srv ~]$ sudo dnf history list
Last metadata expiration check performed 11 days, 19:14:54 ago on Wed Oct 7 02:56:21 2015.
ID | Command line             | Date a           | Action  | Altere
 9 | history undo 8           | 2015-10-06 01:53 | Install | 1 
 8 | erase htop               | 2015-10-06 01:28 | Erase   | 1 
 7 | install htop -y          | 2015-10-06 01:28 | Install | 1 
 6 | remove htop              | 2015-10-06 01:14 | Erase   | 1 
 5 | install htop             | 2015-10-06 01:14 | Install | 1 
 4 | install make gcc kernel- | 2015-09-30 16:21 | Install | 9 
 3 | update                   | 2015-09-30 15:43 | I, U    | 112 
 2 | update                   | 2015-09-16 11:45 | I, O, U | 297 
 1 |                          | 2015-09-16 10:59 | Install | 658 EE
[jrey@fed22-srv ~]$

This can be specially helpful if you need to rollback a change, like clean up dependencies after uninstalling a package or reinstall a package.

[jrey@fed22-srv ~]$ sudo history undo 8

You can also look for duplicated within the installed ones.

[jrey@fed22-srv ~]$ sudo dnf repoquery --duplicated
Last metadata expiration check performed 0:30:42 ago on Tue Oct 6 02:48:41 2015.
[jrey@fed22-srv ~]$

Retrieve all available packages providing a specific software of capability.

[jrey@fed22-srv ~]$ sudo dnf repoquery --whatprovides curl
Last metadata expiration check performed 0:38:00 ago on Tue Oct 6 02:48:41 2015.
[jrey@fed22-srv ~]$

This is a very basic introduction to dnf capabilities but hopefully you have been able to get how it works. My advice is to review DNF documentation for all the details.

The Photon Connection

VMware Photon comes with tdnf (Tiny DNF); this is a development by VMware that comes with compatible repository and package management capabilites. Not every dnf command is available but the basic ones are there.

Package installation and updates.

Screen Shot 2015-10-11 at 19.41.00

Repository management.

Screen Shot 2015-10-11 at 18.54.47

In the future if I find the time I’ll write a new post with some advanced examples of dnf commands. Comments are welcome.


As with the rest of NSX for vSphere components any competent admin would like to configure a remote syslog server for the NSX Controllers, in my homelab I have vRealize Log Insight and recently I decided to configure it on my NSX Controllers and document the procedure here mostly as self-reference.

NSX Manager has the option to configure a remote syslog server using its management web site, but where is the option for the Controllers? Well, if you lurk around NSX interface in vSphere Web Client will quickly notice that the option is somehow missing.  Actually the only option to enable it is using NSX REST API. Let’s see how to do it.

For this post I will use Firefox REST Client Add-on but you can use your favorite REST client. Firstly any REST API call will require at least the Authentication header, in Firefox REST Client click on Authentication drop-down menu, select Basic Authentication and enter the admin credentials.

Screen Shot 2015-09-30 at 02.25.19

Additionally PUT and POST methods will require you to set a custom header with the following values that will define the content of the HTTP Request body. Use these values.

  • Name: Content-Type
  • Value: application/xml

With these two headers set enter the API URL, in my case it is:


To construct this URL you will need the controller ID that can be get in the NSX interface in vSphere Web Client as shown below.

Screen Shot 2015-09-30 at 02.50.38

Select POST as the method. You will need to enter the body for the HTTP Request, in XML format. Use the below code as an example to build the content for the HTTP Request body.

This XML code will indicate the NSX Manager to set the IP address in the syslogServer node as the remote syslog server for the controller in the URL. The protocol, port and log level are also defined.

Screen Shot 2015-09-17 at 01.14.19

Submit the request and if everything is configured as described you will receive a 200 OK status code.

Screen Shot 2015-09-17 at 01.15.06

At this point the syslog server is configured for all NSX Controllers, you can check the status using also an API call with the same URL and selecting GET method.

Comments are welcome.


FirewallD, or Dynamic Firewall Manager, is the replacement for the IPTables firewall in Red Hat Enterprise Linux. The main improvement over IPTables is the capacity to make cahnges without the need to restart the whole firewall service.

FirewallD was first introduced in Fedora 18 and has been the default firewall mechanism for Fedora since then. Finally this year Red Hat decided to include it in RHEL 7, and of course it also made its way to the different RHEL clones like CentOS 7 and Scientific Linux 7.

Checking FirewallD service status

To get the basic status of the service simply use firewall-cmd --state.

[root@centos7 ~]# firewall-cmd --state
[root@centos7 ~]#

If you need to get a more detailed state of the service you can always use systemctl command.

[root@centos7 ~]# systemctl status firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Wed 2014-11-19 06:47:42 EST; 32min ago
 Main PID: 873 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─873 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Nov 19 06:47:41 centos7.vlab.local systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 19 06:47:42 centos7.vlab.local systemd[1]: Started firewalld - dynamic firewall daemon.
[root@centos7 ~]#

To enable or disable FirewallD again use systemctl commands.

systemctl enable firewalld.service
systemctl disable firewalld.service

Managing firewall zones

FirewallD introduces the zones concept, a zone is no more than a way to define the level of trust for a set of connections. A connection definition can only be part of one zone at the same time but zones can be grouped  There is a set of predefined zones:

  • Public – For use in public areas. Only selected incoming connections are accepted.
  • Drop – Any incoming network packets are dropped, there is no reply. Only outgoing network connections are possible.
  • Block – Any incoming network connections are rejected with an icmp-host-prohibited message for IPv4 and icmp6-adm-prohibited for IPv6. Only network connections initiated within this system are possible.
  • External – For use on external networks with masquerading enabled especially for routers. Only selected incoming connections are accepted.
  • DMZ – For computers DMZ network, with limited access to the internal network. Only selected incoming connections are accepted.
  • Work – For use in work areas. Only selected incoming connections are accepted.
  • Home – For use in home areas. Only selected incoming connections are accepted.
  • Trusted – All network connections are accepted.
  • Internal – For use on internal networks. Only selected incoming connections are accepted.

By default all interfaces are assigned to the public zone. Each zone is defined in its own XML file stored in /usr/lib/firewalld/zones. For example the public zone XML file looks like this.

root@centos7 zones]# cat public.xml
<?xml version="1.0" encoding="utf-8"?>
  <description>For use in public areas. You do not trust the other computers on networks to not harm your computer. Only selected incoming connections are accepted.</description>
  <service name="ssh"/>
  <service name="dhcpv6-client"/>
[root@centos7 zones]#

Retrieve a simple list of the existing zones.

[root@centos7 ~]# firewall-cmd --get-zones
block dmz drop external home internal public trusted work
[root@centos7 ~]#

Get a detailed list of the same zones.

firewall-cmd --list-all-zones

Get the default zone.

[root@centos7 ~]# firewall-cmd --get-default-zone
[root@centos7 ~]#

Get the active zones.

[root@centos7 ~]# firewall-cmd --get-active-zones
  interfaces: eno16777736 virbr0
[root@centos7 ~]#

Get the details of a specific zone.

[root@centos7 zones]# firewall-cmd --zone=public --list-all
public (default, active)
  interfaces: eno16777736 virbr0
  services: dhcpv6-client ssh
  masquerade: no
  rich rules:

[root@centos7 zones]#

Change the default zone.

firewall-cmd --set-default-zone=home

Interfaces and sources

Zones can be bound to a network interface and to a specific network addressing or source.

Assign an interface to a different zone, the first command assigns it temporarily and the second makes it permanently.

firewall-cmd --zone=home --change-interface=eth0
firewall-cmd --permanent --zone=home --change-interface=eth0

Retrieve the zone an interface is assigned to.

[root@centos7 zones]# firewall-cmd --get-zone-of-interface=eno16777736
[root@centos7 zones]#

Bound the zone work to a source.

firewall-cmd --permanent --zone=work --add-source=

List the sources assigned to a zone, in this case work.

[root@centos7 ~]# firewall-cmd --permanent --zone=work --list-sources
[root@centos7 ~]#


FirewallD can assign services permanently to a zone, for example to assign http service to the dmz zone. A service can be also assigned to multiple zones.

[root@centos7 ~]# firewall-cmd --permanent --zone=dmz --add-service=http
[root@centos7 ~]# firewall-cmd --reload
[root@centos7 ~]#

List the services assigned to a given zone.

[root@centos7 ~]# firewall-cmd --list-services --zone=dmz
http ssh
[root@centos7 ~]#

Other operations

Besides of Zones, interfaces and Services management FirewallD like other firewalls can perform several network related operations like masquerading, set direct rules and manage ports.

Masquerading and port forwading

Add masquerading to a zone.

firewall-cmd --zone=external --add-masquerade

Query if masquerading is enabled in a zone.

[root@centos7 ~]# firewall-cmd --zone=external --query-masquerade
[root@centos7 ~]#

You can also set port redirection. For example to forward traffic originally intended for port 80/tcp to port 8080/tcp.

firewall-cmd --zone=external --add-forward-port=port=80:proto=tcp:toport=8080

A destination address can also bee added to the above command.

firewall-cmd --zone=external --add-forward-port=port=80:proto=tcp:toport=8080:toaddr=

Set direct rules

Create a firewall rule for 8080/tcp port.

firewall-cmd --direct --add-rule ipv4 filter INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT

Port management

Allow a port temporary in a zone.

firewall-cmd --zone=dmz --add-port=8080/tcp

Hopefully you found the post useful to start working with FirewallD. Comments are welcome.


Being used to have Cockpit in my Fedora 21 Server VMs I decided that having it also on my CentOS machines would be awesome, unfortunately I quickly found that Cockpit was not available in CentOS repositories. Of course I knew that Cockpit comes installed and enabled by default in CentOS 7 Atomic host image so I figured out that those packages had to be hidden in some Atomic related repo.

After looking a bit I finally found in GitHub the sig-atomic-buildscripts repository that belongs to CentOS Project. This repository contains several scripts and files intended to build your own CentOS Atomic host including virt7-testing.repo, the Yum repository file needed for Cockpit.

Clone the GutHub repository.

git clone

Copy virt7-testing.repo file to /etc/yum.repos.d and install Cockpit.

yum install cockpit

Enable Cockpit service.

[root@webtest ~]# systemctl enable cockpit.socket
ln -s '/usr/lib/systemd/system/cockpit.socket' '/etc/systemd/system/'
[root@webtest ~]#

Add Cockpit to the list of trusted services in FirewallD.

[root@webtest ~]# firewall-cmd --permanent --zone=public --add-service=cockpit
[root@webtest ~]#
[root@webtest ~]# firewall-cmd --reload
[root@webtest ~]#
[root@webtest ~]# firewall-cmd --list-services
cockpit dhcpv6-client ssh
[root@webtest ~]#

Start Cockpit socket.

systemctl start cockpit.socket

Do no try to access Cockpit yet, there is an issue about running Cockpit on stock CentOS/RHEL 7. To be able to start it we need first to modify the service file to disable SSL.Edit file /usr/lib/systemd/system/cockpit.service and modify ExecStart line to look like this.

ExecStart=/usr/libexec/cockpit-ws --no-tls

I know this procedure will invalidate Cockpit for a production environment in RHEL7 at least for now but this is for my lab environment and I can live with it.

Reload systemd.

systemctl daemon-reload

Restart Cockpit.

systemctl restart cockpit

Access Cockpit web interface, login as root and have fun :-)

Screen Shot 2015-01-09 at 01.57.51



fedora_infinity_140x140Cockpit is a new web based server manager to administer Linux server, it will provide the system administrators with a user friendly interface to manage their Linux servers, it includes multiserver managing capacity and more importantly it will create no interference or disconnection between the tasks done from the web and from the command line. This last feature is specially useful

By default Cockpit, stable version, comes installed and enabled in Fedora 21 Server. It also can be found in CentOS/RHEL 7 Atomic, Fedora 21 Atomic and Fedora 21 Cloud, and there are plans in the near future to support Arch Linux.

Lets review now some of the features of Cockpit, as said before multiple servers can be managed from the same Cockpit instance.

Screen Shot 2014-12-31 at 19.26.00

Once you access one of managed nodes it will present general overview of the server with real-time charts of CPU, Memory, Disk I/O and Network Traffic.

Screen Shot 2014-12-31 at 19.37.53

On the left pane there are a series of actionable items that will give you access to the different subsystems of the node like Networking, Storage, User Accounts and even the status of the Docker containers running on the server, if the Docker service has been enabled.

System services view.

Screen Shot 2014-12-31 at 19.52.53

When a process is selected Cockpit will display its details.

Screen Shot 2015-01-08 at 12.11.27

Networking area displays traffic for the selected interface, the journal of the networking system and even allows you to create a new bond interface, a new bridge or add a new VLAN tag to the interface.

Screen Shot 2014-12-31 at 19.53.18

The Storage view will display similar info for the disks, and will display detailed information for each of them, review the LVM configuration of the server and perform different storage related operations.

Screen Shot 2014-12-31 at 19.53.45

Journal view lets you review systemd journal. You can go back seven days into the log and filter on the type of messages.

Screen Shot 2014-12-31 at 19.54.29

After using Cockpit for some time in my lab I can say that I genuinely love it, the interface is pretty fast, it uses systemd for everything and it does not interface with my console-based admin habits, on the contrary is a great complement to them.


Every customer usually asks about how to monitor their vCenter Chargeback installations, hence I finally decided to write a small post listing the services and processes of the different Chargeback components.

Windows Service Path to executable
VMware vCenter Chargeback C:\Program Files (x86)\VMware\VMware vCenter Chargeback\apache-tomcat\bin\tomcat6.exe
VMware vCenter Chargeback – VMware Cloud Director DataCollector C:\Program Files (x86)\VMware\VMware vCenter Chargeback\VMware Cloud Director DataCollector\JavaService.exe
VMware vCenter Chargeback – vShield Manager DataCollector C:\Program Files (x86)\VMware\VMware vCenter Chargeback\vShield Manager DataCollector\JavaService.exe
VMware vCenter Chargeback DataCollector-Embedded C:\Program Files (x86)\VMware\VMware vCenter Chargeback\DataCollector-Embedded\JavaService.exe
VMware vCenter Chargeback Load Balancer C:\Program Files (x86)\VMware\VMware vCenter Chargeback\Apache2.2\bin\httpd.exe

Bear in mind that if vShield and vCloud DataCollectors are installed on the same server as Chargeback Server the path will be slightly different:

VMware vCenter Chargeback – vShield Manager DataCollector-Embedded C:\Program Files (x86)\VMware\VMware vCenter Chargeback\vShield Manager DataCollector-Embedded\JavaService.exe
VMware vCenter Chargeback DataCollector-Embedded C:\Program Files (x86)\VMware\VMware vCenter Chargeback\DataCollector-Embedded\JavaService.exe


I found this error last week during a deployment in a customer. The vCenter Infrastructure Navigator appliance does not maintain its configured hostname after a reboot, it gets reset to the default localhost.localdom value.


Setting it again in the administration web interface doesn’t solve problem, it will be lost again after the next reboot.

The problem is in the vami_set_hostname script, it has a HOSTNAME variable set to localhost.localdom and if it fails to make the reverse lookup of the hostname from the IP address using the host command it will be set to the default value.


To fix this edit that file, it can be found on /opt/vmware/share/vami, and set the value of the variable to your hostname. After that reboot the appliance to check that everything works as expected.


vCenter Chargeback Manager gives you the possibility during the installation process to generate an SSL certificate. But this certificate is generated with an expiration period of 60 days.

No problem with that, you can always regenerate it again. Actually Chargeback provides the mechanism to do it. The process can be launched from the vCenter Chargeback Manager Tools folder as it can be seen in the screenshot below.


However this new certificate will come with the same limitation of 60 days of valid period. To easily avoid that we only need to edit the .bat file that generates the certificate and modify the correspondent value.

The script is called Generate_Ssl_Certificate.bat and can be found C:\Program Files(x86)\VMware\VMware vCenter Chargeback\Apache2.2\bin.


Edit the file using your favorite text editor, in my case I’m using Notepad++, and go to line number 72 as in the capture.


Change the –days flag from 60 to your desired value, in the example the value is 365 this is the certificate will expire in one year.

After we can launch the generation process. The batch file will open a cmd window, stop the CBM Load Balancer service and asks for the passphrase of default.key. You’ll have to enter it three times and after that the process will ask for information about the State, City, common name (usually the server FQDN), company name, email, etc.


After that it will generate the new certificate and will start Load Balancer services.



I got aware of this issue last week after installing a Fedora 18 virtual machine on Fusion 5. The installation of the Tools went as expected but when the install process launched the vmware-tools-config,pl script I got the typical error of not being able to find the Linux Kernel headers.

Searching for a valid kernel header path...
The path "" is not a valid path to the 3.7.2-204.fc18.x86_64 kernel headers.
Would you like to change it? [yes]

I installed the kernel headers and devel packages with yum.

[root@fed18 ~]# yum install kernel-headers kernel-devel

Fired up again the configuration script and got the same error. The problem is that snce kernel 3.7 all the kernel header files have been relocated to a new path and because of that the script is not able to find them. To solve it just create a symlink of the version.h file from the new location to the old one.

[root@fed18 src]# ln -s /usr/src/kernels/3.7.2-204.fc18.x86_64/include/generated/uapi/linux/version.h /lib/modules/3.7.2-204.fc18.x86_64/build/include/linux/

With the problem fixed I launched the config script again and the tools finally got configured without problems.

[root@fed18 ~]# 

Making sure services for VMware Tools are stopped.
Stopping Thinprint services in the virtual machine:
 Stopping Virtual Printing daemon: done
Stopping vmware-tools (via systemctl): [ OK ]

The VMware FileSystem Sync Driver (vmsync) allows external third-party backup 
software that is integrated with vSphere to create backups of the virtual 
machine. Do you wish to enable this feature? [no]

Before you can compile modules, you need to have the following installed...
kernel headers of the running kernel

Searching for GCC...
Detected GCC binary at "/bin/gcc".
The path "/bin/gcc" appears to be a valid path to the gcc binary.
Would you like to change it? [no]

Searching for a valid kernel header path...
Detected the kernel headers at 
The path "/lib/modules/3.7.2-204.fc18.x86_64/build/include" appears to be a 
valid path to the 3.7.2-204.fc18.x86_64 kernel headers.
Would you like to change it? [no]