Install EMC NetWorker agent in VMware vCenter Server Appliance 5.5

Download PDF

When running a virtualized vCenter server it is very important to have a working backup. If the vCenter server crashes the virtual landscape cannot be managed or monitored.

Configuring backup is very easy when running VMware vCenter Server on a conventional Microsoft Windows servern because it is a fully-features operating system. In case you’re using the VMware vCenter Server Appliance configuring backup might be more complex. By default this system doesn’t come with pre-installed backup agents because it is assumed that a “agentless” backup solution for virtual machines is used.

Not every company uses backup solutions that are offering “agentless” backups. If this function is missing the backup is missing as well – which is very unfavorable for productive environments. For example – if you’re using EMC NetWorker in an older version agentless backups aren’t possible.

The vCSA is based on SUSE Linux Enterprise Server 11 SP2 which means that you can extend its software using RPM packages – if your backup software comes also in a adequate format. Unlike many other virtual appliances I saw recently it is still possible to gain root access to the system. So it is basically possible to installed additional needed software – of course this is not covered by the VMware support.

Installation of EMC NetWorker

EMC generates generic NetWorker RPMs for RPM-based Linux distros. Amongst others this software package is compatible with Red Hat Enterprise Linux and SUSE Linux Enterprise Server – which means that it is suitable for installation inside the vCSA.

It is necessary to resolve some dependencies for EMC NetWorker. The following software packages need to be installed (might differ depending on your NetWorker version):

  • libcap1*.x86_64.rpm
  • libstdc++33*.x86_64.rpm
  • ksh-93u*.x86_64.rpm

And where do you get these packages from? Bascially you’ll need a valid subscription for downloading SLES packages. SUSE also offers trial versions for their own enterprise products – including SLES: [klick mich!]

Owners of vSphere Standard or a higher edition also had the possibility to get product patches and updates (ended 07/25/2014, see here) at no fee in the “SUSE Linux Enterprise Server for VMware” programme. To use this it was needed to register the ESXi serial number at SUSE. Afterwards it was possible to activate installations by using activation codes: [click me!]

After a short registration it is possible to download a test version including 60 days free update support. Basically only the first DVD is needed – underneath the folder “suse/x86_64” the required RPM packages can be found. It recommended to download the SLES release SP2 the vCSA is based on instead of SP3.

For installing the EMC NetWorker agent the packages lgtoman and lgtoclient are sufficient. The RPM packages extracted from the DVD are copied to the vCSA using SSH/SCP and installed together with the agent:

# zypper localinstall lib*.rpm ksh*.rpm lgtoclnt*.rpm lgtoman*.rpm

Service configuration

Before NetWorker is started for the first time it is important to create the required folder structure and a list of valid backup servers. If NetWorker is installed before those information were served the agent isn’t working properly and might be reinstalled (because the erroneous information are cached).

# mkdir -p /nsr/res
# echo "backupserver.fqdn.loc" > /nsr/res/servers
# chmod -R 755 /nsr/*
# chkconfig rpcbind on
# chkconfig networker on
# service networker start
starting NetWorker daemons:
 nsrexecd

The services rpcbind and networker need to be active to make sure that backups can be created:

# service rpcbind status
Checking for service rpcbind                              running
# service networker status
+--o nsrexecd (PID)

Backup scripts

In many companies it is very common to create offline backups weekly, e.g. at the weekend. Machines are often backed up “online” daily which means that application services aren’t stopped. As a result not all data can be copied consistently because some files (e.g. database files) are in use. These files are backed up during the offline backup.

For controlling offline backups using EMC NetWorker shell scripts are created. Those scripts are executed before and after runing the backup job.

For creating the backup scripts I was guided by the following VMware KB articles:

My scripts:

# vi /opt/start_offline_backup.sh
#!/bin/sh
service vmware-vpxd stop
service vmware-inventoryservice stop
/opt/vmware/vpostgres/1.0/bin/pg_dump INSTANCE -U USER -Fp -c > /tmp/VCDBackUp
/usr/lib/vmware-vpx/inventoryservice/scripts/backup.sh -file /tmp/InventoryServiceDB.DB

ESC ZZ

# vi /opt/stop_offline_backup.sh
#!/bin/sh
service vmware-vpxd start
service vmware-inventoryservice start

ESC ZZ

# chmod +x /opt/st*_offline_backup.sh

The variables INSTANCE and USER need to be customized – the appropriate values can be gathered from the configuration file /etc/vmware-vpx/embedded_db.cfg. It is also necessary to check whether the file system /tmp offers enough storage capacity for holding the backups.

Finally I’d like to mention that this procedure is working pretty fine but it is not covered by the VMware support. It is a good idea to revert those changes before opening a support case (or installing appliance updates!).

Download PDF

POODLE – and how to get rid of it

Download PDF

A couple of days ago another security vulnerability that applies to Linux systems called PODDLE was announced. Less serious than Heartbleed especially web servers that are still allowing SSL generations 2 and 3 are affected. Because of a bad security design it is possible to decrypt transfered data. Often those protocol versions are allowed in the default configuration shipped by many Linux distributions – so administrators should really harden their servers. In the meantime CVE 2014-3566 was created to describe POODLE – the security vulnerability was detected by Google.
To fix the issue it is sufficient to simply disable the older protocol generations. For Apache this is done by altering the appropriate configuration file:

#SSLProtocol All
SSLProtocol All -SSLv2 -SSLv3

This directive enables all SSL protocol versions except the 2. and 3. generation.

Poodle Protector

If you’re maintaining a big amount of systems manual configuring the affected systems also means unnecessary work that can be automated very easily. Because I’m a lazy person I developed a script which can analyse and automatically customize the configuration of Apache servers vulnerable to the POODLE attack. The script (poodle_protector) can be found on GitHub: [click me!]

The script can also restart the appropriate service which makes it really comfortable to use in combination with a central configuration management (like Red Hat Satellite, Spacewalk or SUSE Manager).

The following command analyses the system and simulates which changes would be made (dry-run):

# ./poodle_protector.py -l
I'd like to create a backup of '/etc/apache2/mods-available/ssl.conf as '/etc/apache2/mods-available/ssl.conf.20141016-1303' ...
I'd like to insert 'SSLProtocol All -SSLv2 -SSLv3' into /etc/apache2/mods-available/ssl.conf using the following command: sed -i '/SSLProtocol/ c\SSLProtocol All -SSLv2 -SSLv3' /etc/apache2/mods-available/ssl.conf ...
I'd also like to restart the service using: ['service httpd restart', 'service apache2 restart']

The next command alters the configuration (backups are created before) and restarts the Apache service:

# ./poodle_protector.py -r
httpd: unrecognized service
Restarting web server: apache2 ... waiting .

In this example a Debian system was used – that’s why the httpd service can’t be found.

Download PDF

Red Hat Enterprise Linux 6.6 released

Download PDF

Yesterday Red Hat released another update of its Red Hat Enterprise Linux major release 6: 6.6. Like for the minor updates before many optimizations and some „technical previews“ were implemented.

The changes have been documented well in the release and technical notes:

Beside common kernel driver updates some other interesting customizations were made – I’d like to list some of those:

  • Installation as Hyper-V second generation VM – e.g. under Windows Server 2012 R2 (*), this also includes new Hyper-V Daemons: Hyper-V KVP (Hyper-V Key Value Pair) Hyper-V VSS (Hyper-V Volume Shadow Copy Service)
  • Improved support of additional SCSI signals for better hardware change handling using udev (e.g. size changes, thin-provisioning status, adding new LUNs,…)
  • Implemented a Open vSwitch module for additional Red Hat products, support is only offered in combination with other Red Hat products
  • For device mapper a caching modul (dm-cache) was introduced (*). This module can use faster drives (e.g. SSDs) as cache for slower storage media – details can be found in the lvmcache manpage
  • The software packages keepalived and haproxy are now fully covered by the Red Hat support
  • OpenJDK 8 Java Runtime Environment can be installed optionally (*)
  • Windows 8-certified touch screens are now supported by hid-multitouch
  • Red Hat Enterprise Linux 6.6 is now NSS FIPS-140 Level-1 certified
  • System Security Services Daemon (SSSD) was optimized for better authentification in combination with Microsoft Active Directory
  • gdisk – new tool for GPT partitioniing, the „look and feel“ is alike fdisk
  • rsyslog7 – new, overhauled rsyslog version witn improved encryption and external database support (MySQL, PostgreSQL,…). It is recommended to migrate to this most recent version

(*) = technical preview

Red Hat Enterprise Linux 6.6 is now available to all Red Hat customers with a valid subscription.

Download PDF

Short tip: uninstall pkg applications under OS X

Download PDF

The Apple App Store is not the only way to installation additional applications under OS X – another way is to install .pkg files (which have been created by the Apple Installer Framework).

Unfortunately not all applications installed using these files are present in the “Programs” category inside Finder. Uninstalling these applications using drag & drop inside the recycle bin is impossible.

But there is a tiny tool called General Package Uninstaller – it can be downloaded on GitHub. The program lists installed applications – and uninstalls them:

General Package Uninstaller

General Package Uninstaller

Download PDF

Virtualized cluster storage using VMware ESXi

Download PDF

Many software cluster solutions require shared storage between all cluster nodes. A very prominent example for this might be Oracle RAC (Real Application Clusters). In enterprise environments shared cluster storage is often implemented using SAN storage that is connected to multiple systems.

For (private) test scenarios a SAN storage system might exceed the budget. Fortunately there is a possibility to provided virtualized shared storage. If you’re using VMware ESXi the keyboard for this is “multi-writer“. ESXi uses its own file system called VMFS for local and iSCSI storage – this file system automatically creates locks to make sure that particular files cannot be accessed from multiple virtual machines (unless you’re using Fault Tolerance). By disabling this behavior it is possible to access virtual hard drives (.vmdk files) parallely from up to 8 virtual machines.

The advantages and disadvantages of this configuration are described very detailed in a VMware knowledge base article – I’d like to thank my colleague Johannes who recommend this solution. :)

In every case using a cluster lock manager (e.g. dlm) between the appropriate virtual machines is mandatory – otherwise the parallel access causes data inconsistencies. Implementing the storage between two or multiple machines is quite easy:

  1. Adding a “Thick provisioned – Eager zeroed” virtual hard drive to the first virtual machine. The hard drive needs to reside on a iSCSI or – if all the virtual machines are running on the same host – local SCSI/SAS storage, NFS is not supported! You may want to use the same SCSI-ID on all virtual machines.
  2. Turning off all concerned virtual machines.
  3. Modifying the .vmx configuration file of the first virtual machine by addind the following parameter:scsiX:Y.sharing = “multi-writer”

    The SCSI-ID needs to be set to match the previously created cluster hard drive. For customizing the configuration file you might want to use vSphere client or vi over SSH – I was not successful with setting the parameter using the Web client multiple times.

  4. Adding a pre-existing hard drive to the other virtual machines by specifying the path to the hard drive created previously.
  5. Anpassen der .vmx-Konfigurationsdateien analog zu Schritt 3.

Afterwards the virtual machines can be started. After this customiztation the following ESXi functions and features aren’t supported anymore:

  • snapshots and online expansion of the shared hard drive
  • hibernating the virtual machine
  • cloning
  • Storage vMotion
  • Changed Block Tracking (important for “agentless” backup solutions)
  • vSphere Flash Read Cache

Afterwards the cluster can be configured to use the new hard drive. If you don’t have a cluster yet you can also temporarily create and mount a conventional file system on the particular nodes:

nodeA # mkfs.ext4 /dev/sdb
nodeA # mkdir /cluster ; mount /dev/sdb /cluster
nodeA # echo "mimimi" > /cluster/test
nodeA # umount /cluster
nodeB # mkdir /cluster ; mount /dev/sdb /cluster
nodeB # cat /cluster/test
mimimi
nodeB # umount /cluster
...

I’d like to repeat that the parallel access of multiple virtual machines to the same hard drive without using a cluster lock manager can cause data inconsistencies very easily. With this in mind – happy vClustering! :)

Download PDF

Short tip: configure hard drive standby with systemd

Download PDF

Hard drives can be forced into hibernate using hdparm. While it was quite easy to implement automatic setting this hibernation values by inserting the appropriate command (hdparm -B intervall device) into /etc/rc.local (which was executed after the boot) on sysvinit-based Linux distributions this changed on newer systemd-based systems. It is a good idea to implement this behavior as a service.

First you need to create a system-wide service and activate and start it afterwards:

# vi /usr/lib/systemd/system/sda-spindown.service
[Unit]
Description=Set HDD spindown

[Service]
Type=oneshot
ExecStart=/sbin/hdparm -B 241 /dev/sdb
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

ESC ZZ
# systemctl daemon-reload
# systemctl enable sda-spindown.service
# systemctl start sda-spindown.service

You can verify that the value has been set successfully using the service management:

# systemctl status sda-spindown.service
sda-spindown.service - Fix excessive HDD parking frequency
   Loaded: loaded (/usr/lib/systemd/system/sdd-spindown.service; enabled)
   Active: active (exited) since Sa 2014-10-11 14:27:16 CEST; 3min 32s ago
   Process: 4336 ExecStart=/sbin/hdparm -B 241 /dev/sda (code=exited, status=0/SUCCESS)
...

Many thanks to the following blog that gave me the tip: [click me!]

:)

Download PDF

Short tip: Gather HP SmartArray cache battery information under HP-UX

Download PDF

When a RAID controllers cache battery fails it is a benefit if you can gather the spare part number without any downtime. If your server is equipped with a HP SmartArray controller you can easily get t hose information using the sautil utility. This requires the HP-UX products RAIDSA and RAIDSA-PROVIDER to be installed. For displaying the information the hardware path of the affected controller is gathered and passed to sautil:

# ioscan -funCext_bus
Class       I  H/W Path        Driver       S/W State   H/W Type     Description
=================================================================================
...
ext_bus     1  0/6/0/0/0/0/4/0/0/0      ciss         CLAIMED     INTERFACE    PCIe SAS SmartArray P400 RAID Controller
                              /dev/ciss1

# sautil /dev/ciss1|more
---- ARRAY ACCELERATOR (CACHE) INFORMATION -----------------------------------
  Array Accelerator Board Present?.... yes
  Cache Configuration Status.......... write cache temporarily disabled (code=1)
  Cache Ratio......................... 25% Read / 75% Write
  Total Cache Size (MB)............... 208
    Read Cache........................ 052
    Write Cache....................... 156
    Transfer Buffer................... 000
  Battery Pack Count.................. 1
  Battery Status (pack #1)............ FAILED

In this case the cache is 208 MB which is consistent with the 256 MB P400 model variant. There is also a 512 MB model with a completely different cache battery. If you want to make sure you’re ordering the right part you can check the cache size mentioned in the firmware by using the HP-UX Support Tool Manager. For this you might need to install the HP-UX product Sup-Tool-Mgr. The tool also requires you to specify the controller’s hardware path:

# echo "selclass hwpath 0/6/0/0/0/0/4/0/0/0;info;wait;infolog"|/usr/sbin/cstm|more
...
Controller : HP SmartArray RAID Controller
Device File: /dev/ciss1
Hardware Revision: 'E'
Firmware Revision (Currently Running): 7.22
Firmware Revision Rom: 7.22
Boot Block Revision: 0.04
Total Cache Size (MB) : 256
Battery Pack:    Battery Count : 1
Battery Status:  Battery Number ( 1 ) : BATTERY FAILED
Battery Voltage: Battery Number ( 1 ) : below reference voltage

In thie case it is really the 256 MB model variant which requires a spare part with the number 383280-B21. The 512 MB BBWC model variant needs a spare part with the unique identifier 398648-001 (sources: [klick!] und [klick!]).

Download PDF

DELL OpenManage Integration Virtual Appliance update problems

Download PDF

DELL offers an Linux appliance based on CentOS 5 for customers using VMware products. This appliance is called  “DELL OpenManage Integration for VMware vCenter Appliance” and integrates seamless in VMware vCenter Server (or the appliance). It can be used for monitoring physical servers and also allows controlling the particular servers remotely. Amongst others it is possible to update firmware versions and check warranty information.

Recently I had problems with updating the appliance using the web interface. The update was started and the appliance was restarted but the version number didn’t change at all. In an older documentation I found a hint about a log file named /usr/share/tomcat5/rpmupdate.log – the update process is recorded in this file. Because even the administrator account has no shell access to the appliance it would be necessary to use a live CD. Fortunately this effor isn’t needed because there is a button for creating troubleshooting bundles in the web interface. You can find the mentioned file rpmupdate.log in the ZIP file offered for download. (if you prefer the live CD solution the file can be found underneath /usr/share/tomcatSpectre/logs in appliance version 2.x)

rpmupdate.log in Fehlerbehebungspaket

rpmupdate.log in Fehlerbehebungspaket

In my case YUM wasn’t able to download RPM packages the protocol told very clearly:

$ cat rpmupdate.log
Stopping httpd: [  OK  ]
Loaded plugins: fastestmirror
Cleaning up Everything
Cleaning up list of fastest mirrors
Loaded plugins: fastestmirror
Determining fastest mirrors
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package Spectre-OMSA-Repo.i386 0:1.0.3-1 set to be updated
--> Processing Dependency: dell-omsa-esx40 >= 1.0.1 for package: Spectre-OMSA-Repo
...
Total download size: 610 M
Downloading Packages:
http://linux.dell.com/repo/hardware/vcenter-plugin/latest/spectrebinaries/dell-omsa-esxi51-1.0.1-1.i386.rpm: [Errno 12] Timeout: 
Trying other mirror.
http://linux.dell.com/repo/hardware/vcenter-plugin/latest/spectre/spectre-webclient-package-2.2.0-254.1.noarch.rpm: [Errno 12] Timeout: 
Trying other mirror.
http://linux.dell.com/repo/hardware/vcenter-plugin/latest/centos_updates/kernel-2.6.18-371.8.1.el5.i686.rpm: [Errno 12] Timeout: 
Trying other mirror.
http://linux.dell.com/repo/hardware/vcenter-plugin/latest/spectre/spectre-deps-2.2.0-255.1.i386.rpm: [Errno 12] Timeout: 
Trying other mirror.
http://linux.dell.com/repo/hardware/vcenter-plugin/latest/spectrebinaries/dell-dtk-4.0.1-1.i386.rpm: [Errno 12] Timeout: 
Trying other mirror.

Error Downloading Packages:
  dell-dtk-4.0.1-1.i386: failure: spectrebinaries/dell-dtk-4.0.1-1.i386.rpm from Dell_VC_Plugin: [Errno 256] No more mirrors to try.
  dell-omsa-esxi51-1.0.1-1.i386: failure: spectrebinaries/dell-omsa-esxi51-1.0.1-1.i386.rpm from Dell_VC_Plugin: [Errno 256] No more mirrors to try.
  spectre-deps-2.2.0-255.1.i386: failure: spectre/spectre-deps-2.2.0-255.1.i386.rpm from Dell_VC_Plugin: [Errno 256] No more mirrors to try.
  kernel-2.6.18-371.8.1.el5.i686: failure: centos_updates/kernel-2.6.18-371.8.1.el5.i686.rpm from Dell_VC_Plugin: [Errno 256] No more mirrors to try.
  spectre-webclient-package-2.2.0-254.1.noarch: failure: spectre/spectre-webclient-package-2.2.0-254.1.noarch.rpm from Dell_VC_Plugin: [Errno 256] No more mirrors to try.

My network requires using a proxy server – the configuration was altered using the web interface to set a proxy server. This process was also successful like a test function acknowledged. It seems like it was forgotten to also set the proxy configuration for YUM.

To fix this issue I altered the YUM repository configured in the appliance. This customization requires a live CD because the appliance offers no shell access. Even booting into single-user mode is impossible because GRUB was protected by a password. The VM needs to boot vom a Linux CD image instead of the hard drive. Afterwards the detected hard drive is mounted and the YUM repository configuration gets altered:

# mount /dev/sda1 /crash
# vi /crash/etc/yum.repos.d/spectre.repo
...
proxy=http://ip:port

ESC ZZ

I really disadvise from directly updating the appliance using YUM after doing a chroot because a update started using the web interface seems to do even more steps. In my case the appliance was successfully updated and rebooted twice after the customization. During the first reboot some packages were removed which was definitely not triggered by “yum update“:

Geskriptete Entfernung einiger RPM-Pakete

Geskriptete Entfernung einiger RPM-Pakete

After both reboots the appliance version number has changed:

Erfolgreiches Update

Erfolgreiches Update

Download PDF

Pebble Steel Black Matte

Download PDF
Endlich lieferbar!

Endlich lieferbar!

I was very happy when I heard a couple of weeks ago that the coveted Pebble smartwatch can now also be ordered in Germany after a long time. Until now this was a problem due to duty difficulties. It seems like this difficulties have now been solved – ordering the watch in the Netherlands is now possible. After I’ve been waiting for a long time for this update I really had to order it instantly. The delivery time was quite short in spite of the huge demand. :)

Why Pebble?

But why did I choose the Pebble and not yet another smartwatch like the Samsung Galaxy Gear? Quite simple – I prefer the design concept of the smartwatch financed by Kickstarter for several reasons.

Pebble Steel

Pebble Steel

Other smartwatches try to combine smartphone technology with a compact clock case. As a result you’re wearing more or less powerful ARM devices with the Android operating system and color displays on your wrist. Disadvantages of this are short battery lifes and – in my opinion – overloaded operating systems.

The Pebbler watch is here more minimalist and only offers what I expect from a intelligent watch: displaying the time and establishing a connect to my smartphone. Color displays, a camera, a big memory and WLAN are missing. This might sound counterproductive but results in better battery life (up to 7 days). The pebble provides:

  • STM32F205RE Cortex M3 CPU with 120 Mhz
  • 128 KB memory
  • 4 MB memory for apps and watchfaces (clock face)
  • 4 buttons
  • Bluetooth 4.0
  • 144×168 Sharp e-paper LCD
  • 3 axis accelerometer
  • PeebleOS operating system, a minimalistic FreeRTOS derivate
  • Selectively a plastic (Pebble) or metal case (Pebble Steel) which is waterproof in up to 50 meters
  • Plastic, leather or aluminium bracelet (alternatively you can mount the most 22mm wristbands)

The smartwatch is connected to your iOS or Android device using the official Pebble app. After installation you can use the following functions:

  • Show notifications (E-Mail, Twitter, WhatsApp, Facebook, etc.)
  • Download apps and watchfaces
  • Control music

Developers will like the Pebble Software Development Kits (SDK) because it enables you to develop your own JavaScript and C applications for the smartwatch.

Pebble Apps

Using the Pebble App Store it is possible to sell and download applications. You can find a lot of useful apps – I’m using the following ones:

  • K9ToPebble – Mails received by Kaiten / K9 can be displayed as notifications (incl. text details), the Inbox can also be browsed using the menu
  • Multi Timer – stopwatch, countdown and lap counter, very useful to keep remaining time in mind during presentations

Watchfaces

Red Hat Watchface (Watchface-Generator)

Red Hat Watchface (Watchface-Generator)

It is also very nice to have alternative clock faces. The offer is also very big here – I’m using the following watchfaces:

The Pebble Watchface-Generator is a very useful web application that can easily create clock faces for you – without having programming knowledge. After creating a clock face it can be downloaded by scanning a QR code. The watchface is installed on your watch using the official Pebble app.

Conclusion

After roughly 2 months I’m still very satisfied with the Pebble Steel – I can highly recommend having a look at the Pebble if you’re looking for a “unbloaty” smartwatch with a decent and timeless design without. I’m using this watch nearly every day and don’t want to miss it anymore. :)

Download PDF

Short tip: Mount optical media under Solaris

Download PDF

I’m not administrating Solaris systems very often – it is even more rare that I need to mount optical media on Solaris. This is just a short reminder so that I don’t need to use a search engine everytime.

First of all the CD / DVD drive is determined by the iostat command. Afterwards the drive is mounted using the first device slice.

# iostat -En
c8t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: VMware   Product: Virtual disk     Revision: 1.0  Serial No:
Size: 21,47GB <21474836480 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 4 Predictive Failure Analysis: 0
c7t0d0           Soft Errors: 0 Hard Errors: 5 Transport Errors: 0
Vendor: NECVMWar Product: VMware IDE CDR10 Revision: 1.00 Serial No:
Size: 0,35GB <349175808 bytes>
Media Error: 0 Device Not Ready: 5 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
# mount /dev/c7t0d0s0 /mnt

Depending on the media type is might be necessary to select the file system (e.g. -o hfs).

Download PDF