Zen Load Balancer 3.03 - Error in content3-3 cgi execution, see ZEN logs

UPDATE: Zen Load Balancer 3.05 has been released and resolves the issue mentioned below. http://www.zenloadbalancer.com/downloads/

"... Improvements:
[gui] HTTP farm client request timeout with unit “seconds”
[gui] Refreshing timeout for farms status view
[cluster] Improved RSA synchronization
[gui] Advise message regarding no cluster configuration found
 ..." - http://www.zenloadbalancer.com/changelog/#August_26_2014_Zen_Load_Balancer_v305

---

With the open source 3.03 version of ZenLB, I believe I've isolated one cause for the "Error in content3-3 cgi execution, see ZEN logs" message when attempting to configure a Cluster and using the "Configure RSA connection between nodes" button.

I reported the bug to http://zenloadbalancer.sourceforge.net/bugs and been told “[t]hat is a bug that is solved for the next 3.05 version” by J. Robels.

Reproducing the error:

1. Configure two new Zen load balancers – TestLB1 and TestLB2
2. Connect them in a cluster
3. Disconnect the cluster
4. Rebuild the OS for TestLB2 with an identical configuration – IP address and settings
5. Attempt to configure a cluster again between the Zen load balancers TestLB1 and TestLB2 from TestLB1’s admin interface
6. You will get "Error in content3-3 cgi execution, see ZEN logs" using the "Configure RSA connection between nodes" button
   
   
   

Cause:

• The SSH keys on TestLB1 /root/.ssh/known_hosts file is no longer valid for referencing TestLB2.
   

Workaround prior to ZenLB 3.05:

• Delete the /root/.ssh/known_hosts file from TestLB1.
   
  

You will now get the "SUCCESS! Running process for configure RSA communication. " when pressing the "Configure RSA connection between nodes" button.

DPM 2012 and Beyond Frustration

All of our Hyper-V Clusters, Server 2008 R2 hosts, started having failed backups inside our two independent Data Protection Managers. The problem initially progressed from one node consistently fail backups for virtual machines and the other hosts kept performing backups, until all of our nodes could no longer could make successfully backups of any virtual machines. Our standalone backups via DPM had no issue. These hosts had been configured and unchanged for well over a year - only Windows patches months prior and anti-virus updates were continuously loading.

DPM kept stating for the failed backups that "The VSS application writer or the VSS provider is in a bad state ... ID 30111: VssError:A function call was made when the object was in an incorrect state for that function(0x80042301)) and the local nodes wrote VSS 12362 Application Log Event Errors "A Shadow Copy LUN was not detected in the system and did not arrive" and VSS 12363 Application Log Event Errors "An expected hidden volume arrival did not complete because this LUN was not detected" whenever we attempted to run full virtual machine backup via a Consistency check.

We had tried and didn't work...

• Power cycling all of the equipment involved: Hyper-V Servers (PowerEdge R710's), the iSCSI SAN (EqualLogic PS4000vx's), the switches connecting them (Catalyst 3750X's), and our DPM server
• Unregistering and Registering the EqualLogic VSS provider (eqlvss /unregserver and eqlvss /regserver)
• Removing virtual machines from a protection group (deleting disk data) and adding them back
• Moving virtual machines to a new protection group
• Upgrading the EqualLogic Windows Host Integration Toolkits (HIT kits) on the Hyper-V nodes - upgraded from 4.0 to 4.6
• Installing the EqualLogic HIT kit on one of the virtual machines
• Patching the Hyper-V nodes to all of the latest Windows Updates - even yesterdays released kb 2908783 which resolves issues with corruption of iSCSI LUNs in Windows Server 2008 R2 and 2012

... and still no success.

After much time wasted on what seemed to be magic potions and DPM's hatred of backing up critical data, a random thought of trying to disable our anti-virus on the cluster nodes resolved the issue! Yeah, I know they say to disable anti-virus on everything and everywhere you read, but we have had Microsoft Forefront Client Security on these systems configured and running since we setup these servers 2+ years ago. Apparently, some change in the definitions or just its mood decided to start messing with the iSCSI VSS Hardware process... and messing with my sleep over the last two days.

Good luck!

 

 

DPM 2012 Not Generating E-mail Reports after Upgrading to SP1

We have been using DPM 2012 for quite a while now. We also have the reports set to deliver reports daily/weekly. After we upgraded to SP1, we noticed it no longer was e-mailing us the reports, even though the alerts for errors continued to come. We also could run reports manually, but no automatic e-mails.

Went to clear and recreate the report schedule and set it to e-mail us, and we got this awesome non-descript error ID: 3014. "An error occurred causing the reporting job on to fail. The system files may be corrupt. Retry the reporting task. If the problem persists, repair your DPM installation using the steps described in the System Center 2012 Service Pack 1 DPM Deployment Guide. ID: 3014"

 

I checked out the guide and the basic idea to "repair" is uninstall and reinstall. I don't know about you all, but risking loosing backup data just to fix reporting didn't sit well. So, I proceeded to evaluate what was occurring with the SQL Server Profiler on our system and comparing it to our secondary server.

 

After playing around with it for hours, seemed to narrow down that it was an issue with permissions for the Reporting Services predefined database role called RSExecRole. I went through this guide Create the RSExecRole (http://technet.microsoft.com/en-us/library/cc281308.aspx), used to recreate permissions during a report database move, and we were able to recreate the e-mail subscriptions.  It looks like there must have been some undetected failure during the SP1 upgrade.

BlackBerry Services Not Starting Correctly

I found a couple of articles related to an issue with our Blackberry Enterprise Services not communicating with our user's BlackBerry phones. Apparently, the services didn't start correctly. We had been receiving the following events:

Event ID : 20000
Source : ExchangeConnector, BlackBerry Messaging Agent Localhost Agent 1, BlackBerry Policy Service, BlackBerry Synchronization Service
Description : BIPPeClient::Connect: Error calling host "localhost" 127.0.01 (10061)

The cause is that BlackBerry services need to be started in the correct order or you’ll basically experience the lack connectivity and communication issues that we had been experiencing.
The official Blackberry Support article KB13718 describing how to properly restart or start the Blackberry Services, which states:
"...
The correct order to start the BlackBerry Enterprise Server services is detailed below:

1.BlackBerry Router
2.BlackBerry Dispatcher
3.BlackBerry Controller
4.All remaining BlackBerry Enterprise Server services
..."
-http://btsc.webapps.blackberry.com/btsc/search.do?cmd=displayKC&docType=kc&externalId=KB13718

I found the blog post, by John McLagan, which took it a step further by showing how to assign service dependencies, so that the services are FORCED to start in the correct order by using regedit. You can check his steps out, but you’ll see my steps below that I feel are less precarious then jumping into the registry, but keeps the underlying premise. http://www.johnmclagan.com/blog/2009/2/24/blackberry-messaging-agent-20000-is-showing-in-the-applicati.html

I liked everything John said to do except the regedit part, so I substituted it for the old service control command (Sc.exe). As Microsoft will tell you, "... You can use Sc.exe to help develop services for Windows. Sc.exe, which is provided in the Resource Kit, implements calls to all of the Windows service control application programming interface (API) functions. You can set the parameters to these functions by specifying them on the command line. Sc.exe also displays service status and retrieves the values stored in the status structure fields. ..." http://support.microsoft.com/kb/251192

I feel it’s safer to edit the services using Sc.exe than using the registry (regedit.exe) and it left out the required reboot of the BlackBerry Server system in John's directions. I tested it on our Windows Server 2003 R2 SP2 running Blackberry Enterprise Server for Microsoft Exchange 5.0.1.

1. Open the command prompt
2. Set the Blackberry Router, Dispatch, and Controller services be dependent using the sc (service control command) by typing:
   

   
   sc config "Blackberry Controller" depend= "Blackberry Dispatcher"
   sc config "Blackberry Dispatcher" depend= "Blackberry Router"
   

   

 You should know have the dependencies visible from within the service control panel. You also will now be able to restart the three main services by just restarting the Blackberry Router services.

Note: Our Mail Store Service, Policy Service, and Synchronization Service are set to manual and stop and restart with the other three services, so I didn’t set their dependencies. The Blackberry KB mentions to set dependencies to the other services, but in our case, I didn’t see the need.

If you have a minute, comment if it did or didn’t help with a why and how. Thanks!