Problem: Nbpem may crash after changing policy schedule type from Calendar to Frequency

Solution: The workaround is to be aware of the appropriate values in the schedule. Ultimately, delete the policy to get nbpem to load.

Here is a step by step recap:

8 RMAN jobs were failing with status 25 (cant connect) connection errors while bprd connected successfully. Found no transmission control protocol, host resolution, or network issues.

RMAN log showed that it was failing after sending policy info

dbclient

0:13:09.334 [3020] <2> async_connect: [vnet_connect.c:1477] connect in progress 1 0x1

00:13:09.334 [3020] <2> vnet_pbxConnect: pbxConnectEx Succeeded

00:13:09.334 [3020] <2> do_pbx_service: [vnet_connect.c:2116] via PBX bprd CONNECT FROM 10.162.244.101.61071 TO 10.162.244.242.1556 fd = 15

00:13:09.334 [3020] <2> async_connect: [vnet_connect.c:1644] connect async CONNECT FROM 10.162.244.101.61071 TO 10.162.244.242.1556 fd = 15

00:13:09.334 [3020] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO server01.domain.com 10.162.244.242 bprd VIA pbx

00:13:09.335 [3020] <2> logconnections: BPRD CONNECT FROM 10.162.244.101.61071 TO 10.162.244.242.1556 fd = 15

00:13:09.335 [3020] <2> vnet_check_vxss_client_magic_with_info: [vnet_vxss_helper.c:833] VxSS not supported 0 0x0

00:13:09.335 [3020] <4> serverResponse: entering serverResponse.

00:13:09.335 [3020] <4> serverResponse: initial client_read_timeout = <1800>

00:13:09.335 [3020] <4> readCommMessages: Entering readCommMessages

00:13:10.335 [3020] <4> serverResponse: read comm file:<00:13:09 Initiating backup>

00:13:10.335 [3020] <4> serverResponse: read comm file:<00:13:10 INF – Server status = 25>

00:13:10.335 [3020] <16> serverResponse: ERR – server exited with status 25: cannot connect on socket

00:13:10.335 [3020] <16> CreateNewImage: ERR – serverResponse() failed

00:13:10.335 [3020] <4> closeApi: entering closeApi.

00:13:10.335 [3020] <4> closeApi: INF – EXIT STATUS 6: the backup failed to back up the requested files

A shared memory and nbpem error was in the messages log that started intermittently at the same time the rman jobs started failing.

May  4 15:52:45 server01 SQLAnywhere(nb_servers01): Disconnecting shared memory client, process id not found

May  4 15:52:45 server01 SQLAnywhere(nb_servers01): Disconnected SharedMemory client’s AppInfo: IP=192.168.229.233;HOST=server01.domain.com;OSUSER=root;OS=’Linux 2.6.32-504.30.3.el6.x86_64 #1 SMP Thu Jul 9 15:20:47 EDT 2015 x’;EXE=/usr/openv/netbackup/bin/nbpem;PID=0x2f92;THREAD=0x7f51c5a17700;VERSION=16.0.0.2184;API=ODBC;TIMEZONEADJUSTMENT=-360

After rebooting the appliance, we discovered a NBPEM process coredumping with an assertion error in the nbpem log.

Cleaned NBPEM cache: Procedure for clearing the NetBackup Policy Execution Manager (nbpem) cache on NetBackup 7.x

Then discovered this technote regarding NBPEM assertion errors causing status 25 (cant connect): NetBackup 7.1 NBPEM process is core dumping on start with assertion error after LastBackupData::getWindowTimes and anything connecting to nbpem gets status 25.

Disabled poilicies and restarted services and nbpem stayed up.

One of the NBU admins admitted to copying a policy and a schedule issue at the time the nbpem shared memory error started.

We deleted the corrupt policy; restarted services; restarted jobs.

For more information, go to http://www.veritas.com/docs/000125511

Optimize Your Cloud Spending and Cloud Performance

Working with experienced partners like VAST is the best way to ensure you make the right cloud decisions.


The Three Pillars of Information Technology Rest On A Single Foundation: VAST Support

With 30 years of IT services experience and partnerships with top technology firms, VAST’s services enable you to run your business on a solid base of reliable technology.


Information Technology Services Tailored to Your Technology and Business Needs

VAST offers three kinds of IT services that can be as extensive or focused as your business requires.


Strategic Solutions Driven By Business Needs and Technical Insight

The strategic solutions crafted by VAST help you make the most of the technology you have and ease your transition to a more effective digital business.


The VAST Difference

Since 1989, VAST has been providing enterprise-level IT service and solutions to companies of all sizes and industries across the U.S.