At work we use Symantec Backup Exec 2010 to backup our windows machines, but also to backup some data on our (my) Linux servers. After a kernel update, caused by updating from Debian Squeeze to Debian Wheezy, the Backup Exec client beremote failed to start (sorry for the layout mess, but I want google to find it):
ksh # /opt/VRTSralus/bin/beremote GetIfAddrs(LINUX): failed err = 11 *** glibc detected *** /opt/VRTSralus/bin/beremote: free(): invalid pointer: 0xb5ff84a4 *** ======= Backtrace: ========= /lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x70c91)[0xb58f2c91] /lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x724f8)[0xb58f44f8] /lib/i386-linux-gnu/i686/cmov/libc.so.6(cfree+0x6d)[0xb58f763d] /opt/VRTSralus/bin/libbesocket.so(_Z11freeifaddrsP7ifaddrs+0x2e)[0xb5fd75f6] /opt/VRTSralus/bin/libbesocket.so(_Z10getifaddrsPP7ifaddrs+0x72c)[0xb5fd75a0] /opt/VRTSralus/bin/libbesocket.so(_Z20GetAdaptersAddressesmmPvP21_IP_ADAPTER_ADDRESSESPm+0x69)[0xb5fd7ef1] /opt/VRTSralus/bin/libbesocket.so(_ZN8BESocket13BENetConfigEx18RefreshInformationEPKb+0x41)[0xb5fe1451] /opt/VRTSralus/bin/libbesocket.so(_ZN8BESocket13BENetConfigExC1Eb+0xf4)[0xb5fe12f8] /opt/VRTSralus/bin/libbesocket.so(_ZN8BESocket13BENetConfigEx23GetNetworkConfigurationEb+0x2c)[0xb5fe18a4] /opt/VRTSralus/bin/libbedsvx.so(_Z14VX_EnumSelfDLEP6BE_CFGP8HEAD_DLE+0x300)[0xb61d9d30] /opt/VRTSralus/bin/libndmpcomm.so(+0x24a58)[0xb6057a58] /opt/VRTSralus/bin/libndmpcomm.so(+0x24e41)[0xb6057e41] /opt/VRTSralus/bin/libndmpcomm.so(_Z20NrdsAdvertiserThreadPv+0x12a)[0xb60596f6] /opt/VRTSralus/bin/libvxACEI.so.3(_ZN18ACE_Thread_Adapter8invoke_iEv+0x52)[0xb77229d6] /opt/VRTSralus/bin/libvxACEI.so.3(_ZN18ACE_Thread_Adapter6invokeEv+0x61)[0xb7722941] /opt/VRTSralus/bin/libvxACEI.so.3(ace_thread_adapter+0xe)[0xb76ebd1a] /lib/i386-linux-gnu/i686/cmov/libpthread.so.0(+0x5c39)[0xb6977c39] /lib/i386-linux-gnu/i686/cmov/libc.so.6(clone+0x5e)[0xb59599fe]
So I asked google and discovered, that only Backup Exec 2014 would work with 3.x kernels. Not nice, as we did not want to upgrade all our environment just because of the linux client.
Time for some strace and gdb sessions.
It was not too difficult to detect, that some ioctrl call was the last thing, happening before the crash. Some more searching on the internet revealed, that the following kernel patch is responsible for the crash: http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=41c31f318a5209922d051e293c61e4724daad11c. Interesting parts:
So now a call returns a different code, which seems to cause a crash in beremote. Who coded beremote!?
- err = -EINVAL; + err = -ENOTTY; - return -EINVAL; + return -ENOTTY; - return -EINVAL; + return -ENOTTY;
Nevertheless, I patched the kernel 3.x to return the old value and beremote started working again.
But this might have effects on other software, which expects the new (correct) return code and I did not want to manually build every new kernel on all of my machines.
Looking at the crash output above, it is clear, that the problem is in libbesocket.so.
Time for some disassembler sessions.
It's been a while, since I last read assembler code, but seems like I am old enough, to still know such stuff ;).
EINVAL is defined as 22, hex 0x16. As this is the only place, where such a compare happens, lets patch it! Fire up your favorite hex editor and have a look at offset 0x1ff10:
.text:0001FF05 call _ioctl .text:0001FF0A add esp, 10h .text:0001FF0D test eax, eax .text:0001FF0F jns short loc_1FF26 .text:0001FF11 call ___errno_location .text:0001FF16 cmp dword ptr [eax], 16h .text:0001FF19 jnz loc_2026C
0001ff10: 15 E8 5A DA FE FF 83 38 16 0F 85 4D 03 00 00 C7Ah there is a 0x16!
Time to patch beremote!
Now we replace it with the correct value for ENOTTY, which is 0x19!
And .. now beremote starts again and can do backups, even on newer kernels! So, if you have the same problem and google brought you here: The patch above works for the following file versions:
ksh # shasum beremote libbesocket.so c11b9872438ffbca29f5bf5af9288eb994f92bd0 beremote a320e4236f75185aa5299fba44291c471f2ac235 libbesocket.so patched libbesocket.so: sh # shasum libbesocket.so.patched e52c0d1bb948bdd472dd6192af52626715f7419c libbesocket.so.patched filesizes: 390312 beremote 271624 libbesocket.soWarning:
- Be sure to just apply the patch, if you have exactly those files.
- If you have a different version, have alook above, that is why I wrote down all that stuff. You should be able to search for the above hex values and find 0x16 yourself.
- If not, you should know, where to look in the disassembly.
- If not, maybe you should upgrade to Backup Exec 1024 ;).