分类目录归档:Linux

udev_start导致vip漂移(常见情况:rac在线加盘操作引起)

客户对asm进行扩容,执行udev_start命令之后,所有的vip全部漂移,业务全部中断
20230513203654


优先恢复业务,把所有vip漂移回来

[grid@rac3 ~]$  srvctl relocate vip -i rac1 -n rac1 -f -v
VIP was relocated successfully.
[grid@rac3 ~]$  srvctl relocate vip -i rac2 -n rac2 -f -v
VIP was relocated successfully.
[grid@rac3 ~]$  srvctl relocate vip -i rac3 -n rac3 -f -v
VIP was relocated successfully.
[grid@rac3 ~]$  srvctl relocate vip -i rac4 -n rac4 -f -v
VIP was relocated successfully.

vip恢复正常,业务也恢复正常
20230513203712


出现该问题的原因是由于udev_start命令引起网卡瞬间中断,从而使得vip发生漂移
20230513212316

查看ifcfg配置文件
20230513212440

引起该问题的原因是udev对网卡进行了操作,从而引起该问题,处理建议在对应的ifcfg文件中加上 HOTPLUG=”no” (pulbic,private和其他需要关注的网络)
参考:Network interface going down when dynamically adding disks to storage using udev in RHEL 6 (Doc ID 1569028.1)
20230513213713

发表在 Linux, Oracle RAC | 标签为 , , | 评论关闭

echo 0 > /proc/sys/kernel/hung_task_timeout_secs disables this message

客户反馈数据库无法登录,系统ssh也无法登录,但是可以ping通,通过sqlplus sys/pwd@tns as sysdba方式登录成功,直接对数据库进行shutdown abort操作,然后系统可以正常ssh登录.通过分析发现一些io问题

系统messages日志报错
20230430084031


默认情况下, Linux会最多使用40%[根据系统配置决定]的可用内存作为文件系统缓存。当超过这个阈值后,文件系统会把将缓存中的内存全部写入磁盘, 导致后续的IO请求都是同步的。将缓存写入磁盘时,有一个默认120秒的超时时间。 出现上面的问题的原因是IO子系统的处理速度不够快,不能在120秒将缓存中的数据全部写入磁盘。IO系统响应缓慢,导致越来越多的请求堆积,最终系统内存全部被占用,导致系统失去响应。

检查系统io情况
20230430084643

磁盘在io请求很小的情况下busy 100%,属于不正常情况,让客户安排人检查硬盘情况
20230430084802

发现raid 5中有一块磁盘异常从而引起性能下降,客户安排人员换盘之后,系统恢复正常.

调整系统参数缓解
对于linux系统文件系统缓存可以进行调整参数vm.dirty_background_ratio和vm.dirty_ratio为适当值,比如

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
发表在 Linux | 标签为 | 评论关闭

linux系统文件加密勒索病毒

昨天晚上,一客户联系我们,其linux的dg备库上发现病毒,让我给看看,登录上去之后发现异常进程
import pty;pty.spawn


进一步检查发现很多文件被加密成.locked1
locked1

对应的README.html内容

[yyapp@ncapp ~]$ cat README1.html 
contact email: service@hellowinter.online, prepare 0.12btc, 
if you can't contact my email, please contact some data recovery  company(suggest taobao.com), 
may they can contact to me .
your person id:izeieOMvPH+SDWYAxX6snmD2k306byUOpTP4Djfm9gaekoP0Q9JwWVcG0NI1grBM/DIo22A+sjCm
UfXXDwq/s72bc014WxmIy8jXCowuH4e6hJBjUgnkfoe/NbfPJN1CNQS3EdO6UaxMS3fwUzfnZTvW
63GyygBgZTzq9CfTdDUBmXe3aP30VTisTrtaFCdRnD2JaMUe5fVPUQ/vX39S4Kkng/VeZtOwUek7
24WbH7Z62Jo+7pnXyeB2PJpdvg0Rcy42VXJj7vZYe48xt2/PYYGuNLIjdDGt00qDiBuWLh19Q8us
mBILkB5sypmE6drpzAR74ao9/fh/YOawkppism9bSbKDnLAUQz1MG7z0MqEEwWF+5uMUoxqk+Wmj
zAY6/eu2X5cV/UASWI8TS+U+agRGgo98B2dkVgTGaXrGc/GzX/QNVVrAYAIqe3o3Mx0EG05vwEjf
cv6AUYK3W+QshyNwUJUrptJaAAzdtpz20dAlQtAWrezAHzY14W91puE5NUmBhWO//xv3xu/5F9wd
eBqfIFHFBkb3RxMTuU19UDQtsn/CKObjdiFz9gunugZXbubSnq1m5huk5dpgXvDh5hU1pdPdVNR6
tLldjbQQH+UQKl7wyQvs205UHRQW0OTfoxcnk2DagwqQyCc4U4BnigjMxYnpiLS2p9G1i/Sg3mU1
6ES7lgQ3WzjICcdhisApjFGomWNOwPNrqwlWihCvugUIbe0ST/n57XiU2HcNjvjraR8KINtDHYhp
dS8FQZZA4/G8JTOMOdsQ2CPKXK7BLX+m39/1xe4w6/g492Qb9L6k7xLHdgrMalnNO5yGgd38WEaG
aYAF40ICmOy55hmdl1Gp6docZ5XDB+eB/A5QcoihZkEeSqB39ibLarubyBjS2jv1ZN6uqCw4wwaV
RC22N4miT0aM3GkX+sfT2J3fWo0HFtgE18T9pVhsE73Sf00bW+LT0yh8SpK9IE4wRA4m+jskzg9S
aJLZDWRc4vtYlx93VXT4Z+3G2rnm1FnX2MXySAyhVlvQmRAfPoDC285Jn95259/17e9Y4639jxSs
JvOO8kiJHBQNbbyV1XXsxBtKyl514wbUjn2mccUlJ51EyfssuITjqdMeHoDaO4KzguXNFJgwGzbv
K+y3NX54yuE1Xm9sCI07pJs2WwC8GzErfmXbseTEvUvcXl0qsQFXTFCTNLnSNdXkswN+Hh/5uI5c
dAM73VP+qiTdvj4a0GRXa+riZ136lVEd9pZpy62XYQYkn+LGGJljvPooMH9rM1oIp8tOiPldIm29
0nNLmiaSmaEefY9I6wRAemvNAHw9Wq75pDBY3H3Xjt8ENsmj2MNpchDvYHM/ndckMoReN8cEsouv
LhuhtjjBktuaVz1j9Vj6UM6LEjGe6ZJyx+fjnTI2haMLej6vf0hopck0vJmSuL1mN03gd/QkBsBt
wDxFReExoTfcuhMRSdkrMqJFWLOpKI+XrYaB6DqHbFjqr3ME6RJcP9ynNF8qqF7JXNJsMu9PJ5ml
0hcg71NirMD7iXNUy1YDgzKqULABvL4SeUAjEE/Sa/HvUw+lMgZaM6aodAczTyWVqITVXzcuDXLV
vlrF5uMflQC13qaPTlqgTbB9xv4F/S8joC/c60fd+5WjdjWT6tMXHLlWRPQJrUNW06+fCh9EPjmW
llD6HJXreorpbjB7hWwahu5mSnWhwWqFsHwYbK7tSo98GqXdOEmOH21zPF57UCr0Sff47tDrSEtn
YCKHt47lS/ayCfnx1g9HAFu0NyULUE/UowuW5aPdRyqcRAaA1UAMugRqZB/QkTQVoPsCQRmca352
HE9M4LasANxTk4RT4HHmrBQSCzW0QZ+L2ouDTYgc2ipjXQbnLuZgU1AgIqvjjo+dDz4A9BYeJWEU
4QDg89IfDFpSiU26nvsDHIxh2KP0F5Uvf5n4Q1K/sO3g71G9prxMHLyq+M6UdY5W/zVAzYFuzx/H
Z8jvaQTYHopyUQUHLZ1XvkD+CzRFMruTHyVavu1OL+3xzgILP23IDyoPdp1pyfQbrwN0inDlAEMN
3cJRvMleqKB145p7hItgOpDCwqojMveM+YaT+mPhPCZbV4GsJ/YeP2yzMPG8lXTHG2nu/0Ew08TO
BstwUtAFqTc8C0vgMLR6ZGZ8UtwT0yE7WQm7KPwPiMtzqhNtW4ORtzrSmy1YPjpqZ2LIu5WrqW3Y
hh56D6Kl1fQgFA4x2PuBF1+VJm8jovm4MQkOBjwKr0gpcWqwHsPPGLvTb/tiFRoP3r2+nulgKP7D
zaoJpbhqtp2e18Ip6RC4MWvNRZ8MJwF9X9s1KI7Gqdxp1ePvwmsVFvOzozIUA9WczSlGQ9oMs0Rq
Kf6Q6VypCpOkRHtHpKsB96drqs08dpQ1zRZdLaCzs/r6je7JGFDZyf7iI7qvZjZWBPIJNGR4q+Ms
6ur3xsHm4jRbg5knH+9c7n9hA0Y7HHVweXo8SAmxd2Zbggldiw/qXlnhEg6yUEE3QYvkw9gnmMkO
N7Biclfd6VcOc6vXGtzXLGml09DVNJg4vWVauwldzAEUT15Eoo5aVjqtYLJjDYmWIefKrQoeQ8GS
SmZ7Y66hYZRAQFmBPYNq0T5g0se5j8+tYvldL6u+waqive9cUKG4Au5wYUwDFbY93D9AK73sR7lY
z8oq3AXgT1Leiy3r/O2HNSpb4Qqn6vN3cOxtmmPAPpAhzZ/Ab9iEJCqTp5aqerlJUJWSarQ8DDca
V0gc41vAue9AEc5mNnf/oUILLJv4Kok62PIEAwg3Y/Zw8jv1226QqgAD3jXpVDK52H6nPa6IOqaI
YY5EwUYBcK8FqpJtquzqt7C0NZnIOlSur/og750HieWl5FOc9NpOTNrIW+Fb5Uqhiv2FHR6E874x
IaN3cW2tCtATndFOf5+YQPo1vcEXyZTp+rQjMDqrJdMe8u1nO7ewJF7TAcWLB8PKhejn3aj4S4uC
zMTt7wdp64co8wUusQc11mcpItHfSxE7GViUeZlYnOkb9tzQRmf8ff4I2g2kwwYzrF/OWKgqNDXv
ZfbR1XwXHXlqcyIJJzubxAucYrSaSG6M
[yyapp@ncapp ~]

通过以上信息基本上确认一种类似win的加密勒索病毒,经过分区确认只是加密了yyapp用户有读写权限的数据,其他数据用户数据没有被加密(这个机器是应用服务器,并且做了oracle的备库[没有被加密]),因此基于目前的情况对客户没有太大损失,直接重装应用配置dg即可.通过进一步分区,确认该病毒是通过应用漏洞入侵,建议客户进行应用和系统安全加固.
温馨提示:以前的勒索病毒绝大部分都集中在win平台上,现在可能linux平台也会收到很大影响,建议各位对各自系统进行安全加固,系统和应用打上漏洞补丁和网络安全防护

发表在 Linux | 评论关闭