安装PSU，有可能会导致实例无法启动-白红宇

安装PSU，有可能会导致实例无法启动

阅读量：5223 次

发布时间：2019-06-14

本文共 19199 字，大约阅读时间需要 63 分钟。

最近某次进行压力测试时，需要安装最新的PSU补丁，现场的同事安装完PSU补丁后，结果数据库集群的其中一个实例无法启动。以下为远程处理的过程，简要记录之。

1、查看集群当前状态：

[root@nodedb01 ~]# crsctl status resource -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATADG.dg

ONLINE ONLINE nodedb01

ONLINE ONLINE nodedb02

ora.LISTENER.lsnr

ONLINE ONLINE nodedb01

ONLINE ONLINE nodedb02

ora.TEST.dg

ONLINE ONLINE nodedb01

ONLINE ONLINE nodedb02

ora.asm

ONLINE ONLINE nodedb01 Started

ONLINE ONLINE nodedb02 Started

ora.gsd

OFFLINE OFFLINE nodedb01

OFFLINE OFFLINE nodedb02

ora.net1.network

ONLINE ONLINE nodedb01

ONLINE ONLINE nodedb02

ora.ons

ONLINE ONLINE nodedb01

ONLINE ONLINE nodedb02

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE nodedb02

ora.cvu

1 ONLINE ONLINE nodedb02

ora.nodedb01.vip

1 ONLINE ONLINE nodedb01

ora.nodedb02.vip

1 ONLINE ONLINE nodedb02

ora.oc4j

1 ONLINE ONLINE nodedb02

ora.scan1.vip

1 ONLINE ONLINE nodedb02

ora.test.db

1 ONLINE OFFLINE Instance Shutdown

2 ONLINE ONLINE nodedb02 Open

可见，整个集群状态都是正常的，仅仅是节点1的数据库实例异常，无法启动。

2、尝试手动启动数据库实例，但查看alert日志：

SQL> startup

ORA-03113: end-of-file on communication channel

SQL>

手动启动实例时，报3113错误，同时数据库的alert日志提示：

********************************************************************

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Initial number of CPU is 96

Number of processor cores in the system is 48

Number of processor sockets in the system is 2

alert日志中只有这些信息，启动数据库实例的操作就已经失败。这说明还没有开始读取spfile文件就已经出故障了。

3、检查节点1的补丁信息：

[oracle@nodedb01 ~]$ opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.21

Oracle Home : /u01/app/oracle/product/11.2.0/db_1

Central Inventory : /u01/app/oraInventory

from : /u01/app/oracle/product/11.2.0/db_1/oraInst.loc

OPatch version : 11.2.0.3.21

OUI version : 11.2.0.4.0

Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2019-09-06_17-48-51PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2019-09-06_17-48-51PM.txt

--------------------------------------------------------------------------------

Local Machine Information::

Hostname: nodedb01

ARU platform id: 226

ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Database 11g 11.2.0.4.0

There are 1 products installed in this Oracle Home.

Interim patches (1) :

Patch 29141201 : applied on Fri Sep 06 16:08:23 CST 2019

Unique Patch ID: 22724549

Patch description: "OCW Patch Set Update : 11.2.0.4.190416 (29141201)"

Created on 4 Mar 2019, 09:09:03 hrs PST8PDT

Bugs fixed:

18328800, 19270660, 18691572, 18508710, 20038451, 21251192, 22162062

20365005, 21232394, 17336871, 17387214, 17750548, 17617807, 14497275

20219458, 17733927, 18180541, 23757020, 18962892, 17292250, 17378618

16759171, 20110156, 17843489, 17065496, 13991403, 21694632, 18419139

17273020, 22762046, 17336884, 17155238, 17336889, 18261183, 18053580

20012766, 21245437, 20218012, 17013634, 17886392, 20995001, 24422155

17039197, 16317771, 17947785, 10052729, 20340620, 22353346, 16237657

20317221, 18199185, 15917869, 18399991, 20186278, 18024089, 17374271

16849642, 20246071, 20746251, 14270845, 23621494, 20552947, 18414137

18882642, 17001914, 29323944, 17927970, 14378120, 16346413, 15986647

17336898, 18068871, 21222147, 16206997, 18143836, 21982225, 25490238

19168690, 18343490, 20235511, 21875360, 16613232, 19276791, 17722664

20440643, 12928658, 18226143, 18520351, 16249829, 18952577, 17172091

18229842, 16076412, 18265482, 20676340, 17818075, 20091753, 18231837

14373486, 17483479, 20136892, 20551654, 18120545, 18729166, 13843841

17405302, 21225209, 18709496, 18330979, 18744838, 14525998, 18187697

17087371, 20531190, 20598625, 14385860, 18348155, 19479503, 12928592

17516024, 18370031, 17764053, 19272663, 17551223, 14671408, 18272135

14207615, 21255373, 17500165, 18875012, 14769643, 18464784, 25656952

19558324, 18848125, 19241857, 17955615, 14851828, 14693336, 20315294

16284825, 17352230, 20014326, 17238586, 17089344, 17405605, 17531342

21327402, 26546632, 19398098, 17159489, 17640316, 13823394, 16543190

22024217, 17983675, 20795241, 28973538, 17481314, 18346135, 17598201

16281493, 15986311, 17208793, 19601468, 18999857, 18700935, 14076173

18428146, 16709532, 17435488, 18352845, 18352846, 20408163, 17592037

19616601, 17387779, 17391726, 14777968, 15851860, 16206882, 20141091

15832129, 17305100, 20175174, 21113068, 19885321, 16901346, 17985714

25591658, 18536826, 17780903, 18752378, 18946768, 16876500, 16875342

17769597, 19955755, 16429265, 18336452, 23186035, 17273003, 17209968

25484507, 16988311, 19319357, 20094984, 17046460, 17059927, 18053631

16867761, 18774591, 21442094, 20235486, 19359787, 15869775, 17447588

19642566, 21152052, 16798862, 15920201

--------------------------------------------------------------------------------

OPatch succeeded.

[oracle@nodedb01 ~]$

与正常的节点2对比发现，节点1的补丁不完整，只是显示了一部分补丁信息。

4、非常怀疑是补丁安装异常，接着检查了oracle二进制文件的权限：

[oracle@nodedb01 ~]$ cd $ORACLE_HOME/bin

[oracle@nodedb01 bin]$ ll oracle

-rwsr-s--x 1 oracle oinstall 239885712 Sep 6 17:54 oracle

[oracle@nodedb01 bin]$

oracle二进制文件的属组竟然为oinstall。至此，故障的原因基本已经找到。

5、手动修改属组：

[root@nodedb01 etc]# su - grid

Last login: Fri Sep 6 17:57:38 CST 2019

[grid@nodedb01 ~]$ echo $ORACLE_HOME

/u01/app/11.2.0/grid

[grid@nodedb01 ~]$ /u01/app/11.2.0/grid/bin/setasmgidwrap -o /u01/app/oracle/product/11.2.0/db_1/bin/oracle

[grid@nodedb01 ~]$ exit

6、再次检查补丁情况：

[root@nodedb01 etc]# su - oracle

Last login: Fri Sep 6 18:02:47 CST 2019

[oracle@nodedb01 ~]$ opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.21

Oracle Home : /u01/app/oracle/product/11.2.0/db_1

Central Inventory : /u01/app/oraInventory

from : /u01/app/oracle/product/11.2.0/db_1/oraInst.loc

OPatch version : 11.2.0.3.21

OUI version : 11.2.0.4.0

Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2019-09-06_18-03-06PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2019-09-06_18-03-06PM.txt

--------------------------------------------------------------------------------

Local Machine Information::

Hostname: nodedb01

ARU platform id: 226

ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Database 11g 11.2.0.4.0

There are 1 products installed in this Oracle Home.

Interim patches (2) :

Patch 29141056 : applied on Fri Sep 06 17:53:33 CST 2019

Unique Patch ID: 22768427

Patch description: "Database Patch Set Update : 11.2.0.4.190416 (29141056)"

Created on 17 Feb 2019, 23:52:26 hrs PST8PDT

Sub-patch 28729262; "Database Patch Set Update : 11.2.0.4.190115 (28729262)"

Sub-patch 28204707; "Database Patch Set Update : 11.2.0.4.181016 (28204707)"

Sub-patch 27734982; "Database Patch Set Update : 11.2.0.4.180717 (27734982)"

Sub-patch 27338049; "Database Patch Set Update : 11.2.0.4.180417 (27338049)"

Sub-patch 26925576; "Database Patch Set Update : 11.2.0.4.180116 (26925576)"

Sub-patch 26392168; "Database Patch Set Update : 11.2.0.4.171017 (26392168)"

Sub-patch 26609445; "Database Patch Set Update : 11.2.0.4.170814 (26609445)"

Sub-patch 25869727; "Database Patch Set Update : 11.2.0.4.170718 (25869727)"

Sub-patch 24732075; "Database Patch Set Update : 11.2.0.4.170418 (24732075)"

Sub-patch 24006111; "Database Patch Set Update : 11.2.0.4.161018 (24006111)"

Sub-patch 23054359; "Database Patch Set Update : 11.2.0.4.160719 (23054359)"

Sub-patch 22502456; "Database Patch Set Update : 11.2.0.4.160419 (22502456)"

Sub-patch 21948347; "Database Patch Set Update : 11.2.0.4.160119 (21948347)"

Sub-patch 21352635; "Database Patch Set Update : 11.2.0.4.8 (21352635)"

Sub-patch 20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"

Sub-patch 20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"

Sub-patch 19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"

Sub-patch 19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"

Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"

Sub-patch 18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"

Sub-patch 17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"

Bugs fixed:

17184721, 21174504, 21538558, 16091637, 18092127, 17381384, 15979965

20671094, 16731148, 16314254, 13837378, 18441944, 17291347, 23105538

17835048, 28254374, 13558557, 21656630, 21842740, 17008068, 18382302

17201159, 25427662, 17853498, 21197626, 20717359, 17246576, 18356166

18681862, 18440047, 20569094, 20031873, 16875449, 20387265, 19788842

17296856, 21330264, 14010183, 17648596, 18886413, 17025461, 17551063

24719736, 17267114, 21063322, 22507210, 17912217, 17889583, 18202441

17040764, 17478145, 16524926, 25655390, 19358317, 22730454, 22148226

18747196, 26544823, 18641419, 17036973, 18948177, 17811789, 16542886

14285317, 18009564, 17359610, 16618694, 8322815, 16832076, 18247991

16692232, 22507234, 17570240, 13871092, 24624166, 26631046, 19429927

24348685, 17848897, 17441661, 14034426, 17465741, 20273319, 19207156

16596890, 18510194, 17437634, 21343897, 20506706, 28849751, 18339044

21453153, 17951233, 21795111, 22321741, 18430495, 21787056, 22380919

19469538, 19692824, 20506715, 17811429, 17903598, 19721304, 11786053

18230522, 19554106, 19458377, 21281607, 17612828, 6599380, 22092979

17040527, 22321756, 17811438, 18641461, 18682983, 14657740, 25635149

13364795, 19490948, 21387964, 17346671, 17588480, 22351572, 18235390

18849970, 26474853, 17889549, 19309466, 20627866, 23008056, 16472716

24908321, 20134113, 25775213, 20596234, 18331850, 18641451, 20882568

17019356, 17344412, 19461270, 21179898, 17546761, 24842886, 17231779

14521849, 18203835, 18203838, 18964939, 18203837, 17313525, 22195457

18139690, 16837842, 14106803, 22296366, 17842825, 22657942, 21352646

16360112, 22594718, 20657441, 22195441, 17389192, 26198926, 14565184

19781326, 17019345, 17205719, 18740837, 18440095, 14764829, 14354737

22195448, 17019086, 13944971, 16571443, 21868720, 17186905, 17080436

18673342, 22905130, 17027426, 27374796, 19972569, 19972568, 20144308

19972566, 17282229, 19972564, 16870214, 16410570, 21629064, 19615136

21354456, 26039623, 17390431, 18762750, 23007241, 16613964, 17957017

18098207, 17484762, 18471685, 19730508, 18264060, 21538485, 17323222

17754782, 17600719, 18317531, 17852463, 17596908, 17655634, 18166013

16228604, 20074391, 27053456, 24790914, 19972570, 20856766, 18090142

19891090, 18996843, 16042673, 19854503, 17835627, 22901797, 20334344

17393683, 20861693, 18000422, 17551709, 26575788, 23315889, 20506699

19006849, 18277454, 18456514, 19174430, 17258090, 17174582, 25654936

17242746, 27097854, 16399083, 17824637, 21132297, 22465352, 17762296

22168163, 18604692, 17397545, 16450169, 12364061, 20067212, 18856999

19211724, 19463893, 27351628, 19463897, 21343775, 17853456, 18373438

18673304, 20004021, 28000269, 26030218, 21668627, 16194160, 17477958

23140259, 16538760, 12982566, 24570598, 20828947, 27255377, 18259031

20296213, 21425496, 28855981, 18293054, 17610798, 19699191, 23065323

17311728, 18135678, 18774543, 23294548, 16785708, 10136473, 24560906

19777862, 22551446, 17786518, 25879984, 18315328, 18334586, 12747740

18096714, 19032867, 21641760, 17390160, 18899974, 17232014, 20598042

26679352, 26245237, 16354467, 17484731, 18673325, 16422541, 18155762

19827973, 14015842, 22683225, 17726838, 18554871, 23177648, 18051556

20803583, 18282562, 21972320, 15990359, 17922254, 16855292, 16668584

21343838, 20299015, 17446237, 18043064, 18093615, 23713236, 17694209

17288409, 20475845, 18308268, 17274537, 13955826, 16934803, 17634921

17501491, 16315398, 23725036, 22683212, 17006183, 13829543, 18191164

26746894, 22809871, 17655240, 28819280, 18384391, 19393542, 21538567

16198143, 21847223, 25823754, 17892268, 20142975, 19584068, 17165204

25165496, 18604493, 21756699, 18508861, 18554763, 16901385, 21532755

18189036, 17443671, 17385178, 14829250, 17936109, 20925795, 20509482

17478514, 27441326, 16850630, 13951456, 16595641, 14054676, 15861775

21142837, 16912439, 17299889, 17297939, 23003979, 16833527, 18619917

17798953, 19697993, 17816865, 25914276, 18607546, 17571306, 21286665

16898135, 16819962, 17341326, 26910644, 17851160, 17586955, 20558005

19049453, 21051840, 17587063, 16956380, 18328509, 25423453, 14133975

29033139, 19718981, 18061914, 18522509, 17518652, 21051833, 20294666

18765602, 20860659, 18272672, 20324049, 18199537, 17332800, 13609098

22502493, 18384537, 14338435, 17945983, 16392068, 21067387, 17752995

21051862, 16863422, 17237521, 25505382, 18244962, 19544839, 24433711

24717859, 17156148, 18973907, 23026585, 17449815, 17877323, 18180390

17088068, 17037130, 20004087, 21422580, 19466309, 11733603, 25505371

18610915, 21051858, 18084625, 29027694, 18674024, 26243698, 21051852

18091059, 18306996, 25369547, 16306373, 17787259, 18193833, 19915271

20513399, 20631274, 26439748, 16344544, 25879656, 14692762, 18614015

22782647, 17346091, 18228645, 17721717, 13960236, 18685892, 18436307

11883252, 19888853, 21756677, 17891943, 19475971, 22353199, 16384983

19121551, 25634317, 27825893, 12816846, 17982555, 17761775, 17227073

22243719, 17265217, 25505394, 17071721, 16721594, 18262334, 21756661

17891946, 15913355, 17672719, 17602269, 17239687, 17042658, 25555252

17238511, 21059919, 17811456, 17284817, 17752121, 20879889, 28806384

21380789, 17394950, 17011832, 16579084, 22195465, 14602788, 28790634

18325460, 27567477, 26569225, 24476265, 24476274, 12611721, 18674465

16903536, 17006570, 19689979, 28076295, 16043574, 18783224, 24662775

16494615, 21526048, 17392698, 19197175, 16069901, 17811447, 27870645

28876684, 17308789, 24835538, 22195477, 17865671, 17343514, 19013183

17325413, 18316692, 16180763, 17348614, 14368995, 21983325, 17393915

16285691, 19211433, 20331945, 17883081, 17705023, 24316947, 17614227

19578350, 22195485, 14084247, 24975421, 26078387, 23115139, 13645875

16777840, 19727057, 14852021, 18744139, 18674047, 17716305, 19285025

18482502, 17622427, 19289642, 27534509, 25947799, 22195492, 14458214

20869721, 21172913, 17767676, 18723434, 25505407, 17786278, 19258504

17082983, 17365043, 21351877, 13498382, 18331812, 16065166, 25489607

16685417, 18031668, 22893153, 17551674, 16943711, 19272701, 21517440

25897615, 17649265, 13866822, 18094246, 24528741, 17783588, 14245531

17082359, 26007010, 18280813, 20448824, 23330119, 16268425, 19487147

25600421, 18018515, 17302277, 17215560, 24411921, 19271443, 25764020

14176370, 17016369, 20777150, 16756406, 23330124, 22977256, 20441797

19769489, 28100487, 17545847, 25093656, 18260550, 13853126, 17227277

17551699, 23536835, 25957038, 24652769, 20725343, 19207117, 9756271

18868646, 17614134, 26667023, 17546973, 19680952, 18704244, 26667015

17050888, 18273830, 18828868, 17360606, 16992075, 24563422, 17375354

12905058, 18362222, 21429602, 17254374, 26667032, 28364007, 27086138

17571039, 17468141, 18436647, 17570606, 17235750, 21168487, 17279227

16220077, 16929165