记录工作和生活点点滴滴

Orcal RAC on vsphere

1、介绍

本文通过一个案例简要介绍 Oracle Real Application Clusters (RAC) 在 VMware vSphere® 平台上的安装过程, 帮助 Oracle 数据库管理员在VMware平台上规划和实施Oracle RAC环境. 在VMware环境下,数据库管理员通常需要和VMware的管理员以及存储厂商的管理员合作,以顺利部署一个Oracle RAC环境.

在虚拟化平台上部署Oracle RAC在许多方面与纯物理环境上的部署很类似,比如:

  • 在虚拟机创建完成之后,安装操作系统和Oracle RAC软件的过程与物理环境下基本一致
  • 存储的规划,包括用于数据存放的LUN数量的规划和多路径的设计与物理环境也基本一致.
    差别在于我们可以充分依靠虚拟化软件, 简化许多工作, 从而加快安装进度,减少错误的发生,比如:
  • 利用事先准备好的模板, 多次安装同版本的系统和数据库,同时加强标准化
  • 利用快照技术, 很方便的进行测试和回退
  • 利用虚拟机复制技术,简化多个RAC节点的系统安装、 打补丁、 配置工作
    由于安装过程的细节与软件版本(尤其是Linux版本和Oracle版本) 有较强的关联性, 本文只能尽量把安装步骤描述清楚.读者在自己进行安装的时候,还需要参考相应版本的官方安装文档.

先将本次安装中用到的主要软件和版本罗列如下:

  • VMware ESXi™ 5.1
  • VMware vCenter Server™ 5.1
  • vSphere Client 5.1
  • Oracle Database 11g 11.2.0.3.
  • Oracle Enterprise Linux x86_64 6.3

2、环境介绍

2.1硬件环境

组件 数量 配置
ESXI HOSTS 2 Cisco UCS B440 4x10-Core CPUs,256GB RAM,and 4NICs
Storage 1 NetApp F3240双路控制器
Ethernet 2 Cisco FI6248
FC Swithces 1 Cisco Nexus 7000
Host Based Adapters 5 每个刀片两个, 与网卡共享流量

2.2软件环境

组件 版本 作用
vSphere 5.1 Hypervisor hosting all virtual machines
VMware vCenter™ 5.1 管理 vSphere
Oracle Enterprise Linux x86_64 6.3 Oracle Database server
Oracle Database 11g (with Oracle RAC and Oracle Grid Infrastructure) Enterprise Edition 11.2.0.3
Microsoft Windows Server 2008 R2 用于 vCenter Server

2.3虚拟机配置

虚拟机用途 数量 配置
Oracle RAC nodes 2 20 vCPU, 120GB RAM,Oracle Enterprise Linux x86_64 6.3
vCenter 1 2 vCPUs, 8GB RAM, Windows Server 2008 R2

2.4共享存储设置

Datastore Name Virtual Disk on ESXI Linux路径 虚拟路径 驱动类型 VMDK SIZE(GB) 用途
VMDATASTORE VMDK-Hard disk1 /dev/sda scsi0:0 LSI Logic 150 Oracle Enterprise Linux 5.5 OS and Oracle binaries
CRS1 VMDK – Hard disk 2 Shared disk/dev/sdb1 SCSI 1:0 Paravirtual 20 CRS and voting disk
REDO1 VMDK – Hard disk 3 Shared disk/dev/sdc1 SCSI1:1 Paravirtual 900 REDO and Archieve
DATA01 VMDK – Hard disk 4 Shared disk/dev/sdd1 SCSI2:0 Paravirtual 1000 RAC database DATA
DATA02 VMDK – Hard disk 5 Shared disk/dev/sde1 SCSI2:1 Paravirtual 1000 RAC database DATA
DATA03 VMDK – Hard disk 6 Shared disk/dev/sdf1 SCSI3:0 Paravirtual 1000 RAC database DATA
DATA02 VMDK – Hard disk 7 Shared disk/dev/sdg1 SCSI3:1 Paravirtual 1000 RAC database DATA

2.5 Orcal RAC数据库环境

Node Name ASM Instance RAC Instance SCSI2:1 Database ASM Storage
RAC1 +ASM1 RAC1 RAC ASM file |system
RAC1 +ASM1 RAC1 RAC ASM file |system

2.6网络配置

按照Oracle RAC的实施惯例, 每个RAC节点都提供一个Public网口用于承载业务, 并在其上捆绑虚拟IP地址 (VIP) 用于HA切换..
各RAC节点之间还需要一个私有网络进行节点之间的通信。
在物理环境中, 通常需要考虑链路的冗余以消除单点鼓掌。 而在虚拟化环境中, 冗余问题由底层虚拟化软件处理, 所以对于每个RAC节点, 只是简单地需要两个网口就可以工作, 大大简化了配置和安装工作.
我们这次测试的网络配置如下三:

  • UCS底层软件配置出三个物理网口(均为10Gb) – vmnic0 和 vmnic1用于Oracle Public and Oracle Private网络, vmnic2用于vSphere® vMotion®工作.
  • 两个分布式虚拟交换机 – Oracle Public (VM Net) and Oracle Private (Oracle HB).
    每个RAC节点配置3个IP地址 – 以节点1 (RAC1) 为例:

  • One public static IP address (eth0): 162.16.3.101

  • One private static IP address (eth1): 192.168.2.30
  • One virtual static IP address(VIP): 162.16.3.111(由CRS软件配置和管理)
    还有三个虚拟IP用于节点间负载均衡(SCAN): 162.16.3.200,162.16.3.201,162.16.3.202(由CRS配置和管理)
    物理网口 vmnic0 连接到分布式交换机 VM Net,vmnic1连接到分布式交换机Oracle HB, vMotion 使用标准交换机vSwitch3. 这样区分概念上比较清晰,
    也可以只建一个分布式网络, 用不同的端口组区分数据网络和心跳网。
    Figure 1. Network Configuration

2.7 物理拓扑图

Figure 2. Physical Architecture of Oracle RAC on vSphere

2.8 逻辑拓扑图

Figure 3. Logical Architecture of Two-Node Oracle RAC on vSphere

3、部署流程概述

部署流程大体按照下面的流程图进行.
Figure 4. Deployment Steps for Four-Node Oracle RAC on vSphere with VMFS

4、为第一个RAC节点准备虚拟机环境

本节安装步骤简介如下:

  • 使用vSphere Client创建第一个虚拟机。 添加公网和心跳网卡, 挑选适当的LUN创建仲裁盘CRS (VMDK模式) 。
  • 安装Redhat Linux操作系统, 安装VMware tools, 按照Oracle RAC要求安装相应补丁.
  • 在Linux中配置NTP 客户端.

4.1 创建第一个虚拟机

  1. 使用vSphere Client登录vCenter.
  2. 选中一台主机,在Getting Started 页面单击 Create a new virtual machine.
  3. In the Create New Virtual Machine wizard, select Typical for the Configuration, and click Next.
  4. On the Name and Location page, enter RAC1 for the Name, and click Next.
  5. On the Datastore page, select VMDATASTORE, and click Next.
  6. On the Guest Operating System page, select Linux from the Guest Operating System list. For the Version, select Oracle Linux 6 (64-bit) from the drop-down menu. Click Next.
  7. On the Create a Disk page, set the Virtual Disk Size to 20GB, and click Next.
  8. On the Ready to Complete page, select Edit the virtual machine settings before completion. Click Continue.
  9. On the Virtual Machine Properties page for the RAC1 virtual machine, select 8 CPUs and 128GB Memory.
  10. Click Finish.

4.2 增加两个虚拟网卡

现在需要增加两块网卡. 一块用于公共数据网络, 另一块用于心跳网络. 添加这两块网卡的过程基本相同,差别仅在于添加公网网卡时选择标签: VM Net
而添加心跳网卡时选择 Oracle HB.添加公网网卡的步骤:

  1. Select a virtual machine.
  2. Click Edit settings to display the Virtual Machine Properties pane.
  3. Click Add.
  4. Select Ethernet Adaptor and click Next to display the Add Hardware wizard.
  5. On the Network connection page, select VMXNET 3 from the Adapter Type drop-down menu.
  6. In the Network Connection section of the Network connection page, select Named network with specified label.
  7. In the Named network with specified label drop-down menu, select VM Net.
  8. In the Device Status section of the Network connection page, select Connect at power on.
  9. On the Ready to Complete page, click Finish, and then click OK.
    添加心跳网卡的步骤:

  10. Repeat steps 1 through 6 from the preceding procedure.

  11. In the Named network with specified label drop-down menu, select Oracle HB.
  12. Continue with steps 8 and 9 from the preceding procedure.

4.3 添加仲裁盘

Oracle RAC需要至少一个共享的仲裁盘存放clusterware信息和节点投票信息。 存储管理员需要事先为仲裁盘分配好单独的LUN, 然后我们需要创建单独的SCSI控制器并设为Paravirtual驱动. SCSI 总线 sharing policy 设为 none.

下面描述创建共享存储的步骤.

为虚拟机创建 10GB datastore:

  1. In vSphere Client, select a virtual machine.
  2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
  3. Click Add to display the Add Hardware wizard.
  4. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next.
  5. In the Disk section of the Select a Disk page, select Create a new virtual disk. Set the disk size to 10GB, select the Support clustering features such as Fault Tolerance check box, and select CRS1 to specify a datastore. Also, for vSphere 5, in the Disk Provisioning section select Thick Provision Eager Zeroed. Click Next.
  6. On the Compatibility Mode page, click Next.
  7. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:0). In the Mode section, select Independent. Click Next.
  8. On the Ready to Complete page, click Finish.
  9. Click OK. Add configuration parameters for each Oracle RAC virtual machine as described in Disabling simultaneous write protection provided by VMFS usi
    手工修改虚拟机参数

  10. In vSphere Client, select a virtual machine.

  11. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.3. Click the Options tab, click Advanced, and click General.
  12. Click Configuration Parameters.
  13. On the Configuration Parameters page, add rows and entries for the shared disks in the virtual machine's configuration parameters as shown in the following screenshot.
  14. Repeat steps 1 through 5 for each of the remaining virtual machines.

    You can view the virtual machine properties for the RAC1 virtual machine in vCenter as in the following screenshot.

    web client 显示如下图:

4.4 安装操作系统

将OEL6.4安装介质ISO文件上传到datastore, 然后进行安装。
Note: This document is specific to Linux guest operating system. Oracle RAC 11.2.0.2 and above versions can be deployed and supported on the
在虚拟机上安装Linux操作系统

  1. In vSphere Client, select a virtual machine.
  2. Click Edit settings.
  3. In the Hardware tab of the Virtual Machine Properties pane, select CD/DVD Drive 1.
  4. In the Device Status section of this pane, select the Connect at power on check box.
  5. In the Device Type section of this pane, select Datastore ISO File and enter the path to your Linux ISO image file.
  6. In the Virtual Device Node section of this pane, select CD/DVD Drive 1 from the drop-down menu.
  7. Click OK.
  8. Power on the virtual machine.
  9. Install Linux with the following settings.
    Custom partitioning scheme:
    /dev/sda1 ext3 100GB mounted on /
    Installed packages – use the default package selection, plus the following:
  • kernel-uek.x86_64
  • kmod-oracleasm
  • oracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64 (download from
    http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html.)
  • oracleasm-support-2.1.3-1.el5.x86_64
    systat-7.0.2-3.el5.x86_641. Use below script to install remaining packages:
yum install binutils \
compat-libcap1 \
compat-libstdc++-33 \
compat-libstdc++-33.i686 \
gcc.x86_64 \
gcc-c++.x86_64 \
glibc.x86_64 \
glibc.i686 \
glibc-devel.x86_64 \
glibc-devel.i686 \
unixODBC.i686 \
unixODBC.x86_64 \
unixODBC-devel.x86_64 \
unixODBC-devel.i686 \
libgcc.x86_64 \
libgcc.i686 \
libstdc++.x86_64 \
libstdc++.i686 \
libstdc++-devel.x86_64 \
libstdc++-devel.i686 \
compat-libstdc++-33.x86_64 \
compat-libstdc++-33.i686 \
libaio.x86_64 \
libaio.i686 \
libaio-devel.x86_64 \
libaio-devel.i686 \
sysstat.x86_64 \
ksh \
libXtst-devel.x86_64 \
libXtst.i686 \
compat-db.x86_64 \
xorg-x11-utils.x86_64 \
xorg-x11-xinit.x86_64 \
make
  1. View the installed packages with this command:
    run command
rpm –q
1. Confirm that the following packages have been installed:
binutils-2.20.51.0.2-5.11.el6 (x86_64)
compat-libcap1-1.10-1 (x86_64)
compat-libstdc++-33-3.2.3-69.el6 (x86_64)
compat-libstdc++-33-3.2.3-69.el6.i686
gcc-4.4.4-13.el6 (x86_64)
gcc-c++-4.4.4-13.el6 (x86_64)
glibc-2.12-1.7.el6 (i686)
glibc-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6.i686
ksh
libgcc-4.4.4-13.el6 (i686)
libgcc-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6.i686libstdc++-devel-4.4.4-13.el6 (x86_64)
libstdc++-devel-4.4.4-13.el6.i686 - libaio-0.3.107-10.el6 (x86_64)
libaio-0.3.107-10.el6.i686
libaio-devel-0.3.107-10.el6 (x86_64)
libaio-devel-0.3.107-10.el6.i686
make-3.81-19.el6
sysstat-9.0.4-11.el6 (x86_64)
unixODBC-2.2.14-11.el6 (x86_64) or later
unixODBC-2.2.14-11.el6.i686 or later
unixODBC-devel-2.2.14-11.el6 (x86_64) or later
unixODBC-devel-2.2.14-11.el6.i686 or later

最后安装下面两个包:
oracle-rdbms-server-11gR2-preinstall (从YUM安装, 这个包会配置Oracle RAC所需系统参数和环境变量)
cvuqdisk (Oracle安装介质中提供这个包, 用于安装前检查)
检查配置文件
/etc/sysctl.conf

kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744 (512 x processes for example 6815744 for 13312 processes)
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
kernel.shmall = physical RAM size / pagesize For most systems, this will be the value 2097152. See Note: 301830.1 for more information.
eg:
kernel.shmall=4160749568(62GB)
kernel.shnmax= 34359738368(32GB)
kernel.shmmax = RAM times 0.5 (or higher at customer's discretion - seeNote:567506.1)
/etc/profile
ulimit –u 16384
ulimit –n 65535

依照oracle传统, 创建clusterware主用户 “grid”:

> [root@racnode1 ~]# groupadd -g 1000 oinstall
> [root@racnode1 ~]# groupadd -g 1200 asmadmin
> [root@racnode1 ~]# groupadd -g 1201 asmdba
> [root@racnode1 ~]# groupadd -g 1202 asmoper
> [root@racnode1 ~]# useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d
/home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid
> [root@racnode1 ~]# id grid uid=1100(grid) gid=1000(oinstall)
groups=1000(oinstall), 1200(asmadmin), 1201(asmdba),1202(asmoper)
设置 grid 帐户的口令:
> [root@racnode1 ~]# passwd grid
Changing password for user grid.
New UNIX password:xxxxxxxxxxx
Retype new UNIX password:xxxxxxxxxxxpasswd: all authentication tokens updated successfully

为grid用户创建初始化脚本(/home/grid/.bash_profile):

# ---------------------------------------------------
# .bash_profile
# ---------------------------------------------------
# OS User: grid
# Application: Oracle Grid Infrastructure
# Version: Oracle 11g release 2
# ---------------------------------------------------
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
alias ls="ls -FA"
# ---------------------------------------------------
# ORACLE_SID
# ---------------------------------------------------
# Specifies the Oracle system identifier (SID)
# for the Automatic Storage Management (ASM)instance
# running on this node.
# Each RAC node must have a unique ORACLE_SID.
# (i.e. +ASM1, +ASM2,...)
# ---------------------------------------------------
ORACLE_SID=+ASM1; export ORACLE_SID (注意节点2这里应改为 +ASM2)
# ---------------------------------------------------
# JAVA_HOME
JAVA_HOME=/usr/local/java; export JAVA_HOME
ORACLE_BASE=/u01/app/grid; export ORACLE_BASE
# ---------------------------------------------------
# ORACLE_HOME
# ---------------------------------------------------
# Specifies the directory containing the Oracle
# Grid Infrastructure software. For grid
# infrastructure for a cluster installations, the Grid
# home must not be placed under one of the Oracle base
# directories, or under Oracle home directories of
# Oracle Database installation owners, or in the home
# directory of an installation owner. During
# installation, ownership of the path to the Grid
# home is changed to root. This change causes
# permission errors for other installations.
# ---------------------------------------------------
ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME
# ---------------------------------------------------
# ORACLE_PATH
# ---------------------------------------------------
# Specifies the search path for files used by Oracle
# applications such as SQLPlus. If the full path to
# the file is not specified, or if the file is not# in the current directory, the Oracle application
# uses ORACLE_PATH to locate the file.
# This variable is used by SQLPlus, Forms and Menu.
# ---------------------------------------------------
ORACLE_PATH=/u01/app/oracle/common/oracle/sql; export ORACLE_PATH
# ---------------------------------------------------
# SQLPATH
# ---------------------------------------------------
# Specifies the directory or list of directories that
# SQL*Plus searches for a login.sql file.
# ---------------------------------------------------
# SQLPATH=/u01/app/common/oracle/sql; export SQLPATH
# ---------------------------------------------------
# ORACLE_TERM
# ---------------------------------------------------
# Defines a terminal definition. If not set, it
# defaults to the value of your TERM environment
# variable. Used by all character mode products.
# ---------------------------------------------------
ORACLE_TERM=xterm; export ORACLE_TERM
# ---------------------------------------------------
# NLS_DATE_FORMAT
# ---------------------------------------------------
# Specifies the default date format to use with the
# TO_CHAR and TO_DATE functions. The default value of
# this parameter is determined by NLS_TERRITORY. The
# value of this parameter can be any valid date
# format mask, and the value must be surrounded by
# double quotation marks. For example:
# # NLS_DATE_FORMAT = "MM/DD/YYYY"
# # ---------------------------------------------------
NLS_DATE_FORMAT="DD-MON-YYYY HH24 SS"; export NLS_DATE_FORMAT
# ---------------------------------------------------
# TNS_ADMIN
# ---------------------------------------------------
# Specifies the directory containing the Oracle Net
# Services configuration files like listener.ora,
# tnsnames.ora, and sqlnet.ora.
# ---------------------------------------------------
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
# ---------------------------------------------------
# ORA_NLS11
# ---------------------------------------------------
# Specifies the directory where the language,
# territory, character set, and linguistic definition
# files are stored.
# ---------------------------------------------------
ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11
# ---------------------------------------------------
# PATH
# ---------------------------------------------------
# Used by the shell to locate executable programs;
# must include the $ORACLE_HOME/bin directory.
# ---------------------------------------------------PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH
# ---------------------------------------------------
# LD_LIBRARY_PATH
# ---------------------------------------------------
# Specifies the list of directories that the shared
# library loader searches to locate shared object
# libraries at runtime.
# ---------------------------------------------------
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
# ---------------------------------------------------
# CLASSPATH
# ---------------------------------------------------
# Specifies the directory or list of directories that
# contain compiled Java classes.
# ---------------------------------------------------
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
# ---------------------------------------------------
# THREADS_FLAG
# ---------------------------------------------------
# All the tools in the JDK use green threads as a
# default. To specify that native threads should be
# used, set the THREADS_FLAG environment variable to
# "native". You can revert to the use of green
# threads by setting THREADS_FLAG to the value
# "green".
# ---------------------------------------------------
THREADS_FLAG=native; export THREADS_FLAG
# ---------------------------------------------------
# TEMP, TMP, and TMPDIR
# ---------------------------------------------------
# Specify the default directories for temporary
# files; if set, tools that create temporary files
# create them in one of these directories.
# ---------------------------------------------------
export TEMP=/tmp
export TMPDIR=/tmp
# ---------------------------------------------------
# UMASK
# ---------------------------------------------------
# Set the default file mode creation mask
# (umask) to 022 to ensure that the user performing
# the Oracle software installation creates files
# with 644 permissions.
# ---------------------------------------------------umask 022

创建oracle账号 “oracle”:

[root@racnode1 ~]# groupadd -g 1300 dba
[root@racnode1 ~]# groupadd -g 1301 oper
[root@racnode1 ~]# useradd -m -u 1101 -g oinstall -G dba,oper,asmdba -d
/home/oracle -s /bin/bash -c "Oracle Software Owner" oracle
[root@racnode1 ~]# id oracle
uid=1101(oracle)
gid=1000(oinstall)
groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper)

然后为oracle用户创建登录环境(/home/oracle/.bash_profile)

# ---------------------------------------------------
# .bash_profile
# ---------------------------------------------------
# OS User: oracle
# Application: Oracle Database Software Owner
# Version: Oracle 11g release 2
# ---------------------------------------------------
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
alias ls="ls -FA"
# ---------------------------------------------------
# ORACLE_SID
# ---------------------------------------------------
# Specifies the Oracle system identifier (SID) for
# the Oracle instance running on this node.
# Each RAC node must have a unique ORACLE_SID.
# (i.e. racdb1, racdb2,...)
# ---------------------------------------------------
ORACLE_SID=racdb1; export ORACLE_SID (注意第二个节点改为racdb2)
# ---------------------------------------------------
# ORACLE_UNQNAME
# ---------------------------------------------------
# In previous releases of Oracle Database, you were
# required to set environment variables for
# ORACLE_HOME and ORACLE_SID to start, stop, and
# check the status of Enterprise Manager. With
# Oracle Database 11g release 2 (11.2) and later, you
# need to set the environment variables ORACLE_HOME
# and ORACLE_UNQNAME to use Enterprise Manager.
# Set ORACLE_UNQNAME equal to the database unique
# name.
# ---------------------------------------------------
ORACLE_UNQNAME=racdb; export ORACLE_UNQNAME
# ---------------------------------------------------
# JAVA_HOME
# ---------------------------------------------------# Specifies the directory of the Java SDK and Runtime
# Environment.
# ---------------------------------------------------
JAVA_HOME=/usr/local/java; export JAVA_HOME
# ---------------------------------------------------
# ORACLE_BASE
# ---------------------------------------------------
# Specifies the base of the Oracle directory structure
# for Optimal Flexible Architecture (OFA) compliant
# database software installations.
# ---------------------------------------------------
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
# ---------------------------------------------------
# ORACLE_HOME
# ---------------------------------------------------
# Specifies the directory containing the Oracle
# Database software.
# ---------------------------------------------------
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME
# ---------------------------------------------------
# ORACLE_PATH
# ---------------------------------------------------
# Specifies the search path for files used by Oracle
# applications such as SQLPlus. If the full path to
# the file is not specified, or if the file is not
# in the current directory, the Oracle application
# uses ORACLE_PATH to locate the file.
# This variable is used by SQLPlus, Forms and Menu.
# ---------------------------------------------------
ORACLE_PATH=/u01/app/common/oracle/sql; export ORACLE_PATH
# ---------------------------------------------------
# SQLPATH
# ---------------------------------------------------
# Specifies the directory or list of directories that
# SQL*Plus searches for a login.sql file.
# ---------------------------------------------------
# SQLPATH=/u01/app/common/oracle/sql; export SQLPATH
# ---------------------------------------------------
# ORACLE_TERM
# ---------------------------------------------------
# Defines a terminal definition. If not set, it
# defaults to the value of your TERM environment
# variable. Used by all character mode products.
# ---------------------------------------------------
ORACLE_TERM=xterm; export ORACLE_TERM
# ---------------------------------------------------
# NLS_DATE_FORMAT
# ---------------------------------------------------
# Specifies the default date format to use with the
# TO_CHAR and TO_DATE functions. The default value of
# this parameter is determined by NLS_TERRITORY. The
# value of this parameter can be any valid date
# format mask, and the value must be surrounded by
# double quotation marks. For example:# # NLS_DATE_FORMAT = "MM/DD/YYYY"
# # ---------------------------------------------------
NLS_DATE_FORMAT="DD-MON-YYYY HH24 SS"; export NLS_DATE_FORMAT
# ---------------------------------------------------
# TNS_ADMIN
# ---------------------------------------------------
# Specifies the directory containing the Oracle Net
# Services configuration files like listener.ora,
# tnsnames.ora, and sqlnet.ora.
# ---------------------------------------------------
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
# ---------------------------------------------------
# ORA_NLS11
# ---------------------------------------------------
# Specifies the directory where the language,
# territory, character set, and linguistic definition
# files are stored.
# ---------------------------------------------------
ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11
# ---------------------------------------------------
# PATH
# ---------------------------------------------------
# Used by the shell to locate executable programs;
# must include the $ORACLE_HOME/bin directory.
# ---------------------------------------------------
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH
# ---------------------------------------------------
# LD_LIBRARY_PATH
# ---------------------------------------------------
# Specifies the list of directories that the shared
# library loader searches to locate shared object
# libraries at runtime.
# ---------------------------------------------------
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
# ---------------------------------------------------
# CLASSPATH
# ---------------------------------------------------
# Specifies the directory or list of directories that
# contain compiled Java classes.
# ---------------------------------------------------
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
# ---------------------------------------------------
# THREADS_FLAG
# ---------------------------------------------------# All the tools in the JDK use green threads as a
# default. To specify that native threads should be
# used, set the THREADS_FLAG environment variable to
# "native". You can revert to the use of green
# threads by setting THREADS_FLAG to the value
# "green".
# ---------------------------------------------------
THREADS_FLAG=native; export THREADS_FLAG
# ---------------------------------------------------
# TEMP, TMP, and TMPDIR
# ---------------------------------------------------
# Specify the default directories for temporary
# files; if set, tools that create temporary files
# create them in one of these directories.
# ---------------------------------------------------
export TEMP=/tmp
export TMPDIR=/tmp
# ---------------------------------------------------
# UMASK
# ---------------------------------------------------
# Set the default file mode creation mask
# (umask) to 022 to ensure that the user performing
# the Oracle software installation creates files
# with 644 permissions.
# ---------------------------------------------------
umask 022

创建RAC安装所需目录结构:

mkdir -p /u01/app/11.2.0/grid

chown grid:\oinstall /u01/app/11.2.0/grid

chmod -R 775 /u01/app/11.2.0/grid

mkdir -p /u01/app/oracle

chown -R oracle:\oinstall /u01/app/oracle

chmod -R 775 /u01/app/oracle

mkdir /u01/app/oraInventory

chwon grid:\oinstall /u01/app/oraInventory

chmod –R 775 /u01/app/oraInventory

4.5 安装VMware Tools

VMware Tools 是VMware为虚拟机操作系统提供的工具包, 用于提升虚拟机性能和方便对虚拟机的监控和管理。 VMware Tools 必须在操作系统内安装。 其安装步骤可以参考以下知识
To install VMware Tools

  1. Confirm that the Linux virtual machine is powered on.
  2. In vSphere Client, right-click the virtual machine and select Guest > Install VMware Tools.
  3. In the Linux guest OS, create a mount point and run the following command: mkdir /mnt/cdrom
  4. Mount the CD-ROM with the following command: mount /dev/cdrom /mnt/cdrom5. Install VMware Tools using RPM, with the following command: rpm -ivh /mnt/cdrom/VMwareTools-8.3.2-257589.rpm As an example, “8.3.2-257589” is the version of VMware Tools used here.
  5. Configure VMware Tools, by running the following command: /usr/bin/vmware-config-tools.pl
  6. Unmount the CD-ROM with the following command: umount /mnt/cdrom
  7. Click VM in the virtual machine menu, and click Guest > End VMware Tools Install.

5. 准备其他虚拟机和RAC节点

现在第一个RAC节点(RAC1) 所用的虚拟机就创建好了, 现在可以用这个虚拟机克隆出其他RAC节点所用的虚拟机, 如果前面的步骤都执行正确, 现在就可以把这个虚拟机制成模板, 留作

  • 在vCenter里利用 RAC1 复制新的虚拟机.
  • 在新创建的虚拟机里删除RAC1所带的共享磁盘。
  • 重新添加仲裁盘, 仲裁盘同样需要开启multi-writer开关。
  • 修改客户机IP地址.

5.1 虚拟机克隆

  1. 配置文件/etc/udev/rules.d/70-persistent-net.rules 做备份然后删除
  2. RAC1停机
  3. In vSphere Client, select the virtual machine, RAC1.
  4. Right-click the name of the virtual machine and select Clone.
  5. In the Clone Virtual Machine wizard on the Name and Location page, enter RAC2 for the Name. Click Next.
  6. On the Host / Cluster page, select a host or cluster. Click Next.
  7. On the Specific Host page, select a unique ESX host (that is, one not running a RAC node virtual machine). Click Next.
  8. On the Datastore page, select a VMFS datastore. Click Next.
  9. On the Disk Format page, select Same format as source. Click Next.
  10. On the Guest Customization page, select customize using the customization wizard.
  11. config IP address for eth0 and eth1
  12. On the Ready to Complete page, click Finish.

5.2 配置仲裁盘

虚拟机克隆之后, 指向仲裁盘的路径会失效, 所以需要先在虚机配置里删除这个硬盘重建, 同时把SCSI控制器类型改为paravirtual.
按照下述步骤配置 RAC2:
删除共享盘

  1. In vSphere Client, select a virtual machine.
  2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
  3. Select hard disk 2 and click Remove. In the Removal Options section, select Remove from virtual machine and click OK.
    再新建一次, 注意要找到RAC1所用的共享磁盘

  4. In vSphere Client, select a virtual machine.

  5. Right-click the name of a virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
  6. Click Add to display the Add Hardware wizard.
  7. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next.5. In the Disk section of the Select a Disk page, select Use an existing virtual disk and click Next.
  8. On the Select Existing Disk page, enter the path to the first VMDK of the virtual machine (selected in step 1) used for the CRS and vot
  9. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:0). In the Mode section, select Independent and Persis
  10. On the Ready to Complete page, click Finish.
  11. Click OK.
    修改SCSI控制器类型为 paravirtual

  12. In vSphere Client, select the virtual machine that was selected in the preceding step 1.

  13. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
  14. In the device list, select SCSI controller 1.
  15. In the SCSI Controller Type section, click Change Type.
  16. Select VMware Paravirtual.
  17. Click OK, and click OK again.

5.3 使用fdisk格式化仲裁盘

必须在一个节点上对共享盘进行分区和格式化。 在Linux上可以使用系统自带的fdisk工具进行 。 步骤如下:

**仲裁盘分区: **

  1. 以root用户登录RAC1节点.
  2. Enter the command: fdisk /dev/sdb
  3. Enter n to create a new partition.
  4. Enter p to create a primary partition.
  5. Enter 1 to create the first partition.
  6. Press Enter to take the default value of 1 for the first cylinder.
  7. Press Enter to take the default value of 20480 for the last cylinder.
  8. Enter w to write the new partition table to disk and exit fdisk.
    fdisk命令的执行过程如下图所示:
    > root@vmorarac1 ~# fdisk /dev/sdb
    Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
    Building a new DOS disklabel. Changes will remain in memory only,
    until you decide to write them. After that, of course, the previous
    content won't be recoverable.
    Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
    Command (m for help): n
    Command action
    e extended
    p primary partition (1-4)
    p P
    artition number (1-4): 1
    First cylinder (1-20480, default 1):
    Using default value 1Last cylinder or +size or +sizeM or +sizeK (1-20480, default 20480):
    Using default value 20480
    Command (m for help): w
    The partition table has been altered!
    Calling ioctl() to re-read partition table.
    Syncing disks

6. 安装Oracle Grid软件

现在我们有两个虚机, 可以开始安装grid环境了.

安装Oracle Grid Infrastructure

  1. 用oracleasm在/dev/sdb1 创建griddisk CRS1.
  2. 将Oracle RAC安装包(11.2.0.3) 上传到RAC1并解压
  3. 切换到grid用户启动Oracle Grid安装程序 ./runInstaller
  4. 在第一页安装选项页, 选择 Install and Configure Infrastructure for a Cluster. Click Next.
  5. On the Installation Type page, select Advanced Installation and click Next.
  6. On the Product Languages page, select English and click Next.
  7. On the Grid Plug and Play page, enter rac for the Cluster Name,rac-scan.localdomain for the SCAN Name and 1521 for the SCAN Port. Click Next.
  8. On the Cluster Node Information page, click Add to enter a Hostname and Virtual IP Name. Repeat the process for each of the four hosts, Hostname | Virtual IP Name ---|--- RAC1 | RAC1-vip RAC2 | RAC2-vip
  9. Click Next.
  10. On the Network Interface Usage page, enter the following information for the Interface Name, Subnet, and Interface Type: Interface Name | Subnet|InterfaceType ---|--- |---| eth0 | 162.16.3.0|Public eth1 | 192.168.2.0|Private
  11. Click Next.
  12. On the Storage Option page, select Automatic Storage Management (ASM), and click Next.
  13. On the Create ASM Disk Group page, enter CRS for the Disk Group Name, and select Normal for Redundancy.
  14. In the Add Disks section of the Create ASM Disk Group page, select Candidate Disks. Make the following entries: Disk Path | Size (in MB)|Status ---|--- |---| ORCL:CRSDISK1 | 20479|Candidate
  15. Select the check box to the left of each of these entries, and click Next.
  16. On the ASM Password page, select Use same passwords for these accounts and enter the password “oracle” in the Specify Password and Con
  17. On the Failure Isolation page, select Do not use Intelligent Platform Management Interface (IPMI). Click Next.
  18. On the Operating System Groups page, make the following entries:Field | Selection ---|--- ASM Database Administrator (OSDBA) Group | asmdba ASM Instance Administration Operator (OSOPER) Group | asmoper ASM Instance Administrator (OSASM) Group | asmadmin
  19. Click Next.
  20. On the Installation Location page, enter /u01/app/grid for the Oracle Base, and /u01/app/11.2.0/grid for the Software Location. Click
  21. On the Create Inventory page, enter /u01/app/orainventory for the Inventory Directory. Click Next.
  22. Wait while the setup application conducts the prerequisite checks and installs Grid on each node. After setup is complete, execute the /u01/app/orainventory/orinstRoot.sh /u01/app/11.2.0/grid/root.sh Note: don’t run root.sh scripts on 2 nodes at the same time, or else It will fail.
  23. Click OK, and click Finish.

7. 验证Grid安装的正确性

下面几幅图演示了怎样使用Oracle命令验证clusterware是否已经正常工作:
Figure 5. Check the CRS Status on the First Node as Grid User
Figure 6. Check for Cluster Nodes
Figure 7. Check for Cluster Registry (OCR)
Figure 8. Check Voting Disk

8. 增加数据盘和REDO盘

在这一节, 我们为RAC节点添加数据盘和重做日志(redo log) 盘:
- 现在预先分配好的LUN上创建datastore, 然后用RAC1创建对应的VMDK, SCSI控制器类型设为paravirtual.

  • 把数据盘和重做日志盘添加到RAC2.
  • 在RAC1使用fdisk格式化新添加的磁盘
  • 为每个共享盘设置multi-writer参数.
    下面描述的RAC1的配置过程, RAC2类似.
    添加数据盘和重做日志盘 :
  1. In vSphere Client, select the virtual machine.
  2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
  3. Click Add to display the Add Hardware wizard.
  4. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next.
  5. In the Disk section of the Select a Disk page, select Create a new virtual disk. Set the disk size to 1000GB for DATA or 900GB for REDO
  6. On the Compatibility Mode page, click Next.7. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:1) for REDO , or SCSI (2:0) for DATA. Set the Mode to
  7. On the Ready to Complete page, click Finish.
  8. Click OK.
  9. Repeat steps 1 through 9 for all the DATA datastores. Change step 7 to use SCSI (2:1)/ SCSI (3:0)/ SCSI (3:1) for the other DATA datas
    设置SCSI控制器类型为 paravirtual:

  10. Select the virtual machine that was selected in the preceding step

  11. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
  12. In the device list, select SCSI controller 1.
  13. In the SCSI Bus Sharing section, select None, keep the default selection.
  14. In the SCSI Controller Type section, click Change Type.
  15. Select VMware Paravirtual.
  16. Click OK, and click OK again.
    参照5.3小节, 使用 fdisk 命令格式化所有共享盘
为每个RAC节点的共享盘配置multi-writer参数
细节请参考官方文档 Disabling simultaneous write protection provided by VMFS using the multi-writer flag (http://kb.vmware.com/kb/1034165).
1. In vSphere Client, select a virtual machine.
2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.
3. Click the Options tab, click Advanced, and click General.
4. Click Configuration Parameters.
5. On the Configuration Parameters page, add rows and entries for the shared disks in the virtual machine’s configuration parameters as shown
in the following screenshot


虚拟机硬件配置如下图所示:

9. 以grid用户创建DATA和REDO磁盘组

为安装数据库做准备

  1. 使用oracleasm创建4个数据盘DATA1到DATA4, 使用的设备为/dev/sdd1到/dev/sdg1。 用设备/dev/sdc1创建REDO磁盘REDO1.
  2. 以grid用户登录节点1 (RAC1), 启动ASM配置助手(asmca) , 创建两个磁盘组 DATADG (选择所有DATA标签盘DATA1到DATA4) 和 REDODG (选择磁盘 REDO1).
  3. In the ASM Configuration Assistant, on the Create Disk Group page, enter DATADG for the Disk Group Name. In the Redundancy section, select External (None). Click OK, and click OK again.
  4. On the Create Disk Group page, in the Select Member Disks section, select ORCL: DATA1, ORCL: DATA2, ORCL: DATA3, and ORCL: DATA4. Set the size of each to 1000GB. Click OK.
  5. On the Create Disk Group page, enter REDODG for the Disk Group Name. In the Redundancy section, select External (None). Click OK, and click OK again.
  6. On the Create Disk Group page, in the Select Member Disks section, select ORCL:REDO1, ,. Set the size of each to 900GB. Click OK.

10. 安装 RAC数据库

安装并创建数据库

  1. Log in into RAC1 as oracle user and launch runInstaller from the Oracle software location to install the Oracle binaries.
  2. In the Oracle Installer wizard, on the Configure Security Updates page, leave the Email field blank, and do not select I wish to receive security updates via My Oracle Support. Click Next.
  3. On the Installation Option page, select Install database software only. Click Next.4. On the Grid Options page, select Real Application Clusters database installation. Select the four nodes RAC1, RAC2. Click Next.
  4. On the Product Languages page, select English. Click Next.
  5. On the Database Edition page, select Enterprise Edition (4.29GB). Click Next.
  6. On the Installation Location page, enter /u01/app/oracle for Oracle Base, and /u01/app/oracle/product/11.2.0/dbhome_1 for Software Location. Click Next.
  7. On the Operating System Groups page, select dba for the Database Administrator (OSDBA) Group and oinstall for the Database Operator (OSOPER) Group. Click Next.
  8. Review the configuration on the Summary page, and click Finish.
  9. After the installation is complete, open a terminal window, log in as root, and run the following script: /u01/app/oracle/product/11.2.0/dbhome_1/root.sh Repeat for each of the four nodes, and click OK.
  10. On the Finish page, click Close.
  11. Create the Oracle RAC database using Oracle Database Configuration Assistant (DBCA).

11. 参考资料

The following are resources and references for Oracle and VMware vSphere.
vSphere Installation and Setup

http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-installation-setup-guide.pdf
vSphere Networking
http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-networking-guide.pdf
vSphere Storage
http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-storage-guide.pdf
Guide to configure NTP on ESX servers
http://kb.vmware.com/kb/1003063
Timekeeping in VMware Virtual Machines
http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf
Installing VMware Tools in a Linux virtual machine using Red Hat Package Manager (RPM)
http://kb.vmware.com/kb/1018392
Oracle ASMLib
http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html
Oracle Database Oracle Clusterware and Oracle Real Application Clusters Installation Guide 10g Release 2 (10.2) for Linux
http://download.oracle.com/docs/cd/B19306_01/install.102/b14203/toc.htm
Oracle Clusterware Installation Guide 11g Release 1 (11.1) for Linux
http://www.oracle.com/pls/db111/to_toc?pathname=install.111/b28263/toc.htm
Oracle Real Application Clusters Installation Guide 11g Release 1 (11.1) for Linux and UNIX
http://www.oracle.com/pls/db111/to_toc?pathname=install.111/b28264/toc.htm
未经允许不得转载:许春刚技术博客 » Orcal RAC on vsphere

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址