Installing and upgrading on cluster nodes

Planning and preparing for cluster installation

Beginning the installation of the cluster hardware

Initial steps to carry out in the BIOS or EFI when using a SCSI shared storage device

Initial steps to carry out in the BIOS or EFI when using a fibre channel shared storage device or no shared storage device

Final steps to complete the installation

If you have not already installed Windows Server Enterprise Edition, or Windows Server Datacenter Edition, on the first cluster node, install it before proceeding. After you complete the BIOS or EFI configuration, start the operating system on one cluster node only and complete the configuration of the Cluster service using Cluster Administrator.

With the Cluster Administrator New Server Cluster Wizard, you can choose between Typical (full) configuration and Advanced (minimum) configuration options. Typical configuration is appropriate for most installations and results in a completely configured cluster. Use the Advanced configuration option only for clusters that have complex storage configurations that the New Server Cluster Wizard cannot validate or for configurations in which you do not want the cluster to manage all of the storage. The following examples describe each situation:

• In some complex storage solutions, such as a fiber channel switched fabric that contains several switches, a particular storage unit might have a different identity on each computer in the cluster. Although this is a valid storage configuration, it violates the storage validation heuristics in the New Server Cluster Wizard. If you have this type of storage solution, you might receive an error when you are trying to create a cluster using the Typical configuration option. If your storage configuration is set up correctly, you can disable the storage validation heuristics and avoid this error by restarting the New Server Cluster Wizard, selecting the Advanced configuration option instead.
  
  
• On particular nodes in a cluster, you may want to have some disks that are to be clustered and some disks that are to be kept private. The Typical configuration option configures all
  disks as clustered disks and creates cluster resources for them all. However, with the Advanced configuration option, you can keep certain disks private because this configuration creates a cluster in which only the quorum disk is managed by the cluster (if you chose to use a physical disk as the quorum resource). After the cluster is created, you must then use Cluster Administrator to add any other disks that you want the cluster to manage.
  
  

Important

• If you are using a shared storage device: Before creating a cluster, when you turn the computer on and start the operating system, it is very important that only one node has access to the cluster disk. Otherwise, the cluster disks can become corrupted. To prevent the
  corruption of the cluster disks, shut down all but one cluster node, or use other techniques (for example, LUN masking, selective presentation, or zoning) to protect the cluster disks before creating the cluster. Also, before starting the installation of the second and subsequent nodes, ensure that all disks that are to be managed by the cluster have disk resources associated with them. If these disks do not have disk resources associated with them at this time, the disk data will be corrupted because the disks will not be protected and multiple nodes will attempt to connect to them at the same time.
  
  
• Do not use Manage Your Server or the Configure Your Server Wizard to configure cluster nodes.

Quorum resource options when installing server clusters

With server clusters on the Windows Server operating systems, you can now choose between three ways to set up the quorum resource (the resource that maintains the definitive copy of the cluster configuration data and that must always be available for the cluster to run).

The first is a single node server cluster, which has been available in the past and continues to be supported. A single node cluster is often used for development and testing and can be configured

with, or without, external cluster storage devices. For single node clusters without an external cluster storage device, the local disk is configured as the cluster quorum device.

The second option is a single quorum device server cluster, which has also been available in earlier Windows versions. This model places the cluster configuration data on a shared cluster storage device that all nodes can access. The general topology is:

This is the most common model and is recommended for most situations. You might choose the single quorum device model if all of your cluster nodes are in the same location and you want to take advantage of the fact such a cluster continues supporting users even if only one node is running.

The third option, which is newer for Windows Server, is a "majority node set." A majority node set is a single quorum resource from a Server Cluster perspective; however, the cluster configuration data is actually stored on multiple disks across the cluster. The majority node set resource ensures that the cluster configuration data is kept consistent across the different disks. This allows cluster topologies as follows:

In the majority node set model, every node in the cluster uses a directory on its own local system disk to store the cluster configuration data. If the configuration of the cluster changes, that change is reflected across the different disks. Be aware that it is also possible to have shared storage devices in a majority node set cluster. The exact configuration depends on your installation's requirements.

Only use a majority node set cluster in targeted scenarios, such as:

• Geographically dispersed cluster: A cluster that spans multiple sites.
  
  
• Eliminating single points of failure: Although when using a single cluster storage device the quorum disk itself can be made highly available via RAID, the controller port or the Host Bus Adapters (HBA) itself may be a single point of failure.
  
  
• Clusters with no shared disks: There are some specialized configurations that need tightly consistent cluster features without having shared disks.
  
  
• Clusters that host applications that can fail over, but where there is some other, application-specific way, to replicate or mirror data between nodes.
  
  

Do not configure your cluster as a majority node set cluster unless it is part of a cluster solution offered by your Original Equipment Manufacturer (OEM), Independent Software Vendor (ISV), or Independent Hardware Vendor (IHV).

Cluster model considerations

Node failover behavior

Geographic Considerations

Hardware

Microsoft supports only complete server cluster systems that are compatible with the Windows Server family of products. Confirm that your entire cluster solution is compatible with products in the Windows Server family.

iSCSI is an abbreviation of Internet Small Computer System Interface, an IP-based storage networking standard for linking data storage facilities. By carrying SCSI commands over IP networks, iSCSI is used to facilitate data transfers over intranets and to manage storage over long distances. iSCSI can be used to transmit data over local area networks (LANs), wide area networks (WANs), or the Internet and can enable location-independent data storage and retrieval. The protocol allows clients to send SCSI commands to SCSI storage devices on remote servers. It is a popular Storage Area Network (SAN) protocol, allowing organizations to consolidate storage into data centre storage arrays while providing hosts (such as database and web servers) with the illusion of locally-attached disks. Unlike traditional Fibre Channel, which requires special-purpose cabling, iSCSI can be run over long distances using existing network infrastructure.

Functionality

iSCSI uses TCP/IP (typically TCP ports 860 and 3260). In essence, iSCSI simply allows two hosts to negotiate and then exchange SCSI commands using IP networks. By doing this iSCSI takes a popular high-performance local storage bus and emulates it over WANs, creating a storage area network (SAN). Unlike Fibre Channel as a SAN protocol, iSCSI requires no dedicated cabling; it can be run over existing switching and IP infrastructure. However, the performance of an iSCSI SAN deployment can be severely degraded if not operated on a dedicated network or subnet (LAN or VLAN). As a result, iSCSI is often seen as a low-cost alternative to Fibre Channel, which requires dedicated infrastructure except in its FCoE (Fibre Channel over Ethernet) form.

Although iSCSI can communicate with arbitrary types of SCSI devices, system administrators almost always use it to allow server computers (such as database servers) to access disk volumes on storage arrays. iSCSI SANs often have one of two objectives:

• Storage consolidation

Organizations move disparate storage resources from servers around their network to central locations, often in data centres; this allows for more efficiency in the allocation of storage. In a SAN environment, a server can be allocated a new disk volume without any change to hardware or cabling.

• Disaster recovery

Organizations mirror storage resources from one data centre to a remote data centre, which can serve as a hot standby in the event of a prolonged outage. In particular, iSCSI SANs allow entire disk arrays to be migrated across a WAN with minimal configuration changes, in effect making storage "routable" in the same manner as network traffic.

Network booting

For general data storage on an already-booted computer, any type of generic network interface may be used to access iSCSI devices. However, a generic consumer-grade network interface is not able to boot a diskless computer from a remote iSCSI data source. Instead it is commonplace for a server to load its initial operating system from a tftp server or local boot device, and then use iSCSI for data storage once booting from the local device has finished.

A separate DHCP server may be configured to assist interfaces equipped with network boot capability to be able to boot over iSCSI. In this case the network interface looks for a DHCP server offering a PXE or bootp boot image. This is used to kick off the iSCSI remote boot process, using the booting network interface's MAC address to direct the computer to the correct iSCSI boot target.

Specialized iSCSI interfaces are available with built-in BIOS functionality that allows the interface to be preassigned to an iSCSI target, and be able to boot from it without additional help from a boot server, thereby reducing the network configuration complexity.

Security

• Authentication:
• iSCSI initiators (clients) and targets (storage device on remote servers) prove their identity to each other using the CHAP protocol, which includes a mechanism to prevent cleartext passwords from appearing on the wire. By itself, the CHAP protocol is vulnerable to dictionary attacks, spoofing, or reflection attacks. If followed carefully, the rules for using CHAP within iSCSI prevent most of these attacks.
  
• Additionally, as with all IP-based protocols, IPsec can operate at the network layer.
  
• The iSCSI negotiation protocol is designed to accommodate other authentication schemes, though interoperability issues limit their deployment.

• Logical network isolation:
• To ensure that only valid initiators connect to storage arrays, administrators most commonly run iSCSI only over logically-isolated backchannel networks. In this deployment architecture, only the management ports of storage arrays are exposed to the general-purpose internal network, and the iSCSI protocol itself is run over dedicated network segments or virtual LANs (VLAN).
  
• This mitigates authentication concerns; unauthorized users aren't physically provisioned for iSCSI, and thus can't talk to storage arrays. However, it also creates a transitive trust problem, in that a single compromised host with an iSCSI disk can be used to attack storage resources for other hosts.

• Physical network isolation:
  
• While iSCSI can be logically isolated from the general network using VLANs only, it is still no different from any other network equipment and may use any cable or port as long as there is a completed signal path between source and target. Just a single cabling mistake by an inexperienced network technician can compromise the barrier of logical separation, and an accidental bridging may not be immediately detected because it does not cause network errors.
  
• In order to further differentiate iSCSI from the regular network and prevent cabling mistakes when changing connections, administrators may implement self-defined colour coding and labelling standards, such as only using yellow-coloured cables for the iSCSI connections and only blue cables for the regular network, and clearly labelling ports and switches used only for iSCSI.
• While iSCSI could be implemented as just a VLAN cluster of ports on a large multi-port switch that is also used for general network usage, the administrator may instead choose to use physically separate switches dedicated to iSCSI VLANs only, to further prevent the possibility of an incorrectly connected cable plugged into the wrong port bridging the logical barrier.

• Authorization:
• Because iSCSI aims to consolidate storage for many servers into a single storage array, iSCSI deployments require strategies to prevent unrelated initiators from accessing storage resources.
  
• As a pathological example, a single enterprise storage array could hold data for servers variously regulated by the Sarbanes-Oxley Act for corporate accounting, HIPAA for health benefits information, and PCI DSS for credit card processing.
  
• During an audit, storage systems must demonstrate controls to ensure that a server under one regime cannot access the storage assets of a server under another.
• Typically, iSCSI storage arrays explicitly map initiators to specific target LUNs (Logical Unit Number[1]); an initiator authenticates not to the storage array, but to the specific storage asset it intends to use. However, because the target LUNs for SCSI commands are expressed both in the iSCSI negotiation protocol and in the underlying SCSI protocol, care must be taken to ensure that access control is provided consistently.

Fibre Channel, or FC, is a gigabit-speed network technology primarily used for storage networking. It started use primarily in the supercomputer field, but has become the standard connection type for storage area networks (SAN) in enterprise storage. Despite its name, Fibre Channel signaling can run on both twisted pair copper wire and fiber-optic cables.

Fibre Channel Protocol (FCP) is a transport protocol (similar to TCP used in IP networks) which predominantly transports SCSI commands over Fibre Channel networks.

Fibre Channel topologies

There are three major Fibre Channel topologies, describing how a number of ports are connected together. A port in Fibre Channel terminology is any entity that actively communicates over the network, not necessarily a hardware port. This port is usually implemented in a device such as disk storage, an host bus adapter (HBA) on a server or a Fibre Channel switch. (In the computer storage field, a Fibre Channel switch is a network switch compatible with the Fibre Channel (FC) protocol. It allows the creation of a Fibre Channel fabric, that is currently the core component of most SAN. The fabric is a network of Fibre Channel devices which allows many-to-many communication, device name lookup, security, and redundancy. FC switches implement zoning, a mechanism that disables unwanted traffic between certain fabric nodes.)

-switch Qlogic with optical Fibre Channel connectors installed

Fibre Channel Host Bus Adapter (64-bit PCI-X card)

• Point-to-Point (FC-P2P). Two devices are connected directly to each other. This is the simplest topology, with limited connectivity.
• Arbitrated loop (FC-AL). In this design, all devices are in a loop or ring, similar to token ring networking. Adding or removing a device from the loop causes all activity on the loop to be interrupted. The failure of one device causes a break in the ring. Fibre Channel hubs exist to connect multiple devices together and may bypass failed ports. A loop may also be made by cabling each port to the next in a ring.

o A minimal loop containing only two ports, while appearing to be similar to FC-P2P, differs considerably in terms of the protocol.

o Only one pair of ports can communicate concurrently on a loop.

o Maximum speed of 8GFC.

• Switched fabric (FC-SW). All devices or loops of devices are connected to Fibre Channel switches, similar conceptually to modern Ethernet implementations. Advantages of this topology over FC-P2P or FC-AL include:

o The switches manage the state of the fabric, providing optimized interconnections.

o The traffic between two ports flows through the switches only; it is not transmitted to any other port.

o Failure of a port is isolated and should not affect operation of other ports.

o Multiple pairs of ports may communicate simultaneously in a fabric.

Layer

Fibre Channel does not follow the OSI model layering, but is split similarly into 5 layers, namely:

• FC4 — Protocol Mapping layer, in which application protocols, such as SCSI or IP, are encapsulated into a PDU for delivery to FC2.
• FC3 — Common Services layer, a thin layer that could eventually implement functions like encryption or RAID redundancy algorithms;
• FC2 — Network layer, defined by the FC-PI-2 standard, consists of the core of Fibre Channel, and defines the main protocols;
• FC1 — Data Link layer, which implements line coding of signals;
• FC0 — PHY, includes cabling, connectors etc.;

Layers FC0 through FC2 are also known as FC-PH, the physical layers of Fibre Channel.

Fibre Channel routers operate up to FC4 level (i.e. they may operate as SCSI routers), switches up to FC2, and hubs on FC0 only.

Fibre Channel products are available at 1, 2, 4, 8, 10 and 20 Gbit/s; these protocol flavors are called accordingly 1GFC, 2GFC, 4GFC, 8GFC, 10GFC, or 20GFC. The 16GFC standard was approved by the T11 committee in 2010, and products are expected to be available in 2011. Products based on the 1, 2, 4, 8 and 16GFC standards should be interoperable and backward compatible. The 10 Gbit/s standard and its 20 Gbit/s derivative, however, are not backward compatible with any of the slower speed devices, as they differ considerably on FC1 level in using 64b/66b encoding instead of 8b/10b encoding, and are primarily used as inter-switch links. While the 16GFC standard also uses the 64B/66B encoding, it provides backward compatibility with 4 and 8GFC.

Ports

The following types of ports are defined by Fibre Channel:

• node ports

o N_port is a port on the node (e.g. host or storage device) used with both FC-P2P or FC-SW topologies. Also known as Node port.

o NL_port is a port on the node used with an FC-AL topology. Also known as Node Loop port.

o F_port is a port on the switch that connects to a node point-to-point (i.e. connects to an N_port). Also known as Fabric port. An F_port is not loop capable.

o FL_port is a port on the switch that connects to a FC-AL loop (i.e. to NL_ports). Also known as Fabric Loop port.

o E_port is the connection between two fibre channel switches. Also known as an Expansion port. When E_ports between two switches form a link, that link is referred to as an inter-switch link (ISL).

o EX_port is the connection between a fibre channel router and a fibre channel switch. On the side of the switch it looks like a normal E_port, but on the side of the router it is a EX_port.

o TE_port * a Cisco addition to Fibre Channel, now adopted as a standard. It is an extended ISL or EISL. The TE_port provides not only standard E_port functions but allows for routing of multiple VSANs (Virtual SANs). This is accomplished by modifying the standard Fibre Channel frame (vsan tagging) upon ingress/egress of the VSAN environment. Also known as Trunking E_port.

o VE_Port an INCITS T11 addition, FCIP interconnected E-Port/ISL, i.e. fabrics will merge.

o VEX_Port a INCITS T11 addition, is a FCIP interconnected EX-Port, routing needed via lsan zoning to connect initiator to a target.

• general (catch-all) types

o Auto or auto-sensing port found in Cisco switches, can automatically become an E_, TE_, F_, or FL_port as needed.

o Fx_port a generic port that can become a F_port (when connected to a N_port) or a FL_port (when connected to a NL_port). Found only on Cisco devices where oversubscription is a factor.

o G_port or generic port on a switch can operate as an E_port or F_port. Found on Brocade and McData switches.

o L_port is the loose term used for any arbitrated loop port, NL_port or FL_port. Also known as Loop port.

o U_port is the loose term used for any arbitrated port. Also known as Universal port. Found only on Brocade switches.

(*Note: The term "trunking" is not a standard Fibre Channel term and is used by vendors interchangeably. For example: A trunk (an aggregation of ISLs) in a Brocade device is referred to as a Port Channel by Cisco. Whereas Cisco refers to trunking as an EISL.)