PEX8748-CA80BC G - Avago Technologies

PEX 8748, PCI Express Gen 3 Switch, 48 Lanes, 12 Ports

Highlights

 PEX 8748 General Features

o 48-lane, 12-port PCIe Gen 3 switch

- Integrate 8.0 GT/s SerDes d

27 x 27mm2, 676-pin FCBGA package o

Typical Power: 8.0 Watts

 res PEX 8748 Key Featu

o aSt ndards Compliant

- PCI Express Base Specification, r3.0

) (compatible w/ PCIe r1.0a/1.1 & 2.0

ec, r1.2 - PCI Power Management Sp

- Microsoft Vista Compliant ices - Supports Access Control Serv

- Dynamic link-width control

peed control - Dynamic SerDes s

o ghHi Performance

♦ performancePAK

ng (bandwidth throttling) 9 Read Paci

9 Multicast t Pool 9 Dynamic Buffer/FC Credi

ic - Non-blocking switch fabr

- Full line rate on all ports

- Packet Cut-Thru with 100ns max packet

latency (x16 to x16) - 2KB Max Payload Size

o Flexible Configuration 6 - Ports configurable as x1, x2, x4, x8, x1

- Registers configurable with strapping

2t software pins, EEPROM, I C, or hos

- Lane and polarity reversal

- Compatible with PCIe 1.0a PM

o Multi-Host & Fail-Over Support

sparent (NT) port - Configurable Non-Tran

- Failover with NT port

- Up to 6 upstream/Host ports with 1+1 or

tream ports N+1 failover to other ups

o Quality of Service (QoS)

- Eight traffic classes per port

- Weighted round-robin source

port arbitration

o ailability, Serviceability Reliability, Av

♦ visionPAK

9 Per Port Performance Monitoring

header counters  Per port payload &

9 SerDes Eye Capture

9 PCIe Packet Generator

9 Error Injection and Loopback nals - 3 Hot Plug Ports with native HP Sig

- All ports hot plug capable thru I2C

port) (Hot Plug Controller on every

bit support - ECRC and Poison

- Data Path parity

- Memory (RAM) Error Correction

ignals - INTA# and FATAL_ERR# s

- Advanced Error Reporting

- ailable Port Status bits and GPIO av

• Per port error diagnostics

- JTAG AC/DC boundary scan

The ExpressLane™ PEX 8748 device offers Multi-Host PCI Express

switching capability enabling users to connect multiple hosts to their

respective endpoints via scalable, high bandwidth, non-blocking

interconnection to a wide variety of applications including servers,

storage, communications, and graphics platforms. The PEX 8748 is

well suited for fan-out, aggregation, and peer-to-peer traffic patterns.

Multi-Host Architecture

The PEX 8748 employs an enhanced version of PLX’s field tested PEX 8648

PCIe switch architecture, which allows users to configure the device in legacy

single-host mode or multi-host mode with up to six host ports capable of 1+1

(one active & one backup) or N+1 (N active & one backup) host failover. This

powerful architectural enhancement enables users to build PCIe based systems

to support high-availability, failover, redundant, or clustered systems.

High Performance & Low Packet Latency

The PEX 8748 architecture supports packet cut-thru with a maximum

latency of 100ns (x16 to x16). This, combined with large packet memory,

flexible common buffer/FC credit pool and non-blocking internal switch

architecture, provides full line rate on all ports for performance-hungry

applications such as servers and switch fabrics. The low latency enables

applications to achieve high throughput and performance. In addition to low

latency, the device supports a packet payload size of up to 2048 bytes,

enabling the user to achieve even higher throughput.

Data Integrity

The PEX 8748 provides end-to-end CRC (ECRC) protection and Poison bit

support to enable designs that require end-to-end data integrity. PLX also

supports data path parity and memory (RAM) error correction circuitry

throughout the internal data paths as packets pass through the switch.

Flexible Configuration

The PEX 8748’s 12 ports can be

configured to lane widths of x1,

x2, x4, x8, or x16. Flexible buffer

allocation, along with the device's

flexible packet flow control,

maximizes throughput for

applications where more traffic

flows in the downstream, rather

than upstream, direction. Any

port can be designated as the

upstream port, which can be

changed dynamically. Figure 1

shows some of the PEX 8748’s

common port configurations in

legacy Single-Host mode.

Figure 1. Common Port Configurations

PEX 8748

11 x4 2 x44 x8

10 x4 6x42 x8

Figure 1. Common Port Configurations

PEX 8748

11 x4 2 x44 x8

10 x4 6x42 x8

PEX 8748, PCI Express Gen 3 Switch, 48 Lanes, 12 Ports

CPU

Prim ary Host

Primary H o st

Root

Complex

Root

Complex

End

Point

End

Point End

Point

End

Point End

Point

End

Point

PEX 8748

CPU

Secondary H o st

Secondary Host

Non-Transparent

Port

Figure 3. Non-Transparent Port

CPU

Prim ary Host

Primary H o st

CPU

Prim ary Host

Primary H o st

Root

Complex

Root

Complex

Root

Complex

Root

Complex

End

Point

End

Point End

Point

End

Point End

Point

End

Point

End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point

PEX 8748

CPU

Secondary H o st

Secondary Host

CPU

Secondary H o st

Secondary Host

Non-Transparent

Port

Figure 3. Non-Transparent Port

PEX 8748

X 8748 PEX 8748

PEX 8748

Figure 4a. Multi-Host

Host 1

Host 1 Host 2

Host 2

End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point

Figure 4b. Multi-Host Fail-Over

Host 1

End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point

Host 2

The PEX 8748 can also be configured in Multi-Host mode

where users can choose up to six ports as host/upstream

ports and assign a desired number of downstream ports to

each host. In Multi-Host mode, a virtual switch is created

for each host port and its associated downstream ports

inside the device. The traffic between the ports of a virtual

switch is completely isolated from the traffic in other

virtual switches. Figure 2 illustrates some configurations

of the PEX 8748 in Multi-Host mode where each ellipse

represents a virtual switch inside the device.

The PEX 8748

also provides

several ways to

configure its

registers. The

device can be

configured

through strapping

pins, I2C

interface, host

software, or an

optional serial

EEPROM. This

allows for easy

debug during the

development

phase, p

and driver or software upgrade.

erformance m

ual-Host & Failover Support Non-

ess

used to send interrupts between the

both CPUs) allow inter-processor communication (see

Figure 3).

am host ports, each with its own dedicated

nfigured for 1+1

Hot Plug for High Availability

Hot plug capability allows users to replace hardware

modules and perform maintenance without powering down

ability feature

m ports

t of its

-plug

lly

ompliant with the PCIe power management specification

em.

I Express Base Specification r2.0, and is backwards

press Base Specification r1.1 and

onitoring during the operation phase,

In Single-Host mode, the PEX 8748 supports a

Transparent (NT) Port, which enables the

implementation of

dual-host systems f

redundancy and host

failover capability. Th

NT port allows systems

to isolate host memory

domains by presenting

the processor subsystem

as an endpoint rather than

another memory

system. Base addr

registers are used to

translate addresses;

doorbell registers are

address domains; and scratchpad registers (accessible by

Multi-Host & Failover Support

In Multi-Host mode, PEX 8748 can be configured with up

to six upstre

downstream ports. The device can be co

redundancy or N+1 redundancy. The PEX 8748 allows the

hosts to communicate their status to each other via speci

door-bell registers. In failover mode, if a host fails, the

host designated for failover will disable the upstream port

attached to the failing host and program the downstream

ports of that host to its own domain. Figure 4a shows a t

host system in Multi-Host mode with two virtual switches

inside the device and Figure 4b shows Host 1 disabled

after failure and Host 2 having taken over all of Host 1’s

end-points.

the system. The PEX 8748 hot plug cap

makes it suitable for High Availability (HA)

applications. Three downstream ports include a Standard

Hot Plug Controller. If the PEX 8748 is used in an

application where one or more of its downstrea

connect to PCI Express slots, each port’s Hot Plug

Controller can be used to manage the hot-plug even

associated slot. Every port on the PEX 8748 is equipped

with a hot-plug control/status register to support hot

capability through external logic via the I2C interface.

SerDes Power and Signal Management

The PEX 8748 provides low power capability that is fu

and supports software control of the SerDes outputs

allow optimization of power and signal strength in a

system. Furthermore, the SerDes block supports loop-back

modes and advanced reporting of error conditions,

which enables efficient management of the entire syst

Interoperability

The PEX 8748 is designed to be fully compliant with the

compatible to PCI Ex

Figure 2. Multi-Host Port Configurations

PEX 8748

4 x4

PEX 8748

3 x42 x4

3 x4

4 x4

PEX 8748

8 x4s

4 x4s

PEX 8748

9 x4s

3 x4s

PEX 8748, PCI Express Gen 3 Switch, 48 Lanes, 12 Ports

r1.0a. Additionally, it supports auto-negotiation, lane

reversal, and polarity reversal. Furthermore, the PEX

8748 is tested for Microsoft Vista compliance. All PLX

switches undergo thorough interoperability testing in

PLX’s Interoperability Lab and compliance testing at

the PCI-SIG plug-fest.

performancePAK™

Exclusive to PLX, performancePAK is a suite of unique

eatures which allows PLX’s

ighest performing Gen 2

being made by downstream

ownstream device requests several long

link

ess ports in

n allowing for higher performance in dual-

s opposed to a static buffer

static buffers to each port,

nPAK is a debug diagnostics

suite of integrated hardware and software instruments that

elp bring their systems to market faster.

ally “see” ingress and egress performance on

h the switch using PLX’s

users can view the receiver

and innovative performance f

Gen 2 switches to be the h

switches in the market today. The performancePAK

features consists of the Read Pacing, Multicast, and

Dynamic Buffer Pool.

Read Pacing

The Read Pacing feature allows users to throttle the

amount of read requests

devices. When a d

reads back-to-back, the Root Complex gets tied up in

serving that downstream port. If that port has a narrow

and is therefore slow in receiving these read packets from

the Root Complex, then other downstream ports may

become starved – thus, impacting performance. The Read

Pacing feature enhances performances by allowing for the

adequate servicing of all downstream devices.

Multicast

The Multicast feature enables the copying of data (packets)

from one ingress port to multiple (up to 11) egr

one transactio

graphics, storage, security, and redundant applications,

among others. Multicast relieves the CPU from having to

conduct multiple redundant transactions, resulting in

higher system performance.

Dynamic Buffer Pool

The PEX 8748 employs a dynamic buffer pool for Flo

Control (FC) management. A

scheme which assigns fixed,

PLX’s dynamic buffer allocation scheme utilizes a

common pool of FC Credits which are shared by other

ports. This shared buffer pool is fully programmable by the

user, so FC credits can be allocated among the ports

needed. Not only does this prevent wasted buffers and

inappropriate buffer assignments, any unallocated buffers

remain in the common buffer pool and can then be used

for faster FC credit updates.

visionPAK™

Another PLX exclusive, visio

users can use to h

visionPAK features consist of Performance Monitoring,

SerDes Eye Capture, Error Injection, SerDes Loopback,

and more.

Performance Monitoring

The PEX 8748’s real time performance monitoring allow

users to liter

each port as traffic passes throug

Software Development Kit (SDK). The monitoring is

completely passive and therefore has no affect on overall

system performance. Internal counters provide extensive

granularity down to traffic & packet type and even allo

for the filtering of traffic (i.e. count only Memory Writes).

SerDes Eye Capture

Users can evaluate their system’s signal integrity at the

physical layer using the PEX 8748’s SerDes Eye Capture

feature. Using PLX’s SDK,

eye of any lane on the switch. Users can then modify

SerDes settings and see the impact on the receiver eye.

Figure 5 shows a screenshot of the SerDes Eye Capture

feature in the SDK.

Figure 5. SerDes Eye Capture

PCIe Packet Generator

The PEX 87 et

enerator capable of creating programmable PCIe traffic

and capable of saturating a

48 features a full-fledged PCIe Pack

running at up to Gen 3 speeds

x16 link. Using PLX’s Software Development Kit

(www.plxtech.com/sdk), designers can create custom

traffic scripts for system bring-up and debug. Fully

integrated into the PEX 8748, the Packet Generator

to be a very convenient on-chip debug tool. Furthermo

the Packet Generator can be used to create PCIe traf

test and debug other devices on the system.

proves

re,

fic to

PEX 8748, PCI Express Gen 3 Switch, 48 Lanes, 12 Ports

Memory

CPU

Memory

CPU

x4s

x8s

x1s

SATASATA

Endpoint

PCIe Gen1, Gen2, or Gen3 slots

PEX 8748

PCI

PCH

I/O

PEX 8748

I/O

Mem

CPU

PCH

I/Os

MemMem

CPU

PCHPCH

I/OsI/Os

Mem

CPU

PCH

I/Os

MemMem

CPU

PCHPCH

I/OsI/Os

Mem

CPU

PCH

I/Os

MemMem

CPU

PCHPCH

I/OsI/Os

Mem

CPU

PCH

I/Os

MemMem

CPU

PCHPCH

I/OsI/Os

, users can

inject malformed packets and/or fatal errors into their

system and evaluate a system’s ability to detect and

recover from such errors. The PEX 8748 also supports

Internal Tx, External Tx, Recovered Clock, and Recovered

Data Loopback modes.

Applications

Suitable for host-centric as well as peer-to-peer traffic

patterns, the PEX 8748 can be configured for a wide

variety of form factors and applications.

Host Centric Fan-out

The PEX 8748, with its symmetric or asymmetric lane

configuration capability, allows user-specific tuning to a

variety of host-centric applications. Figure 6 shows a

server design where, in a quad or multi processor system,

users can assign endpoints/slots to CPU cores to distribute

the system load. The packets directed to different CPU

cores will go to different (user assigned) PEX 8748

upstream ports, allowing better queuing and load balancing

capability for

higher

performance.

Conversely, the

PEX 8748 can

also be used in

single-host mode

to simply fan-out

to endpoints.

Figure 6. Host Centric Dual Upstream

Multi-Host Systems

In multi-host mode, the PEX 8748 can be shared by up to

six hosts in a

system. By

creating six

virtual switches,

the PEX 8748

allows six hosts

to fan-out to

their respective

endpoints. This

reduces the

number of

Figure 7. Multi-Host System

switches required for fan-out, saving precious board space

and power consumption. In Figure 7, the PEX 8748 is

being shared by four different servers (hosts) with each

server is running its own applications (I/Os). The PEX

8748 assigns the endpoints to the appropriate host and

isolates them from the other hosts.

Host Failover

The PEX 8748 can also be utilized in applications where

host failover is required. In the below application (Figure

8), two hosts may be active simultaneously and controlling

their own domains while exchange status information

through doorbell registers or I2C interface. The devices can

be programmed to trigger fail-over if the heartbeat

information is not provided.

In the event of a failure, the

surviving device will reset

the endpoints connected to

the failing CPU and

enumerate them in its own

domain without impacting

the operation of endpoints

already in its domain.

Figure 8. Host Fail-Over

N+1 Fail-Over in Storage Systems

The PEX 8748’s Multi-Host feature can also be used to

develop storage array clusters where each host manages a

set of storage devices independent of others (Figure 9).

Users can designate one of the hosts as the failover-host

for all the other hosts while actively managing its own

endpoints. The failover-host will communicate with other

hosts for status/heartbeat information and execute a

failover event if/when it gets triggered.

Error Injection & SerDes Loopback

Using the PEX 8748’s Error Injection feature

x4 x4 x8 x8

x4 x8x8x4

CPU

PEX 8748

8 Disk Chassis

x4 x4

PEX 8712

8 Disk Chassis

x4 x4

PEX 8712

CPU

8 Disk Chassis

x4 x4

PEX 8712

8 Disk Chassis

x4 x4

PEX 8712

8 Disk Chassis

x4 x4

PEX 8716

8 Disk Chassis

x4 x4

PEX 8716

8 Disk Chassis

x4 x4

PEX 8716

Figure 9. N+1 Failover

CPU

PEX 8748

CPU

PEX 8748

EndpointEndpointEndpointEndpoint

x8s

x8s x8s

PEX 8748, PCI Express Gen 3 Switch, 48 Lanes, 12 Ports

n the PEX 8748 are compliant to

xpress system models. The

ce Registers (CSRs) in a virtual

bus

umber, and

Software Model

From a system model viewpoint, each PCI Express port i

a virtual PCI to PCI bridge device and has its own set o

PCI Express configuration registers. It is through the

upstream port that the BIOS or host can configure the

other ports using standard PCI enumeration. The virtual

PCI to PCI bridges withi

the PCI and PCI E

Configuration Spa

primary/secondary PCI to PCI bridge are accessible by

type 0 configuration cycles through the virtual primary

interface (matching bus number, device n

function number).

Interrupt Sources/Events

The PEX 8748 switch supports the INTx interrupt messag

type (compatible with PCI 2.3 Interrupt signals) or

Message Signaled Interrupts (MSI) when enabled.

Interrupts/messages are generated by PEX 8748 for hot

plug events, doorbell interrupts, baseline error reporting

and advanced error reporting.

Development

PLX offers hardw

customer design

module (PEX 87

(available at

Figure 10. PEX 8748 RDK

Tools

are and software tools to enable rapid

activity. These tools consist of a hardware

48 RDK), hardware documentation

w.plxtech.comww ), and a Software

t (also available at www.plxtech.comDevelopment Ki ).

xpressLane PEX 8748 RDK

748

needs to get their

ardware and software development started.

The PEX 8748 RDK (see Figure 10) is a hardware module

containing the PEX 8748 which plugs right into your

system. The PEX 8748 RDK can be used to test and

validate customer software, or used as an evaluation

vehicle for PEX 8748 features and benefits. The PEX 8

RDK provides everything that a user

Software Development Kit (SDK)

PLX’s Software Development Kit is available for

download at www.plxtech.com/sdk. The software

development kit includes drivers, source code, and GUI

interfaces to aid in configuring and debugging the PEX

8748.

Both performancePAK an

PLX’s RDK and SDK, the

hardware- and software-dev

Product Ordering Info

d v

isionPAK are supported by

dustry’s most advanced

lopment kits.

mation

Part Number Description

PEX8748-AA80BC G 48-La ne, 1 2-Port PCI Expres s Switch,

Pb-Free (27x27 mm2)

PEX8748-AA RDK PEX 8748 Rapid Development Kit

PLX Technology Inc. hnology, Inc. All other

ification purposes only

heir

duct

Read Pacing and Dual Cast are trademarks of PLX Tec

product names that appear in this material are for ident

and are acknowledged to be trademarks or registered trademarks of t

respective companies. Information supplied by PLX is believed to be accurate

and reliable, but PLX assumes no responsibility for any errors that may appear in

this material. PLX reserves the right, without notice, to make changes in pro

design or specification.

Visit www.plxtech.com for more information.

Mouser Electronics

Authorized Distributor

Click to View Pricing, Inventory, Delivery & Lifecycle Information:

Avago Technologies:

PEX 8748-CA RDK PEX 8748-CA80BC G PEX8748-CA RDK PEX8748-CA80BC G