Emerging serial chip-to-chip and backplane interconnects are the next level in faster, smaller and more robust device development. And transmission infrastructure standards are next on the agenda.
By Ron Warner and Doron Lapidot
The movement toward serial chip-to-chip and backplane interconnects continues at a frantic pace, particularly in the communications and storage arena. Standardization forums such as the OIF, RapidIO TA, and PCI-SIG have solidified their work and various packet-based protocols are in the process of being adopted by system and chip vendors. Just as the PHY and protocol layers of these new standards are being established, system vendors now have to decide how best to transmit these new protocols over existing transmission infrastructures, both inter- and intra-board.
The prevalent discussion during standards committee meetings has always centered on the best ways to make these serial standards economical to implement on practical boards and chassis and, wherever possible, the reuse of existing transmission infrastructure is a key consideration. For the communications industry, the use of FR-4 on PCBs for short-range, chip-to-chip, and backplanes, and the use of cables (coax, CAT5/5E/6) for long-range, intra-board, or chassis-to-chassis communication, are by far the most popular choices.
The importance of long-range cable transmission is highlighted by activities such as those undertaken by the PCI Express Cabling work group that is expected to release a specification in Q1 2005. At the same time, as system vendors implement the protocols for the first time on prototype vehicles and look for flexibility to implement proprietary logic on top of standard functionality, FPGA-based implementations of these protocols will continue for some time to come.
This article examines several emerging serial I/O standards, specifically focusing on the wireline and wireless network arenas, and discusses some implementations on proprietary FPGA-based SERDES and a cable transmission infrastructure.
Importance of High-Quality SERDES
|Figure 1. PCI Express protocol implementation. Click here  to enlarge.|
Designers often face moving large blocks of data from one location to another over moderate distances at high rates of speed. Historically, this was accomplished by a source synchronous parallel interface, which required large banks of parallel line drivers and receivers. In addition, it has become more and more difficult to ensure the data integrity of these types of interfaces from board to board at the gigabit-plus data rates required today.
With the inception and growing acceptance of SERDES devices, designers can alleviate the concerns inherent in the implementation of a parallel interface. SERDES technology permits smaller, less expensive cables and connectors, while providing a more robust solution in terms of signal integrity when moving large blocks of data at rates of 3.125 Gb/s and beyond.
However, there is more to providing a robust SERDES solution than just the raw data rate. Physical layer parameters such as media type/drive length at high data rates, signal jitter, and overall device power consumption must also be considered if one is to truly assess a SERDES capability.
In addition to a high quality SERDES, it is also essential to offer the associated and required PCS functionality to comply with existing packet-based industry standards such as PCI Express and Serial RapidIO, and emerging standards such as CPRI and OBSAI. These embedded cores, which are implemented in an ASIC technology, are then integrated onto the same die with the FPGA fabric to create a high performance, low-power, system-level solution.
|Figure 2. Serial RapidIO protocol stack implementation.
Click here  to enlarge.
The Value of Programmability
The advantages of stand-alone ASSPs are well documented and understood, but for applications involving emerging specifications, programmability is a key advantage for the designer. Programmability offers system designers the luxury of an early start on architecting and implementing their designs without having to wait for the final version of the specification. Following are examples of various standards and applications for which an integrated ASIC/FPGA technology is well suited.
PCI Express - Conventional PCI, once the standard I/O bus with its roots in the early '90s, is now showing its age. This has led designers to implement newer versions such as PCI-x and PCI-x 2.0, allowing them to maintain the existing software base while achieving greater throughput. But even with these enhancements, processor throughput still outpaces I/O throughput.
PCI Express was conceived to address these ever-increasing bandwidth needs by providing a scalable, point-to-point serial connection between chips, over cable or via connector slots for expansion cards, while maintaining compatibility with conventional PCI at the software layer.
|Figure 3. CPRI protocol stack implementation.
Click here  to enlarge.
A single PCI Express serial link is a dual-simplex connection, specified to speeds of up to 2.5 Gb/s per link that can be scaled in x1, x2, x4, x8, x12 x16 and x32 lane widths to achieve greater bandwidth. A serial implementation is cheaper, can be driven further distances, and alleviates common mode noise and skew concerns inherent in existing source synchronous parallel interfaces (such as conventional PCI), as well as reducing the overall number of connections required. As Figure 1 depicts, PCI Express is a packetized and layered protocol structure. Both the data link layer and transaction layers of the PCI Express stack are good examples of the advantage of programmability.
|Figure 4. OBSAI RP3 protocol stack implementation
Click here  to enlarge.
Depending upon the design, these layers can be customized to support either an end point implementation, a switch, or in many cases in which an FPGA is involved, a bridging function to a legacy communications protocol such as conventional PCI. This provided an economical and configurable PCI Express solution by implementing the fixed functionality of the physical and data link layers in ASIC gates, and leaving the customizable higher layer functionality to be implemented in FPGA gates.
|Figure 5. Tyco/Lattice backplane interconnect setup.
Click here  to enlarge.
Serial RapidIO - Another emerging serial standard is Serial RapidIO. Like PCI Express, Serial RapidIO has roots in the source synchronous world. When combined with the existing RapidIO parallel specification, Serial RapidIO allows designers to standardize on a single interconnect technology for networking, telecommunications, and other embedded applications.
Serial RapidIO is a scalable, point-to-point, low-pin-count interconnect designed to address increasing system bandwidth needs. It leverages industry-standard signaling technology found in Fiber Channel, 10 G Ethernet XAUI interfaces and Infiniband, and operates at 1.25, 2.5, and 3.125 Gb/s per link, providing the required bandwidth for signal processors and backplane applications. The serial specification defines both a single differential link in each direction between devices and support for ganging four links together for higher throughput applications (see Figure 2).
Serial RapidIO also benefits from the inherent flexibility that a programmable device provides. Here, the malleable logical and transport layer functionality can be implemented in FPGA gates, while the fixed functionality of the physical layer is dedicated to the ASIC portion of the device.
|Figure 6. Eye diagrams. Click here  to enlarge.|
The wireless domain, two initiatives are competing to facilitate more rapid development of cellular base stations. The CPRI and OBSAI standards focus on the standardization of serial transmission protocols, with the primary objective being the reduction of overall system cost through the standardization of its components.
|Figure 7. Illustration of intra-symbolic interference (ISI). Click here  to enlarge.|
CPRI - An industry initiative that intends to support a flexible base station architecture, CPRI partitions the base station into two basic building blocks, the REC that handles the baseband functionality, and the RE that provides the RF functionality. Building blocks are interconnected by a serial data link that is 8 b/10 b encoded and intended to use existing high-speed serial standards such as Ethernet and Fiber Channel. Physical layer line rates of 614 Mb/s, 1.228 Gb/s, or 2.456 Gb/s are supported with three different information flows (user plane data, C&M, and synchronization) multiplexed over a single serial interface.
OBSAI - Similarly, OBSAI partitions the base station into baseband and RF blocks, but also defines an additional transport and control block. In contrast to CPRI, the interfaces between each of these are unique reference points, defined as RP1, RP2, and RP3. These building blocks are specified as Ethernet interfaces, but we will focus on the RP3 interface because it is an 8 b/10 b encoded serial link, similar to the CPRI specification mentioned above.
Physical layer line rates of 768 Mb/s and 1.536 Gb/s are supported for the RP3 interface in support of high-speed data transfer and associated control. The protocol stack is again a packet concept utilizing a layered protocol, as shown in Figure 4.
|Figure 8. Illustration of pre-emphasized transmitted data passive equalization. Click here  to enlarge.|
System Interconnect Infrastructure
Achieving high transmission speeds for the emerging serial standards outlined above is a significant challenge for board designers, particularly when they face cost constraints. Different industries have taken disparate approaches to overcoming this price vs. performance trade-off, and in choosing appropriate connectors and transmission media. In the PC industry, for example, to keep overall costs down, interoperability is key. Consequently, the large OEMs joined together and established specifications for applications such as PCI-Express, SATA, SAS, Fiber Channel, FireWire, DVI, HDMI, and the like, not only at the protocol level but also at the physical interconnect level. Within the communications infrastructure industry, however, with applications such as multi-services switches, routers and wireless BTS, standardization has only occurred at the protocol levels and at the UNI. More often than not, physical interconnection is not standards-based and is usually customized.
|Figure 9. A typical response of raw cable, passive equalizer, and equalized cable. Click here  to enlarge.|
The challenge is to get the most cost-effective backbone without compromising system performance. The following issues are usually encountered:
How to use ubiquitous and economical PCB/transmission raw material, whether inter-board or intra-board
How to design the most economical architecture through the optimization of the system form factor.
Choosing a connector with the most suitable footprint that fits into the above interconnect objectives without affecting the system performance goals.
|Figure 10. Cable interconnect. Click here  to enlarge.|
These issues shed light on the pros and cons of two major wired backbones in the market. Following is a discussion on the most common and cost-effective techniques to improve performance of a total system interconnect and extend its length.
Connectors, Media Types, and Performance
|Figure 10a. Cable connector. Click here  to enlarge.|
Connectors - Board-to-board connectors are sub-divided into two groups: open field connectors and control impedance connectors. With open field connectors, performance and use per unit length are directly affected by the pin assignment and signal-to-ground ratio. Examples are the EuroCard-DIN, Z-Pack 2 mm Hard Metric, and FB+.
Because the controlled impedance connector's noise immunity and throughput are much better than those of open field connectors, designers find it prudent to run extremely fast signaling through these connectors while maintaining sufficient margins and acceptable channel loss. As a matter of fact, the system performance bottleneck is no longer the connector itself, but its footprint inside the PCB and the PCB raw material.
|Figure 11. The result of non-optimized cable interconnect. Click here  to enlarge.|
PCB is the most commonly used interconnect component for inner-system backbones. The industry has reached the point at which cost-effective high-Tg FR4 based PCB performs well up to the 5 Gb/s range when used in conjunction with high-speed, controlled-impedance board-to-board connectors. Using this infrastructure, system interconnect achieves transmission of serialized data at 3.125 Gb/s across 1270 mm (~500) and 5 Gb/s across 762 mm (~300) PCB-based interconnect. There is a dependency on silicon driver characteristics as well as silicon receiver sensitivity (see figure 5). The "differential" output voltage for the device used in Figure 6 is 883 mV at 3.125 Gb/s, with output jitter of 36 ps (peak-to-peak). Figure 6 depicts the measured "eye-pattern" at the receiver's inputs while the driver is set without (left "eye") and with (right "eye") pre-emphasis, respectively.
|Figure 12. The result of individually optimized cable interconnect.
Click here  to enlarge.
The DSO together with the bias-T acts as the receiver inputs to avoid affecting system impedance continuity as would connecting the DSO in parallel to the receiver inputs. The two "eye-patterns" measured across 876 mm (34.50) system interconnect with two high-speed and controlled-impedance connector and FPGAs with SERDES interfaces running at 3.125 Gb/s.
The left "eye-pattern" represents the case of "flat" driver output and clearly indicates that even though the signal has relatively large jitter, it still operates with sufficient margin relative to the eye mask to recover the data. While the driver's output is set to 25% pre-emphasis of the signal, the STQ is significantly improved, as depicted in the right "eye-pattern" yielding channel length extension up to 1270 mm (~500) before the receiver's minimum sensitivity is met. Channel length will be significantly affected by which inner layer of a multi-layer board is the connecting one.
The conclusion is that to get the maximum and most cost-effective results of a system interconnect, every factor must be examined as part of total assembly, rather than individually.
High Speed Cables
|Figure 13. The result combined solution for optimized cable interconnect.
Click here  to enlarge.
An additional challenge for many serial high-speed applications involves addressing the need for channel lengths longer than those addressed by PCBs. Transmission across copper media is still cost-effective, as long as the length is within 20 m. Shelf-to-shelf within the cabinet and intra-cabinet applications are examples in which copper cabling would be used.
It already has been shown that an optimized channel with robust silicon and connector can reach ~1200 mm at 3.125 Gb/s, a useful solution within the system's cabinet. Shelf-to-shelf within the cabinet and intra-cabinet high-speed solutions, within a couple of meters, need high-speed cables. The following discussion is based on replacing the backplane with cable and examining overall performance after optimizing the channel. At gigabit speeds, serial interconnect involves data rates that are usually 10 to 20 times faster than parallel interconnect, which challenges every designer in terms of the STQ as well as EMC problems. Clearly, short-reach intra-system interconnects still must be cost-effective, yet must carry high data rates across copper cables with what most likely is differential signaling.
A careful examination of the cable assembly indicates several critical parameters the designer should keep in mind
Cable connectors should terminate the cable while maintaining proper STQ, as well as complying with EMC requirements. The connectors should be robust with all necessary mechanical features such as the accommodation of different wire size. They should have latching, locking, and keying features.
Raw cable is frequently the greatest source of loss and radiated noise
Cable assemblies are long in physical and electrical length compared with system dimensions.
Cable characteristics such as skew, insertion loss, return loss, and DC resistance a key role in the total performance:
Skin effect - known as the tendency of the current to concentrate on the conductor's surface skin. The skin depth becomes thinner as frequency increases and is directly proportional to the square root of the frequency.
DC and low frequency losses - These occur when the finite resistive loss is present in frequencies at which the conductor diameter is less than the skin depth. It is directly proportional to the cable length.
Dispersion - known as the losses in signal amplitude caused by the change in velocity of propagation of signals with an increase in frequency (dispersion) and directly proportional to the frequency.
Dielectric losses - known as the resistive losses in the dielectric material that cause signal dispersion, primarily due to the "saturated" dipoles phenomena in high frequency transmission that causes the dipoles to lose tracking after the alternating fields in the insulation raw material.
Differential skew - known as asymmetrical performance of two wires forming a pair. The differential skew creates two major problems: " eye-pattern" deterioration as a result of rise time degradation, as well as common mode currents that may have adverse effects on EMC.
Crosstalk, which is usually measured as a percentage of the source voltage and described as the energy coupled from active signals into passive adjacent signals, causing false data to trigger the passive receiver.
ISI, which is directly related to the data pattern and causes a time shift of zero crossover point (known also as DDJ). Figure 7 depicts an illustration of an ISI result. As in the case of the board-to-board connector, when considering the cable, it is essential to take the whole system architecture, as well as the silicon device driver and receiver, into consideration. This is likely to be the case in non-standard applications solutions (which are common).
Cable connectors are as important an element as the raw cable itself. The entire cable assembly should be compact and easy to assemble with improved shielding effectiveness, yet should be inexpensive as well. The following information focuses on one type of cable connector, a 1 mm I/O, that satisfies the criteria, including the option of integrating passive equalization and providing as many as ten high-speed differential pairs on a single connector.
Choosing the right raw cable is a key factor in the overall performance of the cable assembly. Several cable structures exist for high speed differential signaling:
UTP - unshielded twisted pair
STP - shielded twisted pair
SPP - shielded parallel pair (also known as flat Twin-ax)
Inherently, differential signaling is associated with two types of impedance values, 100X differential and 150X differential. The 100X differential impedance is the most commonly used in the industry because its overall dimensions are smaller than 150X differential cable, even though 100X differential introduces higher frequency dependent losses.
The cable structure is chosen depending upon the transmission distance. STP inherently has higher skew within the pair relative to SPP, and so for longer distances may accumulate enough skew to affect insertion loss. On the other hand, SPP cables are more rugged and are made of solid wires, and therefore are more difficult to terminate (wire management in the connector); thus, they offer less flexibility than STP.
A full-duplex channel application requires a 1X, 4X, or 12X cable structure; by definition, 1X cable type has one differential pair for each direction, known also as a serial/quad cable. Typically, 1X cable could be found in Fiber Channel (150X) and InfiniBand (100X).
A 4X cable has four serial differential pairs in each direction (total of eight SPP or eight STP bundled and overall shielded in a round jacket), known also as 4-quad cable. Such cable is suitable for XAUI and InfiniBand applications, yet it still is considered to be a serial high-speed channel.
A 12X cable has 24 STP or SPP bundled in the same concept, known also as 12-quad cable. Such cable is suitable for either multiple serial (as in PCI Express) or parallel high-speed interconnects.
On the transmit side, pre-emphasis is used as a technique to overcome the poor jitter and eye opening performance at the receiver end of a long and lossy channel.
Pre-emphasizing/de-emphasizing the signal's amplitude at the driver's output means:
Digitally filter the output based on historical input with FIR or - based on historical input and output - with IIR filters.
Amplify higher frequencies (consecutive bit alteration) to counter cable loss.
Drive signals in full strength after transition and weaken the signals (~30% less) at lower frequencies consecutive bits stay the same. Figure 8 shows how pre-emphasis at the driver side will address ISI at the receiver side, as was shown in Figure 7. Equalizers can be incorporated into the silicon or in the cable assemblies to compensate for the cable-produced attenuation and dispersion. If the output signal is not large enough, or if the DDJ is too large for effective signal recovery, a passive equalizer is used to even out the fundamental frequency response from the byte-rate-equivalent frequency to the bit-rate-equivalent frequency (156.25 MHz to 1.5625 GHz for 3.125 Gb/s XAUI or 10 GE systems).
The equalizer does this in the frequency domain by creating a transfer function that acts effectively like the cable's inverse transfer function. Together, the newly created transfer function behaves relatively flat across the frequency span of which the cable assembly should transfer or at least up to fundamental frequency.
The next matters to examine are practical solutions of copper-based cable assembly as part of a total system interconnect to demonstrate the flexibility of the solution for various networking applications based on high speed data rates such as PCI Express, Serial RapidIO, CPRI/OBSAI, or customized backplane applications. Figure 10 shows the same (PCB-based) backplane interconnect mentioned previously, but here the cable replaces the FR-4 backplane to extend the channel length to illustrate an intra-cabinet solution.
The cable connector offers the following characteristics
Hybrid cable connector (34 signal and four power contacts).
Small and slim connector form factor with push lock metal latch
Card edge contact concept
Flexible cable termination
Various wire gauge size
Optional passive equalizer in the plug
Good electromagnetic compatibility
Low ground impedance
The cable in use consists of 8 x SPP type, 100X differential, 28 AWG.
The longest, most optimal, and cost-effective solution is obtained by using such a connector and combining pre-emphasis and passive equalization.
When none of these techniques are implemented (i.e., neither pre-emphasis at the driver side nor an equalized cable assembly), the "eye pattern" (3.125 Gb/s) at the receiver side after eight meters of cable assembly would look like the waveform depicted in Figure 11. If either pre-emphasis or cable equalization is applied, there will be improvement. But an insufficient "eye opening" will result as shown in the measured results in Figure 12. Because a maximum cable length is desired, combining the pre-emphasis of the SERDES driver's output with an optimized cable assembly with an equalizer embedded in the connector is the best solution. Figure 13 shows the "eye opening" for that combination, which maximizes the cable length and yields the most cost-effective solution. Because the 1 mm I/O cable connector can accommodate eight to 10 high-speed differential pairs, and SERDES has four high speed outputs and inputs, the system interconnect assembly used in this example forms a full PCI Express/Serial RapidIO x4 channel or any other implementation, such as a XAUI port, for cost-effective solutions at speeds as fast as 3.125 Gb/s. And by reducing the data rate to match with CPRI/OBSAI maximum data rates (2.5 Gb/s) enables the designer to extend the cable length to ten meters within safe operating margins. Keep in mind that optimizing cable individually differs from optimizing the combined solution of pre-emphasis and equalizer; therefore, the whole system's interconnect components must be considered essentially as one package.
As the communications and storage systems industries converge on packet-based serial I/O for high-bandwidth and low-pin-count connectivity, system vendors will want silicon and infrastructure vendors to provide them with a one-stop shopping solution to add value to the overall system.
To achieve this, suppliers to system vendors must offer solutions that deliver robust signal integrity and flexible silicon solutions. The push for industry standard infrastructure (connectors, transmission material, and chassis dimensions) will also become more aggressive, as vendors strive for time to market and, more importantly, interoperability. These standards bodies have already begun work in earnest, as evidenced by PCI-SIG and PICMG forums.
1. Transmission of Emerging Serial Standards Over Cable.
About the Author
Ron Warner is a Strategic Marketing Manager at Lattice Semiconductor. Previously, he was an Applications Engineering Manager at Agere/Lucent Technologies and a design engineer at Harris Corporation. Warner received his BSEE from Youngstown State University in Ohio.
Doron Lapidot is a Director of Engineering at Tyco Electronics, responsible for the Circuit & Design (C&D) core competency in Asia Pacific and Europe. Doron is a well-recognized expert in high-speed systems design, modelling and simulations. Doron holds a 'BSc' degree in Electrical Engineering from the 'Technion' Institute of Technology, Haifa, Israel.
Glossary of Acronyms
ASIC - Application-Specific Integrated Circuit
ASSP - Application-Specific Standard Product
BTS - Base Transceiver Station
C&M - Control and Management
CPRI - Common Public Radio
DDJ - Data-Dependent Jitter
DIN - Deutsches Institut fur Normung
DSO - Digital Storage Oscilloscope
DSP - Digital Signal Processing
DVI - Digital Visual Interface
EMC - Electromagnetic Compatibility
FIR - Finite Impulse Response
FPGA - Field-Programmable Gate Array
FPSC - Field-Programmable System Chip
HDMI - High Definition Multimedia Interface
IIR - Infinite Impulse Response
IP - Internet Protocol
ISI - Inter-Symbol Interference
OBSAI - Open Base Station Architecture Initiative
OEM- Original Equipment Manufacturer
OIF - Optical Internetworking Forum
PC - Personal Computer
PCB - Printed Circuit Board
PCI - Peripheral Computer Interface
PCS - Physical Coded Sub-Layer
PICMG - PCI Industrial Computer Manufacturers Group
RE - Radio Equipment
REC - Radio Equipment Control
RP1 - Control Plane
RP2 - User Plane Between Transport To Baseband Blocks
RP3 - User Plane Between Baseband and RF Blocks
SAS - Serial Attached SCSI
SATA - Serial Advanced Technology Attachment
SCSI - Small Computer System Interface
SERDES - SerDes, a serializer/deserializer
Serial RapidIO - Serial interface for serial backplane, DSP and associated serial control plane applications
SIG - Special Interest Group
STQ - Signal Transmission Quality
UNI - User-To-Network Interface
XAUI - An extender for the XGMII 10 Gigabit Media Independent Interface