Predicting the Effect of Collisions

Collisions

With half-duplex (full-duplex will be described below) Ethernet, collisions occur when two or more devices attempt to send a frame at the same time. When frames collide, the frame must be retried, so each device delays (backs-off) a random amount of time and then retries. For each successive collision on a frame, the maximum back-off time doubles. If a frame collides sixteen successive times, the frame will be discarded.

Note: RMCWin provides a histogram of number of frames that have collided a given number of times. See RMC Ethernet Statistics for details.

Collision Domains

Ethernet networks are divided into collision domains. A collision domain is the set of all devices that can collide with one another. The more devices in a collision domain: the higher the probability of a collision. Each collision domain extends through any hubs or repeaters and stop at switches, routers, or end-devices (PCs, RMCs, PLCs). This is best illustrated with some examples:

• A network has four RMCs and one 1756-ENET connected to a hub. This network has a single collision domain with five devices competing for its bandwidth.

• A network has four RMCs and one 1756-ENET connected to a switch. This network has five collision domains, each with two devices competing for its bandwidth. These smaller collision domains will yield a more deterministic and higher performance network. This is why we recommend using a switch instead of a hub.

• A network has four RMCs and one 1756-ENET connection to a switch. The switch is then connected through a router to the rest of the plant network. The control network will have six collision domains: the five in the previous example plus one between the switch and router. All traffic on the plant network will be in separate collision domains.

Full-Duplex Ethernet

Full-duplex Ethernet occurs between exactly two devices (usually a switch and an end-device such as the 1756-ENBT) when both devices support full-duplex (the RMC100 does not support full-duplex Ethernet). Each device in a full-duplex segment can send and receive data at the same time, thereby doubling the bandwidth, and more importantly, eliminating collisions. The 1756-ENBT supports full-duplex Ethernet. Therefore, use of a full-duplex-capable switch is highly recommended to eliminate collisions between the 1756-ENBTs and the switch, which are the most heavily trafficked segments. The 1756-ENET and RMC ENET do not support full-duplex, and therefore will be susceptible to collisions.

Network Utilization

The probability of a collision occurring depends primarily on the utilization of the network in a collision domain. Utilization is the relationship between the available bandwidth (10 or 100 Mbps) and the actual amount of data vying for that bandwidth. The amount of data is a function of the number and size of frames being sent. We will represent utilization as the percent of time the wire is active.

Utilization vs. Collisions

The probability of collisions in a given collision domain depends on the network utilization. As is described below, the maximum delay per frame depends on how many collisions a frame experiences, therefore a good approach is to choose how long of a delay is acceptable and then determine how many collisions per frame are acceptable.

The maximum acceptable collisions per frame, and therefore the maximum acceptable delay per frame will vary from application to application. Typically the absolute limit will be the timeout time for an I/O connection. The ControlLogix sets up its I/O connections to time out if no packets are received within 32 RPIs. For example, for an RPI of 5.0 ms, the connection will reset if a packet is not received in 32 x 5.0 ms or 160 ms. Therefore, the number of collisions per frame should be kept below 11 (using the chart below). When a connection times out, the connection is closed and may take several seconds to be re-established. During this time the PLC and RMC will execute any broken connection handling they have implemented.

Notice that with half-duplex Ethernet, there can be no guarantee that a frame will reach its destination by any given time. However, the probability of such an event may become so low that they are effectively masked by other more-probable events such as a cable being cut or a plant fire. Again, it should be noted that with full-duplex Ethernet, collisions do not exist, and as such, the Ethernet media becomes deterministic.

The following table shows the percentage of the frames that collided n or more times at various network utilizations in lab tests at Delta. These tests were done with a single ControlLogix 1756-ENBT/A module requesting data from multiple RMCs with RPIs of 5.0 ms using a switch. The same utilization with a hub would result in higher collisions since more than two devices can compete for the same bandwidth at once:

The above statistics were captured on a network using a switch. Networks using a hub will experience higher rates of collisions for the same utilization because the probability of a collision increases with more devices.

Delays Incurred by Collisions

The time delayed for each frame depends on the baud rate (10 or 100 Mbps) and frame size. The following chart shows the total minimum and maximum delays incurred by a frame having from zero to sixteen collisions assuming the maximum frame size utilized by a RMC I/O frame. Notice that only the 10 Mbps section of the chart applies to the switch-to-RMC segment of the LAN since the RMC ENET is 10 Mbps.

* Even with no collisions, a frame may have to wait for the wire to be clear before being sent. This is called deferral.

Example:

Suppose the Ethernet statistics for the RMC ENET show that the highest number of collisions for any frame is six (6). How long of a delay is this? Given that the RMC ENET runs at 10 Mbps, the above chart shows that a frame will six collisions would have been successfully delivered between 0.077 ms and 7.86 ms from when it was intended to be sent.

Computing Utilization for RMC/ControlLogix Ethernet Networks

In order to predict the probability of collisions on collision domains of an Ethernet network used by EtherNet/IP between a ControlLogix 1756-ENET or ENBT and RMCs, we must compute the utilization on the various collision domains. The first step in computing utilization is to compute the bandwidth requirement for the collision domain. That is, how many frames are vying for the bandwidth each second?

There are two types of frames used in an I/O connection with an RMC: frames consumed by the RMC, and frames produced by the RMC. Each RMC will produce exactly one frame each RPI, but it will consume one frame per connection each RPI. The bandwidth requirement on each collision domain reached by a consumed or produced frame is 1/RPI.

When a hub is used, there is only one collision domain, so all produced and consumed frames affect the required bandwidth for that domain.

When a switch is used, frames are routed intelligently only to the ports that need them. Frames consumed by the RMC are sent directly from the 1756-ENET/ENBT to the RMC, and thereby only affect only the collision domains of the two devices involved (1756-ENET/ENBT to switch, and switch to RMC). Frames produced by the RMC are multicast. For most inexpensive switches treat multicast packets the same as broadcast packets by forwarding the frames to all ports on the switch. Switches that support IGMP (Internet Group Management Protocol) or IGMP snooping do better by keeping track of which ports have devices that care about the multicast frame and only forwarding to those ports. However, notice that most of these switches require a router to be present in the network for the IGMP snooping to work. Therefore, if your system will not always have a router present, then you should purchase a switch that supports IGMP snooping without a router present. Compare the first two examples to see the difference this can make on utilization.

Note: Delta always recommends using a switch over a hub. The price difference is negligible and the determinism boost is significant. When using a protocol that makes heavy use of multicasting (as EtherNet/IP does), Delta also recommends using a switch that supports IGMP snooping, preferably without requiring an additional router. EtherNet/IP is the only protocol that the RMC ENET currently supports that uses multicast.

The bandwidth required in frames/second for each collision domain is computed by adding up frames/RPI for each different RPI on that collision domain.

Example:

Using the above information, we will give examples of computing the bandwidth required for the most common collision domains in an Ethernet network. For each of these examples assume there are two ControlLogix PLCs (CLX1 and CLX2), and four RMCs (RMC1 through RMC4). Assume that RMC1 and RMC2 are controlled by CLX1 and monitored by CLX2, RMC3 is controlled by CLX1 only, and RMC4 is controlled by CLX2 only. The RPI is 10 ms for RMC1 and RMC2, 5 ms for RMC3, and 7 ms for RMC4. This is summarized in the following chart:

For the first two collisions domains types, we will assume all devices are connected to a switch. For the third, we will assume all devices are connected to a hub.

• The collision domain from the switch to an RMC ENET.

When the switch does not support IGMP, this collision domain will receive all frames consumed by the RMC, plus all frames produced by any RMC on the network.

When the switch does support IGMP, then this collision domain will receive all frames consumed and produced by the RMC. Multicast packets sent by other RMCs are filtered out by the IGMP-capable switch.

Example (IGMP not supported by switch):

In the example above, there are four RMC/switch collision domains: one each for RMC1 to RMC4. All four of these segments will include the four frames produced by the four RMCs. The bandwidth required for these is as follows:

Produced Frames/Second = 2 / 0.010s + 1 / 0.005s + 1 / 0.007s = 543

Next, we need to add the frames consumed by each RMC. This can be different for each RMC/switch collision domain.

RMC1 Frames/Second = 543 + 2 / 0.010s0 = 743

RMC2 Frames/Second = 543 + 2 / 0.010s = 743

RMC3 Frames/Second = 543 + 1 / 0.005s = 743

RMC4 Frames/Second = 543 + 1 / 0.007s = 686

Example (IGMP supported by switch):

In the example above, there are four RMC/switch collision domains: one each for RMC1 to RMC4. The bandwidth of each is computed based on the number of frames consumed by the RMC (one or two in this example) and the number of frames produced by the RMC (always one) each RPI:

RMC1 Frames/Second = (2 + 1) / 0.010s = 300

RMC2 Frames/Second = (2 + 1) / 0.010s = 300

RMC3 Frames/Second = (1 + 1) / 0.005s = 400

RMC4 Frames/Second = (1 + 1) / 0.007s = 286

• The collision domain from the switch to the 1756-ENET/ENBT. It should be noted first that if a 1756-ENBT is used and the switch supports full-duplex Ethernet, then there will be no collisions, and therefore this collision domain need not be considered.

When the switch does not support IGMP, this collision domain sees all frames produced by the ControlLogix for an RMC, plus all frames produced by any RMC on the network.

When the switch does support IGMP, this collision domain sees all frames produced by the ControlLogix for any RMC, plus all frames produced by any RMC destined for this ControlLogix.

Example (IGMP not supported by switch):

The ControlLogix/switch collision domains include all frames produced by any RMC on the network. This was computed for the previous collision domain: 543 frames/second.

In addition to these frames, the ControlLogix/switch collision domains also include frames produced by the ControlLogix and consumed by the RMCs. CLX1 produces frames for RMC1, RMC2, and RMC3. CLX2 produces frames for RMC1, RMC2, and RMC4. Therefore, the bandwidth for the CLX1/switch and CLX2/switch collision domains are as follows:

CLX1 Frames/Second = 543 + 2 / 0.010s + 1 / 0.005s = 943

CLX2 Frames/Second = 543 + 2 / 0.010s + 1 / 0.007s = 886

Notice that if a 1756-ENBT is used and full-duplex is used, then no collisions will occur. If half-duplex is used, but 100 Mbps is used, then the utilization will be 1/10th that of 10 Mbps.

Example (IGMP supported by switch):

There are two ControlLogix/switch collision domains. Each sees frames produced for RMCs by its ControlLogix and frames consumed by that ControlLogix. Therefore, this will be two frames per RMC at each RPI:

CLX1 Frames/Second = 4 / 0.010s + 2 / 0.005s = 800

CLX2 Frames/Second = 4 / 0.010s + 2 / 0.007s = 686

• The collision domain for a network using a hub instead of a switch.

This collision domain will receive all frames consumed and produced by all RMCs.

Example:

This hub collision domain includes all frames. Therefore, we add the bandwidth required by RMC-produced frames (543 frames/second) to the bandwidth required by all frames consumed by RMCs:

Hub Frames/Second = 543 + 4 / 0.010s + 1 / 0.005s + 1 / 0.007s = 1,286

As stated above, utilization is a function of the available bandwidth (10 or 100 Mbps) and the amount of data vying for the bandwidth. The amount of data includes the number and size of frames being sent.

The available bandwidth for a 10 Mbps Ethernet network with 1,792-bit frames and the mandatory 96-bit inter-frame gap is 5,296 frames per second. The 1,792 number is the maximum frame size used in an RMC I/O connection. The available bandwidth on a 100 Mbps Ethernet network is ten times that of 10 Mbps, or 52,966 frames/second.

Finally, use the above frames/second results to compute the utilization by dividing the actual bandwidth requirement by the maximum bandwidth and multiplying by 100%:

Utilization = 100% x actual bandwidth / maximum bandwidth

As noted above, the maximum bandwidths for 10 and 100 Mbps networks are 5,296 and 52,966 frames/second respectively.

Example:

Using the same setup as the previous example and assuming a switch is used, the utilization of the six collisions domains are as follows:

RMC1-3/switch Utilization = 100% x 743 / 5,296 = 14.0%

RMC4/switch Utilization = 100% x 686 / 5,296 = 13.0%

The ControlLogix segments can run at either 10 or 100 Mbps (1756-ENBT only). If 10 Mbps is used, then the utilization would be as follows:

CLX1/switch Utilization = 100% x 943 / 5,296 = 17.8%

CLX2/switch Utilization = 100% x 886 / 5,296 = 16.7%

If 100 Mbps is used, then the utilization would be as follows:

CLX1/switch Utilization = 100% x 943 / 52,966 = 1.8%

CLX2/switch Utilization = 100% x 886 / 52,966 = 1.7%

Again, if full-duplex is used with the 1756-ENBT, there will be no collisions in on the ControlLogix collision domains.

Controlling Collisions

If you are experiencing or anticipate unacceptable levels of collisions, then you have several options for reducing the number of collisions:

• Replace hubs with switches.

• Replace hubs and non-IGMP capable switches with IGMP capable switches.

• Replace half-duplex components with full-duplex components.

• Replace 10 Mbps with 100 Mbps components.

• Increase the RPI.

• Split the network into two or more smaller networks.

For further details, see Setting Up Large EtherNet/IP Networks.

How not to Control Collisions

Do NOT set the Ethernet switch port to the RMC100 to full-duplex. This is why:

1. A collision is defined on a half-duplex 10/100baseT Ethernet segment as happening when the Tx wire pair and Rx wire pair are active at the same time. Notice that there is no electrical contention on 10/100baseT during a collision like there was on a truly shared media like Coax (10base2) Ethernet.

2. When a device says that it is Half Duplex, it is saying that its internal Ethernet controller cannot handle receiving data on the Rx pair at the same time it transmits on the Tx pair.

3. Ethernet has a protocol for half-duplex communication to incur minimal delay on this conceptually shared media.

4. There are 3 pieces to this scheme: (1) before sending data on a half-duplex segment, the device (switch or RMC) must wait until its partner is not sending, (2) if the Rx pair is idle, then the send begins, but the sender monitors for a collision during the first 512 bits, (3) collision during this phase is perfectly NORMAL and triggers a quick retry (see our help topic on just how short this is).

5. So when both sides (switch and RMC in this case) play by the rules, then only one side sends at a time, and when they happen to start talking at the same time, then a timely retry is done automatically.

So, what happens when the switch decides to NOT play by the rules? (i.e. the switch thinks the link is full duplex, but the RMC is still limited to half duplex)

1. The switch will receive any data sent by the RMC without watching to see if the Rx lines are busy when it sends on the Tx lines.

2. Because the switch sends data to the RMC without looking for a collision, the packet may be discarded by the RMC's Ethernet chips, while the switch is blissfully unaware that the packet was thrown away, so it WILL NEVER BE RETRIED! Further, it may or may not interrupt the packet going from the RMC to the switch, causing missing packets in the other direction.

In review, all that is accomplished by setting the switch to Full Duplex is it turns off REPORTING of collisions on the switch side, and makes matters worse by trading normal (quickly retried) collisions for worse errors that do not resend the packet. Because EtherNet/IP is a robust protocol, it can handle a few missed I/O packets without a hiccup. However, if one of the packets that was thrown away was from an MSG block, then it could be multiple seconds before it gets retried by the protocol! A properly configured Ethernet system should never see Jabber, Underrun, Late Collisions, Bad CRC, Extra Data, or Runt errors.

Copyright (c) 1997-2015 by Delta Computer Systems, Inc.