19 \usepackage{svn}

    19 \usepackage[nofancy]{rcsinfo}

    65 \SVN $Date$

    65 \rcsInfo $RCSId$

    66 \SVN $Revision$

    87       Preliminary Documentation}

    90       Documentation}

    96     {\Large Essen, \SVNDate\\[1ex]

    99     {\Large Essen, \rcsInfoLongDate\\[1ex]

    97       Revision \SVNRevision}

   100       Revision \rcsInfoRevision}

   175 \item Comes with EtherCAT-capable drivers for several common Ethernet devices.

   180 \item Comes with EtherCAT-capable native drivers for several common Ethernet

   181 chips, as well as a generic driver for all chips supported by the Linux

   182 kernel.

   179   \item The Ethernet hardware is operated without interrupts.

   186   \item The native drivers operate the hardware without interrupts.

   180 

   187 

   181   \item Drivers for additional Ethernet hardware can easily be implemented

   188   \item Native drivers for additional Ethernet hardware can easily be

   182   using the common device interface (see sec.~\ref{sec:ecdev}) provided by the

   189   implemented using the common device interface (see sec.~\ref{sec:ecdev})

   183   master module.

   190   provided by the master module.

   191 

   192   \item For any other hardware, the generic driver can be used. It uses the

   193   lower layers of the Linux network stack.

   195   \item RTAI\nomenclature{RTAI}{Realtime Application Interface},

   205   \item RTAI\nomenclature{RTAI}{Realtime Application Interface} \cite{rtai},

   197   Systems}, etc.

   207   Systems}, RT-Preempt \cite{rt-preempt}, etc.

   365   \includegraphics[width=.9\textwidth]{images/architecture}

   375   \includegraphics[width=\textwidth]{images/architecture}

   936 hardware. There are device driver modules that handle Ethernet hardware, which

   946 hardware.

   937 a master can use to connect to an EtherCAT bus.

   947 

   948 \paragraph{Native Ethernet Device Drivers} There are native device driver

   949 modules (see sec.~\ref{sec:native-drivers}) that handle Ethernet hardware,

   950 which a master can use to connect to an EtherCAT bus. They offer their

   951 Ethernet hardware to the master module via the device interface (see

   952 sec.~\ref{sec:ecdev}) and must be capable to prepare Ethernet devices either

   953 for EtherCAT (realtime) operation or for ``normal'' operation using the

   954 kernel's network stack. The advantage of this approach is that the master can

   955 operate nearly directly on the hardware, which allows a high performance. The

   956 disadvantage is, that there has to be an EtherCAT-capable version of the

   957 original Ethernet driver.

   958 

   959 \paragraph{Generic Ethernet Device Driver} From master version 1.5, there is a

   960 generic Ethernet device driver module (see sec.~\ref{sec:generic-driver}),

   961 that uses the lower layers of the network stack to connect to the hardware.

   962 The advantage is, that arbitrary Ethernet hardware can be used for EtherCAT

   963 operation, independently of the actual hardware driver (so all Linux Ethernet

   964 drivers are supported without modifications). The disadvantage is, that this

   965 approach does not support realtime extensions like RTAI, because the Linux

   966 network stack is addressed. Moreover the performance is a little worse than

   967 the native approach, because the Ethernet frame data have to traverse the

   968 network stack.

   950 \paragraph{Tasks of a Network Driver}

   981 \paragraph{Tasks of a Network Driver} Network device drivers usually handle

   951 

   982 the lower two layers of the OSI model, that is the physical layer and the

   952 Network device drivers usually handle the lower two layers of the OSI model,

   983 data-link layer. A network device itself natively handles the physical layer

   953 that is the physical layer and the data-link layer. A network device itself

   984 issues: It represents the hardware to connect to the medium and to send and

   954 natively handles the physical layer issues: It represents the hardware to

   985 receive data in the way, the physical layer protocol describes. The network

   955 connect to the medium and to send and receive data in the way, the physical

   986 device driver is responsible for getting data from the kernel's networking

   956 layer protocol describes. The network device driver is responsible for getting

   987 stack and forwarding it to the hardware, that does the physical transmission.

   957 data from the kernel's networking stack and forwarding it to the hardware,

   988 If data is received by the hardware respectively, the driver is notified

   958 that does the physical transmission.  If data is received by the hardware

   989 (usually by means of an interrupt) and has to read the data from the hardware

   959 respectively, the driver is notified (usually by means of an interrupt) and

   990 memory and forward it to the network stack. There are a few more tasks, a

   960 has to read the data from the hardware memory and forward it to the network

   991 network device driver has to handle, including queue control, statistics and

   961 stack. There are a few more tasks, a network device driver has to handle,

   992 device dependent features.

   962 including queue control, statistics and device dependent features.

   993 

   963 

   994 \paragraph{Driver Startup} Usually, a driver searches for compatible devices

   964 \paragraph{Driver Startup}

   995 on module loading.  For PCI drivers, this is done by scanning the PCI bus and

   965 

   996 checking for known device IDs. If a device is found, data structures are

   966 Usually, a driver searches for compatible devices on module loading.

   997 allocated and the device is taken into operation.

   967 For PCI drivers, this is done by scanning the PCI bus and checking for

   998 

   968 known device IDs. If a device is found, data structures are allocated

   999 \paragraph{Interrupt Operation}\index{Interrupt} A network device usually

   969 and the device is taken into operation.

  1000 provides a hardware interrupt that is used to notify the driver of received

   970 

  1001 frames and success of transmission, or errors, respectively. The driver has to

   971 \paragraph{Interrupt Operation}

  1002 register an interrupt service routine

   972 \index{Interrupt}

  1003 (ISR\index{ISR}\nomenclature{ISR}{Interrupt Service Routine}), that is

   973 

  1004 executed each time, the hardware signals such an event. If the interrupt was

   974 A network device usually provides a hardware interrupt that is used to

  1005 thrown by the own device (multiple devices can share one hardware interrupt),

   975 notify the driver of received frames and success of transmission, or

  1006 the reason for the interrupt has to be determined by reading the device's

   976 errors, respectively. The driver has to register an interrupt service

  1007 interrupt register. For example, if the flag for received frames is set, frame

   977 routine (ISR\index{ISR}\nomenclature{ISR}{Interrupt Service Routine}),

  1008 data has to be copied from hardware to kernel memory and passed to the network

   978 that is executed each time, the hardware signals such an event. If the

  1009 stack.

   979 interrupt was thrown by the own device (multiple devices can share one

  1010 

   980 hardware interrupt), the reason for the interrupt has to be determined

  1011 \paragraph{The \lstinline+net_device+ Structure}\index{net\_device} The driver

   981 by reading the device's interrupt register. For example, if the flag

  1012 registers a \lstinline+net_device+ structure for each device to communicate

   982 for received frames is set, frame data has to be copied from hardware

  1013 with the network stack and to create a ``network interface''. In case of an

   983 to kernel memory and passed to the network stack.

  1014 Ethernet driver, this interface appears as \textit{ethX}, where X is a number

   984 

  1015 assigned by the kernel on registration. The \lstinline+net_device+ structure

   985 \paragraph{The \lstinline+net_device+ Structure}

  1016 receives events (either from userspace or from the network stack) via several

   986 \index{net\_device}

  1017 callbacks, which have to be set before registration. Not every callback is

   987 

  1018 mandatory, but for reasonable operation the ones below are needed in any case:

   988 The driver registers a \lstinline+net_device+ structure for each device to

   989 communicate with the network stack and to create a ``network interface''. In

   990 case of an Ethernet driver, this interface appears as \textit{ethX}, where X

   991 is a number assigned by the kernel on registration. The \lstinline+net_device+

   992 structure receives events (either from userspace or from the network stack)

   993 via several callbacks, which have to be set before registration. Not every

   994 callback is mandatory, but for reasonable operation the ones below are needed

   995 in any case:

  1030 \paragraph{The \lstinline+netif+ Interface}

  1053 \paragraph{The \lstinline+netif+ Interface}\index{netif} All other

  1031 \index{netif}

  1054 communication in the direction interface $\to$ network stack is done via the

  1032 

  1055 \lstinline+netif_*()+ calls. For example, on successful device opening, the

  1033 All other communication in the direction interface $\to$ network stack is done

  1056 network stack has to be notified, that it can now pass frames to the

  1034 via the \lstinline+netif_*()+ calls. For example, on successful device opening,

  1035 the network stack has to be notified, that it can now pass frames to the

  1038 stack. Furthermore a network driver usually manages a frame transmission queue.

  1059 stack. Furthermore a network driver usually manages a frame transmission

  1039 If this gets filled up, the network stack has to be told to stop passing

  1060 queue.  If this gets filled up, the network stack has to be told to stop

  1040 further frames for a while. This happens with a call to

  1061 passing further frames for a while. This happens with a call to

  1052 \paragraph{Socket Buffers}

  1073 \paragraph{Socket Buffers}\index{Socket buffer} Socket buffers are the basic

  1053 \index{Socket buffer}

  1074 data type for the whole network stack. They serve as containers for network

  1054 

  1075 data and are able to quickly add data headers and footers, or strip them off

  1055 Socket buffers are the basic data type for the whole network stack. They serve

  1076 again. Therefore a socket buffer consists of an allocated buffer and several

  1056 as containers for network data and are able to quickly add data headers and

  1077 pointers that mark beginning of the buffer (\lstinline+head+), beginning of

  1057 footers, or strip them off again. Therefore a socket buffer consists of an

  1078 data (\lstinline+data+), end of data (\lstinline+tail+) and end of buffer

  1058 allocated buffer and several pointers that mark beginning of the buffer

  1079 (\lstinline+end+). In addition, a socket buffer holds network header

  1059 (\lstinline+head+), beginning of data (\lstinline+data+), end of data

  1080 information and (in case of received data) a pointer to the

  1060 (\lstinline+tail+) and end of buffer (\lstinline+end+). In addition, a socket

  1081 \lstinline+net_device+, it was received on. There exist functions that create

  1061 buffer holds network header information and (in case of received data) a

  1082 a socket buffer (\lstinline+dev_alloc_skb()+), add data either from front

  1062 pointer to the \lstinline+net_device+, it was received on. There exist

  1083 (\lstinline+skb_push()+) or back (\lstinline+skb_put()+), remove data from

  1063 functions that create a socket buffer (\lstinline+dev_alloc_skb()+), add data

  1084 front (\lstinline+skb_pull()+) or back (\lstinline+skb_trim()+), or delete the

  1064 either from front (\lstinline+skb_push()+) or back (\lstinline+skb_put()+),

  1085 buffer (\lstinline+kfree_skb()+).  A socket buffer is passed from layer to

  1065 remove data from front (\lstinline+skb_pull()+) or back

  1086 layer, and is freed by the layer that uses it the last time. In case of

  1066 (\lstinline+skb_trim()+), or delete the buffer (\lstinline+kfree_skb()+).  A

  1087 sending, freeing has to be done by the network driver.

  1067 socket buffer is passed from layer to layer, and is freed by the layer that

  1088 

  1068 uses it the last time. In case of sending, freeing has to be done by the

  1089 %------------------------------------------------------------------------------

  1069 network driver.

  1090 

  1070 

  1091 \section{Native EtherCAT Device Drivers}

  1071 %------------------------------------------------------------------------------

  1092 \label{sec:native-drivers}

  1072 

  1093 

  1073 \section{EtherCAT Device Drivers}

  1094 There are a few requirements, that applies to Ethernet hardware when used with

  1074 \label{sec:drivers}

  1095 a native Ethernet driver with EtherCAT functionality.

  1075 

  1096 

  1076 There are a few requirements for Ethernet network devices to function as

  1097 \paragraph{Dedicated Hardware} For performance and realtime purposes, the

  1077 EtherCAT devices, when connected to an EtherCAT bus.

  1098 EtherCAT master needs direct and exclusive access to the Ethernet hardware.

  1078 

  1099 This implies that the network device must not be connected to the kernel's

  1079 \paragraph{Dedicated Interfaces}

  1100 network stack as usual, because the kernel would try to use it as an ordinary

  1080 

  1101 Ethernet device.

  1081 For performance and realtime purposes, the EtherCAT master needs direct and

  1102 

  1082 exclusive access to the Ethernet hardware. This implies that the network device

  1103 \paragraph{Interrupt-less Operation}\index{Interrupt} EtherCAT frames travel

  1083 must not be connected to the kernel's network stack as usual, because the

  1104 through the logical EtherCAT ring and are then sent back to the master.

  1084 kernel would try to use it as an ordinary Ethernet device.

  1105 Communication is highly deterministic: A frame is sent and will be received

  1085 

  1106 again after a constant time, so there is no need to notify the driver about

  1086 \paragraph{Interrupt-less Operation}

  1107 frame reception: The master can instead query the hardware for received

  1087 \index{Interrupt}

  1108 frames, if it expects them to be already received.

  1088 

  1089 EtherCAT frames travel through the logical EtherCAT ring and are then sent back

  1090 to the master. Communication is highly deterministic: A frame is sent and will

  1091 be received again after a constant time, so there is no need to notify the

  1092 driver about frame reception: The master can instead query the hardware for

  1093 received frames, if it expects them to be already received.

  1126 \paragraph{Ethernet and EtherCAT Devices}

  1141 \paragraph{Ethernet and EtherCAT Devices} Another issue lies in the way Linux

  1127 

  1142 handles devices of the same type.  For example, a

  1128 Another issue lies in the way Linux handles devices of the same type.  For

  1143 PCI\nomenclature{PCI}{Peripheral Component Interconnect, Computer Bus} driver

  1129 example, a PCI\nomenclature{PCI}{Peripheral Component Interconnect, Computer

  1144 scans the PCI bus for devices it can handle. Then it registers itself as the

  1130 Bus} driver scans the PCI bus for devices it can handle. Then it registers

  1145 responsible driver for all of the devices found. The problem is, that an

  1131 itself as the responsible driver for all of the devices found. The problem is,

  1146 unmodified driver can not be told to ignore a device because it will be used

  1132 that an unmodified driver can not be told to ignore a device because it will

  1147 for EtherCAT later. There must be a way to handle multiple devices of the same

  1133 be used for EtherCAT later. There must be a way to handle multiple devices of

  1148 type, where one is reserved for EtherCAT, while the other is treated as an

  1134 the same type, where one is reserved for EtherCAT, while the other is treated

  1149 ordinary Ethernet device.

  1135 as an ordinary Ethernet device.

  1163 \section{Device Selection}

  1177 \section{Generic EtherCAT Device Driver}

  1164 \label{sec:deviceselection}

  1178 \label{sec:generic-driver}

  1165 

  1179 

  1166 After loading the master module, at least one EtherCAT-capable network driver

  1180 Since there are approaches to enable the complete Linux kernel for realtime

  1167 module has to be loaded, that offers its devices to the master (see

  1181 operation \cite{rt-preempt}, it is possible to operate without native

  1168 sec.~\ref{sec:ecdev}. The master module knows the devices to choose from the

  1182 implementations of EtherCAT-capable Ethernet device drivers and use the Linux

  1169 module parameters (see sec.~\ref{sec:mastermod}). If the init script is used

  1183 network stack instead. Fig.~\ref{fig:arch} shows the ``Generic Ethernet Driver

  1170 to start the master, the drivers and devices to use can be specified in the

  1184 Module'', that connects to local Ethernet devices via the network stack. The

  1171 sysconfig file (see sec.~\ref{sec:sysconfig}).

  1185 kernel module is named \lstinline+ec_generic+ and can be loaded after the

  1186 master module like a native EtherCAT-capable Ethernet driver.

  1187 

  1188 The generic device driver scans the network stack for interfaces, that have

  1189 been registered by Ethernet device drivers. It offers all possible devices to

  1190 the EtherCAT master. If the master accepts a device, the generic driver

  1191 creates a packet socket (see \lstinline+man 7 packet+) with

  1192 \lstinline+socket_type+ set to \lstinline+SOCK_RAW+, bound to that device. All

  1193 functions of the device interface (see sec.~\ref{sec:ecdev}) will then operate

  1194 on that socket.

  1195 

  1196 Below are the advantages of this solution:

  1197 

  1198 \begin{itemize}

  1199 \item Any Ethernet hardware, that is covered by a Linux Ethernet driver can be

  1200 used for EtherCAT.

  1201 \item No modifications have to be made to the actual Ethernet drivers.

  1202 \end{itemize}

  1203 

  1204 The generic approach has the following disadvantages:

  1205 

  1206 \begin{itemize}

  1207 \item The performance is a little worse than the native approach, because the

  1208 frame data have to traverse the lower layers of the network stack.

  1209 \item It is not possible to use in-kernel realtime extensions like RTAI with

  1210 the generic driver, because the network stack code uses dynamic memory

  1211 allocations and other things, that could cause the system to freeze in

  1212 realtime context.

  1213 \end{itemize}

  1214 

  1215 %------------------------------------------------------------------------------

  1216 

  1217 \section{Providing Ethernet Devices}

  1218 \label{sec:providing-devices}

  1219 

  1220 After loading the master module, additional module(s) have to be loaded to

  1221 offer devices to the master(s) (see sec.~\ref{sec:ecdev}). The master module

  1222 knows the devices to choose from the module parameters (see

  1223 sec.~\ref{sec:mastermod}). If the init script is used to start the master, the

  1224 drivers and devices to use can be specified in the sysconfig file (see

  1225 sec.~\ref{sec:sysconfig}).

  1226 

  1227 Modules offering Ethernet devices can be

  1228 

  1229 \begin{itemize}

  1230 \item native EtherCAT-capable network driver modules (see

  1231 sec.~\ref{sec:native-drivers}) or

  1232 \item the generic EtherCAT device driver module (see

  1233 sec.~\ref{sec:generic-driver}).

  1234 \end{itemize}

  1199 \section{Patching Network Drivers}

  1262 \section{Patching Native Network Drivers}

  1204 EtherCAT-capable. Unfortunately, there is no standard procedure to enable an

  1267 EtherCAT-capable, using the native approach (see

  1205 Ethernet driver for use with the EtherCAT master, but there are a few common

  1268 sec.~\ref{sec:native-drivers}). Unfortunately, there is no standard procedure

  1206 techniques.

  1269 to enable an Ethernet driver for use with the EtherCAT master, but there are a

  1270 few common techniques.

  2608 Wireshark~\cite{wireshark}, for example.

  2672 Wireshark~\cite{wireshark}, for example. It is also possible to listen to

  2609 

  2673 local network interfaces on the machine running the EtherCAT master directly.

  2610 For convenience, so-called ``debug interfaces'' are supported. Debug

  2674 If the generic Ethernet driver (see sec.~\ref{sec:generic-driver}) is used,

  2611 interfaces are virtual network interfaces allowing to capture EtherCAT traffic

  2675 the network monitor can directly listen on the network interface connected to

  2612 with a network monitor (like Wireshark or tcpdump) running on the master

  2676 the EtherCAT bus.

  2613 machine without using external hardware. To use this functionality, the master

  2677 

  2614 sources have to be configured with the \lstinline+--enable-debug-if+ switch

  2678 When using native Ethernet drivers (see sec.~\ref{sec:native-drivers}), there

  2615 (see sec.~\ref{sec:installation}).

  2679 are no local network interfaces to listen to, because the Ethernet devices

  2680 used for EtherCAT are not registered at the network stack. For that case,

  2681 so-called ``debug interfaces'' are supported, which are virtual network

  2682 interfaces allowing to capture EtherCAT traffic with a network monitor (like

  2683 Wireshark or tcpdump) running on the master machine without using external

  2684 hardware. To use this functionality, the master sources have to be configured

  2685 with the \lstinline+--enable-debug-if+ switch (see

  2686 sec.~\ref{sec:installation}).

  2647 interfaces have to be allocated dynamically. Some Linux realtime extensions do

  2718 interfaces have to be allocated dynamically. Some Linux realtime extensions

  2648 not allow this in realtime context!

  2719 (like RTAI) do not allow this in realtime context!

  3052 \url{http://www.rtai.org}, 2006.

  3135 \url{https://www.rtai.org}, 2010.

  3136 

  3137 \bibitem{rt-preempt} RT PREEMPT HOWTO.

  3138 \url{http://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO}, 2010.