1 %------------------------------------------------------------------------------

     2 %

     3 %  IgH EtherCAT Master Documentation

     4 %

     5 %  $Id$

     6 %

     7 %------------------------------------------------------------------------------

     8 

     9 \documentclass[a4paper,12pt,BCOR6mm,bibtotoc,idxtotoc]{scrbook}

    10 

    11 \usepackage[latin1]{inputenc}

    12 \usepackage[automark,headsepline]{scrpage2}

    13 \usepackage{graphicx}

    14 \usepackage{makeidx}

    15 \usepackage[refpage]{nomencl}

    16 \usepackage{listings}

    17 \usepackage{svn}

    18 \usepackage{textcomp}

    19 \usepackage{url}

    20 \usepackage{hyperref}

    21 

    22 \setlength{\parskip}{0.8ex plus 0.8ex minus 0.5ex}

    23 \setlength{\parindent}{0mm}

    24 

    25 \setcounter{secnumdepth}{\subsubsectionlevel}

    26 

    27 \DeclareFontShape{OT1}{cmtt}{bx}{n}

    28 {

    29   <5><6><7><8><9><10><10.95><12><14.4><17.28><20.74><24.88>cmttb10

    30 }{}

    31 

    32 \lstset{basicstyle=\ttfamily\small,numberstyle=\tiny,aboveskip=4mm,

    33   belowskip=2mm,gobble=2,escapechar=`}

    34 \renewcommand\lstlistlistingname{List of Listings}

    35 

    36 \renewcommand\nomname{Glossary}

    37 

    38 \newcommand{\IgH}{\raisebox{-0.7667ex}

    39   {\includegraphics[height=2.2ex]{images/ighsign}}}

    40 

    41 \SVN $Date$

    42 \SVN $Revision$

    43 

    44 \makeindex

    45 \makeglossary

    46 

    47 %------------------------------------------------------------------------------

    48 

    49 \begin{document}

    50 

    51 \pagenumbering{roman}

    52 \pagestyle{empty}

    53 

    54 \begin{titlepage}

    55   \begin{center}

    56     \rule{\textwidth}{1.5mm}

    57 

    58     {\Huge\bf IgH \includegraphics[height=2.4ex]{images/ethercat}

    59       Master 1.1\\[1ex]

    60       Documentation}

    61 

    62     \vspace{1ex}

    63     \rule{\textwidth}{1.5mm}

    64 

    65     \vspace{\fill}

    66     {\Large Florian Pose, \url{fp@igh-essen.com}\\[1ex]

    67       Ingenieurgemeinschaft \IgH}

    68 

    69     \vspace{\fill}

    70     {\Large Essen, \SVNDate\\[1ex]

    71       Revision \SVNRevision}

    72   \end{center}

    73 \end{titlepage}

    74 

    75 %------------------------------------------------------------------------------

    76 

    77 \tableofcontents

    78 \listoftables

    79 \listoffigures

    80 \lstlistoflistings

    81 

    82 %------------------------------------------------------------------------------

    83 

    84 \newpage

    85 \pagestyle{scrheadings}

    86 

    87 \section*{Conventions}

    88 \addcontentsline{toc}{section}{Conventions}

    89 \markleft{Conventions}

    90 

    91 The following typographic conventions are used:

    92 

    93 \begin{itemize}

    94 \item \textit{Italic face} is used for newly introduced terms, file

    95   names, parameter names and in-text source code elements.

    96 \item \texttt{Typewriter face} is used for code examples and

    97   command line output.

    98 \item \texttt{\textbf{Bold typewriter face}} is used for user input in

    99   command lines.

   100 \end{itemize}

   101 

   102 Data values and addresses are specified as hexadecimal values with the

   103 prefix 0x. Example: 0x88A4. Unless otherwise noted, address values are

   104 specified as byte addresses.

   105 

   106 Concerning bit operations, the phrase ``setting a bit'', stands for

   107 setting the bit to $1$, ``clearing a bit'' means setting it to $0$,

   108 respectively.

   109 

   110 Function names are always printed with parentheses, but without

   111 parameters. So, if a function \textit{ecrt\_request\_master()} has

   112 empty parentheses, this does not mean, that it has no parameters.

   113 

   114 If shell commands have to be entered, this is marked by a prompt:

   115 

   116 \begin{lstlisting}[gobble=2]

   117   host>

   118 \end{lstlisting}

   119 

   120 Further, if a shell command has to be entered as the superuser, the

   121 prompt ends with a mesh:

   122 

   123 \begin{lstlisting}[gobble=2]

   124   host#

   125 \end{lstlisting}

   126 

   127 %------------------------------------------------------------------------------

   128 

   129 \chapter{The IgH EtherCAT Master}

   130 \label{chapter:master}

   131 \pagenumbering{arabic}

   132 

   133 This section will first introduce the master's general features and

   134 the concepts used for master development and will then explain the

   135 master's general architecture and offer details of the different

   136 modules. In addition, it will cover state machine definitions, mailbox

   137 protocol implementation and the user space interface. The last section

   138 will deal with timing aspects.

   139 

   140 %------------------------------------------------------------------------------

   141 

   142 \section{Feature Summary}

   143 \label{sec:summary}

   144 \index{Master!Features}

   145 

   146 The list below gives a short summary of the features of the

   147 implemented EtherCAT master.

   148 

   149 \begin{itemize}

   150 \item The master runs as a kernel module for Linux 2.6.

   151 \item It comes with EtherCAT-capable network driver for RealTek

   152   RTL8139 (and compatible) network interface cards.

   153   \begin{itemize}

   154   \item The Ethernet hardware is operated without interrupts.

   155   \item Drivers for additional Ethernet hardware can easily be

   156     implemented due to a common device interface provided by the

   157     master.

   158   \end{itemize}

   159 \item The master module supports multiple EtherCAT masters on one

   160   machine.

   161 \item The master code supports any Linux realtime extension through

   162   its independent architecture.

   163   \begin{itemize}

   164   \item RTAI\nomenclature{RTAI}{RealTime Application Interface},

   165     ADEOS\nomenclature{ADEOS}{Adaptive Domain Environment for

   166       Operating Systems}, etc.

   167   \item It runs well even without realtime extensions.

   168   \end{itemize}

   169 \item Common ``realtime interface'' for modules, that want to use

   170   EtherCAT functionality.

   171   \begin{itemize}

   172   \item Synchronous and asynchronous sending and receiving of frames

   173     is supported.

   174   \item Avoidance of unnecessary copy operations for process data.

   175   \end{itemize}

   176 \item \textit{Domains} are introduced, to allow grouping of process

   177   data objects.

   178   \begin{itemize}

   179   \item Handling of multiple domains with different sampling rates.

   180   \item Automatic calculation of process data mapping, FMMU and sync manager

   181     configuration within each domain.

   182   \end{itemize}

   183 \item Communication through serveral finite state machines.

   184   \begin{itemize}

   185   \item Bus monitoring possible during realtime operation.

   186   \item Automatic reconfiguration of slaves on bus power failure

   187     during realtime operation.

   188   \item Controlling of single slaves during realtime operation.

   189   \end{itemize}

   190 \item Master idle mode.

   191   \begin{itemize}

   192   \item Automatic scanning of slaves upon topology changes.

   193   \item Bus visualisation and EoE processing without a realtime module

   194     connected.

   195   \end{itemize}

   196 \item Implementation of the CANopen-over-EtherCAT (CoE) protocol.

   197   \begin{itemize}

   198   \item Configuration of CoE-capable slaves via SDO interface.

   199   \end{itemize}

   200 \item Implementation of the Ethernet-over-EtherCAT (EoE) protocol.

   201   \begin{itemize}

   202   \item Each master creates virtual network devices that are

   203     automatically coupled to EoE-cap\-able slaves found.

   204   \item This implementation natively supports either a switched or a

   205     routed EoE network architecture.

   206   \end{itemize}

   207 \item User space interface via the System Filesystem

   208   (Sysfs)\nomenclature{Sysfs}{System Filesystem}.

   209   \begin{itemize}

   210   \item User space tool for bus visualisation.

   211   \item Slave E$^2$PROM image reading and writing.

   212   \end{itemize}

   213 \item Seamless system integration though LSB\nomenclature{LSB}{Linux

   214     Standard Base} compliance.

   215   \begin{itemize}

   216   \item Master and network device configuration via Sysconfig files.

   217   \item Linux Standard Base compatible init script for master control.

   218   \end{itemize}

   219 \item Virtual read-only network interface for monitoring and debugging

   220   purposes.

   221 \end{itemize}

   222 

   223 %------------------------------------------------------------------------------

   224 

   225 \section{License}

   226 \label{sec:license}

   227 

   228 The master code is released under the terms and conditions of the GNU

   229 General Public License\index{GPL} \cite{gpl} (version 2). Other

   230 developers, that want to use EtherCAT with Linux systems, are invited

   231 to use the master code or even participate on development.

   232 

   233 %------------------------------------------------------------------------------

   234 

   235 \section{General Master Architecture}

   236 \label{sec:masterarch}

   237 \index{Master!Architecture}

   238 

   239 The EtherCAT master is integrated into the Linux 2.6 kernel. This was

   240 an early design decision, which has been made for serveral reasons:

   241 

   242 \begin{itemize}

   243 \item Kernel code has significantly better realtime characteristics,

   244   i.~e. less jitter than user space code. It was foreseeable, that a

   245   fieldbus master has a lot of cyclic work to do. Cyclic work is

   246   usually triggered by timer interrupts inside the kernel. The

   247   execution delay of a function that processes timer interrupts is

   248   less, when it resides in kernel space, because there is no need of

   249   time-consuming context switches to a user space process.

   250 \item It was also foreseeable, that the master code has to directly

   251   communicate with the Ethernet hardware. This has to be done in the

   252   kernel anyway (through network device drivers), which is one more

   253   reason for the master code being in kernel space.

   254 \end{itemize}

   255 

   256 A general overview of the master architecture can be seen in

   257 figure~\ref{fig:masterarch}.

   258 

   259 \begin{figure}[htbp]

   260   \centering

   261   \includegraphics[width=.9\textwidth]{images/masterarch}

   262   \caption{Master architecture}

   263   \label{fig:masterarch}

   264 \end{figure}

   265 

   266 \paragraph{Master Module}

   267 \index{Master module}

   268 

   269 The EtherCAT master mainly consists of the master module, containing

   270 one or more EtherCAT masters (section~\ref{sec:mastermod}), the

   271 ``Device Interface'' (section~\ref{sec:ecdev}) and the ``Realtime

   272 Interface'' (section~\ref{sec:ecrt}).

   273 

   274 \paragraph{Device Modules}

   275 \index{Device modules}

   276 

   277 Furthermore there are EtherCAT-capable network device driver

   278 modules\index{Device modules}, that connect to the EtherCAT master via

   279 the device interface. These modified network drivers can handle both

   280 network devices used for EtherCAT operation and ``normal'' Ethernet

   281 devices. The common case is, that the master module offers a single

   282 EtherCAT master: An EtherCAT-capable network device driver module

   283 connects one network device to this master, that is now able to send

   284 and receive EtherCAT frames, while all other network devices handled

   285 by the network driver get connected to the kernel's network stack as

   286 usual.

   287 

   288 \paragraph{Realtime Modules}

   289 

   290 A ``realtime module''\index{Realtime module} is a kernel module, that

   291 uses the EtherCAT master for cyclic exchange of process data with

   292 EtherCAT slaves. Realtime modules are not part of the EtherCAT master

   293 code\footnote{Although there are serveral examples provided in the

   294   \textit{examples} directory, see chapter~\ref{chapter:usage} for

   295   more information}, so anybody wanting to use the master has to write

   296 one. A realtime module can ``request'' a master through the realtime

   297 interface. If this succeeds, the module has the control over the

   298 master. It can now configure slaves and set up a process data image

   299 (see section~\ref{sec:processdata}) for cyclic exchange. This cyclic

   300 code has to be provided by the realtime module, so it is in hands of

   301 the developer, which mechanism to use for this. Moreover he has to

   302 decide, whether or not using a Linux realtime extension.

   303 

   304 \paragraph{Why ``Realtime'' Module?}

   305 

   306 The name shall not imply, that a linux realtime extension is

   307 mandatory: The master runs well even without realtime extensions, as

   308 section~\ref{sec:mini} shows. However, the code using the master is

   309 time-critical, because process data IO has to be done in cyclic work.

   310 Some EtherCAT slaves support watchdog units, that stop driving the

   311 outputs when process data was not exchanged for some time. So the

   312 names ``realtime interface'' and ``realtime module'' are quite

   313 appropriate.

   314 

   315 %------------------------------------------------------------------------------

   316 

   317 \subsection{Handling of Process Data}

   318 \label{sec:processdata}

   319 

   320 \paragraph{Process Data Image}

   321 \index{Process data}

   322 

   323 The slaves offer their inputs and outputs by presenting the master

   324 so-called ``Process Data Objects'' (PDOs\index{PDO}). The available

   325 PDOs can be determined by reading out the slave's TXPDO and RXPDO

   326 E$^2$PROM categories. The realtime module can register the PDOs for

   327 data exchange during cyclic operation. The sum of all registered PDOs

   328 defines the ``process data image'', which is exchanged via the

   329 ``Logical ReadWrite'' datagrams introduced

   330 in~\cite[section~5.4.2.4]{dlspec}.

   331 

   332 \paragraph{Process Data Domains}

   333 \index{Domain}

   334 

   335 The process data image can be easily managed by creating co-called

   336 ``domains'', which group PDOs and allocate the datagrams needed to

   337 exchange them. Domains are mandatory for process data exchange, so

   338 there has to be at least one. They were introduced for the following

   339 reasons:

   340 

   341 \begin{itemize}

   342 \item The maximum size of a ``Logical ReadWrite'' datagram is limited

   343   due to the limited size of an Ethernet frame: The maximum data size

   344   is the Ethernet data field size minus the EtherCAT frame header,

   345   EtherCAT datagram header and EtherCAT datagram footer: $1500 - 2 -

   346   12 - 2 = 1484$ octets. If the size of the process data image exceeds

   347   this limit, multiple frames have to be sent, and the image has to be

   348   partitioned for the use of multiple datagrams. A domain manages this

   349   automatically.

   350 \item Not every PDO has to be exchanged with the same frequency: The

   351   values of PDOs can vary slowly over time (for example temperature

   352   values), so exchanging them with a high frequency would just waste

   353   bus bandwidth. For this reason, multiple domains can be created, to

   354   group different PDOs and so allow separate exchange.

   355 \end{itemize}

   356 

   357 There is no upper limit for the number of domains, but each domain

   358 occupies one FMMU in each slave involved, so the maximum number of

   359 domains is also limited by the slaves' capabilities.

   360 

   361 \paragraph{FMMU Configuration}

   362 \index{FMMU!Configuration}

   363 

   364 A realtime module can register PDOs for process data exchange. Every

   365 PDO is part of a memory area in the slave's physical memory, that is

   366 protected by a sync manager \cite[section~6.7]{dlspec} for

   367 synchronized access. In order to make a sync manager react on a

   368 datagram accessing its memory, it is necessary to access the last byte

   369 covered by the sync manager. Otherwise the sync manager will not react

   370 on the datagram and no data will be exchanged. That is why the whole

   371 synchronized memory area has to be included into the process data

   372 image: For example, if a certain PDO of a slave is registered for

   373 exchange with a certain domain, one FMMU will be configured to map the

   374 complete sync-manager-protected memory, the PDO resides in. If a

   375 second PDO of the same slave is registered for process data exchange

   376 within the same domain, and this PDO resides in the same

   377 sync-manager-protected memory as the first PDO, the FMMU configuration

   378 is not touched, because the appropriate memory is already part of the

   379 domain's process data image.  If the second PDO belongs to another

   380 sync-manager-protected area, this complete area is also included into

   381 the domains process data image. See figure~\ref{fig:fmmus} for an

   382 overview, how FMMU's are configured to map physical memory to logical

   383 process data images.

   384 

   385 \begin{figure}[htbp]

   386   \centering

   387   \includegraphics[width=\textwidth]{images/fmmus}

   388   \caption{FMMU configuration for serveral domains}

   389   \label{fig:fmmus}

   390 \end{figure}

   391 

   392 \paragraph{Process Data Pointers}

   393 

   394 The figure also demonstrates the way, the realtime module can access the

   395 exchanged process data: At PDO registration, the realtime module has

   396 to provide the address of a process data pointer. Upon calculation of

   397 the domain image and allocation of process data memory, this pointer

   398 is redirected to the appropriate location inside the domain's process

   399 data memory and can later be easily dereferenced by the module code.

   400 

   401 %------------------------------------------------------------------------------

   402 

   403 \subsection{Operation Modes}

   404 \index{Master modes}

   405 

   406 The EtherCAT master has serveral modes of operation:

   407 

   408 \begin{description}

   409 \item[Orphaned Mode] This mode takes effect, when the master has no

   410   EtherCAT-capable network device connected. No bus communication is

   411   possible, so this mode is not of further interest.

   412 \item[Idle Mode]\index{Idle mode} takes effect when the master is

   413   unused (i.~e. there is no realtime module, that reserved the

   414   master). In this case, the master has the ability to scan the bus by

   415   itsself and generously allow bus access from user space.  This mode

   416   is meant for maintenance and visualisation.

   417 \item[Operation Mode]\index{Operation mode} The master is reserved for

   418   exclusive access by a realtime module. In this mode, the master is

   419   adjusted for availability and monitoring. Access from user space is

   420   very restrictive and mostly limited to reading direction.

   421 

   422 \end{description}

   423 

   424 Figure~\ref{fig:modes} shows the three modes and the possible mode

   425 transitions.

   426 

   427 \begin{figure}[htbp]

   428   \centering

   429   \includegraphics[width=.9\textwidth]{images/modes}

   430   \caption{Master modes and transitions}

   431   \label{fig:modes}

   432 \end{figure}

   433 

   434 \subsubsection{Idle Mode}

   435 \index{Idle mode}

   436 

   437 The master enters idle mode upon connection of a device module (see

   438 section~\ref{sec:device}) or releasing by a realtime module. The

   439 master owns a kernel workqueue and a suitable work structure, which is

   440 used to cyclically process the ``Idle state machine'' (see

   441 section~\ref{sec:fsm-idle}). This state machine automatically scans

   442 the bus for slaves (and re-scans upon topology changes), configures

   443 slaves for idle operation and executes pending operations from the

   444 user space interface (for example E$^2$PROM writing). On device

   445 disconnection or realtime request, the idle mode is stopped by

   446 cancelling the work and flushing the workqueue.

   447 

   448 \subsubsection{Operation Mode}

   449 \index{Operation mode}

   450 

   451 Operation mode is entered when a realtime module requests the master.

   452 The idle mode is stopped and the bus is scanned by getting the number

   453 of slaves and executing the ``Slave scan state machine'' (see

   454 section~\ref{sec:fsm-scan}) for each slave. The master is now ready to

   455 create domains and accept PDO registrations and slave configurations.

   456 After that, cyclic communication can be done by the realtime module.

   457 

   458 \paragraph{Master Phases}

   459 

   460 Every realtime module should use the master in three phases:

   461 

   462 \begin{enumerate}

   463 \item \textit{Startup} - The master is requested and the bus is

   464   validated. Domains are created and PDOs are registered. Slave

   465   configurations are applied.

   466 \item \textit{Operation} - Cyclic code is run, process data is

   467   exchanged and the master state machine is executed.

   468 \item \textit{Shutdown} - Cyclic code is stopped and the master

   469   is released.

   470 \end{enumerate}

   471 

   472 %------------------------------------------------------------------------------

   473 

   474 \section{Device Modules}

   475 \label{sec:device}

   476 \index{Device modules}

   477 

   478 Device modules are network device driver modules that handle Ethernet

   479 devices, which the master can use to connect to an EtherCAT bus.

   480 

   481 Section~\ref{sec:networkdrivers} offers an overview of general Linux

   482 network driver modules, while section~\ref{sec:requirements} will show

   483 the requirements to an EtherCAT-enabled network driver. Finally,

   484 sections~\ref{sec:seldev} to~\ref{sec:patching} show how to fulfill

   485 these requirements and implement such a driver module.

   486 

   487 %------------------------------------------------------------------------------

   488 

   489 \subsection{Network Driver Basics}

   490 \label{sec:networkdrivers}

   491 \index{Network drivers}

   492 

   493 EtherCAT relies on Ethernet hardware and the master needs a physical

   494 Ethernet device to communicate with the bus. Therefore it is necessary

   495 to understand how Linux handles network devices and their drivers,

   496 respectively.

   497 

   498 \paragraph{Tasks of a Network Driver}

   499 

   500 Network device drivers handle the lower two layers of the OSI model,

   501 that is the physical layer and the data-link layer. A network device

   502 itself natively handles the physical layer issues: It represents the

   503 hardware to connect to the medium and to send and receive data in the

   504 way, the physical layer protocol describes. The network device driver

   505 is responsible for getting data from the kernel's networking stack and

   506 forwarding it to the hardware, that does the physical transmission.

   507 If data is received by the hardware respectively, the driver is

   508 notified (usually by means of an interrupt) and has to read the data

   509 from the hardware memory and forward it to the network stack. There

   510 are a few more tasks, a network device driver has to handle, including

   511 queue control, statistics and device dependent features.

   512 

   513 \paragraph{Driver Startup}

   514 

   515 Usually, a driver searches for compatible devices on module loading.

   516 For PCI drivers, this is done by scanning the PCI bus and checking for

   517 known device IDs. If a device is found, data structures are allocated

   518 and the device is taken into operation.

   519 

   520 \paragraph{Interrupt Operation}

   521 \index{Interrupt}

   522 

   523 A network device usually provides a hardware interrupt that is used to

   524 notify the driver of received frames and success of transmittion, or

   525 errors, respectively. The driver has to register an interrupt service

   526 routine (ISR\index{ISR}\nomenclature{ISR}{Interrupt Service Routine}),

   527 that is executed each time, the hardware signals such an event. If the

   528 interrupt was thrown by the own device (multiple devices can share one

   529 hardware interrupt), the reason for the interrupt has to be determined

   530 by reading the device's interrupt register. For example, if the flag

   531 for received frames is set, frame data has to be copied from hardware

   532 to kernel memory and passed to the network stack.

   533 

   534 \paragraph{The net\_device structure}

   535 \index{net\_device}

   536 

   537 The driver registers a \textit{net\_device} structure for each device

   538 to communicate with the network stack and to create a ``network

   539 interface''. In case of an Ethernet driver, this interface appears as

   540 \textit{ethX}, where X is a number assigned by the kernel on

   541 registration. The \textit{net\_device} structure receives events

   542 (either from user space or from the network stack) via serveral

   543 callbacks, which have to be set before registration. Not every

   544 callback is mandatory, but for reasonable operation the ones below are

   545 needed in any case:

   546 

   547 \begin{description}

   548 \item[int (*open)(struct net\_device *)] This function is called when

   549   network communication has to be started, for example after a command

   550   \textit{ifconfig ethX up} from user space. Frame reception has to be

   551   enabled by the driver.

   552 \item[int (*stop)(struct net\_device *)] The purpose of this callback

   553   is to ``close'' the device, i.~e. make the hardware stop receiving

   554   frames.

   555 \item[int (*hard\_start\_xmit)(struct sk\_buff *, struct net\_device

   556   *)] This function is cal\-led for each frame that has to be

   557   transmitted.  The network stack passes the frame as a pointer to an

   558   \textit{sk\_buff} structure (``socket buffer''\index{Socket buffer},

   559   see below), which has to be freed after sending.

   560 \item[struct net\_device\_stats *(*get\_stats)(struct net\_device *)]

   561   This call has to return a pointer to the device's

   562   \textit{net\_device\_stats} structure, which permanently has to be

   563   filled with frame statistics. This means, that everytime a frame is

   564   received, sent, or an error happened, the appropriate counter in

   565   this structure has to be increased.

   566 \end{description}

   567 

   568 The actual registration is done with the \textit{register\_netdev()}

   569 call, unregistering is done with \textit{unregister\_netdev()}.

   570 

   571 \paragraph{The netif Interface}

   572 \index{netif}

   573 

   574 All other communication in the direction interface $\to$ network stack

   575 is done via the \textit{netif\_*} calls. For example, on successful

   576 device opening, the network stack has to be notified, that it can now

   577 pass frames to the interface. This is done by calling

   578 \textit{netif\_start\_queue()}. After this call, the

   579 \textit{hard\_start\_xmit()} callback can be called by the network

   580 stack. Furthermore a network driver usually manages a frame

   581 transmission queue. If this gets filled up, the network stack has to

   582 be told to stop passing further frames for a while. This happens with

   583 a call to \textit{netif\_stop\_queue()}. If some frames have been

   584 sent, and there is enough space again to queue new frames, this can be

   585 notified with \textit{netif\_wake\_queue()}. Another important call is

   586 \textit{netif\_receive\_skb()}\footnote{This function is part of the

   587   NAPI (``New API''), that replaces the ``old'' kernel 2.4 technique

   588   for interfacing to the network stack (with \textit{netif\_rx()}).

   589   NAPI is a technique to improve network performance on Linux. Read

   590   more in

   591   http://www.cyberus.ca/\textasciitilde{}hadi/usenix-paper.tgz}: It

   592 passes a frame to the network stack, that was just received by the

   593 device.  Frame data has to be packed into a so-called ``socket

   594 buffer'' for that (see below).

   595 

   596 \paragraph{Socket Buffers}

   597 \index{Socket buffer}

   598 

   599 Socket buffers are the basic data type for the whole network stack.

   600 They serve as containers for network data and are able to quickly add

   601 data headers and footers, or strip them off again. Therefore a socket

   602 buffer consists of an allocated buffer and serveral pointers that mark

   603 beginning of the buffer (\textit{head}), beginning of data

   604 (\textit{data}), end of data (\textit{tail}) and end of buffer

   605 (\textit{end}). In addition, a socket buffer holds network header

   606 information and (in case of received data) a pointer to the

   607 \textit{net\_device}, it was received on. There exist functions that

   608 create a socket buffer (\textit{dev\_alloc\_skb()}), add data either

   609 from front (\textit{skb\_push()}) or back (\textit{skb\_put()}),

   610 remove data from front (\textit{skb\_pull()}) or back

   611 (\textit{skb\_trim()}), or delete the buffer (\textit{kfree\_skb()}).

   612 A socket buffer is passed from layer to layer, and is freed by the

   613 layer that uses it the last time. In case of sending, freeing has to

   614 be done by the network driver.

   615 

   616 %------------------------------------------------------------------------------

   617 

   618 \subsection{EtherCAT Network Drivers}

   619 \label{sec:requirements}

   620 

   621 There are a few requirements for Ethernet network devices to function

   622 as EtherCAT devices, when connected to an EtherCAT bus.

   623 

   624 \paragraph{Dedicated Interfaces}

   625 

   626 For performance and realtime purposes, the EtherCAT master needs

   627 direct and exclusive access to the Ethernet hardware. This implies

   628 that the network device must not be connected to the kernel's network

   629 stack as usual, because the kernel would try to use it as an ordinary

   630 Ethernet device.

   631 

   632 \paragraph{Interrupt-less Operation}

   633 \index{Interrupt}

   634 

   635 EtherCAT frames travel through the logical EtherCAT ring and are then

   636 sent back to the master. Communication is highly deterministic: A

   637 frame is sent and will be received again after a constant time.

   638 Therefore, there is no need to notify the driver about frame

   639 reception: The master can instead query the hardware for received

   640 frames.

   641 

   642 Figure~\ref{fig:interrupt} shows two workflows for cyclic frame

   643 transmittion and reception with and without interrupts.

   644 

   645 \begin{figure}[htbp]

   646   \centering

   647   \includegraphics[width=.8\textwidth]{images/interrupt}

   648   \caption{Interrupt Operation versus Interrupt-less Operation}

   649   \label{fig:interrupt}

   650 \end{figure}

   651 

   652 In the left workflow ``Interrupt Operation'', the data from the last

   653 cycle is first processed and a new frame is assembled with new

   654 datagrams, which is then sent.  The cyclic work is done for now.

   655 Later, when the frame is received again by the hardware, an interrupt

   656 is triggered and the ISR is executed. The ISR will fetch the frame

   657 data from the hardware and initiate the frame dissection: The

   658 datagrams will be processed, so that the data is ready for processing

   659 in the next cycle.

   660 

   661 In the right workflow ``Interrupt-less Operation'', there is no

   662 hardware interrupt enabled.  Instead, the hardware will be polled by

   663 the master by executing the ISR. If the frame has been received in the

   664 meantime, it will be dissected. The situation is now the same as at

   665 the beginning of the left workflow: The received data is processed and

   666 a new frame is assembled and sent. There is nothing to do for the rest

   667 of the cycle.

   668 

   669 The interrupt-less operation is desirable, because there is simply no

   670 need for an interrupt. Moreover hardware interrupts are not conducive

   671 in improving the driver's realtime behaviour: Their undeterministic

   672 incidences contribute to increasing the jitter. Besides, if a realtime

   673 extension (like RTAI) is used, some additional effort would have to be

   674 made to priorize interrupts.

   675 

   676 \paragraph{Ethernet and EtherCAT Devices}

   677 

   678 Another issue lies in the way Linux handles devices of the same type.

   679 For example, a PCI\nomenclature{PCI}{Peripheral Component

   680   Interconnect, Computer Bus} driver scans the PCI bus for devices it

   681 can handle. Then it registers itself as the responsible driver for all

   682 of the devices found. The problem is, that an unmodified driver can

   683 not be told to ignore a device because it will be used for EtherCAT

   684 later. There must be a way to handle multiple devices of the same

   685 type, where one is reserved for EtherCAT, while the other is treated

   686 as an ordinary Ethernet device.

   687 

   688 For all this reasons, the author has decided that the only acceptable

   689 solution is to modify standard Ethernet drivers in a way that they

   690 keep their normal functionality, but gain the ability to treat one or

   691 more of the devices as EtherCAT-capable.

   692 

   693 Below are the advantages of this solution:

   694 

   695 \begin{itemize}

   696 \item No need to tell the standard drivers to ignore certain devices.

   697 \item One networking driver for EtherCAT and non-EtherCAT devices.

   698 \item No need to implement a network driver from scratch and running

   699   into issues, the former developers already solved.

   700 \end{itemize}

   701 

   702 The chosen approach has the following disadvantages:

   703 

   704 \begin{itemize}

   705 \item The modified driver gets more complicated, as it must handle

   706   EtherCAT and non-EtherCAT devices.

   707 \item Many additional case differentiations in the driver code.

   708 \item Changes and bugfixes on the standard drivers have to be ported

   709   to the Ether\-CAT-capable versions from time to time.

   710 \end{itemize}

   711 

   712 %------------------------------------------------------------------------------

   713 

   714 \subsection{Device Selection}

   715 \label{sec:seldev}

   716 

   717 After loading the master module, at least one EtherCAT-capable network

   718 driver module has to be loaded, that connects one of its devices to

   719 the master. To specify an EtherCAT device and the master to connect

   720 to, all EtherCAT-capable network driver modules should provide two

   721 module parameters:

   722 

   723 \begin{description}

   724 \item[ec\_device\_index] PCI device index of the device that is

   725   connected to the EtherCAT bus. If this parameter is left away, all

   726   devices found are treated as ordinary Ethernet devices. Default:

   727   $-1$

   728 \item[ec\_master\_index] Index of the master to connect to. Default:

   729   $0$

   730 \end{description}

   731 

   732 The following command loads the EtherCAT-capable RTL8139 device

   733 driver, telling it to handle the second device as an EtherCAT device

   734 and connecting it to the first master:

   735 

   736 \begin{lstlisting}

   737   host# `\textbf{modprobe ec\_8139too ec\_device\_index=1}`

   738 \end{lstlisting}

   739 

   740 Usually, this command does not have to be entered manually, but is

   741 called by the EtherCAT init script. See section~\ref{sec:init} for

   742 more information.

   743 

   744 %------------------------------------------------------------------------------

   745 

   746 \subsection{The Device Interface}

   747 \label{sec:ecdev}

   748 \index{Device interface}

   749 

   750 An anticipation to the section about the master module

   751 (section~\ref{sec:mastermod}) has to be made in order to understand

   752 the way, a network device driver module can connect a device to a

   753 specific EtherCAT master.

   754 

   755 The master module provides a ``device interface'' for network device

   756 drivers. To use this interface, a network device driver module must

   757 include the header

   758 \textit{devices/ecdev.h}\nomenclature{ecdev}{EtherCAT Device}, coming

   759 with the EtherCAT master code. This header offers a function interface

   760 for EtherCAT devices which is explained below. All functions of the

   761 device interface are named with the prefix \textit{ecdev}.

   762 

   763 \paragraph{Device Registration}

   764 

   765 A network device driver can connect a physical device to an EtherCAT

   766 master with the \textit{ecdev\_register()} function.

   767 

   768 \begin{lstlisting}[language=C]

   769   ec_device_t *ecdev_register(unsigned int master_index,

   770                               struct net_device *net_dev,

   771                               ec_isr_t isr,

   772                               struct module *module);

   773 \end{lstlisting}

   774 

   775 The first parameter \textit{master\_index} must be the index of the

   776 EtherCAT master to connect to (see section~\ref{sec:mastermod}),

   777 followed by \textit{net\_dev}, the pointer to the corresponding

   778 net\_device structure, which represents the network device to connect.

   779 The third parameter \textit{isr} must be a pointer to the interrupt

   780 service routine (ISR\index{ISR}) handling the device. The master will

   781 later execute the ISR in order to receive frames and to update the

   782 device status. The last parameter \textit{module} must be the pointer

   783 to the device driver module, which is usually accessible via the macro

   784 \textit{THIS\_MODULE} (see next paragraph). On success, the function

   785 returns a pointer to an \textit{ec\_device\_t} object, which has to be

   786 specified when calling further functions of the device interface.

   787 Therefore the device module has to store this pointer for future use.

   788 In error case, the \textit{ecdev\_register()} returns \textit{NULL},

   789 which means that the device could not be registered.  The reason for

   790 this is printed to \textit{syslog}\index{syslog}. In this case, the

   791 device module is supposed to abort the module initialisation and let

   792 the \textit{insmod} command fail.

   793 

   794 \paragraph{Implicit Dependencies}

   795 

   796 The reason for the module pointer has to be specified at device

   797 registration is a non-trivial one: The master has to know about the

   798 module, because there will be an implicit dependency between the

   799 device module and a later connected realtime module: When a realtime

   800 module connects to the master, the use count of the master module will

   801 be increased, so that the master module can not be unloaded for the

   802 time of the connection. This is reasonable, and so automatically done

   803 by the kernel. The kernel knows about this dependency, because the

   804 realtime module uses kernel symbols provided by the master module.

   805 Moreover it is mandatory, that the device module can be unloaded

   806 neither, because it is implicitely used by the realtime module, too.

   807 Unloading it would lead to a fatal situation, because the master would

   808 have no device to send and receive frames for the realtime module.

   809 This dependency can not be detected automatically, because the

   810 realtime module does not use any symbols of the device module.

   811 Therefore the master explicitly increments the use counter of the

   812 connected device module upon connection of a realtime module and

   813 decrements it, if the realtime module disconnects. In this manner, it

   814 is impossible to unload a device module while the master is in use.

   815 This is done with the kernel function pair \textit{try\_module\_get()}

   816 \index{try\_module\_get@\textit{try\_module\_get()}} and

   817 \textit{module\_put()} \index{module\_put@\textit{module\_put()}}. The

   818 first one increases the use count of a module and only fails, if the

   819 module is currenly being unloaded. The last one decreases the use

   820 count again and never fails. Both functions take a pointer to the

   821 module as their argument, which the device module therefore has to

   822 specify upon device registration.

   823 

   824 \paragraph{Device Unregistering}

   825 

   826 The unregistration of a device is usually done in the device module's

   827 cleanup function, by calling the \textit{ecdev\_unregister()} function

   828 and specifying the master index and a pointer to the device object

   829 again.

   830 

   831 \begin{lstlisting}[language=C]

   832   void ecdev_unregister(unsigned int master_index,

   833                         ec_device_t *device);

   834 \end{lstlisting}

   835 

   836 This function can fail too (if the master index is invalid, or the

   837 given device was not registered), but due to the fact, that this

   838 failure can not be dealt with appropriately, because the device module

   839 is unloading anyway, the failure code would not be of any interest. So

   840 the function has a void return value.

   841 

   842 \paragraph{Starting the Master}

   843 

   844 When a device has been initialized completely and is ready to send and

   845 receive frames, the master has to be notified about this by calling

   846 the \textit{ecdev\_start()} function.

   847 

   848 \begin{lstlisting}[language=C]

   849   int ecdev_start(unsigned int master_index);

   850 \end{lstlisting}

   851 

   852 The master will then enter ``Idle Mode'' and start scanning the bus

   853 (and possibly handling EoE slaves). Moreover it will make the bus

   854 accessible via Sysfs interface and react to user interactions. The

   855 function takes one parameter \textit{master\_index}, which has to be

   856 the same as at the call to \textit{ecdev\_register()}. The return

   857 value will be non-zero if the starting process failed. In this case

   858 the device module is supposed to abort the init sequence and make the

   859 init function return an error code.

   860 

   861 \paragraph{Stopping the Master}

   862 

   863 Before a device can be unregistered, the master has to be stopped by

   864 calling the \textit{ecdev\_stop()} function. It will stop processing

   865 messages of EoE slaves and leave ``Idle Mode''. The only parameter is

   866 \textit{master\_index}. This function can not fail.

   867 

   868 \begin{lstlisting}[language=C]

   869   void ecdev_stop(unsigned int master_index);

   870 \end{lstlisting}

   871 

   872 A subsequent call to \textit{ecdev\_unregister()} will now unregister

   873 the device savely.

   874 

   875 \paragraph{Receiving Frames}

   876 

   877 The interrupt service routine handling device events usually has a

   878 section where new frames are fetched from the hardware and forwarded

   879 to the kernel network stack via \textit{netif\_receive\_skb()}. For an

   880 EtherCAT-capable device, this has to be replaced by calling the

   881 \textit{ecdev\_receive()} function to forward the received data to the

   882 connected EtherCAT master instead.

   883 

   884 \begin{lstlisting}[language=C]

   885   void ecdev_receive(ec_device_t *device,

   886                      const void *data,

   887                      size_t size);

   888 \end{lstlisting}

   889 

   890 This function takes 3 arguments, a pointer to the device object

   891 (\textit{device}), a pointer to the received data, and the size of the

   892 received data. The data range has to include the Ethernet headers

   893 starting with the destination address and reach up to the last octet

   894 of EtherCAT data, excluding the FCS. Most network devices handle the

   895 FCS in hardware, so it is not seen by the driver code and therefore

   896 doesn't have to be cut off manually.

   897 

   898 \paragraph{Handling the Link Status}

   899 

   900 Information about the link status (i.~e. if there is a carrier signal

   901 detected on the physical port) is also important to the master. This

   902 information is usually gathered by the ISR and should be forwarded to

   903 the master by calling the \textit{ecdev\_link\_state()} function. The

   904 master then can react on this and warn the realtime module of a lost

   905 link.

   906 

   907 \begin{lstlisting}[language=C]

   908   void ecdev_link_state(ec_device_t *device,

   909                         uint8_t new_state);

   910 \end{lstlisting}

   911 

   912 The parameter \textit{device} has to be a pointer to the device object

   913 returned by \textit{ecdev\_\-register()}. With the second parameter

   914 \textit{new\_state}, the new link state is passed: 1, if the link went

   915 up, and 0, if it went down.

   916 

   917 %------------------------------------------------------------------------------

   918 

   919 \subsection{Patching Network Drivers}

   920 \label{sec:patching}

   921 \index{Network drivers}

   922 

   923 This section will demonstrate, how to make a standard Ethernet driver

   924 EtherCAT-capable. The below code examples are taken out of the

   925 modified RealTek RTL8139 driver coming with the EtherCAT master

   926 (\textit{devices/8139too.c}). The driver was originally developed by

   927 Donald Becker, and is currently maintained by Jeff Garzik.

   928 

   929 Unfortunately, there is no standard procedure to enable an Ethernet

   930 driver for use with the EtherCAT master, but there are a few common

   931 techniques, that are described in this section.

   932 

   933 \begin{enumerate}

   934 \item A first simple rule is, that \textit{netif\_*()}-calls must be

   935   strictly avoided for all EtherCAT devices. As mentioned before,

   936   EtherCAT devices have no connection to the network stack, and

   937   therefore must not call its interface functions.

   938 \item Another important thing is, that EtherCAT devices should be

   939   operated without interrupts. So any calls of registering interrupt

   940   handlers and enabling interrupts at hardware level must be avoided,

   941   too.

   942 \item The master does not use a new socket buffer for each send

   943   operation: Instead there is a fix one allocated on master

   944   initialization. This socket buffer is filled with an EtherCAT frame

   945   with every send operation and passed to the

   946   \textit{hard\_start\_xmit()} callback. For that it is necessary,

   947   that the socket buffer is not be freed by the network driver as

   948   usual.

   949 \end{enumerate}

   950 

   951 As mentioned before, the driver will handle both EtherCAT and ordinary

   952 Ethernet devices. This implies, that for each device-dependent

   953 operation, it has to be checked if an EtherCAT device is involved, or

   954 just an Ethernet device. For means of simplicity, this example driver

   955 will only handle one EtherCAT device. This makes the case

   956 differentiations easier.

   957 

   958 \paragraph{Global Variables}

   959 

   960 First of all, there have to be additional global variables declared,

   961 as shown in the listing:

   962 

   963 \begin{lstlisting}[language=C,numbers=left]

   964   static int ec_device_index = -1;

   965   static int ec_device_master_index = 0;

   966   static ec_device_t *rtl_ec_dev;

   967   struct net_device *rtl_ec_net_dev = NULL;

   968 \end{lstlisting}

   969 

   970 \begin{description}

   971 \item[\normalfont\textcircled{\tiny 1} -- \textcircled{\tiny 2}] To

   972   comply to the requirements for parameters of EtherCAT device modules

   973   described in section~\ref{sec:seldev}, there have to be additional

   974   parameter variables: \textit{ec\_\-device\_\-index} holds the index

   975   of the EtherCAT device and defaults to $-1$ (no EtherCAT device),

   976   while \textit{ec\_device\_master\_index} stores index of the master,

   977   the single device will be connected to. Default: $0$

   978 \item[\normalfont\textcircled{\tiny 3}] \textit{rtl\_ec\_dev} will be

   979   the pointer to the later registered RealTek EtherCAT device, which

   980   can be used as a parameter for device methods.

   981 \item[\normalfont\textcircled{\tiny 4}] \textit{rtl\_ec\_net\_dev} is

   982   a pointer to the \textit{net\_device} structure of the dedicated

   983   device and is set while scanning the PCI bus and finding the device

   984   with the specified index. This is done inside the

   985   \textit{pci\_module\_init()} function executed as the first thing on

   986   module loading.

   987 \end{description}

   988 

   989 \paragraph{Module Initialization}

   990 

   991 Below is the (shortened) coding of the device driver's module init

   992 function:

   993 

   994 \begin{lstlisting}[language=C,numbers=left]

   995   static int __init rtl8139_init_module(void)

   996   {

   997           if (pci_module_init(&rtl8139_pci_driver) < 0) {

   998                   printk(KERN_ERR "Failed to init PCI mod.\n");

   999                   goto out_return;

  1000           }

  1001 

  1002           if (rtl_ec_net_dev) {

  1003                   printk(KERN_INFO "Registering"

  1004                          " EtherCAT device...\n");

  1005                   if (!(rtl_ec_dev =

  1006                           ecdev_register(ec_device_master_index,

  1007                                          rtl_ec_net_dev,

  1008                                          rtl8139_interrupt,

  1009                                          THIS_MODULE))) {

  1010                           printk(KERN_ERR "Failed to reg."

  1011                                           " EtherCAT device!\n");

  1012                           goto out_unreg_pci;

  1013                   }

  1014 

  1015                   printk(KERN_INFO "Starting EtherCAT"

  1016                          " device...\n");

  1017                   if (ecdev_start(ec_device_master_index)) {

  1018                           printk(KERN_ERR "Failed to start"

  1019                                  " EtherCAT device!\n");

  1020                           goto out_unreg_ec;

  1021                   }

  1022           } else {

  1023                   printk(KERN_WARNING "No EtherCAT device"

  1024                          " registered!\n");

  1025           }

  1026 

  1027           return 0;

  1028 

  1029       out_unreg_ec:

  1030           ecdev_unregister(ec_device_master_index, rtl_ec_dev);

  1031       out_unreg_pci:

  1032           pci_unregister_driver(&rtl8139_pci_driver);

  1033       out_return:

  1034           return -1;

  1035   }

  1036 \end{lstlisting}

  1037 

  1038 \begin{description}

  1039 \item[\normalfont\textcircled{\tiny 3}] This call initializes all

  1040   RTL8139-compatible devices found on the pci bus. If a device with

  1041   index \textit{ec\_device\_index} is found, a pointer to its

  1042   \textit{net\_device} structure is stored in

  1043   \textit{rtl\_ec\_net\_dev} for later use (see next listings).

  1044 \item[\normalfont\textcircled{\tiny 8}] If the specified device was

  1045   found, \textit{rtl\_ec\_net\_dev} is non-zero.

  1046 \item[\normalfont\textcircled{\tiny 11}] The device is connected to

  1047   the specified master with a call to \textit{ecdev\_register()}. If

  1048   this fails, module loading is aborted.

  1049 \item[\normalfont\textcircled{\tiny 23}] The device registration was

  1050   successful and the master is started. This can fail, which aborts

  1051   module loading.

  1052 \item[\normalfont\textcircled{\tiny 29}] If no EtherCAT device was

  1053   found, a warning is output.

  1054 \end{description}

  1055 

  1056 \paragraph{Device Searching}

  1057 

  1058 During the PCI initialization phase, a variable \textit{board\_idx} is

  1059 increased for each RTL8139-compatible device found. The code below is

  1060 executed for each device:

  1061 

  1062 \begin{lstlisting}[language=C,numbers=left]

  1063   if (board_idx == ec_device_index) {

  1064           rtl_ec_net_dev = dev;

  1065           strcpy(dev->name, "ec0");

  1066   }

  1067 \end{lstlisting}

  1068 

  1069 \begin{description}

  1070 \item[\normalfont\textcircled{\tiny 1}] The device with the specified

  1071   index will be the EtherCAT device.

  1072 \end{description}

  1073 

  1074 \paragraph{Avoiding Device Registration}

  1075 

  1076 Later in the PCI initialization phase, the net\_devices get

  1077 registered. This has to be avoided for EtherCAT devices and so this is

  1078 a typical example for an EtherCAT case differentiation:

  1079 

  1080 \begin{lstlisting}[language=C,numbers=left]

  1081   if (dev != rtl_ec_net_dev) {

  1082           i = register_netdev(dev);

  1083           if (i) goto err_out;

  1084   }

  1085 \end{lstlisting}

  1086 

  1087 \begin{description}

  1088 \item[\normalfont\textcircled{\tiny 1}] If the current net\_device is

  1089   not the EtherCAT device, it is registered at the network stack.

  1090 \end{description}

  1091 

  1092 \paragraph{Avoiding Interrupt Registration}

  1093 

  1094 In the next two listings, there is an interrupt requested and the

  1095 device's interrupts are enabled. This also has to be encapsulated by

  1096 if-clauses, because interrupt operation is not wanted for EtherCAT

  1097 devices.

  1098 

  1099 \begin{lstlisting}[language=C,numbers=left]

  1100   if (dev != rtl_ec_net_dev) {

  1101           retval = request_irq(dev->irq, rtl8139_interrupt,

  1102                                SA_SHIRQ, dev->name, dev);

  1103           if (retval) return retval;

  1104   }

  1105 \end{lstlisting}

  1106 

  1107 \begin{lstlisting}[language=C,numbers=left]

  1108   if (dev != rtl_ec_net_dev) {

  1109           /* Enable all known interrupts by setting

  1110              the interrupt mask. */

  1111           RTL_W16(IntrMask, rtl8139_intr_mask);

  1112   }

  1113 \end{lstlisting}

  1114 

  1115 \paragraph{Frame Sending}

  1116 

  1117 The listing below shows an exerpt of the function representing the

  1118 \textit{hard\_start\_xmit()} callback of the net\_device.

  1119 

  1120 \begin{lstlisting}[language=C,numbers=left]

  1121   /* Note: the chip doesn't have auto-pad! */

  1122   if (likely(len < TX_BUF_SIZE)) {

  1123           if (len < ETH_ZLEN)

  1124                   memset(tp->tx_buf[entry], 0, ETH_ZLEN);

  1125           skb_copy_and_csum_dev(skb, tp->tx_buf[entry]);

  1126           if (dev != rtl_ec_net_dev) {

  1127                   dev_kfree_skb(skb);

  1128           }

  1129   } else {

  1130           if (dev != rtl_ec_net_dev) {

  1131                   dev_kfree_skb(skb);

  1132           }

  1133           tp->stats.tx_dropped++;

  1134           return 0;

  1135   }