%------------------------------------------------------------------------------
%
% IgH EtherCAT Master Documentation
%
% $Id$
%
%------------------------------------------------------------------------------
\documentclass[a4paper,12pt,BCOR6mm,bibtotoc,idxtotoc]{scrbook}
\usepackage[latin1]{inputenc}
\usepackage[automark,headsepline]{scrpage2}
\usepackage{graphicx}
\usepackage{makeidx}
\usepackage[refpage]{nomencl}
\usepackage{listings}
\usepackage{svn}
\usepackage{textcomp}
\usepackage{url}
\usepackage[pdfpagelabels,plainpages=false]{hyperref}
\setlength{\parskip}{0.8ex plus 0.8ex minus 0.5ex}
\setlength{\parindent}{0mm}
\setcounter{secnumdepth}{\subsubsectionlevel}
\DeclareFontShape{OT1}{cmtt}{bx}{n}
{
<5><6><7><8><9><10><10.95><12><14.4><17.28><20.74><24.88>cmttb10
}{}
\lstset{basicstyle=\ttfamily\small,numberstyle=\tiny,aboveskip=4mm,
belowskip=2mm,gobble=2,escapechar=`}
\renewcommand\lstlistlistingname{List of Listings}
% Workaround for lstlistoflistings bug
\makeatletter% --> De-TeX-FAQ
\renewcommand*{\lstlistoflistings}{%
\begingroup
\if@twocolumn
\@restonecoltrue\onecolumn
\else
\@restonecolfalse
\fi
\lol@heading
\setlength{\parskip}{\z@}%
\setlength{\parindent}{\z@}%
\setlength{\parfillskip}{\z@ \@plus 1fil}%
\@starttoc{lol}%
\if@restonecol\twocolumn\fi
\endgroup
}
\makeatother% --> \makeatletter
\renewcommand\nomname{Glossary}
\newcommand{\IgH}{\raisebox{-0.7667ex}
{\includegraphics[height=2.2ex]{images/ighsign}}}
\SVN $Date$
\SVN $Revision$
\newcommand{\masterversion}{1.1.1}
\makeindex
\makenomenclature
%------------------------------------------------------------------------------
\begin{document}
\pagenumbering{roman}
\pagestyle{empty}
\begin{titlepage}
\begin{center}
\rule{\textwidth}{1.5mm}
{\Huge\bf IgH \includegraphics[height=2.4ex]{images/ethercat}
Master \masterversion\\[1ex]
Documentation}
\vspace{1ex}
\rule{\textwidth}{1.5mm}
\vspace{\fill}
{\Large Florian Pose, \url{fp@igh-essen.com}\\[1ex]
Ingenieurgemeinschaft \IgH}
\vspace{\fill}
{\Large Essen, \SVNDate\\[1ex]
Revision \SVNRevision}
\end{center}
\end{titlepage}
%------------------------------------------------------------------------------
\tableofcontents
\listoftables
\listoffigures
\lstlistoflistings
%------------------------------------------------------------------------------
\newpage
\pagestyle{scrheadings}
\section*{Conventions}
\addcontentsline{toc}{section}{Conventions}
\markleft{Conventions}
The following typographic conventions are used:
\begin{itemize}
\item \textit{Italic face} is used for newly introduced terms, file
names, parameter names and in-text source code elements.
\item \texttt{Typewriter face} is used for code examples and
command line output.
\item \texttt{\textbf{Bold typewriter face}} is used for user input in
command lines.
\end{itemize}
Data values and addresses are specified as hexadecimal values with the
prefix 0x. Example: 0x88A4. Unless otherwise noted, address values are
specified as byte addresses.
Concerning bit operations, the phrase ``setting a bit'', stands for
setting the bit to $1$, ``clearing a bit'' means setting it to $0$,
respectively.
Function names are always printed with parentheses, but without
parameters. So, if a function \textit{ecrt\_request\_master()} has
empty parentheses, this does not mean, that it has no parameters.
If shell commands have to be entered, this is marked by a prompt:
\begin{lstlisting}[gobble=2]
`\$`
\end{lstlisting}
Further, if a shell command has to be entered as the superuser, the
prompt ends with a mesh:
\begin{lstlisting}[gobble=2]
#
\end{lstlisting}
%------------------------------------------------------------------------------
\chapter{The IgH EtherCAT Master}
\label{chapter:master}
\pagenumbering{arabic}
This section will first introduce the master's general features and
the concepts used for master development and will then explain the
master's general architecture and offer details of the different
modules. In addition, it will cover state machine definitions, mailbox
protocol implementation and the user space interface. The last section
will deal with timing aspects.
%------------------------------------------------------------------------------
\section{Feature Summary}
\label{sec:summary}
\index{Master!Features}
The list below gives a short summary of the features of the
implemented EtherCAT master.
\begin{itemize}
\item The master runs as a kernel module for Linux 2.6.
\item It comes with EtherCAT-capable network driver for RealTek
RTL8139 (and compatible) network interface cards.
\begin{itemize}
\item The Ethernet hardware is operated without interrupts.
\item Drivers for additional Ethernet hardware can easily be
implemented due to a common device interface provided by the
master.
\end{itemize}
\item The master module supports multiple EtherCAT masters on one
machine.
\item The master code supports any Linux realtime extension through
its independent architecture.
\begin{itemize}
\item RTAI\nomenclature{RTAI}{RealTime Application Interface},
ADEOS\nomenclature{ADEOS}{Adaptive Domain Environment for
Operating Systems}, etc.
\item It runs well even without realtime extensions.
\end{itemize}
\item Common ``realtime interface'' for modules, that want to use
EtherCAT functionality.
\begin{itemize}
\item Synchronous and asynchronous sending and receiving of frames
is supported.
\item Avoidance of unnecessary copy operations for process data.
\end{itemize}
\item \textit{Domains} are introduced, to allow grouping of process
data objects.
\begin{itemize}
\item Handling of multiple domains with different sampling rates.
\item Automatic calculation of process data mapping, FMMU and sync manager
configuration within each domain.
\end{itemize}
\item Communication through serveral finite state machines.
\begin{itemize}
\item Bus monitoring possible during realtime operation.
\item Automatic reconfiguration of slaves on bus power failure
during realtime operation.
\item Controlling of single slaves during realtime operation.
\end{itemize}
\item Master idle mode.
\begin{itemize}
\item Automatic scanning of slaves upon topology changes.
\item Bus visualisation and EoE processing without a realtime module
connected.
\end{itemize}
\item Implementation of the CANopen-over-EtherCAT (CoE) protocol.
\begin{itemize}
\item Configuration of CoE-capable slaves via Sdo interface.
\end{itemize}
\item Implementation of the Ethernet-over-EtherCAT (EoE) protocol.
\begin{itemize}
\item Each master creates virtual network devices that are
automatically coupled to EoE-cap\-able slaves found.
\item This implementation natively supports either a switched or a
routed EoE network architecture.
\end{itemize}
\item User space interface via the System Filesystem
(Sysfs)\nomenclature{Sysfs}{System Filesystem}.
\begin{itemize}
\item User space tool for bus visualisation.
\item Slave E$^2$PROM image reading and writing.
\end{itemize}
\item Seamless system integration though LSB\nomenclature{LSB}{Linux
Standard Base} compliance.
\begin{itemize}
\item Master and network device configuration via Sysconfig files.
\item Linux Standard Base compatible init script for master control.
\end{itemize}
\item Virtual read-only network interface for monitoring and debugging
purposes.
\end{itemize}
%------------------------------------------------------------------------------
\section{License}
\label{sec:license}
The master code is released under the terms and conditions of the GNU
General Public License\index{GPL} \cite{gpl} (version 2). Other
developers, that want to use EtherCAT with Linux systems, are invited
to use the master code or even participate on development.
%------------------------------------------------------------------------------
\section{General Master Architecture}
\label{sec:masterarch}
\index{Master!Architecture}
The EtherCAT master is integrated into the Linux 2.6 kernel. This was
an early design decision, which has been made for serveral reasons:
\begin{itemize}
\item Kernel code has significantly better realtime characteristics,
i.~e. less jitter than user space code. It was foreseeable, that a
fieldbus master has a lot of cyclic work to do. Cyclic work is
usually triggered by timer interrupts inside the kernel. The
execution delay of a function that processes timer interrupts is
less, when it resides in kernel space, because there is no need of
time-consuming context switches to a user space process.
\item It was also foreseeable, that the master code has to directly
communicate with the Ethernet hardware. This has to be done in the
kernel anyway (through network device drivers), which is one more
reason for the master code being in kernel space.
\end{itemize}
A general overview of the master architecture can be seen in
figure~\ref{fig:masterarch}.
\begin{figure}[htbp]
\centering
\includegraphics[width=.9\textwidth]{images/masterarch}
\caption{Master architecture}
\label{fig:masterarch}
\end{figure}
\paragraph{Master Module}
\index{Master module}
The EtherCAT master mainly consists of the master module, containing
one or more EtherCAT masters (section~\ref{sec:mastermod}), the
``Device Interface'' (section~\ref{sec:ecdev}) and the ``Realtime
Interface'' (section~\ref{sec:ecrt}).
\paragraph{Device Modules}
\index{Device modules}
Furthermore there are EtherCAT-capable network device driver
modules\index{Device modules}, that connect to the EtherCAT master via
the device interface. These modified network drivers can handle both
network devices used for EtherCAT operation and ``normal'' Ethernet
devices. The common case is, that the master module offers a single
EtherCAT master: An EtherCAT-capable network device driver module
connects one network device to this master, that is now able to send
and receive EtherCAT frames, while all other network devices handled
by the network driver get connected to the kernel's network stack as
usual.
\paragraph{Realtime Modules}
A ``realtime module''\index{Realtime module} is a kernel module, that
uses the EtherCAT master for cyclic exchange of process data with
EtherCAT slaves. Realtime modules are not part of the EtherCAT master
code\footnote{Although there are serveral examples provided in the
\textit{examples} directory, see chapter~\ref{chapter:usage} for
more information}, so anybody wanting to use the master has to write
one. A realtime module can ``request'' a master through the realtime
interface. If this succeeds, the module has the control over the
master. It can now configure slaves and set up a process data image
(see section~\ref{sec:processdata}) for cyclic exchange. This cyclic
code has to be provided by the realtime module, so it is in hands of
the developer, which mechanism to use for this. Moreover he has to
decide, whether or not using a Linux realtime extension.
\paragraph{Why ``Realtime'' Module?}
The name shall not imply, that a linux realtime extension is
mandatory: The master runs well even without realtime extensions, as
section~\ref{sec:mini} shows. However, the code using the master is
time-critical, because process data IO has to be done in cyclic work.
Some EtherCAT slaves support watchdog units, that stop driving the
outputs when process data was not exchanged for some time. So the
names ``realtime interface'' and ``realtime module'' are quite
appropriate.
%------------------------------------------------------------------------------
\subsection{Handling of Process Data}
\label{sec:processdata}
\paragraph{Process Data Image}
\index{Process data}
The slaves offer their inputs and outputs by presenting the master
so-called ``Process Data Objects'' (Pdos\index{Pdo}). The available
Pdos can be determined by reading out the slave's TXPDO and RXPDO
E$^2$PROM categories. The realtime module can register the Pdos for
data exchange during cyclic operation. The sum of all registered Pdos
defines the ``process data image'', which is exchanged via the
``Logical ReadWrite'' datagrams introduced
in~\cite[section~5.4.2.4]{dlspec}.
\paragraph{Process Data Domains}
\index{Domain}
The process data image can be easily managed by creating co-called
``domains'', which group Pdos and allocate the datagrams needed to
exchange them. Domains are mandatory for process data exchange, so
there has to be at least one. They were introduced for the following
reasons:
\begin{itemize}
\item The maximum size of a ``Logical ReadWrite'' datagram is limited
due to the limited size of an Ethernet frame: The maximum data size
is the Ethernet data field size minus the EtherCAT frame header,
EtherCAT datagram header and EtherCAT datagram footer: $1500 - 2 -
12 - 2 = 1484$ octets. If the size of the process data image exceeds
this limit, multiple frames have to be sent, and the image has to be
partitioned for the use of multiple datagrams. A domain manages this
automatically.
\item Not every Pdo has to be exchanged with the same frequency: The
values of Pdos can vary slowly over time (for example temperature
values), so exchanging them with a high frequency would just waste
bus bandwidth. For this reason, multiple domains can be created, to
group different Pdos and so allow separate exchange.
\end{itemize}
There is no upper limit for the number of domains, but each domain
occupies one FMMU in each slave involved, so the maximum number of
domains is also limited by the slaves' capabilities.
\paragraph{FMMU Configuration}
\index{FMMU!Configuration}
A realtime module can register Pdos for process data exchange. Every
Pdo is part of a memory area in the slave's physical memory, that is
protected by a sync manager \cite[section~6.7]{dlspec} for
synchronized access. In order to make a sync manager react on a
datagram accessing its memory, it is necessary to access the last byte
covered by the sync manager. Otherwise the sync manager will not react
on the datagram and no data will be exchanged. That is why the whole
synchronized memory area has to be included into the process data
image: For example, if a certain Pdo of a slave is registered for
exchange with a certain domain, one FMMU will be configured to map the
complete sync-manager-protected memory, the Pdo resides in. If a
second Pdo of the same slave is registered for process data exchange
within the same domain, and this Pdo resides in the same
sync-manager-protected memory as the first Pdo, the FMMU configuration
is not touched, because the appropriate memory is already part of the
domain's process data image. If the second Pdo belongs to another
sync-manager-protected area, this complete area is also included into
the domains process data image. See figure~\ref{fig:fmmus} for an
overview, how FMMU's are configured to map physical memory to logical
process data images.
\begin{figure}[htbp]
\centering
\includegraphics[width=\textwidth]{images/fmmus}
\caption{FMMU configuration for serveral domains}
\label{fig:fmmus}
\end{figure}
\paragraph{Process Data Pointers}
The figure also demonstrates the way, the realtime module can access the
exchanged process data: At Pdo registration, the realtime module has
to provide the address of a process data pointer. Upon calculation of
the domain image and allocation of process data memory, this pointer
is redirected to the appropriate location inside the domain's process
data memory and can later be easily dereferenced by the module code.
%------------------------------------------------------------------------------
\subsection{Operation Modes}
\index{Master modes}
The EtherCAT master has serveral modes of operation:
\begin{description}
\item[Orphaned Mode] This mode takes effect, when the master has no
EtherCAT-capable network device connected. No bus communication is
possible, so this mode is not of further interest.
\item[Idle Mode]\index{Idle mode} takes effect when the master is
unused (i.~e. there is no realtime module, that reserved the
master). In this case, the master has the ability to scan the bus by
itsself and generously allow bus access from user space. This mode
is meant for maintenance and visualisation.
\item[Operation Mode]\index{Operation mode} The master is reserved for
exclusive access by a realtime module. In this mode, the master is
adjusted for availability and monitoring. Access from user space is
very restrictive and mostly limited to reading direction.
\end{description}
Figure~\ref{fig:modes} shows the three modes and the possible mode
transitions.
\begin{figure}[htbp]
\centering
\includegraphics[width=.9\textwidth]{images/modes}
\caption{Master modes and transitions}
\label{fig:modes}
\end{figure}
\subsubsection{Idle Mode}
\index{Idle mode}
The master enters idle mode upon connection of a device module (see
section~\ref{sec:device}) or releasing by a realtime module. The
master owns a kernel workqueue and a suitable work structure, which is
used to cyclically process the ``Idle state machine'' (see
section~\ref{sec:fsm-idle}). This state machine automatically scans
the bus for slaves (and re-scans upon topology changes), configures
slaves for idle operation and executes pending operations from the
user space interface (for example E$^2$PROM writing). On device
disconnection or realtime request, the idle mode is stopped by
cancelling the work and flushing the workqueue.
\subsubsection{Operation Mode}
\index{Operation mode}
Operation mode is entered when a realtime module requests the master.
The idle mode is stopped and the bus is scanned by getting the number
of slaves and executing the ``Slave scan state machine'' (see
section~\ref{sec:fsm-scan}) for each slave. The master is now ready to
create domains and accept Pdo registrations and slave configurations.
After that, cyclic communication can be done by the realtime module.
\paragraph{Master Phases}
Every realtime module should use the master in three phases:
\begin{enumerate}
\item \textit{Startup} - The master is requested and the bus is
validated. Domains are created and Pdos are registered. Slave
configurations are applied.
\item \textit{Operation} - Cyclic code is run, process data is
exchanged and the master state machine is executed.
\item \textit{Shutdown} - Cyclic code is stopped and the master
is released.
\end{enumerate}
%------------------------------------------------------------------------------
\section{Device Modules}
\label{sec:device}
\index{Device modules}
Device modules are network device driver modules that handle Ethernet
devices, which the master can use to connect to an EtherCAT bus.
Section~\ref{sec:networkdrivers} offers an overview of general Linux
network driver modules, while section~\ref{sec:requirements} will show
the requirements to an EtherCAT-enabled network driver. Finally,
sections~\ref{sec:seldev} to~\ref{sec:patching} show how to fulfill
these requirements and implement such a driver module.
%------------------------------------------------------------------------------
\subsection{Network Driver Basics}
\label{sec:networkdrivers}
\index{Network drivers}
EtherCAT relies on Ethernet hardware and the master needs a physical
Ethernet device to communicate with the bus. Therefore it is necessary
to understand how Linux handles network devices and their drivers,
respectively.
\paragraph{Tasks of a Network Driver}
Network device drivers handle the lower two layers of the OSI model,
that is the physical layer and the data-link layer. A network device
itself natively handles the physical layer issues: It represents the
hardware to connect to the medium and to send and receive data in the
way, the physical layer protocol describes. The network device driver
is responsible for getting data from the kernel's networking stack and
forwarding it to the hardware, that does the physical transmission.
If data is received by the hardware respectively, the driver is
notified (usually by means of an interrupt) and has to read the data
from the hardware memory and forward it to the network stack. There
are a few more tasks, a network device driver has to handle, including
queue control, statistics and device dependent features.
\paragraph{Driver Startup}
Usually, a driver searches for compatible devices on module loading.
For PCI drivers, this is done by scanning the PCI bus and checking for
known device IDs. If a device is found, data structures are allocated
and the device is taken into operation.
\paragraph{Interrupt Operation}
\index{Interrupt}
A network device usually provides a hardware interrupt that is used to
notify the driver of received frames and success of transmittion, or
errors, respectively. The driver has to register an interrupt service
routine (ISR\index{ISR}\nomenclature{ISR}{Interrupt Service Routine}),
that is executed each time, the hardware signals such an event. If the
interrupt was thrown by the own device (multiple devices can share one
hardware interrupt), the reason for the interrupt has to be determined
by reading the device's interrupt register. For example, if the flag
for received frames is set, frame data has to be copied from hardware
to kernel memory and passed to the network stack.
\paragraph{The net\_device structure}
\index{net\_device}
The driver registers a \textit{net\_device} structure for each device
to communicate with the network stack and to create a ``network
interface''. In case of an Ethernet driver, this interface appears as
\textit{ethX}, where X is a number assigned by the kernel on
registration. The \textit{net\_device} structure receives events
(either from user space or from the network stack) via serveral
callbacks, which have to be set before registration. Not every
callback is mandatory, but for reasonable operation the ones below are
needed in any case:
\begin{description}
\item[int (*open)(struct net\_device *)] This function is called when
network communication has to be started, for example after a command
\textit{ifconfig ethX up} from user space. Frame reception has to be
enabled by the driver.
\item[int (*stop)(struct net\_device *)] The purpose of this callback
is to ``close'' the device, i.~e. make the hardware stop receiving
frames.
\item[int (*hard\_start\_xmit)(struct sk\_buff *, struct net\_device
*)] This function is cal\-led for each frame that has to be
transmitted. The network stack passes the frame as a pointer to an
\textit{sk\_buff} structure (``socket buffer''\index{Socket buffer},
see below), which has to be freed after sending.
\item[struct net\_device\_stats *(*get\_stats)(struct net\_device *)]
This call has to return a pointer to the device's
\textit{net\_device\_stats} structure, which permanently has to be
filled with frame statistics. This means, that everytime a frame is
received, sent, or an error happened, the appropriate counter in
this structure has to be increased.
\end{description}
The actual registration is done with the \textit{register\_netdev()}
call, unregistering is done with \textit{unregister\_netdev()}.
\paragraph{The netif Interface}
\index{netif}
All other communication in the direction interface $\to$ network stack
is done via the \textit{netif\_*} calls. For example, on successful
device opening, the network stack has to be notified, that it can now
pass frames to the interface. This is done by calling
\textit{netif\_start\_queue()}. After this call, the
\textit{hard\_start\_xmit()} callback can be called by the network
stack. Furthermore a network driver usually manages a frame
transmission queue. If this gets filled up, the network stack has to
be told to stop passing further frames for a while. This happens with
a call to \textit{netif\_stop\_queue()}. If some frames have been
sent, and there is enough space again to queue new frames, this can be
notified with \textit{netif\_wake\_queue()}. Another important call is
\textit{netif\_receive\_skb()}\footnote{This function is part of the
NAPI (``New API''), that replaces the ``old'' kernel 2.4 technique
for interfacing to the network stack (with \textit{netif\_rx()}).
NAPI is a technique to improve network performance on Linux. Read
more in
http://www.cyberus.ca/\textasciitilde{}hadi/usenix-paper.tgz}: It
passes a frame to the network stack, that was just received by the
device. Frame data has to be packed into a so-called ``socket
buffer'' for that (see below).
\paragraph{Socket Buffers}
\index{Socket buffer}
Socket buffers are the basic data type for the whole network stack.
They serve as containers for network data and are able to quickly add
data headers and footers, or strip them off again. Therefore a socket
buffer consists of an allocated buffer and serveral pointers that mark
beginning of the buffer (\textit{head}), beginning of data
(\textit{data}), end of data (\textit{tail}) and end of buffer
(\textit{end}). In addition, a socket buffer holds network header
information and (in case of received data) a pointer to the
\textit{net\_device}, it was received on. There exist functions that
create a socket buffer (\textit{dev\_alloc\_skb()}), add data either
from front (\textit{skb\_push()}) or back (\textit{skb\_put()}),
remove data from front (\textit{skb\_pull()}) or back
(\textit{skb\_trim()}), or delete the buffer (\textit{kfree\_skb()}).
A socket buffer is passed from layer to layer, and is freed by the
layer that uses it the last time. In case of sending, freeing has to
be done by the network driver.
%------------------------------------------------------------------------------
\subsection{EtherCAT Network Drivers}
\label{sec:requirements}
There are a few requirements for Ethernet network devices to function
as EtherCAT devices, when connected to an EtherCAT bus.
\paragraph{Dedicated Interfaces}
For performance and realtime purposes, the EtherCAT master needs
direct and exclusive access to the Ethernet hardware. This implies
that the network device must not be connected to the kernel's network
stack as usual, because the kernel would try to use it as an ordinary
Ethernet device.
\paragraph{Interrupt-less Operation}
\index{Interrupt}
EtherCAT frames travel through the logical EtherCAT ring and are then
sent back to the master. Communication is highly deterministic: A
frame is sent and will be received again after a constant time.
Therefore, there is no need to notify the driver about frame
reception: The master can instead query the hardware for received
frames.
Figure~\ref{fig:interrupt} shows two workflows for cyclic frame
transmittion and reception with and without interrupts.
\begin{figure}[htbp]
\centering
\includegraphics[width=.8\textwidth]{images/interrupt}
\caption{Interrupt Operation versus Interrupt-less Operation}
\label{fig:interrupt}
\end{figure}
In the left workflow ``Interrupt Operation'', the data from the last
cycle is first processed and a new frame is assembled with new
datagrams, which is then sent. The cyclic work is done for now.
Later, when the frame is received again by the hardware, an interrupt
is triggered and the ISR is executed. The ISR will fetch the frame
data from the hardware and initiate the frame dissection: The
datagrams will be processed, so that the data is ready for processing
in the next cycle.
In the right workflow ``Interrupt-less Operation'', there is no
hardware interrupt enabled. Instead, the hardware will be polled by
the master by executing the ISR. If the frame has been received in the
meantime, it will be dissected. The situation is now the same as at
the beginning of the left workflow: The received data is processed and
a new frame is assembled and sent. There is nothing to do for the rest
of the cycle.
The interrupt-less operation is desirable, because there is simply no
need for an interrupt. Moreover hardware interrupts are not conducive
in improving the driver's realtime behaviour: Their undeterministic
incidences contribute to increasing the jitter. Besides, if a realtime
extension (like RTAI) is used, some additional effort would have to be
made to priorize interrupts.
\paragraph{Ethernet and EtherCAT Devices}
Another issue lies in the way Linux handles devices of the same type.
For example, a PCI\nomenclature{PCI}{Peripheral Component
Interconnect, Computer Bus} driver scans the PCI bus for devices it
can handle. Then it registers itself as the responsible driver for all
of the devices found. The problem is, that an unmodified driver can
not be told to ignore a device because it will be used for EtherCAT
later. There must be a way to handle multiple devices of the same
type, where one is reserved for EtherCAT, while the other is treated
as an ordinary Ethernet device.
For all this reasons, the author has decided that the only acceptable
solution is to modify standard Ethernet drivers in a way that they
keep their normal functionality, but gain the ability to treat one or
more of the devices as EtherCAT-capable.
Below are the advantages of this solution:
\begin{itemize}
\item No need to tell the standard drivers to ignore certain devices.
\item One networking driver for EtherCAT and non-EtherCAT devices.
\item No need to implement a network driver from scratch and running
into issues, the former developers already solved.
\end{itemize}
The chosen approach has the following disadvantages:
\begin{itemize}
\item The modified driver gets more complicated, as it must handle
EtherCAT and non-EtherCAT devices.
\item Many additional case differentiations in the driver code.
\item Changes and bugfixes on the standard drivers have to be ported
to the Ether\-CAT-capable versions from time to time.
\end{itemize}
%------------------------------------------------------------------------------
\subsection{Device Selection}
\label{sec:seldev}
After loading the master module, at least one EtherCAT-capable network
driver module has to be loaded, that connects one of its devices to
the master. To specify an EtherCAT device and the master to connect
to, all EtherCAT-capable network driver modules should provide two
module parameters:
\begin{description}
\item[ec\_device\_index] PCI device index of the device that is
connected to the EtherCAT bus. If this parameter is left away, all
devices found are treated as ordinary Ethernet devices. Default:
$-1$
\item[ec\_master\_index] Index of the master to connect to. Default:
$0$
\end{description}
The following command loads the EtherCAT-capable RTL8139 device
driver, telling it to handle the second device as an EtherCAT device
and connecting it to the first master:
\begin{lstlisting}
# `\textbf{modprobe ec\_8139too ec\_device\_index=1}`
\end{lstlisting}
Usually, this command does not have to be entered manually, but is
called by the EtherCAT init script. See section~\ref{sec:init} for
more information.
%------------------------------------------------------------------------------
\subsection{The Device Interface}
\label{sec:ecdev}
\index{Device interface}
An anticipation to the section about the master module
(section~\ref{sec:mastermod}) has to be made in order to understand
the way, a network device driver module can connect a device to a
specific EtherCAT master.
The master module provides a ``device interface'' for network device
drivers. To use this interface, a network device driver module must
include the header
\textit{devices/ecdev.h}\nomenclature{ecdev}{EtherCAT Device}, coming
with the EtherCAT master code. This header offers a function interface
for EtherCAT devices which is explained below. All functions of the
device interface are named with the prefix \textit{ecdev}.
\paragraph{Device Registration}
A network device driver can connect a physical device to an EtherCAT
master with the \textit{ecdev\_register()} function.
\begin{lstlisting}[language=C]
ec_device_t *ecdev_register(unsigned int master_index,
struct net_device *net_dev,
ec_isr_t isr,
struct module *module);
\end{lstlisting}
The first parameter \textit{master\_index} must be the index of the
EtherCAT master to connect to (see section~\ref{sec:mastermod}),
followed by \textit{net\_dev}, the pointer to the corresponding
net\_device structure, which represents the network device to connect.
The third parameter \textit{isr} must be a pointer to the interrupt
service routine (ISR\index{ISR}) handling the device. The master will
later execute the ISR in order to receive frames and to update the
device status. The last parameter \textit{module} must be the pointer
to the device driver module, which is usually accessible via the macro
\textit{THIS\_MODULE} (see next paragraph). On success, the function
returns a pointer to an \textit{ec\_device\_t} object, which has to be
specified when calling further functions of the device interface.
Therefore the device module has to store this pointer for future use.
In error case, the \textit{ecdev\_register()} returns \textit{NULL},
which means that the device could not be registered. The reason for
this is printed to \textit{syslog}\index{syslog}. In this case, the
device module is supposed to abort the module initialisation and let
the \textit{insmod} command fail.
\paragraph{Implicit Dependencies}
The reason for the module pointer has to be specified at device
registration is a non-trivial one: The master has to know about the
module, because there will be an implicit dependency between the
device module and a later connected realtime module: When a realtime
module connects to the master, the use count of the master module will
be increased, so that the master module can not be unloaded for the
time of the connection. This is reasonable, and so automatically done
by the kernel. The kernel knows about this dependency, because the
realtime module uses kernel symbols provided by the master module.
Moreover it is mandatory, that the device module can be unloaded
neither, because it is implicitely used by the realtime module, too.
Unloading it would lead to a fatal situation, because the master would
have no device to send and receive frames for the realtime module.
This dependency can not be detected automatically, because the
realtime module does not use any symbols of the device module.
Therefore the master explicitly increments the use counter of the
connected device module upon connection of a realtime module and
decrements it, if the realtime module disconnects. In this manner, it
is impossible to unload a device module while the master is in use.
This is done with the kernel function pair \textit{try\_module\_get()}
\index{try\_module\_get@\textit{try\_module\_get()}} and
\textit{module\_put()} \index{module\_put@\textit{module\_put()}}. The
first one increases the use count of a module and only fails, if the
module is currenly being unloaded. The last one decreases the use
count again and never fails. Both functions take a pointer to the
module as their argument, which the device module therefore has to
specify upon device registration.
\paragraph{Device Unregistering}
The unregistration of a device is usually done in the device module's
cleanup function, by calling the \textit{ecdev\_unregister()} function
and specifying the master index and a pointer to the device object
again.
\begin{lstlisting}[language=C]
void ecdev_unregister(unsigned int master_index,
ec_device_t *device);
\end{lstlisting}
This function can fail too (if the master index is invalid, or the
given device was not registered), but due to the fact, that this
failure can not be dealt with appropriately, because the device module
is unloading anyway, the failure code would not be of any interest. So
the function has a void return value.
\paragraph{Starting the Master}
When a device has been initialized completely and is ready to send and
receive frames, the master has to be notified about this by calling
the \textit{ecdev\_start()} function.
\begin{lstlisting}[language=C]
int ecdev_start(unsigned int master_index);
\end{lstlisting}
The master will then enter ``Idle Mode'' and start scanning the bus
(and possibly handling EoE slaves). Moreover it will make the bus
accessible via Sysfs interface and react to user interactions. The
function takes one parameter \textit{master\_index}, which has to be
the same as at the call to \textit{ecdev\_register()}. The return
value will be non-zero if the starting process failed. In this case
the device module is supposed to abort the init sequence and make the
init function return an error code.
\paragraph{Stopping the Master}
Before a device can be unregistered, the master has to be stopped by
calling the \textit{ecdev\_stop()} function. It will stop processing
messages of EoE slaves and leave ``Idle Mode''. The only parameter is
\textit{master\_index}. This function can not fail.
\begin{lstlisting}[language=C]
void ecdev_stop(unsigned int master_index);
\end{lstlisting}
A subsequent call to \textit{ecdev\_unregister()} will now unregister
the device savely.
\paragraph{Receiving Frames}
The interrupt service routine handling device events usually has a
section where new frames are fetched from the hardware and forwarded
to the kernel network stack via \textit{netif\_receive\_skb()}. For an
EtherCAT-capable device, this has to be replaced by calling the
\textit{ecdev\_receive()} function to forward the received data to the
connected EtherCAT master instead.
\begin{lstlisting}[language=C]
void ecdev_receive(ec_device_t *device,
const void *data,
size_t size);
\end{lstlisting}
This function takes 3 arguments, a pointer to the device object
(\textit{device}), a pointer to the received data, and the size of the
received data. The data range has to include the Ethernet headers
starting with the destination address and reach up to the last octet
of EtherCAT data, excluding the FCS. Most network devices handle the
FCS in hardware, so it is not seen by the driver code and therefore
doesn't have to be cut off manually.
\paragraph{Handling the Link Status}
Information about the link status (i.~e. if there is a carrier signal
detected on the physical port) is also important to the master. This
information is usually gathered by the ISR and should be forwarded to
the master by calling the \textit{ecdev\_link\_state()} function. The
master then can react on this and warn the realtime module of a lost
link.
\begin{lstlisting}[language=C]
void ecdev_link_state(ec_device_t *device,
uint8_t new_state);
\end{lstlisting}
The parameter \textit{device} has to be a pointer to the device object
returned by \textit{ecdev\_\-register()}. With the second parameter
\textit{new\_state}, the new link state is passed: 1, if the link went
up, and 0, if it went down.
%------------------------------------------------------------------------------
\subsection{Patching Network Drivers}
\label{sec:patching}
\index{Network drivers}
This section will demonstrate, how to make a standard Ethernet driver
EtherCAT-capable. The below code examples are taken out of the
modified RealTek RTL8139 driver coming with the EtherCAT master
(\textit{devices/8139too.c}). The driver was originally developed by
Donald Becker, and is currently maintained by Jeff Garzik.
Unfortunately, there is no standard procedure to enable an Ethernet
driver for use with the EtherCAT master, but there are a few common
techniques, that are described in this section.
\begin{enumerate}
\item A first simple rule is, that \textit{netif\_*()}-calls must be
strictly avoided for all EtherCAT devices. As mentioned before,
EtherCAT devices have no connection to the network stack, and
therefore must not call its interface functions.
\item Another important thing is, that EtherCAT devices should be
operated without interrupts. So any calls of registering interrupt
handlers and enabling interrupts at hardware level must be avoided,
too.
\item The master does not use a new socket buffer for each send
operation: Instead there is a fix one allocated on master
initialization. This socket buffer is filled with an EtherCAT frame
with every send operation and passed to the
\textit{hard\_start\_xmit()} callback. For that it is necessary,
that the socket buffer is not be freed by the network driver as
usual.
\end{enumerate}
As mentioned before, the driver will handle both EtherCAT and ordinary
Ethernet devices. This implies, that for each device-dependent
operation, it has to be checked if an EtherCAT device is involved, or
just an Ethernet device. For means of simplicity, this example driver
will only handle one EtherCAT device. This makes the case
differentiations easier.
\paragraph{Global Variables}
First of all, there have to be additional global variables declared,
as shown in the listing:
\begin{lstlisting}[language=C,numbers=left]
static int ec_device_index = -1;
static int ec_device_master_index = 0;
static ec_device_t *rtl_ec_dev;
struct net_device *rtl_ec_net_dev = NULL;
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 1} -- \textcircled{\tiny 2}] To
comply to the requirements for parameters of EtherCAT device modules
described in section~\ref{sec:seldev}, there have to be additional
parameter variables: \textit{ec\_\-device\_\-index} holds the index
of the EtherCAT device and defaults to $-1$ (no EtherCAT device),
while \textit{ec\_device\_master\_index} stores index of the master,
the single device will be connected to. Default: $0$
\item[\normalfont\textcircled{\tiny 3}] \textit{rtl\_ec\_dev} will be
the pointer to the later registered RealTek EtherCAT device, which
can be used as a parameter for device methods.
\item[\normalfont\textcircled{\tiny 4}] \textit{rtl\_ec\_net\_dev} is
a pointer to the \textit{net\_device} structure of the dedicated
device and is set while scanning the PCI bus and finding the device
with the specified index. This is done inside the
\textit{pci\_module\_init()} function executed as the first thing on
module loading.
\end{description}
\paragraph{Module Initialization}
Below is the (shortened) coding of the device driver's module init
function:
\begin{lstlisting}[language=C,numbers=left]
static int __init rtl8139_init_module(void)
{
if (pci_module_init(&rtl8139_pci_driver) < 0) {
printk(KERN_ERR "Failed to init PCI mod.\n");
goto out_return;
}
if (rtl_ec_net_dev) {
printk(KERN_INFO "Registering"
" EtherCAT device...\n");
if (!(rtl_ec_dev =
ecdev_register(ec_device_master_index,
rtl_ec_net_dev,
rtl8139_interrupt,
THIS_MODULE))) {
printk(KERN_ERR "Failed to reg."
" EtherCAT device!\n");
goto out_unreg_pci;
}
printk(KERN_INFO "Starting EtherCAT"
" device...\n");
if (ecdev_start(ec_device_master_index)) {
printk(KERN_ERR "Failed to start"
" EtherCAT device!\n");
goto out_unreg_ec;
}
} else {
printk(KERN_WARNING "No EtherCAT device"
" registered!\n");
}
return 0;
out_unreg_ec:
ecdev_unregister(ec_device_master_index, rtl_ec_dev);
out_unreg_pci:
pci_unregister_driver(&rtl8139_pci_driver);
out_return:
return -1;
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 3}] This call initializes all
RTL8139-compatible devices found on the pci bus. If a device with
index \textit{ec\_device\_index} is found, a pointer to its
\textit{net\_device} structure is stored in
\textit{rtl\_ec\_net\_dev} for later use (see next listings).
\item[\normalfont\textcircled{\tiny 8}] If the specified device was
found, \textit{rtl\_ec\_net\_dev} is non-zero.
\item[\normalfont\textcircled{\tiny 11}] The device is connected to
the specified master with a call to \textit{ecdev\_register()}. If
this fails, module loading is aborted.
\item[\normalfont\textcircled{\tiny 23}] The device registration was
successful and the master is started. This can fail, which aborts
module loading.
\item[\normalfont\textcircled{\tiny 29}] If no EtherCAT device was
found, a warning is output.
\end{description}
\paragraph{Device Searching}
During the PCI initialization phase, a variable \textit{board\_idx} is
increased for each RTL8139-compatible device found. The code below is
executed for each device:
\begin{lstlisting}[language=C,numbers=left]
if (board_idx == ec_device_index) {
rtl_ec_net_dev = dev;
strcpy(dev->name, "ec0");
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 1}] The device with the specified
index will be the EtherCAT device.
\end{description}
\paragraph{Avoiding Device Registration}
Later in the PCI initialization phase, the net\_devices get
registered. This has to be avoided for EtherCAT devices and so this is
a typical example for an EtherCAT case differentiation:
\begin{lstlisting}[language=C,numbers=left]
if (dev != rtl_ec_net_dev) {
i = register_netdev(dev);
if (i) goto err_out;
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 1}] If the current net\_device is
not the EtherCAT device, it is registered at the network stack.
\end{description}
\paragraph{Avoiding Interrupt Registration}
In the next two listings, there is an interrupt requested and the
device's interrupts are enabled. This also has to be encapsulated by
if-clauses, because interrupt operation is not wanted for EtherCAT
devices.
\begin{lstlisting}[language=C,numbers=left]
if (dev != rtl_ec_net_dev) {
retval = request_irq(dev->irq, rtl8139_interrupt,
SA_SHIRQ, dev->name, dev);
if (retval) return retval;
}
\end{lstlisting}
\begin{lstlisting}[language=C,numbers=left]
if (dev != rtl_ec_net_dev) {
/* Enable all known interrupts by setting
the interrupt mask. */
RTL_W16(IntrMask, rtl8139_intr_mask);
}
\end{lstlisting}
\paragraph{Frame Sending}
The listing below shows an exerpt of the function representing the
\textit{hard\_start\_xmit()} callback of the net\_device.
\begin{lstlisting}[language=C,numbers=left]
/* Note: the chip doesn't have auto-pad! */
if (likely(len < TX_BUF_SIZE)) {
if (len < ETH_ZLEN)
memset(tp->tx_buf[entry], 0, ETH_ZLEN);
skb_copy_and_csum_dev(skb, tp->tx_buf[entry]);
if (dev != rtl_ec_net_dev) {
dev_kfree_skb(skb);
}
} else {
if (dev != rtl_ec_net_dev) {
dev_kfree_skb(skb);
}
tp->stats.tx_dropped++;
return 0;
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 6} + \textcircled{\tiny 10}] The
master uses a fixed socket buffer for transmission, which is reused
and may not be freed.
\end{description}
\paragraph{Frame Receiving}
During ordinary frame reception, a socket buffer is created and filled
with the received data. This is not necessary for an EtherCAT device:
\begin{lstlisting}[language=C,numbers=left]
if (dev != rtl_ec_net_dev) {
/* Malloc up new buffer, compatible with net-2e. */
/* Omit the four octet CRC from the length. */
skb = dev_alloc_skb (pkt_size + 2);
if (likely(skb)) {
skb->dev = dev;
skb_reserve(skb, 2); /* 16 byte align
the IP fields. */
eth_copy_and_sum(skb, &rx_ring[ring_off + 4],
pkt_size, 0);
skb_put(skb, pkt_size);
skb->protocol = eth_type_trans(skb, dev);
dev->last_rx = jiffies;
tp->stats.rx_bytes += pkt_size;
tp->stats.rx_packets++;
netif_receive_skb (skb);
} else {
if (net_ratelimit())
printk(KERN_WARNING
"%s: Memory squeeze, dropping"
" packet.\n", dev->name);
tp->stats.rx_dropped++;
}
} else {
ecdev_receive(rtl_ec_dev,
&rx_ring[ring_offset + 4], pkt_size);
dev->last_rx = jiffies;
tp->stats.rx_bytes += pkt_size;
tp->stats.rx_packets++;
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 28}] If the device is an EtherCAT
device, no socket buffer is allocated. Instead a pointer to the data
(which is still in the device's receive ring) is passed to the
EtherCAT master. Unnecessary copy operations are avoided.
\item[\normalfont\textcircled{\tiny 30} -- \textcircled{\tiny 32}] The
device's statistics are updated as usual.
\end{description}
\paragraph{Link State}
The link state (i.~e. if there is a carrier signal detected on the
receive port) is determined during execution of the ISR. The listing
below shows the different processing for Ethernet and EtherCAT
devices:
\begin{lstlisting}[language=C,numbers=left]
if (dev != rtl_ec_net_dev) {
if (tp->phys[0] >= 0) {
mii_check_media(&tp->mii, netif_msg_link(tp),
init_media);
}
} else {
void __iomem *ioaddr = tp->mmio_addr;
uint16_t link = RTL_R16(BasicModeStatus)
& BMSR_LSTATUS;
ecdev_link_state(rtl_ec_dev, link ? 1 : 0);
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 3}] The ``media check'' is done
via the media independent interface (MII\nomenclature{MII}{Media
Independent Interface}), a standard interface for Fast Ethernet
devices.
\item[\normalfont\textcircled{\tiny 7} -- \textcircled{\tiny 10}] For
EtherCAT devices, the link state is fetched manually from the
appropriate device register, and passed to the EtherCAT master by
calling \textit{ecdev\_\-link\_\-state()}.
\end{description}
\paragraph{Module Cleanup}
Below is the module's cleanup function:
\begin{lstlisting}[language=C,numbers=left]
static void __exit rtl8139_cleanup_module (void)
{
printk(KERN_INFO "Cleaning up RTL8139-EtherCAT"
" module...\n");
if (rtl_ec_net_dev) {
printk(KERN_INFO "Stopping device...\n");
ecdev_stop(ec_device_master_index);
printk(KERN_INFO "Unregistering device...\n");
ecdev_unregister(ec_device_master_index,
rtl_ec_dev);
rtl_ec_dev = NULL;
}
pci_unregister_driver(&rtl8139_pci_driver);
printk(KERN_INFO "RTL8139-EtherCAT module"
" cleaned up.\n");
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 6}] Stopping and unregistration is
only done, if a device was registered before.
\item[\normalfont\textcircled{\tiny 8}] The master is first stopped,
so it does not access the device any more.
\item[\normalfont\textcircled{\tiny 10}] After this, the device is
unregistered. The master is now ``ophaned''.
\end{description}
%------------------------------------------------------------------------------
\section{The Master Module}
\label{sec:mastermod}
\index{Master module}
The EtherCAT master is designed to run as a kernel module. Moreover
the master kernel module \textit{ec\_master} can handle multiple
masters at the same time: The number of masters has to be passed to
the module with the parameter \textit{ec\_master\_count}, that
defaults to $1$. A certain master can later be addressed by its index.
For example, if the master module has been loaded with the command
\begin{lstlisting}
# `\textbf{modprobe ec\_master ec\_master\_count=2}`
\end{lstlisting}
the two masters can be addressed by their indices 0 and 1 respectively
(see figure~\ref{fig:masters}). This master index mandatory for
certain functions of the master interfaces.
\begin{figure}[htbp]
\centering
\includegraphics[width=.5\textwidth]{images/masters}
\caption{Multiple masters in one module}
\label{fig:masters}
\end{figure}
\paragraph{Master Log Messages}
The master module gives information about it's state and events via
the syslog interface. The module loading command above should result
in the syslog messages below (or similar):
\begin{lstlisting}
EtherCAT: Master driver, 1.1 (stable) - rev. 513,
compiled by fp at Aug 09 2006 09:43:50
EtherCAT: Initializing 2 EtherCAT master(s)...
EtherCAT: Initializing master 0.
EtherCAT: Initializing master 1.
EtherCAT: Master driver initialized.
\end{lstlisting}
The master provides information about it's version number, subversion
revision number and compile information, like the date of compilation
and the user, who compiled. All messages are prefixed either with
\texttt{EtherCAT:}, \texttt{EtherCAT WARNING:} or \texttt{EtherCAT
ERROR:}, which makes searching the logs easier.
%------------------------------------------------------------------------------
\subsection{Class Reference}
\label{sec:classes}
This section is not intended to be a complete reference of master
classes and functions\footnote{The comprehensive master reference can
be obtained at http://etherlab.org/download/download-en.html}, but
will give a general survey of the master's classes, and how they
interact.
Figure~\ref{fig:uml-all} shows an UML class diagram of the master
classes.
\begin{figure}[htbp]
\centering
\includegraphics[width=\textwidth]{images/uml-all}
\caption{UML class diagram with associations}
\label{fig:uml-all}
\end{figure}
The following subsections introduce serveral classes with their
attributes and methods.
%------------------------------------------------------------------------------
\subsubsection{The Master Class}
\label{sec:class-master}
\index{Master!Class}
Figure~\ref{fig:uml-master} shows an UML class diagram of the master
class. There is a short explanation of the attributes and methods
below.
\begin{figure}[htbp]
\centering
\includegraphics[width=.8\textwidth]{images/uml-master}
\caption{Master UML class diagram}
\label{fig:uml-master}
\end{figure}
\paragraph{Master Attributes}
\begin{description}
\item[list] is a listhead structure that is needed to manage the list
of masters in the master module (see section~\ref{sec:mastermod}).
\item[reserved] is a flag, that marks the master as reserved for a
realtime module, so that a call to \textit{ecrt\_request\_master()}
fails, if another module is already using the master.
\item[index] contains the number of the master. The first master will
get index 0, the second index 1, and so on.
\item[kobj] In order to make the master object available via Sysfs
(see section~\ref{sec:sysfs}), this structure is needed inside the
master object (see section~\ref{sec:sysfs}).
\item[slaves] is the list of slaves. It consists of objects of the
\textit{ec\_slave\_t} class (see section~\ref{sec:class-slave}).
\item[slave\_count] is the number of slaves in the list.
\item[device] points to the network device that is used by the master
to send and receive frames (see section~\ref{sec:class-device}). It
is \textit{NULL}, if no device is connected.
\item[datagram\_queue] is a list of datagrams (see
section~\ref{sec:class-datagram}) that have to be sent by the
master, or have already been sent and wait to be received again.
Upon reception or error, the datagrams are dequeued.
\item[datagram\_index] contains the index value for the next datagram.
The master stores this incrementing index into every datagram, to
make it easier to assign a received datagram to the one sent before.
\item[domains] contains the list of domains created by the realtime
module (section~\ref{sec:class-domain}).
\item[debug\_level] controls, how much debugging output is printed by
the master: 0 means no debugging output, 1 means to output certain
executing marks and actions, and 2 means to output frame contents in
addition. This value can be changed at runtime via the Sysfs
interface (see section~\ref{sec:sysfs}).
\item[stats] is a statistics object that contains certain counters
(like the number of missed frames). These statistics are output on
demand, but at most once a second.
\item[workqueue] is the kernel workqueue used for idle mode.
\item[idle\_work] is the work object, that is queued.
\item[fsm] The attribute \textit{fsm} represents the master's finite
state machine, that does all the slave processing. See
sections~\ref{sec:class-fsm} and~\ref{sec:fsm} for further details.
\item[mode] contains the current master mode, if it is orphaned, idle,
or in operation mode.
\item[eoe\_timer] is the kernel timer used for EoE\index{EoE}
processing.
\item[eoe\_running] marks the state of EoE processing.
\item[eoe\_handlers] is the list of EoE handlers (see
section~\ref{sec:class-eoe}).
\item[internal\_lock] is a spinlock used in idle mode, that controls
the concurrency of the idle and EoE processes.
\item[request\_cb] The ``request lock'' callback function, the master
has to provide for foreign instances, which want to access the
master (see section~\ref{sec:concurr}).
\item[release\_cb] The callback function that will release the master
lock.
\item[cb\_data] This value will be passed as an argument to each
callback.
\item[eeprom\_write\_enable] flag can be written via Sysfs to enable
the general writing of E$^2$PROM contents.
\end{description}
\paragraph{Public Master Methods}
\begin{description}
\item[ec\_master\_init()] is the master's constructor. It initializes
all attributes, creates the workqueue, creates EoE handlers and the
state machine object, and adds the kernel object to the Sysfs
hierarchy.
\item[ec\_master\_clear()] is the destructor and undoes all these
actions.
\item[ec\_master\_reset()] clears the master, but initializes it
again. This is needed, when a realtime module disconnects: Slaves
and other attributes are cleared and are later rebuilt by the idle
process.
\item[ec\_master\_idle\_start/stop()] These methods enable or disable
the idle process.
\item[ec\_master\_eoe\_start/stop()] These methods do the same for the
EoE timer.
\item[ec\_master\_receive\_datagrams()] This method is called by the
device, which uses it to pass received frames to the master. The
frame is dissected and the contained datagrams are assigned to the
datagram objects in the datagram queue, which are dequeued on
reception or error.
\item[ec\_master\_queue\_datagram()] This method is used to queue a
new datagram for sending and receiving.
\item[ec\_master\_output\_stats()] This method is cyclically called to
output a summary of the \textit{stats} attribute at most once a
second.
\item[ec\_master\_clear\_slaves()] clears the list of slaves. This is
needed on connection/disconnection of a realtime module or at a
topology change in idle mode, when all slaves objects are rebuilt.
\end{description}
\paragraph{Private Master Methods}
A few of a master's methods are private, meaning, that they can only
be called from other master methods:
\begin{description}
\item[ec\_master\_send\_datagrams()] searches the datagram queue for
unsent datagrams, allocates frames to send them, does the actual
sending and marks the datagrams as sent.
\item[ec\_master\_idle\_run()] is the work function for the idle mode.
It executes the idle state machine, described in
section~\ref{sec:fsm-idle}.
\item[ec\_master\_eoe\_run()] is called by the EoE timer and is
responsible for communicating with EoE-capable slaves. See
section~\ref{sec:eoeimp} for more information.
\end{description}
\paragraph{Master Methods (Realtime Interface)}
The master methods belonging to the Eth\-er\-CAT realtime
interface\index{ecrt@\textit{ecrt}}\nomenclature{ecrt}{EtherCAT
Realtime Interface} begin with the prefix \textit{ecrt} instead of
\textit{ec}. The functions of the realtime interface are explained in
section~\ref{sec:ecrt-master}.
%------------------------------------------------------------------------------
\subsubsection{The Slave Class}
\label{sec:class-slave}
\index{Slave!Class}
Figure~\ref{fig:uml-slave} shows an UML class diagram of the slave
class. There is a short explanation of the attributes and methods
below.
\begin{figure}[htbp]
\centering
\includegraphics[width=.8\textwidth]{images/uml-slave}
\caption{Slave UML class diagram}
\label{fig:uml-slave}
\end{figure}
\paragraph{Slave Attributes}
\begin{description}
\item[list] The master holds a slave list, therefore the slave class
must contain this structure used as an anchor for the linked
list.
\item[kobj] This pointer serves as base object for the slave's Sysfs
representation.
\item[master] is the pointer to the master owning this slave object.
\item[ring\_position] is the logical position in the logical ring
topology.
\item[station\_address] is the configured station address. This is
always the ring position~+~$1$).
\item[coupler\_index] is the index of the last bus coupler.
\item[coupler\_subindex] is the slave's position, counted from the
last bus coupler. See section~\ref{sec:addr} for more information.
\item[base\_*] These attributes contain base information about the
slave, that are read from the ``DL Information'' attribute.
\item[dl\_*] These fields store information of the ``DL Status''
attribute, for example states of the the communication ports.
\item[sii\_*] These attributes contain values from the ``Slave
Information Interface'' \cite[section~6.4]{dlspec}, mostly identity
and mailbox information, but also the list of sync manager
configurations and Pdos.
\item[registered] This flag is set, if one or more Pdos of the slave
have been registered for process data exchange. Otherwise a warning
is output, because the slave is unused.
\item[fmmus] Is an array of FMMU configurations, that have to be
applied to the slave.
\item[fmmu\_count] contains number of FMMUs used.
\item[eeprom\_*] These fields contain E$^2$PROM contents and the
extracted category information \cite[section~5.4]{alspec}.
\item[new\_eeprom\_data] If this pointer is not \textit{NULL}, it
points to new E$^2$PROM contents, that have to be written to the
slave.
\item[new\_eeprom\_size] This field represents the size of the new
E$^2$PROM data.
\item[requested\_state] is the requested slave state.
\item[current\_state] is the current slave state.
\item[error\_flag] is used by the operation and idle state machines
to indicate, that a state transisition has failed and should not be
tried again until an external event happens.
\item[online] This flag contains the online state of the slave (i.~e.
if it currently responds to the masters commands). Changes of the
online state are always reported.
\item[varsize\_fields] is only suitable for slaves that provide Pdos
of variable size (like slaves that manage a sub-fieldbus) and
contains information about what size this fields actually should
have.
\end{description}
\paragraph{Public Slave Methods}
\begin{description}
\item[ec\_slave\_init()] The slave's constructor.
\item[ec\_slave\_clear()] The slave's destructor.
\item[ec\_prepare\_fmmu()] prepares an FMMU configuration. The FMMU is
configured for a certain sync manager and domain.
\item[ec\_fetch\_*()] Serveral methods to extract information of the
E$^2$PROM category contents.
\item[ec\_slave\_locate\_string()] extracts a string out of a STRING
category and allocates string memory.
\item[ec\_slave\_calc\_sync\_size()] calculates the size of sync
manager contents, because they can be variable due to variable-sized
Pdos.
\item[ec\_slave\_info()] This method prints all slave information into
a buffer for Sysfs reading.
\item[ec\_slave\_mbox\_*()] These functions prepare datagrams for
mailbox communication, or process mailbox responses, respectively.
\end{description}
\paragraph{Private Slave Methods}
\begin{description}
\item[ec\_slave\_write\_eeprom()] This function accepts E$^2$PROM data
from user space, does a quick validation of the contents and
schedules them for writing through the idle state machine.
\end{description}
\paragraph{Slave Methods (Realtime Interface)}
\begin{description}
\item[ecrt\_slave\_conf\_sdo*()] These methods accept Sdo
configurations, that are applied on slave activation (i.~e.
everytime the slave is configured). The methods differ only in the
data size of the Sdo (8, 16 or 32 bit).
\item[ecrt\_slave\_pdo\_size()] This method specifies the size of a
variable-sized Pdo.
\end{description}
%------------------------------------------------------------------------------
\subsubsection{The Device Class}
\label{sec:class-device}
\index{Device!Class}
The device class is responsible for communicating with the connected
EtherCAT-enabled network driver. Figure~\ref{fig:uml-device} shows its
UML class diagram.
\begin{figure}[htbp]
\centering
\includegraphics[width=.3\textwidth]{images/uml-device}
\caption{Device UML class diagram}
\label{fig:uml-device}
\end{figure}
\paragraph{Device Attributes}
\begin{description}
\item[master] A pointer to the master, which owns the device object.
\item[dev] This is the pointer to the \textit{net\_device} structure
of the connected network device driver.
\item[open] This flag stores, if the network device is ``opened'' and
ready for transmitting and receiving frames.
\item[tx\_skb] The transmittion socket buffer. Instead of allocating a
new socket buffer for each frame, the same socket buffer is recycled
and used for every frame.
\item[isr] The pointer to the network device's interrupt service
routine. \textit{ec\_isr\_t} is a type definition in the device
interface, which looks like below:
\begin{lstlisting}[gobble=4,language=C]
typedef irqreturn_t (*ec_isr_t)(int, void *,
struct pt_regs *);
\end{lstlisting}
\item[module] A pointer to the network driver module, to increase and
decrease the use counter (see paragraph ``Implicit Dependencies'' in
section~\ref{sec:ecdev}).
\item[link\_state] The current link state. It can be 0 ``down'' or 1
``up''.
\item[dbg] Every device objects contains a debugging interface (see
sectios~\ref{sec:class-debug} and~\ref{sec:debug}).
\end{description}
\paragraph{Public Device Methods}
\begin{description}
\item[ec\_device\_init()] The device constructor.
\item[ec\_device\_clear()] The device destructor.
\item[ec\_device\_open()] ``Opens'' the device for transmittion and
reception of frames. This is equivalent to the \textit{ifconfig up}
command for ordinary Ethernet devices.
\item[ec\_device\_close()] Stops frame transmittion and reception.
This is equivalent to the \textit{ifconfig down} command for
ordinary Ethernet devices.
\item[ec\_device\_call\_isr()] Calls the interrupt service routine of
the device.
\item[ec\_device\_tx\_data()] Returns a pointer into the memory of the
transmittion socket buffer \textit{tx\_skb}. This is used by the
master to assemble a new EtherCAT frame.
\item[ec\_device\_send()] Sends an assembled frame by passing it to
the device's \textit{hard\_\-start\_\-xmit()} callback.
\end{description}
\paragraph{Device Methods (Device Interface)}
The device methods belonging to the device interface are explained in
section~\ref{sec:ecdev}.
%------------------------------------------------------------------------------
\subsubsection{The Datagram Class}
\label{sec:class-datagram}
\index{Datagram!Class}
So send and receive a datagram, an object of the
\textit{ec\_datagram\_t} class is needed. It can be initialized with a
datagram type \cite[section~5.4]{dlspec} and length (optionally filled
with data) and appended to the master's datagram queue.
Figure~\ref{fig:uml-datagram} shows its UML class diagram.
\begin{figure}[htbp]
\centering
\includegraphics[width=.3\textwidth]{images/uml-datagram}
\caption{Datagram UML class diagram}
\label{fig:uml-datagram}
\end{figure}
\paragraph{Datagram Attributes}
\begin{description}
\item[list] This attribute is needed to make a list of datagrams, as
used in the domain class (see section~\ref{sec:class-domain}).
\item[queue] This attribute is the anchor to the master's datagram
queue, which is implemented as a linked list.
\item[type] The datagram type. \textit{ec\_\-datagram\_\-type\_\-t} is
an enumeration, which can have the values
\textit{EC\_\-DATAGRAM\_\-APRD}, \textit{EC\_\-DATAGRAM\_\-APWR},
\textit{EC\_\-DATAGRAM\_\-NPRD}, \textit{EC\_\-DATAGRAM\_\-NPWR},
\textit{EC\_\-DATAGRAM\_\-BRD}, \textit{EC\_\-DATAGRAM\_\-BWR} or
\textit{EC\_\-DATAGRAM\_\-LRW}.
\item[address] The slave address. For all addressing schemes take 4
bytes, \textit{ec\_address\_t} is a union type:
\begin{lstlisting}[gobble=4,language=C]
typedef union {
struct {
uint16_t slave; /**< configured or
autoincrement
address */
uint16_t mem; /**< physical memory
address */
} physical; /**< physical address */
uint32_t logical; /**< logical address */
} ec_address_t;
\end{lstlisting}
\item[data] The actual data of the datagram. These are either filled
in before sending (at writing access) or are inserted by the
adressed slave(s) (at reading access). In any case, the data memory
must be dynamically allocated. Besides, this can be done before
cyclic processing with the \textit{ec\_datagram\_prealloc()} method
(see below).
\item[mem\_size] The size of the allocated memory, \textit{data}
points to.
\item[data\_size] The size of the actual data in the \textit{data}
memory.
\item[index] The sequential EtherCAT datagram index. This value is set
by the master on sending, to easier assign a received datagram to a
queued datagram object.
\item[working\_counter] The working counter of the datagram. This is
set to zero on sending and filled with the real value of the working
counter on datagram reception.
\item[state] The state of the datagram.
\textit{ec\_\-datagram\_\-state\_\-t} is an enumeration and can be
\textit{EC\_\-DATA\-GRAM\_\-INIT},
\textit{EC\_\-DATA\-GRAM\_\-QUEU\-ED},
\textit{EC\_\-DATA\-GR\-AM\_\-SEN\-T},
\textit{EC\_\-DATA\-GRAM\_\-REC\-EIVED},
\textit{EC\_\-DATA\-GRAM\_\-TIMED\_\-OUT} or
\textit{EC\_\-DA\-TA\-GRAM\_\-ERR\-OR}.
\item[t\_sent] This attribute is set to the timestamp, when the
datagram was sent, to later detect a timeout.
\end{description}
\paragraph{Public Datagram Methods}
\begin{description}
\item[ec\_datagram\_init()] The datagram's constructor.
\item[ec\_datagram\_clear()] The datagram's destructor.
\item[ec\_datagram\_prealloc()] Allocates memory for the datagram
data. This is especially needed, if the datagram structure will
later be used in a context, where no dynamic memory allocation is
allowed.
\item[ec\_datagram\_nprd()] Initializes a ``Node-Addressed Physical
Read'' datagram \cite[section~5.4.1.2]{dlspec}.
\item[ec\_datagram\_npwr()] Initializes a ``Node-Addressed Physical
Write'' datagram \cite[section~5.4.2.2]{dlspec}.
\item[ec\_datagram\_aprd()] Initializes a ``Auto-Increment Physical
Read'' datagram \cite[section~5.4.1.1]{dlspec}.
\item[ec\_datagram\_apwr()] Initializes a ``Auto-Increment Physical
Write'' datagram \cite[section~5.4.2.1]{dlspec}.
\item[ec\_datagram\_brd()] Initializes a ``Broadcast Read'' datagram
\cite[section~5.4.1.3]{dlspec}.
\item[ec\_datagram\_bwr()] Initializes a ``Broadcast Write'' datagram
\cite[section~5.4.2.3]{dlspec}.
\item[ec\_datagram\_lrw()] Initializes a ``Logical ReadWrite''
datagram \cite[section~5.4.3.4]{dlspec}.
\end{description}
%------------------------------------------------------------------------------
\subsubsection{The Domain Class}
\label{sec:class-domain}
\index{Domain!Class}
The domain class encapsules Pdo registration and management of the
process data image and its exchange. The UML class diagram can be seen
in figure~\ref{fig:uml-domain}.
\begin{figure}[htbp]
\centering
\includegraphics[width=.4\textwidth]{images/uml-domain}
\caption{Domain UML class diagram}
\label{fig:uml-domain}
\end{figure}
\paragraph{Domain Attributes}
\begin{description}
\item[kobj] This \textit{kobject} structure is needed for the Sysfs
representation of the domain.
\item[list] The master manages a list of domains, so this list anchor
is needed.
\item[index] The domain's index. The first domain will get index 0,
the second index 1, and so on.
\item[master] A pointer to the master owning the domain.
\item[data\_size] The size of the domain's process data image.
\item[datagram] A linked list with the datagram objects, the domain
needs for process data exchange (see
section~\ref{sec:class-datagram}).
\item[base\_address] This attribute stores the logical offset, to
which the domain's process data are mapped.
\item[response\_count] The sum of the datagrams' working counters at
the last process data exchange. Changes are always reported.
\item[data\_regs] The (linked) list of Pdo registrations. The realtime
module requests the exchange of certain Pdos and supplies the
address of process data pointers, that will later point to the
respective locations in the process data image. These ``data
registrations'' are saved in the \textit{data\_regs} list.
\item[working\_counter\_changes] This field stores the number of
working counter changes since the last notification. This helps to
reduce syslog output in case of frequent changes.
\item[t\_last] The timestamp of the last working counter change
notification.
\end{description}
\paragraph{Public Domain Methods}
\begin{description}
\item[ec\_domain\_init()] The domain's constructor.
\item[ec\_domain\_clear()] The domain's destructor.
\item[ec\_domain\_alloc()] Allocates the process data image and the
respective datagrams based on the process data registrations.
\item[ec\_domain\_queue()] Queues the domain's datagrams for exchange
via the master.
\end{description}
\paragraph{Private Domain Methods}
\begin{description}
\item[ec\_domain\_reg\_pdo\_entry()] This method is used to do a Pdo
registration. It finds the appropriate sync manager covering the Pdo
data, calculates its offset in the sync-manager-protected memory and
prepares the FMMU configurations for the related slave. Then the Pdo
registration is appended to the list.
\item[ec\_domain\_clear\_data\_regs()] Clearing all process data
registrations is needed in serveral places and therefore has been
sourced out to an own method.
\item[ec\_domain\_add\_datagram()] This methods allocates a datagram
and appends it to the list. This is done during domain allocation.
\end{description}
\paragraph{Domain Methods (Realtime Interface)}
The domain methods belonging to the realtime interface are introduced
in section~\ref{sec:ecrt-domain}.
%------------------------------------------------------------------------------
\subsubsection{The Finite State Machine Class}
\label{sec:class-fsm}
\index{FSM!Class}
This class encapsules all state machines, except the EoE state
machine. Its UML class diagram can be seen in
figure~\ref{fig:uml-fsm}.
\begin{figure}[htbp]
\centering
\includegraphics[width=.9\textwidth]{images/uml-fsm}
\caption{Finite State Machine UML class diagram}
\label{fig:uml-fsm}
\end{figure}
\paragraph{FSM Attributes}
\begin{description}
\item[master] A pointer to the master owning the FSM object.
\item[slave] Serveral sub state machines are executed for single
slaves. This pointer stores the current slave for these FSMs.
\item[datagram] The FSM class has its own datagram, which is used in
every state and sub-state.
\item[master\_state] This function pointer stores the current state
function for one of the master's state machines.
\item[master\_slaves\_responding] This attribute is used in the
operation state machine (see section~\ref{sec:fsm-op}) to store the
number of slaves, that responded to the last broadcast command.
\item[master\_slave\_states] This attribute stores the slave states,
that were determined by the last broadcast command.
\item[master\_validation] This flag is used by the operation state
machine and is non-zero, if a bus validation has to be done.
\item[slave\_state] This function pointer stores the current state of
the slave scan state machine (see section~\ref{sec:fsm-scan}) or the
slave configuration state machine (see section~\ref{sec:fsm-conf}).
\item[sii\_state] This function pointer stores the current state of
the SII state machine (see section~\ref{sec:fsm-sii}).
\item[sii\_offset] This attribute is used by the SII state machine to
store the word address for the current read or write cycle.
\item[sii\_mode] If this attribute is zero, the SII access is done
with ``auto-increment'' datagrams \cite[section~5.4]{dlspec}.
If it is non-zero, ``station-address'' datagrams are used.
\item[sii\_value] This attribute stores the value to write, or the
read value, respectively.
\item[sii\_start] A timestamp attribute, that stores the beginning
time of an SII operation to detect a timeout.
\item[change\_state] This function pointer stores the current state of
the state change state machine.
\item[change\_new] This attribute stores the requested state for the
state change state machine.
\item[change\_start] A timestamp attribute to detect a timeout while
changing slave states.
\item[coe\_state] This function pointer stores the current state of
the CoE state machines.
\item[sdodata] This is an Sdo data object that stores information
about the current Sdo to write.
\item[coe\_start] A timestamp attribute to detect timeouts during CoE
configuration.
\end{description}
\paragraph{Public FSM Methods}
\begin{description}
\item[ec\_fsm\_init()] Constructor of the FSM class.
\item[ec\_fsm\_clear()] Destructor of the FSM class.
\item[ec\_fsm\_reset()] Resets the whole FSM object. This is needed to
restart the master state machines.
\item[ec\_fsm\_execute()] Executes one state of the current state
machine and then returns.
\item[ec\_fsm\_startup()] Initializes the master startup state
machine, which determines the number of slaves and executes the
slave scan state machine for each slave.
\item[ec\_fsm\_startup\_running()] Returns non-zero, if the startup
state machine did not terminate yet.
\item[ec\_fsm\_startup\_success()] Returns non-zero, if the startup
state machine terminated with success.
\item[ec\_fsm\_configuration()] Initializes the master configuration
state machine, which executes the slave configuration state machine
for each slave.
\item[ec\_fsm\_configuration\_running()] Returns non-zero, if the
configuration state machine did not terminate yet.
\item[ec\_fsm\_configuration\_success()] Returns non-zero, if the
configuration state machine terminated with success.
\end{description}
\paragraph{FSM State Methods}
The rest of the methods showed in the UML class diagram are state
methods of the state machines. These states are described in
section~\ref{sec:fsm}.
%------------------------------------------------------------------------------
\subsubsection{The EoE Class}
\label{sec:class-eoe}
\index{EoE!Class}
Objects of the \textit{ec\_eoe\_t} class are called EoE handlers. Each
EoE handler represents a virtual network interface and can be coupled
to a EoE-capable slave on demand. The UML class diagram can be seen in
figure~\ref{fig:uml-eoe}.
\begin{figure}[htbp]
\centering
\includegraphics[width=.4\textwidth]{images/uml-eoe}
\caption{EoE UML class diagram}
\label{fig:uml-eoe}
\end{figure}
\paragraph{EoE Attributes}
\begin{description}
\item[list] The master class maintains a list of EoE handlers.
Therefore this list anchor is needed.
\item[slave] If an EoE handler is coupled to a slave, this pointer
points to the related slave object. Otherwise it is \textit{NULL}.
\item[datagram] Every EoE handler owns a datagram object to exchange
data with the coupled slave via its state machine.
\item[state] This function pointer points to the current state of the
EoE state machine (see section~\ref{sec:eoeimp}).
\item[dev] A pointer to the \textit{net\_device} structure that
represents the network interface to the kernel.
\item[stats] The statistics object for the network interface.
\item[opened] This flag stores, if the network interface was opened.
No EoE processing will be done, if the device is not opened.
\item[t\_last] This timestamp attribute stores the time of the last
bit rate measurement.
\item[rx\_skb] A pointer to the current receive socket buffer. On
every first fragment of a received frame, a new receive socket
buffer is allocated. On every last fragment, this buffer will be
passed to the network stack.
\item[rx\_skb\_offset] This attribute stores the offset for the next
fragment data in the receive socket buffer.
\item[rx\_skb\_size] This attribute stores the current data size of
the receive socket buffer.
\item[rx\_expected\_fragment] The expected number of the next
fragment. If a fragment with an invalid number is received, the
whole frame will be dropped.
\item[rx\_counter] This is the sum of the octets received since the
last bit rate measurement.
\item[rx\_rate] This attribute stores the receive bit rate in bps.
\item[tx\_queue] Each EoE handler maintains a transmittion queue for
frames, that come in via the network interface. This queue is
implemented with a linked list and protected by a spinlock.
\item[tx\_queue\_active] This flag stores, if the transmittion queue
is currently accepting frames from the network stack. If the queue
gets filled up, frame transmittion is suspended with a call to
\textit{netif\_stop\_queue()}. If the fill state decreases below the
half capacity, frame transmittion is restarted with
\textit{netif\_wake\_queue()}.
\item[tx\_queued\_frames] The number of frames in the transmittion
queue.
\item[tx\_queue\_lock] The spinlock used to protect the transmittion
queue. This is needed, because the queue is accessed both from
network stack context and from the master's EoE timer.
\item[tx\_frame] The frame that is currently sent. The
\textit{ec\_eoe\_frame\_t} structure combines the socket buffer
structure with a list head to append it to the transmittion queue.
\item[tx\_frame\_number] The EoE protocol demands to maintain a
sequencial frame number, that must be increased with every frame
sent.
\item[tx\_fragment\_number] The sequencial number of the next fragment
to transmit.
\item[tx\_offset] Current frame data offset for the next fragment to
transmit.
\item[tx\_counter] The number of octets transferred since the last bit
rate measurement.
\item[tx\_rate] The recent transmittion bit rate in bps.
\end{description}
\paragraph{Public EoE Methods}
\begin{description}
\item[ec\_eoe\_init()] The EoE handler's constructor. The network
interface is allocated and registered.
\item[ec\_eoe\_clear()] The EoE handler's destructor. The network
interface is unregistered and all allocated memory is freed.
\item[ec\_eoe\_run()] Executes the EoE state machine (see
section~\ref{sec:eoeimp}) for this handler.
\item[ec\_eoe\_active()] Returns true, if the handler has a slave
coupled and the network interface is opened.
\end{description}
\paragraph{Private EoE Methods}
\begin{description}
\item[ec\_eoe\_flush()] Clears the transmittion queue and drops all
frames queued for sending.
\item[ec\_eoe\_send()] Sends one fragment of the current frame.
\end{description}
\paragraph{EoE State Methods}
The rest of the private methods are state functions for the EoE state
machine, which is discussed in section~\ref{sec:eoeimp}.
%------------------------------------------------------------------------------
\subsubsection{The Debug Class}
\label{sec:class-debug}
The debug class maintains a virtual network interface. All frames that
are sent and received by the master will be forwarded to this network
interface, so that bus monitoring can be done with third party tools
(see section~\ref{sec:debug}). Figure~\ref{fig:uml-debug} shows the
UML class diagram.
\begin{figure}[htbp]
\centering
\includegraphics[width=.3\textwidth]{images/uml-debug}
\caption{Debug UML class diagram}
\label{fig:uml-debug}
\end{figure}
\paragraph{Debug Attributes}
\begin{description}
\item[dev] A pointer to the allocated \textit{net\_device} structure
that represents the network interface in the kernel.
\item[stats] An object for interface statistics.
\item[opened] Stores the state of the device. Frames will only be
forwarded, if the device was opened with the \textit{ifconfig up}
command (or something similar).
\end{description}
\paragraph{Public Debug Methods}
\begin{description}
\item[ec\_debug\_init()] The constructor.
\item[ec\_debug\_clear()] The destructor.
\item[ec\_debug\_send()] This method forwards a frame to the virtual
network interface. It dynamically allocates a new socket buffer and
passes it to the network stack.
\end{description}
%------------------------------------------------------------------------------
\subsection{The Realtime Interface}
\label{sec:ecrt}
\index{Realtime interface}
The realtime interface provides functions and data structures for
realtime modules to access and use an EtherCAT master.
\subsubsection{Master Requesting and Releasing}
Before a realtime module can access am EtherCAT master provided by the
master module, it has to reserve one for exclusive use. After use, it
has to release the requested master and make it available for other
modules. This is done with the following functions:
\begin{lstlisting}[language=C]
ec_master_t *ecrt_request_master(unsigned int master_index);
void ecrt_release_master(ec_master_t *master);
\end{lstlisting}
The \textit{ecrt\_request\_master()} function has to be the first
function a module has to call, when using EtherCAT. The function takes
the index of the master as its argument. The first master has index 0,
the $n$th master has index $n - 1$. The number of existent masters has
to be specified when loading the master module (see
section~\ref{sec:mastermod}). The function tries to reserve the
specified master and scans for slaves. It returns a pointer to the
reserved master object upon success, or \textit{NULL} if an error
occured.
The \textit{ecrt\_release\_master()} function releases a reserved
master after use. It takes the pointer to the master object returned
by \textit{ecrt\_request\_master()} as its argument and can never
fail.
\subsubsection{Master Methods}
\label{sec:ecrt-master}
\paragraph{Domain Creation}
For process data exchange, at least one process data domain is needed
(see section~\ref{sec:processdata}).
\begin{lstlisting}[language=C]
ec_domain_t *ecrt_master_create_domain(ec_master_t *master);
\end{lstlisting}
The \textit{ecrt\_master\_create\_domain()} method creates a new
process data domain and returns a pointer to the new domain object.
This object can be used for registering process data objects and
exchange process data in cyclic operation. On failure, the function
returns \textit{NULL}.
\paragraph{Slave Handlers}
To access a certain slave, there is a method to get a slave handler:
\begin{lstlisting}[language=C]
ec_slave_t *ecrt_master_get_slave(const ec_master_t *,
const char *);
\end{lstlisting}
The \textit{ecrt\_master\_get\_slave()} method returns a pointer to a
certain slave object, specified by its ASCII address (see
section~\ref{sec:addr}). If the address is invalid, \textit{NULL} is
returned.
\paragraph{Master Activation}
When all domains are created, and all process data objects are
registered, the master can be activated:
\begin{lstlisting}[language=C]
int ecrt_master_activate(ec_master_t *master);
void ecrt_master_deactivate(ec_master_t *master);
\end{lstlisting}
By calling the \textit{ecrt\_master\_activate()} method, all slaves
are configured according to the prior method calls and are brought
into OP state. In this case, the method has a return value of 0.
Otherwise (wrong configuration or bus failure) the method returns
non-zero.
The \textit{ecrt\_master\_deactivate()} method is the counterpart to
the activate call: It brings all slaves back into INIT state again.
This method should be called prior to
\textit{ecrt\_\-master\_\-release()}.
\paragraph{Locking Callbacks}
For concurrent master access, the realtime module has to provide a
locking mechanism (see section~\ref{sec:concurr}):
\begin{lstlisting}[language=C]
void ecrt_master_callbacks(ec_master_t *master,
int (*request_cb)(void *),
void (*release_cb)(void *),
void *cb_data);
\end{lstlisting}
The ``request lock'' and ``release lock'' callbacks can be set with
the \textit{ecrt\_master\_call\-backs()} method. It takes two function
pointers and a data value as additional arguments. The arbitrary data
value will be passed as argument on every callback. Asynchronous
master access (like EoE processing) is only possible if these
callbacks have been set.
\paragraph{Preparation of Cyclic Data Exchange}
Cyclic operation mostly consists of the three steps input, processing
and output. In EtherCAT terms this would mean: Receive datagrams,
evaluate process data and send datagrams. The first cycle differs from
this principle, because no datagrams have been sent yet, so there is
nothing to receive. To avoid having a case differantiation (in terms
of an \textit{if} clause), the following method exists:
\begin{lstlisting}[language=C]
void ecrt_master_prepare(ec_master_t *master);
\end{lstlisting}
As a last thing before cyclic operation, a call to the
\textit{ecrt\_master\_prepare()} method should be issued. It makes all
process data domains queue their datagrams and issues a send command,
so that the first receive call in cyclic operation will not fail.
\paragraph{Frame Sending and Receiving}
To send all queued datagrams and to later receive the sent datagrams
there are two methods:
\begin{lstlisting}[language=C]
void ecrt_master_send(ec_master_t *master);
void ecrt_master_receive(ec_master_t *master);
\end{lstlisting}
The \textit{ecrt\_master\_send()} method takes all datagrams, that
have been queued for transmission, packs them into frames, and passes
them to the network device for sending.
The \textit{ecrt\_master\_receive()} queries the network device for
received frames (by calling the ISR\index{ISR}), extracts received
datagrams and dispatches the results to the datagram objects in the
queue. Received datagrams, and the ones that timed out, will be
marked, and then dequeued.
\paragraph{Running the Operation State Machine}
The master's operation state machine (see section~\ref{sec:fsm-op})
monitors the bus in cyclic operation and reconfigures slaves, if
necessary. Therefore, the following method should be called
cyclically:
\begin{lstlisting}[language=C]
void ecrt_master_run(ec_master_t *master);
\end{lstlisting}
The \textit{ecrt\_master\_run()} method executes the master's
operation state machine step by step. It returns after processing one
state and queuing a datagram. Calling this function is not mandatory,
but highly recommended.
\paragraph{Master Monitoring}
It is also highly recommended to evaluate the master's error state. In
this way it is possible to notice lost network links, failed bus
segments, and other issues:
\begin{lstlisting}[language=C]
int ecrt_master_state(const ec_master_t *master);
\end{lstlisting}
The \textit{ecrt\_master\_state()} method returns the master's error
state. The following states are defined as part of the realtime
interface:
\begin{description}
\item[EC\_MASTER\_OK] means, that no error has occurred.
\item[EC\_MASTER\_LINK\_ERROR] means, that the network link is
currently down.
\item[EC\_MASTER\_BUS\_ERROR] means, that one or more slaves do not
respond.
\end{description}
\subsubsection{Domain Methods}
\label{sec:ecrt-domain}
\paragraph{Pdo Registration}
To access data of a slave's Pdo in cyclic operation, it is necessary
to make it part of a process data domain:
\begin{lstlisting}[language=C]
ec_slave_t *ecrt_domain_register_pdo(ec_domain_t *domain,
const char *address,
uint32_t vendor_id,
uint32_t product_code,
const char *pdo_name
void **data_ptr);
int ecrt_domain_register_pdo_list(ec_domain_t *domain,
const ec_pdo_reg_t *pdos);
\end{lstlisting}
The \textit{ecrt\_domain\_register\_pdo()} method registers a certain
Pdo as part of the domain and takes the address of the process data
pointer. This pointer will be set on master activation and then can be
parameter to the \textit{EC\_READ\_*} and \textit{EC\_WRITE\_*} macros
described below.
A perhaps easier way to register multiple Pdos at the same time is to
fill an array of \textit{ec\_pdo\_reg\_t} and hand it to the
\textit{ecrt\_domain\_register\_pdo\_list()} method. Attention: This
array has to be terminated by an empty structure (\textit{\{\}})!
\paragraph{Evaluating Domain Data}
To evaluate domain data, the following method has to be used:
\begin{lstlisting}[language=C]
void ecrt_domain_process(ec_domain_t *domain);
\end{lstlisting}
The \textit{ecrt\_domain\_process()} method sets the domains state and
requeues its datagram for sending.
\paragraph{Domain State}
Similar to the master state, a domain has an own error state:
\begin{lstlisting}[language=C]
int ecrt_domain_state(const ec_domain_t *domain);
\end{lstlisting}
The \textit{ecrt\_domain\_state()} method returns the domain's error
state. It is non-zero if \underline{not} all process data values could
be exchanged, and zero otherwise.
\subsubsection{Slave Methods}
\label{sec:ecrt-slave}
\paragraph{Sdo Configuration}
To configure slave Sdos, the function interface below can be used:
\begin{lstlisting}[language=C]
int ecrt_slave_conf_sdo8(ec_slave_t *slave,
uint16_t sdo_index,
uint8_t sdo_subindex,
uint8_t value);
int ecrt_slave_conf_sdo16(ec_slave_t *slave,
uint16_t sdo_index,
uint8_t sdo_subindex,
uint16_t value);
int ecrt_slave_conf_sdo32(ec_slave_t *slave,
uint16_t sdo_index,
uint8_t sdo_subindex,
uint32_t value);
\end{lstlisting}
The \textit{ecrt\_slave\_conf\_sdo*()} methods prepare the
configuration of a certain Sdo. The index and subindex of the Sdo, and
the value have to be specified. The configuration is done each time,
the slave is reconfigured. The methods only differ in the Sdo's data
type. If the configuration could be prepared, zero is returned. If an
error occured, non-zero is returned.
\paragraph{Variable-sized Pdos}
For specifying the size of variable-sized Pdos, the following method
can be used:
\begin{lstlisting}[language=C]
int ecrt_slave_pdo_size(ec_slave_t *slave,
const char *pdo_name,
size_t size);
\end{lstlisting}
The \textit{ecrt\_slave\_pdo\_size()} method takes the name of the Pdo
and the size. It returns zero on success, otherwise non-zero.
\subsubsection{Process Data Access}
\label{sec:macros}
The endianess of the process data could differ from that of the CPU.
Therefore, process data access has to be done by the macros below,
that are also provided by the realtime interface:
\begin{lstlisting}[language=C]
#define EC_READ_BIT(DATA, POS)
#define EC_WRITE_BIT(DATA, POS, VAL)
#define EC_READ_U8(DATA)
#define EC_READ_S8(DATA)
#define EC_READ_U16(DATA)
#define EC_READ_S16(DATA)
#define EC_READ_U32(DATA)
#define EC_READ_S32(DATA)
#define EC_WRITE_U8(DATA, VAL)
#define EC_WRITE_S8(DATA, VAL)
#define EC_WRITE_U16(DATA, VAL)
#define EC_WRITE_S16(DATA, VAL)
#define EC_WRITE_U32(DATA, VAL)
#define EC_WRITE_S32(DATA, VAL)
\end{lstlisting}
There are macros for bitwise access (\textit{EC\_READ\_BIT()},
\textit{EC\_WRITE\_BIT()}), and bytewise access
(\textit{EC\_READ\_*()}, \textit{EC\_WRITE\_*()}). The bytewise macros
carry the data type in their name. Example: \textit{EC\_WRITE\_S16()}
writes a 16 bit signed value to EtherCAT data. The \textit{DATA}
parameter is supposed to be a process data pointer, as provided at Pdo
registration.
The macros use the kernel's endianess conversion macros, that are
preprocessed to empty macros in case of equal endianess. This is the
definition for the \textit{EC\_\-READ\_\-U16()} macro:
\begin{lstlisting}[language=C]
#define EC_READ_U16(DATA) \
((uint16_t) le16_to_cpup((void *) (DATA)))
\end{lstlisting}
The \textit{le16\_to\_cpup()} macro converts a little-endian, 16 bit
value to the CPU's architecture and takes a pointer to the input value
as its argument. If the CPU's architecture is little-endian, too (for
example on X86 and compatible), nothing has to be converted. In this
case, the macro is replaced with an empty macro by the preprocessor
and so there is no unneeded function call or case differentiation in
the code.
For keeping it portable, it is highly recommended to make use of these
macros.
%------------------------------------------------------------------------------
\subsection{Slave Addressing}
\label{sec:addr}
\index{Slave!Addressing}
The master offers the serveral slave addressing schemes (for Pdo
registration or configuration) via the realtime interface. For this
reason, slave addresses are ASCII\nomenclature{ASCII}{American
Standard Code for Information Interchange}-coded and passed as a
string. The addressing schemes are independent of the EtherCAT
protocol and represent an additional feature of the master.
Below, the allowed addressing schemes are described. The descriptions
are followed by a regular expression formally defining the addressing
scheme, and one or more examples.
\begin{description}
\item[Position Addressing] This is the normal addressing scheme, where
each slave is addressed by its ring position. The first slave has
address 0, and the $n$th slave has address $n - 1$. This addressing
scheme is useful for small busses that have a fixed number of slaves.\\
RegEx: \texttt{[0-9]+} --- Example: \texttt{"42"}
\item[Advanced Position Addressing] Bus couplers segment the bus into
(physical) blocks. Though the logical ring positions keep being the
same, it is easier to address a slave with its block number and the
relative position inside the block. This addressing is done by
passing the (zero-based) index of the bus coupler (not the coupler's
ring position), followed by a colon and the relative position of the
actual slave starting at the bus coupler.\\
RegEx: \texttt{[0-9]+:[0-9]+} --- Examples: \texttt{"0:42"},
\texttt{"2:7"}
\item[Alias Addressing] Each slave can have a ``secondary slave
address'' or ``alias address''\footnote{Information about how to set
the alias can be found in section~\ref{sec:eepromaccess}} stored
in its E$^2$PROM. The alias is evaluated by the master and can be
used to address the slave, which is useful when a clearly defined
slave has to be addressed and the ring position is not known or can
change over time. This scheme is used by starting the address string
with a mesh (\#) followed by the alias address. The latter can also
be provided as hexadecimal value, prefixed with \textit{0x}.\\
RegEx: \texttt{\#(0x[0-9A-F]+|[0-9]+)} --- Examples:
\texttt{"\#6622"}, \texttt{"\#0xBEEF"}
\item[Advanced Alias Addressing] This is a mixture of the ``Alias
Addressing'' and ``Advanced Position Addressing'' schemes. A certain
slave is addressed by specifying its relative position after an
aliased slave. This is very useful, if a complete block of slaves
can vary its position in the bus. The bus coupler preceeding the
block should get an alias. The block slaves can then be addressed by
specifying this alias and their position inside the block. This
scheme is used by starting the address string with a mesh (\#)
followed by the alias address (which can be hexadecimal), then a
colon and the relative posision of the slave to
address.\\
RegEx: \texttt{\#(0x[0-9A-F]+|[0-9]+):[0-9]+} --- Examples:
\texttt{"\#0xBEEF:7"}, \texttt{"\#6:2"}
\end{description}
In anticipation of section~\ref{sec:ecrt}, the functions accepting
these address strings are \textit{ecrt\_\-master\_\-get\_slave()},
\textit{ecrt\_domain\_register\_pdo()} and
\textit{ecrt\_domain\_register\_pdo\_list()} (the latter through the
\textit{ec\_pdo\_reg\_t} structure).
%------------------------------------------------------------------------------
\subsection{Concurrent Master Access}
\label{sec:concurr}
\index{Concurrency}
In some cases, one master is used by serveral instances, for example
when a realtime module does cyclic process data exchange, and there
are EoE-capable slaves that require to exchange Ethernet data with the
kernel (see section~\ref{sec:eoeimp}). For this reason, the master is
a shared resource, and access to it has to be sequenctialized. This is
usually done by locking with semaphores, or other methods to protect
critical sections.
The master itself can not provide locking mechanisms, because it has
no chance to know the appropriate kind of lock. Imagine, the realtime
module uses RTAI functionality, then ordinary kernel semaphores would
not be sufficient. For that, an important design decision was made:
The realtime module that reserved a master must have the total
control, therefore it has to take responsibility for providing the
appropriate locking mechanisms. If another instance wants to access
the master, it has to request the master lock by callbacks, that have
to be set by the realtime module. Moreover the realtime module can
deny access to the master if it consideres it to be awkward at the
moment.
\begin{figure}[htbp]
\centering
\includegraphics[width=.6\textwidth]{images/master-locks}
\caption{Concurrent master access}
\label{fig:locks}
\end{figure}
Figure~\ref{fig:locks} exemplary shows, how two processes share one
master: The realtime module's cyclic thread uses the master for
process data exchange, while the master-internal EoE process uses it
to communicate with EoE-capable slaves. Both have to aquire the master
lock before access: The realtime thread can access the lock natively,
while the EoE process has to use the master callbacks.
Section~\ref{sec:concurrency} gives an example, of how to implement
this.
%------------------------------------------------------------------------------
\section{The Master's State Machines}
\label{sec:fsm}
\index{FSM}
Many parts of the EtherCAT master are implemented as \textit{finite
state machines} (FSMs\nomenclature{FSM}{Finite State Machine}).
Though this leads to a higher grade of complexity in some aspects, is
opens many new possibilities.
The below short code example exemplary shows how to read all slave
states and moreover illustrates the restrictions of ``sequential''
coding:
\begin{lstlisting}[language=C,numbers=left]
ec_datagram_brd(datagram, 0x0130, 2); // prepare datagram
if (ec_master_simple_io(master, datagram)) return -1;
slave_states = EC_READ_U8(datagram->data); // process datagram
\end{lstlisting}
The \textit{ec\_master\_simple\_io()} function provides a simple
interface for synchronously sending a single datagram and receiving
the result\footnote{For all communication issues have been meanwhile
sourced out into state machines, the function is deprecated and
stopped existing. Nevertheless it is adequate for showing it's own
restrictions.}. Internally, it queues the specified datagram,
invokes the \textit{ec\_master\_send\_datagrams()} function to send a
frame with the queued datagram and then waits actively for its
reception.
This sequential approach is very simple, reflecting in only three
lines of code. The disadvantage is, that the master is blocked for the
time it waits for datagram reception. There is no difficulty when only
one instance is using the master, but if more instances want to
(synchronously\footnote{At this time, synchronous master access will
be adequate to show the advantages of an FSM. The asynchronous
approach will be discussed in section~\ref{sec:eoeimp}}) use the
master, it is inevitable to think about an alternative to the
sequential model.
Master access has to be sequentialized for more than one instance
wanting to send and receive datagrams synchronously. With the present
approach, this would result in having one phase of active waiting for
each instance, which would be non-acceptable especially in realtime
circumstances, because of the huge time overhead.
A possible solution is, that all instances would be executed
sequentially to queue their datagrams, then give the control to the
next instance instead of waiting for the datagram reception. Finally,
bus IO is done by a higher instance, which means that all queued
datagrams are sent and received. The next step is to execute all
instances again, which then process their received datagrams and issue
new ones.
This approach results in all instances having to retain their state,
when giving the control back to the higher instance. It is quite
obvious to use a \textit{finite state machine} model in this case.
Section~\ref{sec:fsmtheory} will introduce some of the theory used,
while the listings below show the basic approach by coding the example
from above as a state machine:
\begin{lstlisting}[language=C,numbers=left]
// state 1
ec_datagram_brd(datagram, 0x0130, 2); // prepare datagram
ec_master_queue(master, datagram); // queue datagram
next_state = state_2;
// state processing finished
\end{lstlisting}
After all instances executed their current state and queued their
datagrams, these are sent and received. Then the respective next
states are executed:
\begin{lstlisting}[language=C,numbers=left]
// state 2
if (datagram->state != EC_DGRAM_STATE_RECEIVED) {
next_state = state_error;
return; // state processing finished
}
slave_states = EC_READ_U8(datagram->data); // process datagram
// state processing finished.
\end{lstlisting}
See section~\ref{sec:statemodel} for an introduction to the
state machine programming concept used in the master code.
%------------------------------------------------------------------------------
\subsection{State Machine Theory}
\label{sec:fsmtheory}
\index{FSM!Theory}
A finite state machine \cite{automata} is a model of behavior with
inputs and outputs, where the outputs not only depend on the inputs,
but the history of inputs. The mathematical definition of a finite
state machine (or finite automaton) is a six-tuple $(\Sigma, \Gamma,
S, s_0, \delta, \omega)$, with
\begin{itemize}
\item the input alphabet $\Sigma$, with $\Sigma \neq
\emptyset$, containing all input symbols,
\item the output alphabet $\Gamma$, with $\Gamma \neq
\emptyset$, containing all output symbols,
\item the set of states $S$, with $S \neq \emptyset$,
\item the set of initial states $s_0$ with $s_0 \subseteq S, s_0 \neq
\emptyset$
\item the transition function $\delta: S \times \Sigma \rightarrow S
\times \Gamma$
\item the output function $\omega$.
\end{itemize}
The state transition function $\delta$ is often specified by a
\textit{state transition table}, or by a \textit{state transition
diagram}. The transition table offers a matrix view of the state
machine behavior (see table~\ref{tab:statetrans}). The matrix rows
correspond to the states ($S = \{s_0, s_1, s_2\}$) and the columns
correspond to the input symbols ($\Gamma = \{a, b, \varepsilon\}$).
The table contents in a certain row $i$ and column $j$ then represent
the next state (and possibly the output) for the case, that a certain
input symbol $\sigma_j$ is read in the state $s_i$.
\begin{table}[htbp]
\caption{A typical state transition table}
\label{tab:statetrans}
\vspace{2mm}
\centering
\begin{tabular}{l|ccc}
& $a$ & $b$ & $\varepsilon$\\ \hline
$s_0$ & $s_1$ & $s_1$ & $s_2$\\
$s_1$ & $s_2$ & $s_1$ & $s_0$\\
$s_2$ & $s_0$ & $s_0$ & $s_0$\\ \hline
\end{tabular}
\end{table}
The state diagram for the same example looks like the one in
figure~\ref{fig:statetrans}. The states are represented as circles or
ellipses and the transitions are drawn as arrows between them. Close
to a transition arrow can be the condition that must be fulfilled to
allow the transition. The initial state is marked by a filled black
circle with an arrow pointing to the respective state.
\begin{figure}[htbp]
\centering
\includegraphics[width=.5\textwidth]{images/statetrans}
\caption{A typical state transition diagram}
\label{fig:statetrans}
\end{figure}
\paragraph{Deterministic and non-deterministic state machines}
A state machine can be deterministic, meaning that for one state and
input, there is one (and only one) following state. In this case, the
state machine has exactly one starting state. Non-deterministic state
machines can have more than one transitions for a single state-input
combination. There is a set of starting states in the latter case.
\paragraph{Moore and Mealy machines}
There is a distinction between so-called \textit{Moore machines}, and
\textit{Mealy machines}. Mathematically spoken, the distinction lies
in the output function $\omega$: If it only depends on the current
state ($\omega: S \rightarrow \Gamma$), the machine corresponds to the
``Moore Model''. Otherwise, if $\omega$ is a function of a state and
the input alphabet ($\omega: S \times \Sigma \rightarrow \Gamma$) the
state machine corresponds to the ``Mealy model''. Mealy machines are
the more practical solution in most cases, because their design allows
machines with a minimum number of states. In practice, a mixture of
both models is often used.
\paragraph{Misunderstandings about state machines}
There is a phenomenon called ``state explosion'', that is oftenly
taken as a counter-argument against general use of state machines in
complex environments. It has to be mentioned, that this point is
misleading~\cite{fsmmis}. State explosions happen usually as a result
of a bad state machine design: Common mistakes are storing the present
values of all inputs in a state, or not dividing a complex state
machine into simpler sub state machines. The EtherCAT master uses
serveral state machines, that are executed hierarchically and so serve
as sub state machines. These are also described below.
%------------------------------------------------------------------------------
\subsection{The Master's State Model}
\label{sec:statemodel}
This section will introduce the techniques used in the master to
implement state machines.
\paragraph{State Machine Programming}
There are certain ways to implement a state machine in \textit{C}
code. An obvious way is to implement the different states and actions
by one big case differentiation:
\begin{lstlisting}[language=C,numbers=left]
enum {STATE_1, STATE_2, STATE_3};
int state = STATE_1;
void state_machine_run(void *priv_data) {
switch (state) {
case STATE_1:
action_1();
state = STATE_2;
break;
case STATE_2:
action_2()
if (some_condition) state = STATE_1;
else state = STATE_3;
break;
case STATE_3:
action_3();
state = STATE_1;
break;
}
}
\end{lstlisting}
For small state machines, this is an option. The disadvantage is, that
with an increasing number of states the code soon gets complex and an
additional case differentiation is executed each run. Besides, lots of
indentation is wasted.
The method used in the master is to implement every state in an own
function and to store the current state function with a function
pointer:
\begin{lstlisting}[language=C,numbers=left]
void (*state)(void *) = state1;
void state_machine_run(void *priv_data) {
state(priv_data);
}
void state1(void *priv_data) {
action_1();
state = state2;
}
void state2(void *priv_data) {
action_2();
if (some_condition) state = state1;
else state = state2;
}
void state3(void *priv_data) {
action_3();
state = state1;
}
\end{lstlisting}
In the master code, state pointers of all state machines\footnote{All
except for the EoE state machine, because multiple EoE slaves have
to be handled in parallel. For this reason each EoE handler object
has its own state pointer.} are gathered in a single object of the
\textit{ec\_fsm\_t} class. This is advantageous, because there is
always one instance of every state machine available and can be
started on demand.
\paragraph{Mealy and Moore}
If a closer look is taken to the above listing, it can be seen that
the actions executed (the ``outputs'' of the state machine) only
depend on the current state. This accords to the ``Moore'' model
introduced in section~\ref{sec:fsmtheory}. As mentioned, the ``Mealy''
model offers a higher flexibility, which can be seen in the listing
below:
\begin{lstlisting}[language=C,numbers=left]
void state7(void *priv_data) {
if (some_condition) {
action_7a();
state = state1;
}
else {
action_7b();
state = state8;
}
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 3} + \textcircled{\tiny 7}] The
state function executes the actions depending on the state
transition, that is about to be done.
\end{description}
The most flexible alternative is to execute certain actions depending
on the state, followed by some actions dependent on the state
transition:
\begin{lstlisting}[language=C,numbers=left]
void state9(void *priv_data) {
action_9();
if (some_condition) {
action_9a();
state = state7;
}
else {
action_9b();
state = state10;
}
}
\end{lstlisting}
This model is oftenly used in the master. It combines the best aspects
of both approaches.
\paragraph{Using Sub State Machines}
To avoid having too much states, certain functions of the EtherCAT
master state machine have been sourced out into sub state machines.
This helps to encapsule the related workflows and moreover avoids the
``state explosion'' phenomenon described in
section~\ref{sec:fsmtheory}. If the master would instead use one big
state machine, the number of states would be a multiple of the actual
number. This would increase the level of complexity to a
non-manageable grade.
\paragraph{Executing Sub State Machines}
If a state machine starts to execute a sub state machine, it usually
remains in one state until the sub state machine terminates. This is
usually done like in the listing below, which is taken out of the
slave configuration state machine code:
\begin{lstlisting}[language=C,numbers=left]
void ec_fsm_slaveconf_safeop(ec_fsm_t *fsm)
{
fsm->change_state(fsm); // execute state change
// sub state machine
if (fsm->change_state == ec_fsm_error) {
fsm->slave_state = ec_fsm_end;
return;
}
if (fsm->change_state != ec_fsm_end) return;
// continue state processing
...
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 3}] \textit{change\_state} is the
state pointer of the state change state machine. The state function,
the pointer points on, is executed\ldots
\item[\normalfont\textcircled{\tiny 6}] \ldots either until the state
machine terminates with the error state \ldots
\item[\normalfont\textcircled{\tiny 11}] \ldots or until the state
machine terminates in the end state. Until then, the ``higher''
state machine remains in the current state and executes the sub
state machine again in the next cycle.
\end{description}
\paragraph{State Machine Descriptions}
The below sections describe every state machine used in the EtherCAT
master. The textual descriptions of the state machines contain
references to the transitions in the corresponding state transition
diagrams, that are marked with an arrow followed by the name of the
successive state. Transitions caused by trivial error cases (i.~e. no
response from slave) are not described explicitly. These transitions
are drawn as dashed arrows in the diagrams.
%------------------------------------------------------------------------------
\subsection{The Operation State Machine}
\label{sec:fsm-op}
\index{FSM!Operation}
The Operation state machine is executed by calling the
\textit{ecrt\_master\_run()} method in cyclic realtime code. Its
purpose is to monitor the bus and to reconfigure slaves after a bus
failure or power failure. Figure~\ref{fig:fsm-op} shows its transition
diagram.
\begin{figure}[htbp]
\centering
\includegraphics[width=.8\textwidth]{images/fsm-op}
\caption{Transition diagram of the operation state machine}
\label{fig:fsm-op}
\end{figure}
\begin{description}
\item[START] This is the beginning state of the operation state
machine. There is a datagram issued, that queries the ``AL Control
Response'' attribute \cite[section~5.3.2]{alspec} of all slaves via
broadcast. In this way, all slave states and the number of slaves
responding can be determined. $\rightarrow$~BROADCAST
\item[BROADCAST] The broadcast datagram is evaluated. A change in the
number of responding slaves is treates as a topology change. If the
number of slaves is not as expected, the bus is marked as
``tainted''. In this state, no slave reconfiguration is possible,
because the assignment of known slaves and those present on the bus
is ambiguous. If the number of slaves is considered as right, the
bus is marked for validation, because it turned from tainted to
normal state and it has to be checked, if all slaves are valid. Now,
the state of every single slave has to be determined. For that, a
(unicast) datagram is issued, that queries the first slave's ``AL
Control Response'' attribute. $\rightarrow$~READ STATES
\item[READ STATES] If the current slave did not respond to its
configured station address, it is marked as offline, and the next
slave is queried. $\rightarrow$~READ STATES
If the slave responded, it is marked as online and its current state
is stored. The next slave is queried. $\rightarrow$~READ STATES
If all slaves have been queried, and the bus is marked for
validation, the validation is started by checking the first slaves
vendor ID. $\rightarrow$~VALIDATE VENDOR
If no validation has to be done, it is checked, if all slaves are in
the state they are supposed to be. If not, the first of slave with
the wrong state is reconfigured and brought in the required state.
$\rightarrow$~CONFIGURE SLAVES
If all slaves are in the correct state, the state machine is
restarted. $\rightarrow$~START
\item[CONFIGURE SLAVES] The slave configuration state machine is
executed until termination. $\rightarrow$~CONFIGURE SLAVES
If there are still slaves in the wrong state after another check,
the first of these slaves is configured and brought into the correct
state again. $\rightarrow$~CONFIGURE SLAVES
If all slaves are in the correct state, the state machine is
restarted. $\rightarrow$~START
\item[VALIDATE VENDOR] The SII state machine is executed until
termination. If the slave has the wrong vendor ID, the state machine
is restarted. $\rightarrow$~START
If the slave has the correct vendor ID, its product ID is queried.
$\rightarrow$~VALIDATE PRODUCT
\item[VALIDATE PRODUCT] The SII state machine is executed until
termination. If the slave has the wrong product ID, the state
machine is restarted. $\rightarrow$~START
If the slave has the correct product ID, the next slave's vendor ID
is queried. $\rightarrow$~VALIDATE VENDOR
If all slaves have the correct vendor IDs and product codes, the
configured station addresses can be safely rewritten. This is done
for the first slave marked as offline.
$\rightarrow$~REWRITE ADDRESSES
\item[REWRITE ADDRESSES] If the station address was successfully
written, it is sear\-ched for the next slave marked as offline. If
there is one, its address is reconfigured, too.
$\rightarrow$~REWRITE ADDRESSES
If there are no more slaves marked as offline, the state machine is
restarted. $\rightarrow$~START
\end{description}
%------------------------------------------------------------------------------
\subsection{The Idle State Machine}
\label{sec:fsm-idle}
\index{FSM!Idle}
The Idle state machine is executed by a kernel workqueue, if no
realtime module is connected. Its purpose is to make slave information
available to user space, operate EoE-capable slaves, read and write
E$^2$PROM contents and test slave functionality.
Figure~\ref{fig:fsm-idle} shows its transition diagram.
\begin{figure}[htbp]
\centering
\includegraphics[width=.8\textwidth]{images/fsm-idle}
\caption{Transition diagram of the idle state machine}
\label{fig:fsm-idle}
\end{figure}
\begin{description}
\item[START] The beginning state of the idle state machine. Similar to
the operation state machine, a broadcast datagram is issued, to
query all slave states and the number of slaves.
$\rightarrow$~BROADCAST
\item[BROADCAST] The number of responding slaves is evaluated. If it
has changed since the last time, this is treated as a topology
change and the internal list of slaves is cleared and rebuild
completely. The slave scan state machine is started for the first
slave. $\rightarrow$~SCAN FOR SLAVES
If no topology change happened, every single slave state is fetched.
$\rightarrow$~READ STATES
\item[SCAN FOR SLAVES] The slave scan state machine is executed until
termination. $\rightarrow$~SCAN FOR SLAVES
If there is another slave to scan, the slave scan state machine is
started again. $\rightarrow$~SCAN FOR SLAVES
If all slave information has been fetched, slave addresses are
calculated and EoE processing is started. Then, the state machine is
restarted. $\rightarrow$~START
\item[READ STATES] If the slave did not respond to the query, it is
marked as offline. The next slave is queried.
$\rightarrow$~READ STATES
If the slave responded, it is marked as online. And the next slave
is queried. $\rightarrow$~READ STATES
If all slave states have been determined, it is checked, if any
slaves are not in the state they supposed to be. If this is true,
the slave configuration state machine is started for the first of
them. $\rightarrow$~CONFIGURE SLAVES
If all slaves are in the correct state, it is checked, if any
E$^2$PROM write operations are pending. If this is true, the first
pending operation is executed by starting the SII state machine for
writing access. $\rightarrow$~WRITE EEPROM
If all these conditions are false, there is nothing to do and the
state machine is restarted. $\rightarrow$~START
\item[CONFIGURE SLAVES] The slave configuration state machine is
executed until termination. $\rightarrow$~CONFIGURE SLAVES
After this, it is checked, if another slave needs a state change. If
this is true, the slave state change state machine is started for
this slave. $\rightarrow$~CONFIGURE SLAVES
If all slaves are in the correct state, it is determined, if any
E$^2$PROM write operations are pending. If this is true, the first
pending operation is executed by starting the SII state machine for
writing access. $\rightarrow$~WRITE EEPROM
If all prior conditions are false, the state machine is restarted.
$\rightarrow$~START
\item[WRITE EEPROM] The SII state machine is executed until
termination. $\rightarrow$~WRITE EEPROM
If the current word has been written successfully, and there are
still word to write, the SII state machine is started for the next
word. $\rightarrow$~WRITE EEPROM
If all words have been written successfully, the new E$^2$PROM
contents are evaluated and the state machine is restarted.
$\rightarrow$~START
\end{description}
%------------------------------------------------------------------------------
\subsection{The Slave Scan State Machine}
\label{sec:fsm-scan}
\index{FSM!Slave Scan}
The slave scan state machine, which can be seen in
figure~\ref{fig:fsm-slavescan}, leads through the process of fetching
all slave information.
\begin{figure}[htbp]
\centering
\includegraphics[width=.6\textwidth]{images/fsm-slavescan}
\caption{Transition diagram of the slave scan state machine}
\label{fig:fsm-slavescan}
\end{figure}
\begin{description}
\item[START] In the beginning state of the slave scan state machine,
the station address is written to the slave, which is always the
ring position~+~$1$. In this way, the address 0x0000 (default
address) is not used, which makes it easy to detect unconfigured
slaves. $\rightarrow$~ADDRESS
\item[ADDRESS] The writing of the station address is verified. After
that, the slave's ``AL Control Response'' attribute is queried.
$\rightarrow$~STATE
\item[STATE] The AL state is evaluated. A warning is output, if the
slave has still the \textit{Change} bit set. After that, the slave's
``DL Information'' attribute is queried.
$\rightarrow$~BASE
\item[BASE] The queried base data are evaluated: Slave type, revision
and build number, and even more important, the number of supported
sync managers and FMMUs are stored. After that, the slave's data
link layer information is read from the ``DL Status'' attribute at
address 0x0110. $\rightarrow$~DATALINK
\item[DATALINK] In this state, the DL information is evaluated: This
information about the communication ports contains, if the link is
up, if the loop has been closed and if there is a carrier detected
on the RX side of each port.
Then, the state machine starts measuring the size of the slave's
E$^2$PROM contents. This is done by subsequently reading out each
category header, until the last category is reached (type 0xFFFF).
This procedure is started by querying the first category header at
word address 0x0040 via the SII state machine.
$\rightarrow$~EEPROM SIZE
\item[EEPROM SIZE] The SII state machine is executed until
termination. $\rightarrow$~EEPROM SIZE
If the category type does not mark the end of the categories, the
position of the next category header is determined via the length of
the current category, and the SII state machine is started again.
$\rightarrow$~EEPROM SIZE
If the size of the E$^2$PROM contents has been determined, memory is
allocated, to read all the contents. The SII state machine is
started to read the first word. $\rightarrow$~EEPROM DATA
\item[EEPROM DATA] The SII state machine is executed until
termination. $\rightarrow$~EEPROM DATA
Two words have been read. If more than one word is needed, the two
words are written in the allocated memory. Otherwise only one word
(the last word) is copied. If more words are to read, the SII state
machine is started again to read the next two words.
$\rightarrow$~EEPROM DATA
The complete E$^2$PROM contents have been read. The slave's identity
object and mailbox information are evaluated. Moreover the category
types STRINGS, GENERAL, SYNC and PDO are evaluated. The slave
scanning has been completed. $\rightarrow$~END
\item[END] Slave scanning has been finished.
\end{description}
%------------------------------------------------------------------------------
\subsection{The Slave Configuration State Machine}
\label{sec:fsm-conf}
\index{FSM!Slave Configuration}
The slave configuration state machine, which can be seen in
figure~\ref{fig:fsm-slaveconf}, leads through the process of
configuring a slave and bringing it to a certain state.
\begin{figure}[htbp]
\centering
\includegraphics[width=.6\textwidth]{images/fsm-slaveconf}
\caption{Transition diagram of the slave configuration state
machine}
\label{fig:fsm-slaveconf}
\end{figure}
\begin{description}
\item[INIT] The state change state machine has been initialized to
bring the slave into the INIT state. Now, the slave state change
state machine is executed until termination. $\rightarrow$~INIT
If the slave state change failed, the configuration has to be
aborted. $\rightarrow$~END
The slave state change succeeded and the slave is now in INIT state.
If this is the target state, the configuration is finished.
$\rightarrow$~END
If the slave does not support any sync managers, the sync manager
configuration can be skipped. The state change state machine is
started to bring the slave into PREOP state.
$\rightarrow$~PREOP
Sync managers are configured conforming to the sync manager category
information provided in the slave's E$^2$PROM. The corresponding
datagram is issued. $\rightarrow$~SYNC
\item[SYNC] If the sync manager configuration datagram is accepted,
the sync manager configuration was successful. The slave may now
enter the PREOP state, and the state change state machine is
started. $\rightarrow$~PREOP
\item[PREOP] The state change state machine is executed until
termination. $\rightarrow$~PREOP
If the state change failed, the configuration has to be aborted.
$\rightarrow$~END
If the PREOP state was the target state, the configuration is
finished. $\rightarrow$~END
If the slave supports no FMMUs, the FMMU configuration can be
skipped. If the slave has Sdos to configure, it is begun with
sending the first Sdo. $\rightarrow$~SDO\_CONF
If no Sdo configurations are provided, the slave can now directly be
brought into the SAFEOP state and the state change state machine is
started again. $\rightarrow$~SAFEOP
Otherwise, all supported FMMUs are configured according to the Pdos
requested via the master's realtime interface. The appropriate
datagram is issued. $\rightarrow$~FMMU
\item[FMMU] The FMMU configuration datagram was accepted. If the slave
has Sdos to configure, it is begun with sending the first Sdo.
$\rightarrow$~SDO\_CONF
Otherwise, the slave can now be brought into the SAFEOP state. The
state change state machine is started.
$\rightarrow$~SAFEOP
\item[SDO\_CONF] The CoE state machine is executed until termination.
$\rightarrow$~SDO\_CONF
If another Sdo has to be configured, a new Sdo download sequence is
begun. $\rightarrow$~SDO\_CONF
Otherwise, the slave can now be brought into the SAFEOP state. The
state change state machine is started.
$\rightarrow$~SAFEOP
\item[SAFEOP] The state change state machine is executed until
termination. $\rightarrow$~SAFEOP
If the state change failed, the configuration has to be aborted.
$\rightarrow$~END
If the SAFEOP state was the target state, the configuration is
finished. $\rightarrow$~END
The slave can now directly be brought into the OP state and the
state change state machine is started a last time.
$\rightarrow$~OP
\item[OP] The state change state machine is executed until
termination. $\rightarrow$~OP
If the state change state machine terminates, the slave
configuration is finished, regardless of its success.
$\rightarrow$~END
\item[END] The termination state.
\end{description}
%------------------------------------------------------------------------------
\subsection{The State Change State Machine}
\label{sec:fsm-change}
\index{FSM!State Change}
The state change state machine, which can be seen in
figure~\ref{fig:fsm-change}, leads through the process of changing a
slave's state. This implements the states and transitions described in
\cite[section~6.4.1]{alspec}.
\begin{figure}[htbp]
\centering
\includegraphics[width=.9\textwidth]{images/fsm-change}
\caption{Transition diagram of the state change state machine}
\label{fig:fsm-change}
\end{figure}
\begin{description}
\item[START] The beginning state, where a datagram with the state
change command is written to the slave's ``AL Control Request''
attribute. Nothing can fail. $\rightarrow$~CHECK
\item[CHECK] After the state change datagram has been sent, the ``AL
Control Response'' attribute is queried with a second datagram.
$\rightarrow$~STATUS
\item[STATUS] The read memory contents are evaluated: While the
parameter \textit{State} still contains the old slave state, the
slave is busy with reacting on the state change command. In this
case, the attribute has to be queried again.
$\rightarrow$~STATUS
In case of success, the \textit{State} parameter contains the new
state and the \textit{Change} bit is cleared. The slave is in the
requested state. $\rightarrow$~END
If the slave can not process the state change, the \textit{Change}
bit is set: Now the master tries to get the reason for this by
querying the \textit{AL Status Code} parameter.
$\rightarrow$~CODE
\item[END] If the state machine ends in this state, the slaves's state
change has been successful.
\item[CODE] The status code query has been sent. Reading the
\textit{AL Status Code} might fail, because not all slaves support
this parameter. Anyway, the master has to acknowledge the state
change error by writing the current slave state to the ``AL Control
Request'' attribute with the \textit{Acknowledge} bit set.
$\rightarrow$~ACK
\item[ACK] After that, the ``AL Control Response'' attribute is
queried for the state of the acknowledgement.
$\rightarrow$~CHECK ACK
\item[CHECK ACK] If the acknowledgement has been accepted by the
slave, the old state is kept. Still, the state change was
unsuccessful. $\rightarrow$~ERROR
If the acknowledgement is ignored by the slave, a timeout happens.
In any case, the overall state change was unsuccessful.
$\rightarrow$~ERROR
If there is still now response from the slave, but the timer did not
run out yet, the slave's ``AL Control Response'' attribute is
queried again. $\rightarrow$~CHECK ACK
\item[ERROR] If the state machine ends in this state, the slave's
state change was unsuccessful.
\end{description}
%------------------------------------------------------------------------------
\subsection{The SII State Machine}
\label{sec:fsm-sii}
\index{FSM!SII}
The SII\index{SII} state machine (shown in figure~\ref{fig:fsm-sii})
implements the process of reading or writing E$^2$PROM data via the
Slave Information Interface described in \cite[section~5.4]{alspec}.
\begin{figure}[htbp]
\centering
\includegraphics[width=.9\textwidth]{images/fsm-sii}
\caption{Transition diagram of the SII state machine}
\label{fig:fsm-sii}
\end{figure}
\begin{description}
\item[READ\_START] The beginning state for reading access, where the
read request and the requested address are written to the SII
attribute. Nothing can fail up to now.
$\rightarrow$~READ\_CHECK
\item[READ\_CHECK] When the SII read request has been sent
successfully, a timer is started. A check/fetch datagram is issued,
that reads out the SII attribute for state and data.
$\rightarrow$~READ\_FETCH
\item[READ\_FETCH] Upon reception of the check/fetch datagram, the
\textit{Read Operation} and \textit{Busy} parameters are checked:
\begin{itemize}
\item If the slave is still busy with fetching E$^2$PROM data into
the interface, the timer is checked. If it timed out, the reading
is aborted ($\rightarrow$~ERROR), if not, the check/fetch datagram
is issued again. $\rightarrow$~READ\_FETCH
\item If the slave is ready with reading data, these are copied from
the datagram and the read cycle is completed.
$\rightarrow$~END
\end{itemize}
\end{description}
The write access states behave nearly the same:
\begin{description}
\item[WRITE\_START] The beginning state for writing access,
respectively. A write request, the target address and the data word
are written to the SII attribute. Nothing can fail.
$\rightarrow$~WRITE\_CHECK
\item[WRITE\_CHECK] When the SII write request has been sent
successfully, the timer is started. A check datagram is issued, that
reads out the SII attribute for the state of the write operation.
$\rightarrow$~WRITE\_CHECK2
\item[WRITE\_CHECK2] Upon reception of the check datagram, the
\textit{Write Operation} and \textit{Busy} parameters are checked:
\begin{itemize}
\item If the slave is still busy with writing E$^2$PROM data, the
timer is checked. If it timed out, the operation is aborted
($\rightarrow$~ERROR), if not, the check datagram is issued again.
$\rightarrow$~WRITE\_CHECK2
\item If the slave is ready with writing data, the write cycle is
completed. $\rightarrow$~END
\end{itemize}
\end{description}
%------------------------------------------------------------------------------
\section{Mailbox Protocol Implementations}
\index{Mailbox}
The EtherCAT master implements the EoE and the CoE mailbox
protocols. See the below section for details.
%------------------------------------------------------------------------------
\subsection{Ethernet-over-EtherCAT (EoE)}
\label{sec:eoeimp}
\index{EoE}
The EtherCAT master implements the Ethernet-over-EtherCAT mailbox
protocol to enable the tunneling of Ethernet frames to special slaves,
that can either have physical Ethernet ports to forward the frames to,
or have an own IP stack to receive the frames.
\paragraph{Virtual Network Interfaces}
The master creates a virtual EoE network interface for every
EoE-capable slave. These interfaces are called \textit{eoeX}, where X
is a number provided by the kernel on interface registration. Frames
sent to these interfaces are forwarded to the associated slaves by the
master. Frames, that are received by the slaves, are fetched by the
master and forwarded to the virtual interfaces.
This bears the following advantages:
\begin{itemize}
\item Flexibility: The user can decide, how the EoE-capable slaves are
interconnected with the rest of the world.
\item Standard tools can be used to monitor the EoE activity and to
configure the EoE interfaces.
\item The Linux kernel's layer-2-bridging implementation (according to
the IEEE 802.1D MAC Bridging standard) can be used natively to
bridge Ethernet traffic between EoE-capable slaves.
\item The Linux kernel's network stack can be used to route packets
between EoE-capable slaves and to track security issues, just like
having physical network interfaces.
\end{itemize}
\paragraph{EoE Handlers}
The virtual EoE interfaces and the related functionality is encapsuled
in the \textit{ec\_eoe\_t} class (see section~\ref{sec:class-eoe}).
So the master does not create the network interfaces directly: This is
done inside the constructor of the \textit{ec\_eoe\_t} class. An
object of this class is called ``EoE handler'' below. An EoE handler
additionaly contains a frame queue. Each time, the kernel passes a new
socket buffer for sending via the interface's
\textit{hard\_start\_xmit()} callback, the socket buffer is queued for
transmittion by the EoE state machine (see below). If the queue gets
filled up, the passing of new socket buffers is suspended with a call
to \textit{netif\_stop\_queue()}.
\paragraph{Static Handler Creation}
The master creates a pool of EoE handlers at startup, that are coupled
to EoE-capable slaves on demand. The lifetime of the corresponding
network interfaces is equal to the lifetime of the master module.
This approach is opposed to creating the virtual network interfaces on
demand (i.~e. on running across a new EoE-capable slave). The latter
approach was considered as difficult, because of serveral reasons:
\begin{itemize}
\item The \textit{alloc\_netdev()} function can sleep and must be
called from a non-interrupt context. This reduces the flexibility of
choosing an appropriate method for cyclic EoE processing.
\item Unregistering network interfaces requires them to be ``down'',
which can not be guaranteed upon sudden disappearing of an
EoE-capable slave.
\item The connection to the EoE-capable slaves must be as continuous
as possible. Especially the transition from idle to operation mode
(and vice versa) causes the rebuilding of the internal data
structures. These transitions must be as transparent as possible for
the instances using the network interfaces.
\end{itemize}
\paragraph{Number of Handlers}
The master module has a parameter \textit{ec\_eoeif\_count} to specify
the number of EoE interfaces (and handlers) per master to create. This
parameter can either be specified when manually loading the master
module, or (when using the init script) by setting the
\$EOE\_INTERFACES variable in the sysconfig file (see
section~\ref{sec:sysconfig}). Upon loading of the master module, the
virtual interfaces become available:
\begin{lstlisting}
# `\textbf{ifconfig -a}`
eoe0 Link encap:Ethernet HWaddr 00:11:22:33:44:06
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
eoe1 Link encap:Ethernet HWaddr 00:11:22:33:44:07
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
...
\end{lstlisting}
\paragraph{Coupling of EoE Slaves}
During execution of the slave scan state machine (see
section~\ref{sec:fsm-scan}), the master determines the supported
mailbox protocols. This is done by examining the ``Supported Mailbox
Protocols'' mask field at word address 0x001C of the SII\index{SII}.
If bit 1 is set, the slave supports the EoE protocol. After slave
scanning, the master runs through all slaves again and couples each
EoE-capable slave to a free EoE handler. It can happen, that there are
not enough EoE handlers to cover all EoE-capable slaves. In this case,
the number of EoE handlers must be increased accordingly.
\paragraph{EoE State Machine}
\index{FSM!EoE}
Every EoE handler owns an EoE state machine, that is used to send
frames to the coupled slave and receive frames from the it via the EoE
communication primitives. This state machine is showed in
figure~\ref{fig:fsm-eoe}.
\begin{figure}[htbp]
\centering
\includegraphics[width=.7\textwidth]{images/fsm-eoe}
\caption{Transition diagram of the EoE state machine}
\label{fig:fsm-eoe}
\end{figure}
\begin{description}
\item[RX\_START] The beginning state of the EoE state machine. A
mailbox check datagram is sent, to query the slave's mailbox for new
frames. $\rightarrow$~RX\_CHECK
\item[RX\_CHECK] The mailbox check datagram is received. If the
slave's mailbox did not contain data, a transmit cycle is started.
$\rightarrow$~TX\_START
If there are new data in the mailbox, a datagram is sent to fetch
the new data. $\rightarrow$~RX\_FETCH
\item[RX\_FETCH] The fetch datagram is received. If the mailbox data
do not contain a ``EoE Fragment request'' command, the data are
dropped and a transmit sequence is started.
$\rightarrow$~TX\_START
If the received Ethernet frame fragment is the first fragment, a new
socket buffer is allocated. In either case, the data are copied into
the correct position of the socket buffer.
If the fragment is the last fragment, the socket buffer is forwarded
to the network stack and a transmit sequence is started.
$\rightarrow$~TX\_START
Otherwise, a new receive sequence is started to fetch the next
fragment. $\rightarrow$~RX\_\-START
\item[TX\_START] The beginning state of a transmit sequence. It is
checked, if the transmittion queue contains a frame to send. If not,
a receive sequence is started. $\rightarrow$~RX\_START
If there is a frame to send, it is dequeued. If the queue was
inactive before (because it was full), the queue is woken up with a
call to \textit{netif\_wake\_queue()}. The first fragment of the
frame is sent. $\rightarrow$~TX\_SENT
\item[TX\_SENT] It is checked, if the first fragment was sent
successfully. If the current frame consists of further fragments,
the next one is sent. $\rightarrow$~TX\_SENT
If the last fragment was sent, a new receive sequence is started.
$\rightarrow$~RX\_START
\end{description}
\paragraph{EoE Processing}
To execute the EoE state machine of every active EoE handler, there
must be a cyclic process. The easiest thing would be to execute the
EoE state machines synchronously to the operation state machine (see
section~\ref{sec:fsm-op}) with every realtime cycle. This approach has
the following disadvantages:
\begin{itemize}
\item Only one EoE fragment can be sent or received every few cycles.
This causes the data rate to be very low, because the EoE state
machines are not executed in the time between the realtime
cycles. Moreover, the data rate would be dependent on the frequency
of the realtime process.
\item The receiving and forwarding of frames to the kernel requires
the dynamic allocation of frames. Some realtime extensions do not
support calling memory allocation functions in realtime context, so
the EoE state machine may not be executed with each realtime cycle.
\end{itemize}
To overcome these problems, an own cyclic process is needed to
asynchronously execute the EoE state machines. For that, the master
owns a kernel timer, that is executed each timer interrupt. This
guarantees a constant bandwidth, but poses the new problem of
concurrent access to the master. The locking mechanisms needed for
this are introduced in section~\ref{sec:concurr}.
Section~\ref{sec:concurrency} gives practical implementation examples.
\paragraph{Idle Mode}
EoE data must also be exchanged idle mode, to guarantee the continuous
availability of the connection to the EoE-capable slaves. Although
there is no realtime module connected in this case, the master is
still accessed by the idle state machine (see
section~\ref{sec:fsm-idle}), that is executed by the master's
workqueue. With the EoE timer running in addition, there is still
concurrency, that has to be protected by a lock. Therefore the master
owns an internal spinlock that is used protect master access during
idle mode.
\paragraph{Automatic Configuration}
By default, slaves are left in INIT state during idle mode. If an EoE
interface is set to running state (i.~e. with the \textit{ifconfig up}
command), the requested slave state of the related slave is
automatically set to OP, whereupon the idle state machine will attempt
to configure the slave and put it into operation.
%------------------------------------------------------------------------------
\subsection{CANopen-over-EtherCAT (CoE)}
\label{sec:coeimp}
\index{CoE}
The CANopen-over-EtherCAT protocol \cite[section~5.6]{alspec} is used
to configure slaves on application level. Each CoE-capable slave
provides a list of Sdos for this reason.
\paragraph{Sdo Configuration}
The Sdo configurations have to be provided by the realtime module.
This is done via the \textit{ecrt\_slave\_conf\_sdo*()} methods (see
section~\ref{sec:ecrt-slave}), that are part of the realtime
interface. The slave stores the Sdo configurations in a linked list,
but does not apply them at once.
\paragraph{Sdo Download State Machine}
The best time to apply Sdo configurations is during the slave's PREOP
state, because mailbox communication is already possible and slave's
application will start with updating input data in the succeeding
SAFEOP state. Therefore the Sdo configuration has to be part of the
slave configuration state machine (see section~\ref{sec:fsm-conf}): It
is implemented via an Sdo download state machine, that is executed
just before entering the slave's SAFEOP state. In this way, it is
guaranteed that the Sdo configurations are applied each time, the
slave is reconfigured.
The transition diagram of the Sdo Download state machine can be seen
in figure~\ref{fig:fsm-coedown}.
\begin{figure}[htbp]
\centering
\includegraphics[width=.9\textwidth]{images/fsm-coedown}
\caption{Transition diagram of the CoE download state machine}
\label{fig:fsm-coedown}
\end{figure}
\begin{description}
\item[START] The beginning state of the CoE download state
machine. The ``Sdo Download Normal Request'' mailbox command is
sent. $\rightarrow$~REQUEST
\item[REQUEST] It is checked, if the CoE download request has been
received by the slave. After that, a mailbox check command is issued
and a timer is started. $\rightarrow$~CHECK
\item[CHECK] If no mailbox data is available, the timer is checked.
\begin{itemize}
\item If it timed out, the Sdo download is aborted.
$\rightarrow$~ERROR
\item Otherwise, the mailbox is queried again.
$\rightarrow$~CHECK
\end{itemize}
If the mailbox contains new data, the response is fetched.
$\rightarrow$~RESPONSE
\item[RESPONSE] If the mailbox response could not be fetched, the data
is invalid, the wrong protocol was received, or a ``Abort Sdo
Transfer Request'' was received, the Sdo download is aborted.
$\rightarrow$~ERROR
If a ``Sdo Download Normal Response'' acknowledgement was received,
the Sdo download was successful. $\rightarrow$~END
\item[END] The Sdo download was successful.
\item[ERROR] The Sdo download was aborted due to an error.
\end{description}
%------------------------------------------------------------------------------
\section{User Space}
\label{sec:user}
\index{User space}
For the master runs as a kernel module, accessing it is natively
limited to analyzing syslog messages and controlling using modutils.
It is necessary to implement further interfaces, that make it easier
to access the master from user space and allow a finer influence. It
should be possible to view and to change special parameters at runtime.
Bus visualization is a second point: For development and debugging
purposes it would be nice, if one could show the connected slaves with
a single command.
Another aspect is automatic startup and configuration. If the master
is to be integrated into a running system, it must be able to
automatically start with a persistent configuration.
A last thing is monitoring EtherCAT communication. For debugging
purposes, there had to be a way to analyze EtherCAT datagrams. The
best way would be with a popular network analyzer, like Wireshark
\cite{wireshark} (the former Ethereal) or others.
This section covers all those points and introduces the interfaces and
tools to make all that possible.
%------------------------------------------------------------------------------
\subsection{The Sysfs Interface}
\label{sec:sysfs}
The system filesystem (Sysfs\index{Sysfs}) was introduced with Linux
kernel 2.5 and is a well-defined interface for drivers to export
information to user space. It serves also as an relief for the process
filesystem (Procfs), where over the years much non-process information
was concentrated.
Sysfs exports information about devices, classes and busses via a
virtual filesystem, usually mounted to \textit{/sys}. The EtherCAT
master slightly differs from this concept, because the only physical
device is the network adapter it uses for bus communication, which is
already represented in Sysfs. For the EtherCAT bus is no system bus
like PCI (with device and driver structures), it would not make any
sense to represent it as bus structure in Sysfs.
Therefore, the EtherCAT master is represented as a new directory
directly unter the Sysfs root. Every master gets its own Sysfs entry
named \textit{ethercatX}, where X is the index of the master. Two
masters would result in the directories \textit{/sys/ethercat0} and
\textit{/sys/ethercat1}, respectively.
The Sysfs base class in the kernel code is the \textit{kobject}
structure. Each object structure, that is to be represented in Sysfs,
has to contain such a structure, because due to the concurrent access
(through ``normal'' kernel code and Sysfs code) the object deletion
gets a little more complicated: The object may not be freed until no
instance uses it any more. Therefore, each kobject maintains a
reference counter. If the reference counter gets zero, the object is
finally freed. A kobject can be registered to appear as a directory in
Sysfs with a call to \textit{kobject\_add()}. Each kobject type can
define attributes, that appear as files in the kobject's
directory. Callback functions have to be provided for reading (and
perhaps writing) access.
\subsubsection{Master Attributes}
\label{sec:sysfs-master}
Below is a typical listing of the masters Sysfs directory (that is a
file system representation of the master's kobject):
\begin{lstlisting}
`\$` `\textbf{ls /sys/ethercat0}`
debug_level slave000 slave003 slave006
eeprom_write_enable slave001 slave004 slave007
info slave002 slave005 slave008
\end{lstlisting}
The following attributes exist in the master directory:
\begin{description}
\item[debug\_level] (read/write) This attribute contains the master's
debug level, which controls, how much information is printed into
syslog. The values 0 (no additional debug messages), 1 (a few
additional debug messages) and 2 (all additional debug messages) are
defined. Writing is done with command like
\begin{lstlisting}[gobble=4]
# `\textbf{echo 1 > /sys/ethercat0/debug\_level}`
\end{lstlisting}
and is receipted with a syslog message by the master:
\begin{lstlisting}[gobble=4]
EtherCAT: Master debug level set to 1.
\end{lstlisting}
\item[enable\_eeprom\_writing] (read/write) See
section~\ref{sec:eepromaccess} for how to use this attribute.
\item[info] (read only) This attribute contains information about the
master. Example contents are below:
\begin{lstlisting}[gobble=4]
`\$` `\textbf{cat /sys/ethercat0/info}`
Mode: IDLE
Slaves: 9
Timing (min/avg/max) [us]:
Idle cycle: 4 / 4.38 / 34
EoE cycle: 9 / 11.91 / 23
EoE statistics (RX/TX) [bps]:
eoe0: 0 / 3184
\end{lstlisting}
The mode can be \textit{ORPHANED}, \textit{IDLE} or
\textit{OPERATION}. The other parameters are self-explanatory.
\end{description}
\subsubsection{Domain Attributes}
\label{sec:sysfs-domain}
In operation mode, each created domain is represented as a directory
\textit{domainX}, where X is the domain index. Below is a listing of
the domain directory contents:
\begin{lstlisting}
`\$` `\textbf{ls /sys/ethercat0/domain0}`
image_size
\end{lstlisting}
The domain directories currently only export the domain's image size.
It is planned to export the whole process data mapping for debugging
purposes.
\subsubsection{Slave Attributes}
\label{sec:sysfs-slave}
Each slave on the bus is represented in its own directory
\textit{slaveXXX}, where XXX is the slave's 3-digit ring position in
the EtherCAT bus. Below is a listing of a slave directory:
\begin{lstlisting}
`\$` `\textbf{ls /sys/ethercat0/slave003}`
eeprom info state
\end{lstlisting}
\begin{description}
\item[eeprom] (read/write) See section~\ref{sec:eepromaccess} for how
to use this attribute.
\item[info] (read only) This attribute contains a bunch of information
about the slave. Below is an example output:
\begin{lstlisting}[gobble=4]
`\$` `\textbf{cat /sys/ethercat0/slave003/info}`
Name: EL4132 2K. Ana. Ausgang +/-10V
Vendor ID: 0x00000002
Product code: 0x10243052
State: INIT
Ring position: 3
Advanced position: 1:3
Data link status:
Port 0 (EBUS) Link down, Loop open, Signal detected
Port 1 (EBUS) Link down, Loop open, Signal detected
Port 2 (EBUS) Link down, Loop closed, No signal
Port 3 (EBUS) Link down, Loop closed, No signal
Mailboxes:
RX mailbox: 0x1800/246, TX mailbox: 0x18F6/246
Supported protocols: CoE, FoE
SII data:
Group: AnaOut
Image: TERM_AO
Order#: EL4132
Sync-Managers:
0: 0x1800, length 246, control 0x26, enable
1: 0x18F6, length 246, control 0x22, enable
2: 0x1000, length 0, control 0x24, enable
3: 0x1100, length 0, control 0x20, enable
Pdos:
RXPDO "Channel 1" (0x1600), Sync-Manager 2
"Output" 0x6411:1, 16 bit
RXPDO "Channel 2" (0x1601), Sync-Manager 2
"Output" 0x6411:2, 16 bit
\end{lstlisting}
This is nearly all of the SII category information needed to
configure the slave, supplemented with state and addressing
information.
\item[state] (read/write) This attribute contains the slave's state.
It can be read or written:
\begin{lstlisting}[gobble=4]
# `\textbf{cat /sys/ethercat0/slave003/state}`
OP
# `\textbf{echo SAFEOP > /sys/ethercat0/slave003/state}`
\end{lstlisting}
This command should also be receipted with a syslog message:
\begin{lstlisting}[gobble=4]
EtherCAT: Accepted new state SAFEOP for slave 3.
EtherCAT: Changing state of slave 3 from OP to SAFEOP.
EtherCAT: Slave states: INIT, SAFEOP, OP.
\end{lstlisting}
After the new requested state was accepted from user space, the
operation state machine (see section~\ref{sec:fsm-op}) or the idle
state machine (section~\ref{sec:fsm-idle}) notices, that the
requested slave state differs from the current one, and therefore
executes the slave configuration state machine, until the slave has
reached the requested state.
\end{description}
%------------------------------------------------------------------------------
\subsubsection{E$^2$PROM Access}
\label{sec:eepromaccess}
\index{E$^2$PROM!Access}
It is possible to directly read or write the complete E$^2$PROM
contents of the slaves. This was introduced for the reasons below:
\begin{itemize}
\item The format of the E$^2$PROM data is still in development and
categories can be added in the future. With read and write access,
the complete memory contents can be easily backed up and restored.
\item Some E$^2$PROM data fields have to be altered (like the alias
address). A quick writing must be possible for that.
\item Through read access, analyzing category data is possible from
user space.
\end{itemize}
Reading out E$^2$PROM data is as easy as reading other
attributes. Though the data are in binary format, analyzation is
easier with a tool like \textit{hexdump}:
\begin{lstlisting}
`\$` `\textbf{cat /sys/ethercat0/slave003/eeprom | hexdump}`
0000000 0103 0000 0000 0000 0000 0000 0000 008c
0000010 0002 0000 3052 07f0 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
...
\end{lstlisting}
Backing up E$^2$PROM contents gets as easy as copying a file:
\begin{lstlisting}
`\$` `\textbf{cp /sys/ethercat0/slave003/eeprom slave003.eep}`
\end{lstlisting}
Writing access is only possible as \textit{root}. Moreover writing has
to be explicitly enabled and is only allowed in idle mode. This is a
safety measure, because without the correct memory contents, a slave
is unusable. Writing E$^2$PROM contents in operation mode is not
provided yet.
E$^2$PROM writing is enabled with the command below:
\begin{lstlisting}
# `\textbf{echo 1 > /sys/ethercat0/eeprom\_write\_enable}`
\end{lstlisting}
The success can be seen in the syslog messages again:
\begin{lstlisting}
EtherCAT: Slave EEPROM writing enabled.
\end{lstlisting}
Now, it is possible to write E$^2$PROM contents to a slave. The master
will accept data through the \textit{eeprom} file and will perform a
short validation of the contents, before starting the write operation.
This validation checks the complete size and the category headers.
\begin{lstlisting}
# `\textbf{cat slave003.eep > /sys/ethercat0/slave003/eeprom}`
\end{lstlisting}
The write operation can take a few seconds.
\begin{lstlisting}
EtherCAT: EEPROM writing scheduled for slave 3, 88 words.
EtherCAT: Writing EEPROM of slave 3...
EtherCAT: Finished writing EEPROM of slave 3.
\end{lstlisting}
%------------------------------------------------------------------------------
\subsection{User Space Tools}
\index{User space!Tools}
There is a user space tool called \textit{lsec}\index{lsec} (``List
EtherCAT'') to visualize the EtherCAT bus. Running it usually results
in an output like this:
\begin{lstlisting}
`\$` `\textbf{lsec}`
EtherCAT bus listing for master 0:
0 1:0 OP EK1100 Ethernet Kopplerklemme (2A E-Bus)
1 1:1 INIT EL4132 2K. Ana. Ausgang +/-10V
2 1:2 INIT EL4132 2K. Ana. Ausgang +/-10V
3 1:3 SAFEOP EL4132 2K. Ana. Ausgang +/-10V
4 1:4 INIT EL5101 Incremental Encoder Interface
5 1:5 INIT EL1014 4K. Dig. Eingang 24V, 10s
6 1:6 OP EL6601 1 Port Switch (Ethernet, CoE)
7 1:7 INIT EL5101 Incremental Encoder Interface
8 1:8 INIT EL5001 1K. SSI Encoder
\end{lstlisting}
Every slave is displayed as one text row. The first column shows its
ring position, the second displays the ``advanced position address''
(see section~\ref{sec:addr}) and the third column displays the current
slave state. The last column is the slave's name, as it appears in the
``general'' E$^2$PROM category.
The lsec program is a Perl script, that evaluates the Sysfs
\textit{info} attributes of the slaves (see
section~\ref{sec:sysfs-slave}). This is done for master $0$ by
default, but the master index can be specified via command line:
\begin{lstlisting}
`\$` `\textbf{lsec -h}`
Usage: ec_list [OPTIONS]
-m <IDX> Query master <IDX>.
-h Show this help.
\end{lstlisting}
This script has proved as useful for troubleshooting: If it displays
slaves, the master is up and running, and the bus connection is
present, too. It is also useful when building up a bus: It can verify
the list of slaves and help to create a process data image (see
chapter~\ref{chapter:usage}).
%------------------------------------------------------------------------------
\subsection{System Integration}
\label{sec:system}
To integrate the EtherCAT master into a running system, it has to be
guaranteed, that it is started on system startup. In addition, there has
to be a persistent configuration, that is also applied on startup.
\subsubsection{The EtherCAT Init Script}
\label{sec:init}
\index{Init script}
The EtherCAT master provides an ``init script'', that conforms to the
requirements of the ``Linux Standard Base'' (LSB\index{LSB},
\cite{lsb}). The script is installed to \textit{etc/init.d/ethercat}
below the installation prefix and has to be copied to the appropriate
location (see section~\ref{sec:make}), before the master can be
inserted as a service. The different Linux distributions offer
different ways to mark the service for starting and stopping in
certain runlevels (for example, SUSE Linux provides the
\textit{insserv} command).
To provide service dependencies (i.~e. which services have to be
started before) right inside the init script code, LSB defines a
special comment block. System tools can extract this information to
insert the EtherCAT init script at the correct place in the startup
sequence:
\begin{lstlisting}
### BEGIN INIT INFO
# Provides: ethercat
# Required-Start: $local_fs $syslog $network
# Should-Start: $time
# Required-Stop: $local_fs $syslog $network
# Should-Stop: $time
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Short-Description: EtherCAT master modules
# Description:
### END INIT INFO
\end{lstlisting}
The init script can also be used for manually starting and stopping
the EtherCAT master. It has to be executed with one of the parameters
\texttt{start}, \texttt{stop}, \texttt{restart} or \texttt{status}.
\begin{lstlisting}
# `\textbf{/etc/init.d/ethercat restart}`
Shutting down EtherCAT master done
Starting EtherCAT master done
\end{lstlisting}
\subsubsection{The EtherCAT Sysconfig File}
\label{sec:sysconfig}
\index{Sysconfig file}
For persistent configuration, the init script uses a sysconfig file
installed to \textit{etc/sysconfig/ethercat} (below the installation
prefix), that is mandatory for the init script. The sysconfig file
contains all configuration variables needed to operate a master:
\begin{description}
\item[DEVICE\_INDEX] This variable must contain the PCI index of the
EtherCAT device. Setting this is mandatory for the EtherCAT init
script. Default: $-1$
\item[EOE\_INTERFACES] The number of virtual Ethernet-over-EtherCAT
interfaces, every master creates on startup. See
section~\ref{sec:eoeimp}. Default: $0$
\item[EOE\_BRIDGE] If this variable is set, all EoE interfaces will be
added to a network bridge according to IEEE 802.1D after master
startup. The variable must contain the name of the bridge. To use
this functionality, the kernel must be configured with the
CONFIG\_BRIDGE option and the \textit{bridge-utils} package must be
installed (i.~e. the \textit{brctl} command is needed).
\item[EOE\_IP\_ADDRESS] The IP address of the EoE bridge. Setting this
together with \$EOE\_IP\_NETMASK will let the local host communicate
with devices on the EoE bridge.
\item[EOE\_IP\_NETMASK] IP netmask of the EoE bridge.
\item[EOE\_EXTRA\_INTERFACES] The list of extra interfaces to include
in the EoE brid\-ge. Set this to interconnect the EoE bridge with
other local interfaces. If \$EOE\_\-BRIDGE is empty or undefined,
setting this variable has no effect. Important: The IP address of
the listed interfaces will be cleared. Setting
\$EOE\_\-IP\_\-ADDRESS and \$EOE\_IP\_NETMASK will re-enable them
for IP traffic.
\item[EOE\_GATEWAY] The IP address of the default gateway. If this
variable is set, the gateway will be renewed after bridge
installation. This is necessary, if the default gateway's interface
is one of the \$EOE\_EXTRA\_INTERFACES.
\end{description}
%------------------------------------------------------------------------------
\subsection{Monitoring and Debugging}
\label{sec:debug}
\index{Monitoring}
For debugging purposes, every EtherCAT master registeres a read-only
network interface \textit{ecX}, where X is a number, provided by the
kernel on device registration. While it is ``up'', the master forwards
every frame sent and received to this interface.
This makes it possible to connect an network monitor (like Wireshark
or tcpdump) to the debug interface and monitor the EtherCAT frames.
It has to be considered, that can be frame rate can be very high. The
idle state machine usually runs every kernel timer interrupt (up to
$1$~kHz) and with a connected realtime module, the rate can be even
higher.
\paragraph{Attention:} The socket buffers needed for the operation of
the debugging interface have to be allocated dynamically. Some Linux
realtime extensions do not allow this in realtime context!
%------------------------------------------------------------------------------
\section{Timing Aspects}
\label{sec:timing}
Although EtherCAT's timing is highly deterministic and therefore
timing issues are rare, there are a few aspects that can (and should
be) dealt with.
%------------------------------------------------------------------------------
\subsection{Realtime Interface Profiling}
\label{sec:timing-profile}
\index{Realtime!Profiling}
One of the most important timing aspects are the runtimes of the
realtime interface functions, that are called in cyclic context. These
functions make up an important part of the overall timing of the
realtime module. To measure the timing of the functions, the following
code was used:
\begin{lstlisting}[gobble=2,language=C]
c0 = get_cycles();
ecrt_master_receive(master);
c1 = get_cycles();
ecrt_domain_process(domain1);
c2 = get_cycles();
ecrt_master_run(master);
c3 = get_cycles();
ecrt_master_send(master);
c4 = get_cycles();
\end{lstlisting}
Between each call of an interface function, the CPU timestamp counter
is read. The counter differences are converted to microseconds with
help of the \textit{cpu\_khz} variable, that contains the number of
increments per millisecond.
For the actual measuring, a system with a $2.0$~GHz CPU was used, that
ran the above code in an RTAI thread with a cycle time of $100$
\textmu s. The measuring was repeated $n = 100$ times and the results
were averaged. These can be seen in table~\ref{tab:profile}.
\begin{table}[htpb]
\centering
\caption{Profiling of a Realtime Cycle on a $2.0$~GHz Processor}
\label{tab:profile}
\vspace{2mm}
\begin{tabular}{l|r|r}
Element & Mean Duration [\textmu s] & Standard Deviancy [\textmu s] \\
\hline
\textit{ecrt\_master\_receive()} & 8.04 & 0.48\\
\textit{ecrt\_domain\_process()} & 0.14 & 0.03\\
\textit{ecrt\_master\_run()} & 0.29 & 0.12\\
\textit{ecrt\_master\_send()} & 2.18 & 0.17\\ \hline
Complete Cycle & 10.65 & 0.69\\ \hline
\end{tabular}
\end{table}
It is obvious, that the the functions accessing hardware make up the
lion's share. The \textit{ec\_master\_receive()} executes the ISR of
the Ethernet device, analyzes datagrams and copies their contents into
the memory of the datagram objects. The \textit{ec\_master\_send()}
assembles a frame out of different datagrams and copies it to the
hardware buffers. Interestingly, this makes up only a quarter of the
receiving time.
The functions that only operate on the masters internal data
structures are very fast ($\Delta t < 1$~\textmu s). Interestingly the
runtime of \textit{ec\_domain\_process()} has a small standard
deviancy relative to the mean value, while this ratio is about twice
as big for \textit{ec\_master\_run()}: This probably results from the
latter function having to execute code depending on the current state
and the different state functions are more or less complex.
For a realtime cycle makes up about $10$~\textmu s, the theoretical
frequency can be up to $100$~kHz. For two reasons, this frequency
keeps being theoretical:
\begin{enumerate}
\item The processor must still be able to run the operating system
between the realtime cycles.
\item The EtherCAT frame must be sent and received, before the next
realtime cycle begins. The determination of the bus cycle time is
difficult and covered in section~\ref{sec:timing-bus}.
\end{enumerate}
%------------------------------------------------------------------------------
\subsection{Bus Cycle Measuring}
\label{sec:timing-bus}
\index{Bus cycle}
For measuring the time, a frame is ``on the wire'', two timestamps
must be be taken:
\begin{enumerate}
\item The time, the Ethernet hardware begins with physically sending
the frame.
\item The time, the frame is completely received by the Ethernet
hardware.
\end{enumerate}
Both times are difficult to determine. The first reason is, that the
interrupts are disabled and the master is not notified, when a frame
is sent or received (polling would distort the results). The second
reason is, that even with interrupts enabled, the time from the event
to the notification is unknown. Therefore the only way to confidently
determine the bus cycle time is an electrical measuring.
Anyway, the bus cycle time is an important factor when designing
realtime code, because it limits the maximum frequency for the cyclic
part of the realtime module. In practice, these timing parameters are
highly dependent on the hardware and often a trial and error method
must be used to determine the limits of the system.
The central question is: What happens, if the cycle frequency is too
high? The answer is, that the EtherCAT frames that have been sent at
the end of the cycle are not yet received, when the next cycle starts.
First this is noticed by \textit{ecrt\_domain\_process()}, because the
working counter of the process data datagrams were not increased. The
function will notify the user via syslog\footnote{To limit syslog
output, a mechanism has been implementet, that outputs a summarized
notification at maximum once a second.}. In this case, the process
data keeps being the same as in the last cycle, because it is not
erased by the domain. When the domain datagrams are queued again, the
master notices, that they are already queued (and marked as sent). The
master will mark them as unsent again and output a warning, that
datagrams were ``skipped''.
On the mentioned $2.0$~GHz system, the possible cycle frequency can be
up to $25$~kHz without skipped frames. This value can surely be
increased by choosing faster hardware. Especially the RealTek network
hardware could be replaced by a faster one. Besides, implementing a
dedicated ISR for EtherCAT devices would also contribute to increasing
the latency. These are two points on the author's to-do list.
%------------------------------------------------------------------------------
\chapter{Using the EtherCAT Master}
\label{chapter:usage}
\index{Master!Usage}
This chapter will give practical examples of how to use the EtherCAT
master via the realtime interface by writing a realtime module.
Section~\ref{sec:make} shows how to compile and install the master,
while the sections~\ref{sec:mini} to~\ref{sec:concurrency} give
examples for different realtime modules.
%------------------------------------------------------------------------------
\section{Compiling and Installing}
\label{sec:make}
\index{Master!Compilation}
The current EtherCAT master code is available at~\cite{etherlab} or
can be obtained from the EtherLab\textsuperscript{\textregistered} CD.
The \textit{tar.bz2} file has to be unpacked with the commands below
(or similar):
\begin{lstlisting}
`\$` `\textbf{tar xjf ethercat-\masterversion.tar.bz2}`
`\$` `\textbf{cd ethercat-\masterversion/}`
\end{lstlisting}
The tarball was created with GNU Autotools, so the build process
follows the below commands:
\begin{lstlisting}
`\$` `\textbf{./configure}`
`\$` `\textbf{make modules}`
\end{lstlisting}
The default installation prefix is \textit{/opt/etherlab}. It can be
changed with the \texttt{--prefix} argument.
Linux kernel sources are needed for compilation\footnote{If a realtime
extension shall to be used, the kernel should be patched before
compiling the EtherCAT master.}. To compile the EtherCAT master
modules for a different kernel than the running kernel, the target
kernel version can be specified with the \texttt{--with-linux}
argument. Example:
\begin{lstlisting}
`\$` `\textbf{./configure --with-linux="2.6.17-ipipe"}`
`\$` `\textbf{make modules}`
\end{lstlisting}
The below commands have to be entered as \textit{root}: The first one
will install the kernel modules to the kernel's modules directory. The
second one will install EtherCAT headers, the init script, the
sysconfig file and the user space tools to the prefix path.
\begin{lstlisting}
# `\textbf{make modules\_install}`
# `\textbf{make install}`
\end{lstlisting}
If the target kernel's modules directory is not under
\textit{/lib/modules}, a different destination directory can be
specified with the \textit{DESTDIR} make variable. For example:
\begin{lstlisting}
# `\textbf{make DESTDIR=/vol/nfs/root modules\_install}`
\end{lstlisting}
This command will install the compiled kernel modules to
\textit{/vol/nfs/root/lib/modules}, prepended by the kernel release.
If the EtherCAT master shall be run as a service
(recommended\footnote{Even if the EtherCAT master shall not be loaded
on system startup, the use of the init script is recommended for
manual (un-)loading.}), the init script and the sysconfig file have
to be copied to the appropriate locations. The below example is
suitable for SUSE Linux. It may vary for other distributions.
\begin{lstlisting}
# `\textbf{cd /opt/etherlab}`
# `\textbf{cp etc/sysconfig/ethercat /etc/sysconfig/}`
# `\textbf{cp etc/init.d/ethercat /etc/init.d/}`
# `\textbf{insserv ethercat}`
\end{lstlisting}
Now the sysconfig file \texttt{/etc/sysconfig/ethercat} (see
section~\ref{sec:sysconfig}) has to be customized: This is mainly done
by uncommenting and adjusting the \$DEVICE\_INDEX variable. It has to
be set to the index of the compatible network device to use with
EtherCAT, where the order of devices is dependent on their position in
the PCI bus:
\begin{lstlisting}[numbers=left,basicstyle=\ttfamily\scriptsize]
# `\textbf{lspci}`
00:00.0 Host bridge: VIA Technologies, Inc. VT8363/8365 [KT133/KM133] (rev 03)
00:01.0 PCI bridge: VIA Technologies, Inc. VT8363/8365 [KT133/KM133 AGP]
00:04.0 ISA bridge: VIA Technologies, Inc. VT82C686 [Apollo Super South] (rev 40)
00:04.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C...
00:04.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller...
00:04.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller...
00:04.4 Bridge: VIA Technologies, Inc. VT82C686 [Apollo Super ACPI] (rev 40)
00:09.0 Ethernet controller: D-Link System Inc RTL8139 Ethernet (rev 10)
00:0a.0 Ethernet controller: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 08)
00:0b.0 Ethernet controller: D-Link System Inc RTL8139 Ethernet (rev 10)
00:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:11.0 Unknown mass storage controller: Promise Technology, Inc. PDC20265...
\end{lstlisting}
In the above output of the \textit{lspci} command, two compatible
network devices can be found in lines~\textcircled{\tiny 9} and
\textcircled{\tiny 11}. The \$DEVICE\_INDEX variable should be set to
$0$ or $1$, respectively.
After the basic configuration is done, the master can be started with
the below command:
\begin{lstlisting}
# `\textbf{/etc/init.d/ethercat start}`
\end{lstlisting}
The operation of the master can be observed by looking at the
syslog\index{syslog} messages, which should look like the ones below.
If EtherCAT slaves are connected to the master's EtherCAT device, the
activity indicators should begin to flash.
\begin{lstlisting}[numbers=left]
EtherCAT: Master driver, 1.1 (stable) - rev. 513,
compiled by fp at Aug 09 2006 10:23:20
EtherCAT: Initializing 1 EtherCAT master(s)...
EtherCAT: Initializing master 0.
EtherCAT: Master driver initialized.
ec_8139too Fast Ethernet driver 0.9.27 Revision 513,
compiled by fp at Aug 09 2006 10:23:20
ec_device_index is 0
ACPI: PCI Interrupt 0000:01:00.0[A] -> Link [LNKC]
-> GSI 11 (level, low) -> IRQ 11
ec0: RealTek RTL8139 at 0xd047c000, 00:c0:26:00:c6:aa, IRQ 11
ec0: Identified 8139 chip type 'RTL-8100B/8139D'
Registering EtherCAT device...
Starting EtherCAT device...
EtherCAT: Link state changed to UP.
EtherCAT: Starting Idle mode.
EtherCAT: 11 slaves responding.
EtherCAT: Slave states: INIT, OP.
EtherCAT: Scanning bus.
EtherCAT: Bus scanning completed.
EtherCAT: No EoE handlers coupled.
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 1}] The master module is loaded,
and one master is initialized.
\item[\normalfont\textcircled{\tiny 6}] The EtherCAT-capable RTL8139
device driver is loaded. It connects its first network device to the
master.
\item[\normalfont\textcircled{\tiny 16}] The master starts idle mode
and begins scanning the bus for slaves.
\end{description}
%------------------------------------------------------------------------------
\section{A Minimal Example Module}
\label{sec:mini}
\index{Examples!Minimal}
This section will explain the usage of the EtherCAT master from a
minimal kernel module. The complete module code is obtainable as a
part of the EtherCAT master code release (see~\cite{etherlab}, file
\textit{examples/mini/mini.c}).
The minimal example uses a kernel timer (software interrupt) to handle
cyclic code. After the timer function is executed, it re-adds itself
with a delay of one \textit{jiffy}\index{jiffies}, which results in a
timer frequency of \textit{HZ}\nomenclature{HZ}{Kernel macro
containing the timer interrupt frequency}
The module-global variables, needed to operate the master can be seen
in listing~\ref{lst:minivar}.
\begin{lstlisting}[language=C,numbers=left,caption={Minimal
variables},label=lst:minivar]
struct timer_list timer;
ec_master_t *master = NULL;
ec_domain_t *domain1 = NULL;
void *r_dig_in, *r_ana_out;
ec_pdo_reg_t domain1_pdos[] = {
{"1", Beckhoff_EL1014_Inputs, &r_dig_in},
{"2", Beckhoff_EL4132_Ouput1, &r_ana_out},
{}
};
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 1}] There is a timer object
declared, that is needed to tell the kernel to install a timer and
execute a certain function, if it runs out. This is done by a
variable of the \textit{timer\_list} structure.
\item[\normalfont\textcircled{\tiny 3} -- \textcircled{\tiny 4}] There
is a pointer declared, that will later point to a requested EtherCAT
master. Additionally there is a pointer to a domain object needed,
that will manage process data IO.
\item[\normalfont\textcircled{\tiny 6}] The pointers \textit{r\_*}
will later point to the \underline{r}aw process data values inside
the domain memory. The addresses they point to will be set during a
call to \textit{ec\_\-master\_\-activate()}, that will create the
domain memory and configure the mapped process data image.
\item[\normalfont\textcircled{\tiny 8} -- \textcircled{\tiny 12}] The
configuration of the mapping of certain Pdos in a domain can easily
be done with the help of an initialization array of the
\textit{ec\_pdo\_reg\_t} type, defined as part of the realtime
interface. Each record must contain the ASCII bus-address of the
slave (see section~\ref{sec:addr}), the slave's vendor ID and
product code, and the index and subindex of the Pdo to map (these
four fields can be specified in junction, by using one of the
defines out of the \textit{include/ecdb.h} header). The last field
has to be the address of the process data pointer, so it can later
be redirected appropriately. Attention: The initialization array
must end with an empty record (\textit{\{\}})!
\end{description}
The initialization of the minimal realtime module is done by the
``Minimal init function'' in listing~\ref{lst:miniinit}.
\begin{lstlisting}[language=C,numbers=left,caption={Minimal init
function},label={lst:miniinit}]
int __init init_mini_module(void)
{
if (!(master = ecrt_request_master(0))) {
goto out_return;
}
if (!(domain1 = ecrt_master_create_domain(master))) {
goto out_release_master;
}
if (ecrt_domain_register_pdo_list(domain1,
domain1_pdos)) {
goto out_release_master;
}
if (ecrt_master_activate(master)) {
goto out_release_master;
}
ecrt_master_prepare(master);
init_timer(&timer);
timer.function = run;
timer.expires = jiffies + 10;
add_timer(&timer);
return 0;
out_release_master:
ecrt_release_master(master);
out_return:
return -1;
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 3}] It is tried to request the
first EtherCAT master (index 0). On success, the
\textit{ecrt\_\-request\_\-master()} function returns a pointer to
the reserved master, that can be used as an object to following
functions calls. On failure, the function returns \textit{NULL}.
\item[\normalfont\textcircled{\tiny 7}] In order to exchange process
data, a domain object has to be created. The
\textit{ecrt\_\-master\_\-create\_domain()} function also returns a
pointer to the created domain, or \textit{NULL} in error case.
\item[\normalfont\textcircled{\tiny 11}] The registration of domain
Pdos with an initialization array results in a single function call.
Alternatively the data fields could be registered with individual
calls of \textit{ecrt\_domain\_register\_pdo()}.
\item[\normalfont\textcircled{\tiny 16}] After the configuration of
process data mapping, the master can be activated for cyclic
operation. This will configure all slaves and bring them into
OP state.
\item[\normalfont\textcircled{\tiny 20}] This call is needed to avoid
a case differentiation in cyclic operation: The first operation in
cyclic mode is a receive call. Due to the fact, that there is
nothing to receive during the first cycle, there had to be an
\textit{if}-statement to avoid a warning. A call to
\textit{ec\_master\_prepare()} sends a first datagram containing a
process data exchange datagram, so that the first receive call will
not fail.
\item[\normalfont\textcircled{\tiny 22} -- \textcircled{\tiny 25}] The
master is now ready for cyclic operation. The kernel timer that
cyclically executes the \textit{run()} function is initialized and
started.
\end{description}
The coding of a cleanup function fo the minimal module can be seen in
listing~\ref{lst:miniclean}.
\begin{lstlisting}[language=C,numbers=left,caption={Minimal cleanup
function},label={lst:miniclean}]
void __exit cleanup_mini_module(void)
{
del_timer_sync(&timer);
ecrt_master_deactivate(master);
ecrt_release_master(master);
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 3}] To cleanup the module, it it
necessary to stop the cyclic processing. This is done by a call to
\textit{del\_timer\_sync()} which safely removes a queued timer
object. It is assured, that no cyclic work will be done after this
call returns.
\item[\normalfont\textcircled{\tiny 4}] This call deactivates the
master, which results in all slaves being brought to their INIT
state again.
\item[\normalfont\textcircled{\tiny 5}] This call releases the master,
removes any existing configuration and silently starts the idle
mode. The value of the master pointer is invalid after this call and
the module can be safely unloaded.
\end{description}
The final part of the minimal module is that for the cyclic work. Its
coding can be seen in listing~\ref{lst:minirun}.
\begin{lstlisting}[language=C,numbers=left,caption={Minimal cyclic
function},label={lst:minirun}]
void run(unsigned long data)
{
static uint8_t dig_in_0;
ecrt_master_receive(master);
ecrt_domain_process(domain1);
dig_in_0 = EC_READ_BIT(r_dig_in, 0);
EC_WRITE_S16(r_ana_out, dig_in_0 * 0x3FFF);
ecrt_master_run(master);
ecrt_master_send(master);
timer.expires += 1; // frequency = HZ
add_timer(&timer);
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 5}] The cyclic processing starts
with receiving datagrams, that were sent in the last cycle. The
frames containing these datagrams have to be received by the network
interface card prior to this call.
\item[\normalfont\textcircled{\tiny 6}] The process data of domain 1
has been automatically copied into domain memory while datagram
reception. This call checks the working counter for changes and
re-queues the domain's datagram for sending.
\item[\normalfont\textcircled{\tiny 8}] This is an example for reading
out a bit-oriented process data value (i.~e. bit 0) via the
\textit{EC\_READ\_BIT()} macro. See section~\ref{sec:macros} for
more information about those macros.
\item[\normalfont\textcircled{\tiny 9}] This line shows how to write a
signed, 16-bit process data value. In this case, the slave is able
to output voltages of $-10$~V to $+10$~V with a resolution of 16
bit. This write command outputs either $0$~V or $+5$~V, depending
of the value of \textit{dig\_in\_0}.
\item[\normalfont\textcircled{\tiny 11}] This call runs the master's
operation state machine (see section~\ref{sec:fsm-op}). A single
state is processed, and datagrams are queued. Mainly bus observation
is done: The bus state is determined and in case of slaves that lost
their configuration, reconfiguration is tried.
\item[\normalfont\textcircled{\tiny 12}] This method sends all queued
datagrams, in this case the domain's datagram and one of the master
state machine. In best case, all datagrams fit into one frame.
\item[\normalfont\textcircled{\tiny 14} -- \textcircled{\tiny 15}]
Kernel timers are implemented as ``one-shot'' timers, so they have
to be re-added after each execution. The time of the next execution
is specified in \textit{jiffies} and will happen at the time of the
next system timer interrupt. This results in the \textit{run()}
function being executed with a frequency of \textit{HZ}.
\end{description}
%------------------------------------------------------------------------------
\section{An RTAI Example Module}
\label{sec:rtai}
\index{Examples!RTAI}
The whole code can be seen in the EtherCAT master code release
(see~\cite{etherlab}, file \textit{examples/rtai/rtai\_sample.c}).
Listing~\ref{lst:rtaivar} shows the defines and global variables
needed for a minimal RTAI module with EtherCAT processing.
\begin{lstlisting}[language=C,numbers=left,caption={RTAI task
declaration},label={lst:rtaivar}]
#define FREQUENCY 10000
#define TIMERTICKS (1000000000 / FREQUENCY)
RT_TASK task;
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 1} -- \textcircled{\tiny 2}] RTAI
takes the cycle period as nanoseconds, so the easiest way is to
define a frequency and convert it to a cycle time in nanoseconds.
\item[\normalfont\textcircled{\tiny 4}] The \textit{task} variable
later contains information about the running RTAI task.
\end{description}
Listing~\ref{lst:rtaiinit} shows the module init function for the RTAI
module. Most lines are the same as in listing~\ref{lst:miniinit},
differences come up when starting the cyclic code.
\begin{lstlisting}[language=C,numbers=left,caption={RTAI module init
function},label={lst:rtaiinit}]
int __init init_mod(void)
{
RTIME requested_ticks, tick_period, now;
if (!(master = ecrt_request_master(0))) {
goto out_return;
}
if (!(domain1 = ecrt_master_create_domain(master))) {
goto out_release_master;
}
if (ecrt_domain_register_pdo_list(domain1,
domain1_pdos)) {
goto out_release_master;
}
if (ecrt_master_activate(master)) {
goto out_release_master;
}
ecrt_master_prepare(master);
requested_ticks = nano2count(TIMERTICKS);
tick_period = start_rt_timer(requested_ticks);
if (rt_task_init(&task, run, 0, 2000, 0, 1, NULL)) {
goto out_stop_timer;
}
now = rt_get_time();
if (rt_task_make_periodic(&task, now + tick_period,
tick_period)) {
goto out_stop_task;
}
return 0;
out_stop_task:
rt_task_delete(&task);
out_stop_timer:
stop_rt_timer();
out_deactivate:
ecrt_master_deactivate(master);
out_release_master:
ecrt_release_master(master);
out_return:
return -1;
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 24} -- \textcircled{\tiny 25}] The
nanoseconds are converted to RTAI timer ticks and an RTAI timer is
started. \textit{tick\_period} will be the ``real'' number of ticks
used for the timer period (which can be different to the requested
one).
\item[\normalfont\textcircled{\tiny 27}] The RTAI task is initialized
by specifying the cyclic function, the parameter to hand over, the
stack size, priority, a flag that tells, if the function will use
floating point operations and a signal handler.
\item[\normalfont\textcircled{\tiny 32}] The task is made periodic by
specifying a start time and a period.
\end{description}
The cleanup function of the RTAI module in listing~\ref{lst:rtaiclean}
is nearly as simple as that of the minimal module.
\begin{lstlisting}[language=C,numbers=left,caption={RTAI module
cleanup function},label={lst:rtaiclean}]
void __exit cleanup_mod(void)
{
rt_task_delete(&task);
stop_rt_timer();
ecrt_master_deactivate(master);
ecrt_release_master(master);
rt_sem_delete(&master_sem);
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 2}] The RTAI task will be stopped
and deleted.
\item[\normalfont\textcircled{\tiny 3}] After that, the RTAI timer can
be stopped.
\end{description}
The rest is the same as for the minimal module.
Worth to mention is, that the cyclic function of the RTAI module
(listing~\ref{lst:rtairun}) has a slightly different architecture. The
function is not executed until returning for every cycle, but has an
infinite loop in it, that is placed in a waiting state for the rest of
each cycle.
\begin{lstlisting}[language=C,numbers=left,caption={RTAI module cyclic
function},label={lst:rtairun}]
void run(long data)
{
while (1) {
ecrt_master_receive(master);
ecrt_domain_process(domain1);
k_pos = EC_READ_U32(r_ssi_input);
ecrt_master_run(master);
ecrt_master_send(master);
rt_task_wait_period();
}
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 3}] The \textit{while (1)} loop
executes for the lifetime of the RTAI task.
\item[\normalfont\textcircled{\tiny 12}] The
\textit{rt\_task\_wait\_period()} function sets the process into a
sleeping state until the beginning of the next cycle. It also
checks, if the cyclic function has to be terminated.
\end{description}
%------------------------------------------------------------------------------
\section{Concurrency Example}
\label{sec:concurrency}
\index{Examples!Concurrency}
As mentioned before, there can be concurrent access to the EtherCAT
master. The realtime module and a EoE\index{EoE} process can compete
for master access, for example. In this case, the module has to
provide the locking mechanism, because it depends on the module's
architecture which lock has to be used. The module makes this locking
mechanism available to the master through the master's locking
callbacks.
In case of RTAI, the lock can be an RTAI semaphore, as shown in
listing~\ref{lst:convar}. A normal linux semaphore would not be
appropriate, because it could not block the RTAI task due to RTAI
running in a higher domain than the linux kernel (see~\cite{rtai}).
\begin{lstlisting}[language=C,numbers=left,caption={RTAI semaphore for
concurrent access},label={lst:convar}]
SEM master_sem;
\end{lstlisting}
The module has to implement the two callbacks for requesting and
releasing the master lock. An exemplary coding can be seen in
listing~\ref{lst:conlock}.
\begin{lstlisting}[language=C,numbers=left,caption={RTAI locking
callbacks for concurrent access},label={lst:conlock}]
int request_lock(void *data)
{
rt_sem_wait(&master_sem);
return 0;
}
void release_lock(void *data)
{
rt_sem_signal(&master_sem);
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 1}] The \textit{request\_lock()}
function has a data parameter. The master always passes the value,
that was specified when registering the callback function. This can
be used for handing the master pointer. Notice, that it has an
integer return value (see line 4).
\item[\normalfont\textcircled{\tiny 3}] The call to
\textit{rt\_sem\_wait()} either returns at once, when the semaphore
was free, or blocks until the semaphore is freed again. In any case,
the semaphore finally is reserved for the process calling the
request function.
\item[\normalfont\textcircled{\tiny 4}] When the lock was requested
successfully, the function should return 0. The module can prohibit
requesting the lock by returning non-zero (see paragraph ``Tuning
the jitter'' below).
\item[\normalfont\textcircled{\tiny 7}] The \textit{release\_lock()}
function gets the same argument passed, but has a void return value,
because is always succeeds.
\item[\normalfont\textcircled{\tiny 9}] The \textit{rt\_sem\_signal()}
function frees the semaphore, that was prior reserved with
\textit{rt\_sem\_wait()}.
\end{description}
In the module's init function, the semaphore must be initialized, and
the callbacks must be passed to the EtherCAT master:
\begin{lstlisting}[language=C,numbers=left,caption={Module init
function for concurrent access},label={lst:coninit}]
int __init init_mod(void)
{
RTIME tick_period, requested_ticks, now;
rt_sem_init(&master_sem, 1);
if (!(master = ecrt_request_master(0))) {
goto out_return;
}
ecrt_master_callbacks(master, request_lock,
release_lock, NULL);
// ...
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 5}] The call to
\textit{rt\_sem\_init()} initializes the semaphore and sets its
value to 1, meaning that only one process can reserve the semaphore
without blocking.
\item[\normalfont\textcircled{\tiny 11}] The callbacks are passed to
the master with a call to \textit{ecrt\_master\_callbacks()}. The
last parameter is the argument, that the master should pass with
each call to a callback function. Here it is not used and set to
\textit{NULL}.
\end{description}
For the cyclic function being only one competitor for master access,
it has to request the lock like any other process. There is no need to
use the callbacks (which are meant for processes of lower priority),
so it can access the semaphore directly:
\begin{lstlisting}[language=C,numbers=left,caption={RTAI cyclic
function for concurrent access},label={lst:conrun}]
void run(long data)
{
while (1) {
rt_sem_wait(&master_sem);
ecrt_master_receive(master);
ecrt_domain_process(domain1);
k_pos = EC_READ_U32(r_ssi_input);
ecrt_master_run(master);
ecrt_master_send(master);
rt_sem_signal(&master_sem);
rt_task_wait_period();
}
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 4}] Every access to the master has
to be preceeded by a call to \textit{rt\_sem\_wait()}, because
another instance might currently access the master.
\item[\normalfont\textcircled{\tiny 14}] When cyclic processing
finished, the semaphore has to be freed again, so that other
processes have the possibility to access the master.
\end{description}
A little change has to be made to the cleanup function in case of
concurrent master access.
\begin{lstlisting}[language=C,numbers=left,caption={RTAI module
cleanup function for concurrent access},label={lst:conclean}]
void __exit cleanup_mod(void)
{
rt_task_delete(&task);
stop_rt_timer();
ecrt_master_deactivate(master);
ecrt_release_master(master);
rt_sem_delete(&master_sem);
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 7}] Upon module cleanup, the
semaphore has to be deleted, so that memory can be freed.
\end{description}
\paragraph{Tuning the Jitter}
\index{Jitter}
Concurrent access leads to higher jitter of the realtime process,
because there are situations, in which the realtime process has to
wait for a process of lower priority to finish accessing the master.
In most cases this is acceptable, because a master access cycle
(receive/process/send) only takes $10$~\textmu s to $20$~\textmu s on
recent systems, what would be the maximum additional jitter. However
some applications demand a minimum jitter. For this reason the master
access can be prohibited by the realtime module: If the time, another
process wants to access the master, is to close to the beginning of
the next realtime cycle, the module can disallow, that the lock is
taken. In this case, the request callback has to return $1$, meaning
that the lock has not been taken. The foreign process must abort its
master access and try again next time.
This measure helps to significantly reducing the jitter produced by
concurrent master access. Below are exerpts of an example coding:
\begin{lstlisting}[language=C,numbers=left,caption={Variables for
jitter reduction},label={lst:redvar}]
#define FREQUENCY 10000 // RTAI task frequency in Hz
// ...
cycles_t t_last_cycle = 0;
const cycles_t t_critical = cpu_khz * 1000 / FREQUENCY
- cpu_khz * 30 / 1000;
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 3}] The variable
\textit{t\_last\_cycle} holds the timer ticks at the beginning of
the last realtime cycle.
\item[\normalfont\textcircled{\tiny 4}] \textit{t\_critical} contains
the number of ticks, that may have passed since the beginning of the
last cycle, until there is no more foreign access possible. It is
calculated by substracting the ticks for $30$~\textmu s from the
ticks for a complete cycle.
\end{description}
\begin{lstlisting}[language=C,numbers=left,caption={Cyclic function
with reduced jitter},label={lst:redrun}]
void run(long data)
{
while (1) {
t_last_cycle = get_cycles();
rt_sem_wait(&master_sem);
// ...
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 4}] The ticks of the beginning of
the current realtime cycle are taken before reserving the semaphore.
\end{description}
\begin{lstlisting}[language=C,numbers=left,caption={Request callback
for reduced jitter},label={lst:redreq}]
int request_lock(void *data)
{
// too close to the next RT cycle: deny access.
if (get_cycles() - t_last_cycle > t_critical)
return -1;
// allow access
rt_sem_wait(&master_sem);
return 0;
}
\end{lstlisting}
\begin{description}
\item[\normalfont\textcircled{\tiny 4}] If the time of request is too
close to the next realtime cycle (here: \textless~$30$~\textmu s
before the estimated beginning), the locking is denied. The
requesting process must abort its cycle.
\end{description}
%------------------------------------------------------------------------------
\begin{thebibliography}{99}
\bibitem{etherlab} Ingenieurgemeinschaft IgH: EtherLab -- Open Source
Toolkit for rapid realtime code generation under Linux with
Simulink/RTW and EtherCAT technology. URL: http://etherlab.org,
July~31, 2006.
\bibitem{dlspec} IEC 61158-4-12: Data-link Protocol Specification.
International Electrotechnical Comission (IEC), 2005.
\bibitem{alspec} IEC 61158-6-12: Application Layer Protocol
Specification. International Electrotechnical Comission (IEC), 2005.
\bibitem{gpl} GNU General Public License, Version 2. URL:
http://www.gnu.org/licenses/gpl.txt. August~9, 2006.
\bibitem{lsb} Linux Standard Base. URL:
http://www.freestandards.org/en/LSB. August~9, 2006.
\bibitem{wireshark} Wireshark. URL: http://www.wireshark.org.
August~9, 2006.
\bibitem{automata} {\it Hopcroft, J.~E. / Ullman, J.~D.}: Introduction
to Automata Theory, Languages and Computation. Adison-Wesley,
Reading, Mass.~1979.
\bibitem{fsmmis} {\it Wagner, F. / Wolstenholme, P.}: State machine
misunderstandings. In: IEE journal ``Computing and Control
Engineering'', 2004.
\bibitem{rtai} RTAI. The RealTime Application Interface for Linux from
DIAPM. URL: http://www.rtai.org, 2006.
\end{thebibliography}
\printnomenclature
\addcontentsline{toc}{chapter}{\nomname}
\markleft{\nomname}
\printindex
\markleft{Index}
%------------------------------------------------------------------------------
\end{document}
%------------------------------------------------------------------------------