1

%------------------------------------------------------------------------------

     2

%

     3

%  IgH EtherCAT Master Documentation

     4

%

     5

%  $Id$

     6

%

     7

%------------------------------------------------------------------------------

     8

     9

\documentclass[a4paper,12pt,BCOR6mm,bibtotoc,idxtotoc]{scrbook}

    10

    11

\usepackage[latin1]{inputenc}

    12

\usepackage[automark,headsepline]{scrpage2}

    13

\usepackage{graphicx}

    14

\usepackage{makeidx}

    15

\usepackage[refpage]{nomencl}

    16

\usepackage{listings}

    17

\usepackage{svn}

    18

\usepackage{textcomp}

    19

\usepackage{url}

    20

\usepackage[pdfpagelabels,plainpages=false]{hyperref}

    21

    22

\setlength{\parskip}{0.8ex plus 0.8ex minus 0.5ex}

    23

\setlength{\parindent}{0mm}

    24

    25

\setcounter{secnumdepth}{\subsubsectionlevel}

    26

    27

\DeclareFontShape{OT1}{cmtt}{bx}{n}

    28

{

    29

  <5><6><7><8><9><10><10.95><12><14.4><17.28><20.74><24.88>cmttb10

    30

}{}

    31

    32

\lstset{basicstyle=\ttfamily\small,numberstyle=\tiny,aboveskip=4mm,

    33

  belowskip=2mm,gobble=2,escapechar=`}

    34

\renewcommand\lstlistlistingname{List of Listings}

    35

    36

\renewcommand\nomname{Glossary}

    37

    38

\newcommand{\IgH}{\raisebox{-0.7667ex}

    39

  {\includegraphics[height=2.2ex]{images/ighsign}}}

    40

    41

\SVN $Date$

    42

\SVN $Revision$

    43

    44

\newcommand{\masterversion}{1.1.1}

    45

    46

\makeindex

    47

\makeglossary

    48

    49

%------------------------------------------------------------------------------

    50

    51

\begin{document}

    52

    53

\pagenumbering{roman}

    54

\pagestyle{empty}

    55

    56

\begin{titlepage}

    57

  \begin{center}

    58

    \rule{\textwidth}{1.5mm}

    59

    60

    {\Huge\bf IgH \includegraphics[height=2.4ex]{images/ethercat}

    61

      Master \masterversion\\[1ex]

    62

      Documentation}

    63

    64

    \vspace{1ex}

    65

    \rule{\textwidth}{1.5mm}

    66

    67

    \vspace{\fill}

    68

    {\Large Florian Pose, \url{fp@igh-essen.com}\\[1ex]

    69

      Ingenieurgemeinschaft \IgH}

    70

    71

    \vspace{\fill}

    72

    {\Large Essen, \SVNDate\\[1ex]

    73

      Revision \SVNRevision}

    74

  \end{center}

    75

\end{titlepage}

    76

    77

%------------------------------------------------------------------------------

    78

    79

\tableofcontents

    80

\listoftables

    81

\listoffigures

    82

\lstlistoflistings

    83

    84

%------------------------------------------------------------------------------

    85

    86

\newpage

    87

\pagestyle{scrheadings}

    88

    89

\section*{Conventions}

    90

\addcontentsline{toc}{section}{Conventions}

    91

\markleft{Conventions}

    92

    93

The following typographic conventions are used:

    94

    95

\begin{itemize}

    96

\item \textit{Italic face} is used for newly introduced terms, file

    97

  names, parameter names and in-text source code elements.

    98

\item \texttt{Typewriter face} is used for code examples and

    99

  command line output.

   100

\item \texttt{\textbf{Bold typewriter face}} is used for user input in

   101

  command lines.

   102

\end{itemize}

   103

   104

Data values and addresses are specified as hexadecimal values with the

   105

prefix 0x. Example: 0x88A4. Unless otherwise noted, address values are

   106

specified as byte addresses.

   107

   108

Concerning bit operations, the phrase ``setting a bit'', stands for

   109

setting the bit to $1$, ``clearing a bit'' means setting it to $0$,

   110

respectively.

   111

   112

Function names are always printed with parentheses, but without

   113

parameters. So, if a function \textit{ecrt\_request\_master()} has

   114

empty parentheses, this does not mean, that it has no parameters.

   115

   116

If shell commands have to be entered, this is marked by a prompt:

   117

   118

\begin{lstlisting}[gobble=2]

   119

  `\$`

   120

\end{lstlisting}

   121

   122

Further, if a shell command has to be entered as the superuser, the

   123

prompt ends with a mesh:

   124

   125

\begin{lstlisting}[gobble=2]

   126

  #

   127

\end{lstlisting}

   128

   129

%------------------------------------------------------------------------------

   130

   131

\chapter{The IgH EtherCAT Master}

   132

\label{chapter:master}

   133

\pagenumbering{arabic}

   134

   135

This section will first introduce the master's general features and

   136

the concepts used for master development and will then explain the

   137

master's general architecture and offer details of the different

   138

modules. In addition, it will cover state machine definitions, mailbox

   139

protocol implementation and the user space interface. The last section

   140

will deal with timing aspects.

   141

   142

%------------------------------------------------------------------------------

   143

   144

\section{Feature Summary}

   145

\label{sec:summary}

   146

\index{Master!Features}

   147

   148

The list below gives a short summary of the features of the

   149

implemented EtherCAT master.

   150

   151

\begin{itemize}

   152

\item The master runs as a kernel module for Linux 2.6.

   153

\item It comes with EtherCAT-capable network driver for RealTek

   154

  RTL8139 (and compatible) network interface cards.

   155

  \begin{itemize}

   156

  \item The Ethernet hardware is operated without interrupts.

   157

  \item Drivers for additional Ethernet hardware can easily be

   158

    implemented due to a common device interface provided by the

   159

    master.

   160

  \end{itemize}

   161

\item The master module supports multiple EtherCAT masters on one

   162

  machine.

   163

\item The master code supports any Linux realtime extension through

   164

  its independent architecture.

   165

  \begin{itemize}

   166

  \item RTAI\nomenclature{RTAI}{RealTime Application Interface},

   167

    ADEOS\nomenclature{ADEOS}{Adaptive Domain Environment for

   168

      Operating Systems}, etc.

   169

  \item It runs well even without realtime extensions.

   170

  \end{itemize}

   171

\item Common ``realtime interface'' for modules, that want to use

   172

  EtherCAT functionality.

   173

  \begin{itemize}

   174

  \item Synchronous and asynchronous sending and receiving of frames

   175

    is supported.

   176

  \item Avoidance of unnecessary copy operations for process data.

   177

  \end{itemize}

   178

\item \textit{Domains} are introduced, to allow grouping of process

   179

  data objects.

   180

  \begin{itemize}

   181

  \item Handling of multiple domains with different sampling rates.

   182

  \item Automatic calculation of process data mapping, FMMU and sync manager

   183

    configuration within each domain.

   184

  \end{itemize}

   185

\item Communication through serveral finite state machines.

   186

  \begin{itemize}

   187

  \item Bus monitoring possible during realtime operation.

   188

  \item Automatic reconfiguration of slaves on bus power failure

   189

    during realtime operation.

   190

  \item Controlling of single slaves during realtime operation.

   191

  \end{itemize}

   192

\item Master idle mode.

   193

  \begin{itemize}

   194

  \item Automatic scanning of slaves upon topology changes.

   195

  \item Bus visualisation and EoE processing without a realtime module

   196

    connected.

   197

  \end{itemize}

   198

\item Implementation of the CANopen-over-EtherCAT (CoE) protocol.

   199

  \begin{itemize}

   200

  \item Configuration of CoE-capable slaves via SDO interface.

   201

  \end{itemize}

   202

\item Implementation of the Ethernet-over-EtherCAT (EoE) protocol.

   203

  \begin{itemize}

   204

  \item Each master creates virtual network devices that are

   205

    automatically coupled to EoE-cap\-able slaves found.

   206

  \item This implementation natively supports either a switched or a

   207

    routed EoE network architecture.

   208

  \end{itemize}

   209

\item User space interface via the System Filesystem

   210

  (Sysfs)\nomenclature{Sysfs}{System Filesystem}.

   211

  \begin{itemize}

   212

  \item User space tool for bus visualisation.

   213

  \item Slave E$^2$PROM image reading and writing.

   214

  \end{itemize}

   215

\item Seamless system integration though LSB\nomenclature{LSB}{Linux

   216

    Standard Base} compliance.

   217

  \begin{itemize}

   218

  \item Master and network device configuration via Sysconfig files.

   219

  \item Linux Standard Base compatible init script for master control.

   220

  \end{itemize}

   221

\item Virtual read-only network interface for monitoring and debugging

   222

  purposes.

   223

\end{itemize}

   224

   225

%------------------------------------------------------------------------------

   226

   227

\section{License}

   228

\label{sec:license}

   229

   230

The master code is released under the terms and conditions of the GNU

   231

General Public License\index{GPL} \cite{gpl} (version 2). Other

   232

developers, that want to use EtherCAT with Linux systems, are invited

   233

to use the master code or even participate on development.

   234

   235

%------------------------------------------------------------------------------

   236

   237

\section{General Master Architecture}

   238

\label{sec:masterarch}

   239

\index{Master!Architecture}

   240

   241

The EtherCAT master is integrated into the Linux 2.6 kernel. This was

   242

an early design decision, which has been made for serveral reasons:

   243

   244

\begin{itemize}

   245

\item Kernel code has significantly better realtime characteristics,

   246

  i.~e. less jitter than user space code. It was foreseeable, that a

   247

  fieldbus master has a lot of cyclic work to do. Cyclic work is

   248

  usually triggered by timer interrupts inside the kernel. The

   249

  execution delay of a function that processes timer interrupts is

   250

  less, when it resides in kernel space, because there is no need of

   251

  time-consuming context switches to a user space process.

   252

\item It was also foreseeable, that the master code has to directly

   253

  communicate with the Ethernet hardware. This has to be done in the

   254

  kernel anyway (through network device drivers), which is one more

   255

  reason for the master code being in kernel space.

   256

\end{itemize}

   257

   258

A general overview of the master architecture can be seen in

   259

figure~\ref{fig:masterarch}.

   260

   261

\begin{figure}[htbp]

   262

  \centering

   263

  \includegraphics[width=.9\textwidth]{images/masterarch}

   264

  \caption{Master architecture}

   265

  \label{fig:masterarch}

   266

\end{figure}

   267

   268

\paragraph{Master Module}

   269

\index{Master module}

   270

   271

The EtherCAT master mainly consists of the master module, containing

   272

one or more EtherCAT masters (section~\ref{sec:mastermod}), the

   273

``Device Interface'' (section~\ref{sec:ecdev}) and the ``Realtime

   274

Interface'' (section~\ref{sec:ecrt}).

   275

   276

\paragraph{Device Modules}

   277

\index{Device modules}

   278

   279

Furthermore there are EtherCAT-capable network device driver

   280

modules\index{Device modules}, that connect to the EtherCAT master via

   281

the device interface. These modified network drivers can handle both

   282

network devices used for EtherCAT operation and ``normal'' Ethernet

   283

devices. The common case is, that the master module offers a single

   284

EtherCAT master: An EtherCAT-capable network device driver module

   285

connects one network device to this master, that is now able to send

   286

and receive EtherCAT frames, while all other network devices handled

   287

by the network driver get connected to the kernel's network stack as

   288

usual.

   289

   290

\paragraph{Realtime Modules}

   291

   292

A ``realtime module''\index{Realtime module} is a kernel module, that

   293

uses the EtherCAT master for cyclic exchange of process data with

   294

EtherCAT slaves. Realtime modules are not part of the EtherCAT master

   295

code\footnote{Although there are serveral examples provided in the

   296

  \textit{examples} directory, see chapter~\ref{chapter:usage} for

   297

  more information}, so anybody wanting to use the master has to write

   298

one. A realtime module can ``request'' a master through the realtime

   299

interface. If this succeeds, the module has the control over the

   300

master. It can now configure slaves and set up a process data image

   301

(see section~\ref{sec:processdata}) for cyclic exchange. This cyclic

   302

code has to be provided by the realtime module, so it is in hands of

   303

the developer, which mechanism to use for this. Moreover he has to

   304

decide, whether or not using a Linux realtime extension.

   305

   306

\paragraph{Why ``Realtime'' Module?}

   307

   308

The name shall not imply, that a linux realtime extension is

   309

mandatory: The master runs well even without realtime extensions, as

   310

section~\ref{sec:mini} shows. However, the code using the master is

   311

time-critical, because process data IO has to be done in cyclic work.

   312

Some EtherCAT slaves support watchdog units, that stop driving the

   313

outputs when process data was not exchanged for some time. So the

   314

names ``realtime interface'' and ``realtime module'' are quite

   315

appropriate.

   316

   317

%------------------------------------------------------------------------------

   318

   319

\subsection{Handling of Process Data}

   320

\label{sec:processdata}

   321

   322

\paragraph{Process Data Image}

   323

\index{Process data}

   324

   325

The slaves offer their inputs and outputs by presenting the master

   326

so-called ``Process Data Objects'' (PDOs\index{PDO}). The available

   327

PDOs can be determined by reading out the slave's TXPDO and RXPDO

   328

E$^2$PROM categories. The realtime module can register the PDOs for

   329

data exchange during cyclic operation. The sum of all registered PDOs

   330

defines the ``process data image'', which is exchanged via the

   331

``Logical ReadWrite'' datagrams introduced

   332

in~\cite[section~5.4.2.4]{dlspec}.

   333

   334

\paragraph{Process Data Domains}

   335

\index{Domain}

   336

   337

The process data image can be easily managed by creating co-called

   338

``domains'', which group PDOs and allocate the datagrams needed to

   339

exchange them. Domains are mandatory for process data exchange, so

   340

there has to be at least one. They were introduced for the following

   341

reasons:

   342

   343

\begin{itemize}

   344

\item The maximum size of a ``Logical ReadWrite'' datagram is limited

   345

  due to the limited size of an Ethernet frame: The maximum data size

   346

  is the Ethernet data field size minus the EtherCAT frame header,

   347

  EtherCAT datagram header and EtherCAT datagram footer: $1500 - 2 -

   348

  12 - 2 = 1484$ octets. If the size of the process data image exceeds

   349

  this limit, multiple frames have to be sent, and the image has to be

   350

  partitioned for the use of multiple datagrams. A domain manages this

   351

  automatically.

   352

\item Not every PDO has to be exchanged with the same frequency: The

   353

  values of PDOs can vary slowly over time (for example temperature

   354

  values), so exchanging them with a high frequency would just waste

   355

  bus bandwidth. For this reason, multiple domains can be created, to

   356

  group different PDOs and so allow separate exchange.

   357

\end{itemize}

   358

   359

There is no upper limit for the number of domains, but each domain

   360

occupies one FMMU in each slave involved, so the maximum number of

   361

domains is also limited by the slaves' capabilities.

   362

   363

\paragraph{FMMU Configuration}

   364

\index{FMMU!Configuration}

   365

   366

A realtime module can register PDOs for process data exchange. Every

   367

PDO is part of a memory area in the slave's physical memory, that is

   368

protected by a sync manager \cite[section~6.7]{dlspec} for

   369

synchronized access. In order to make a sync manager react on a

   370

datagram accessing its memory, it is necessary to access the last byte

   371

covered by the sync manager. Otherwise the sync manager will not react

   372

on the datagram and no data will be exchanged. That is why the whole

   373

synchronized memory area has to be included into the process data

   374

image: For example, if a certain PDO of a slave is registered for

   375

exchange with a certain domain, one FMMU will be configured to map the

   376

complete sync-manager-protected memory, the PDO resides in. If a

   377

second PDO of the same slave is registered for process data exchange

   378

within the same domain, and this PDO resides in the same

   379

sync-manager-protected memory as the first PDO, the FMMU configuration

   380

is not touched, because the appropriate memory is already part of the

   381

domain's process data image.  If the second PDO belongs to another

   382

sync-manager-protected area, this complete area is also included into

   383

the domains process data image. See figure~\ref{fig:fmmus} for an

   384

overview, how FMMU's are configured to map physical memory to logical

   385

process data images.

   386

   387

\begin{figure}[htbp]

   388

  \centering

   389

  \includegraphics[width=\textwidth]{images/fmmus}

   390

  \caption{FMMU configuration for serveral domains}

   391

  \label{fig:fmmus}

   392

\end{figure}

   393

   394

\paragraph{Process Data Pointers}

   395

   396

The figure also demonstrates the way, the realtime module can access the

   397

exchanged process data: At PDO registration, the realtime module has

   398

to provide the address of a process data pointer. Upon calculation of

   399

the domain image and allocation of process data memory, this pointer

   400

is redirected to the appropriate location inside the domain's process

   401

data memory and can later be easily dereferenced by the module code.

   402

   403

%------------------------------------------------------------------------------

   404

   405

\subsection{Operation Modes}

   406

\index{Master modes}

   407

   408

The EtherCAT master has serveral modes of operation:

   409

   410

\begin{description}

   411

\item[Orphaned Mode] This mode takes effect, when the master has no

   412

  EtherCAT-capable network device connected. No bus communication is

   413

  possible, so this mode is not of further interest.

   414

\item[Idle Mode]\index{Idle mode} takes effect when the master is

   415

  unused (i.~e. there is no realtime module, that reserved the

   416

  master). In this case, the master has the ability to scan the bus by

   417

  itsself and generously allow bus access from user space.  This mode

   418

  is meant for maintenance and visualisation.

   419

\item[Operation Mode]\index{Operation mode} The master is reserved for

   420

  exclusive access by a realtime module. In this mode, the master is

   421

  adjusted for availability and monitoring. Access from user space is

   422

  very restrictive and mostly limited to reading direction.

   423

   424

\end{description}

   425

   426

Figure~\ref{fig:modes} shows the three modes and the possible mode

   427

transitions.

   428

   429

\begin{figure}[htbp]

   430

  \centering

   431

  \includegraphics[width=.9\textwidth]{images/modes}

   432

  \caption{Master modes and transitions}

   433

  \label{fig:modes}

   434

\end{figure}

   435

   436

\subsubsection{Idle Mode}

   437

\index{Idle mode}

   438

   439

The master enters idle mode upon connection of a device module (see

   440

section~\ref{sec:device}) or releasing by a realtime module. The

   441

master owns a kernel workqueue and a suitable work structure, which is

   442

used to cyclically process the ``Idle state machine'' (see

   443

section~\ref{sec:fsm-idle}). This state machine automatically scans

   444

the bus for slaves (and re-scans upon topology changes), configures

   445

slaves for idle operation and executes pending operations from the

   446

user space interface (for example E$^2$PROM writing). On device

   447

disconnection or realtime request, the idle mode is stopped by

   448

cancelling the work and flushing the workqueue.

   449

   450

\subsubsection{Operation Mode}

   451

\index{Operation mode}

   452

   453

Operation mode is entered when a realtime module requests the master.

   454

The idle mode is stopped and the bus is scanned by getting the number

   455

of slaves and executing the ``Slave scan state machine'' (see

   456

section~\ref{sec:fsm-scan}) for each slave. The master is now ready to

   457

create domains and accept PDO registrations and slave configurations.

   458

After that, cyclic communication can be done by the realtime module.

   459

   460

\paragraph{Master Phases}

   461

   462

Every realtime module should use the master in three phases:

   463

   464

\begin{enumerate}

   465

\item \textit{Startup} - The master is requested and the bus is

   466

  validated. Domains are created and PDOs are registered. Slave

   467

  configurations are applied.

   468

\item \textit{Operation} - Cyclic code is run, process data is

   469

  exchanged and the master state machine is executed.

   470

\item \textit{Shutdown} - Cyclic code is stopped and the master

   471

  is released.

   472

\end{enumerate}

   473

   474

%------------------------------------------------------------------------------

   475

   476

\section{Device Modules}

   477

\label{sec:device}

   478

\index{Device modules}

   479

   480

Device modules are network device driver modules that handle Ethernet

   481

devices, which the master can use to connect to an EtherCAT bus.

   482

   483

Section~\ref{sec:networkdrivers} offers an overview of general Linux

   484

network driver modules, while section~\ref{sec:requirements} will show

   485

the requirements to an EtherCAT-enabled network driver. Finally,

   486

sections~\ref{sec:seldev} to~\ref{sec:patching} show how to fulfill

   487

these requirements and implement such a driver module.

   488

   489

%------------------------------------------------------------------------------

   490

   491

\subsection{Network Driver Basics}

   492

\label{sec:networkdrivers}

   493

\index{Network drivers}

   494

   495

EtherCAT relies on Ethernet hardware and the master needs a physical

   496

Ethernet device to communicate with the bus. Therefore it is necessary

   497

to understand how Linux handles network devices and their drivers,

   498

respectively.

   499

   500

\paragraph{Tasks of a Network Driver}

   501

   502

Network device drivers handle the lower two layers of the OSI model,

   503

that is the physical layer and the data-link layer. A network device

   504

itself natively handles the physical layer issues: It represents the

   505

hardware to connect to the medium and to send and receive data in the

   506

way, the physical layer protocol describes. The network device driver

   507

is responsible for getting data from the kernel's networking stack and

   508

forwarding it to the hardware, that does the physical transmission.

   509

If data is received by the hardware respectively, the driver is

   510

notified (usually by means of an interrupt) and has to read the data

   511

from the hardware memory and forward it to the network stack. There

   512

are a few more tasks, a network device driver has to handle, including

   513

queue control, statistics and device dependent features.

   514

   515

\paragraph{Driver Startup}

   516

   517

Usually, a driver searches for compatible devices on module loading.

   518

For PCI drivers, this is done by scanning the PCI bus and checking for

   519

known device IDs. If a device is found, data structures are allocated

   520

and the device is taken into operation.

   521

   522

\paragraph{Interrupt Operation}

   523

\index{Interrupt}

   524

   525

A network device usually provides a hardware interrupt that is used to

   526

notify the driver of received frames and success of transmittion, or

   527

errors, respectively. The driver has to register an interrupt service

   528

routine (ISR\index{ISR}\nomenclature{ISR}{Interrupt Service Routine}),

   529

that is executed each time, the hardware signals such an event. If the