1

%------------------------------------------------------------------------------

     2

%

     3

%  IgH EtherCAT Master Documentation

     4

%

     5

%  $Id$

     6

%

     7

%------------------------------------------------------------------------------

     8

     9

\documentclass[a4paper,12pt,BCOR6mm,bibtotoc,idxtotoc]{scrbook}

    10

    11

\usepackage[latin1]{inputenc}

    12

\usepackage[automark,headsepline]{scrpage2}

    13

\usepackage{graphicx}

    14

\usepackage{makeidx}

    15

\usepackage[refpage]{nomencl}

    16

\usepackage{listings}

    17

\usepackage{svn}

    18

\usepackage{textcomp}

    19

\usepackage{url}

    20

\usepackage[pdfpagelabels,plainpages=false]{hyperref}

    21

    22

\setlength{\parskip}{0.8ex plus 0.8ex minus 0.5ex}

    23

\setlength{\parindent}{0mm}

    24

    25

\setcounter{secnumdepth}{\subsubsectionlevel}

    26

    27

\DeclareFontShape{OT1}{cmtt}{bx}{n}

    28

{

    29

  <5><6><7><8><9><10><10.95><12><14.4><17.28><20.74><24.88>cmttb10

    30

}{}

    31

    32

\lstset{basicstyle=\ttfamily\small,numberstyle=\tiny,aboveskip=4mm,

    33

  belowskip=2mm,gobble=2,escapechar=`}

    34

\renewcommand\lstlistlistingname{List of Listings}

    35

    36

\renewcommand\nomname{Glossary}

    37

    38

\newcommand{\IgH}{\raisebox{-0.7667ex}

    39

  {\includegraphics[height=2.2ex]{images/ighsign}}}

    40

    41

\SVN $Date$

    42

\SVN $Revision$

    43

    44

\makeindex

    45

\makeglossary

    46

    47

%------------------------------------------------------------------------------

    48

    49

\begin{document}

    50

    51

\pagenumbering{roman}

    52

\pagestyle{empty}

    53

    54

\begin{titlepage}

    55

  \begin{center}

    56

    \rule{\textwidth}{1.5mm}

    57

    58

    {\Huge\bf IgH \includegraphics[height=2.4ex]{images/ethercat}

    59

      Master 1.1\\[1ex]

    60

      Documentation}

    61

    62

    \vspace{1ex}

    63

    \rule{\textwidth}{1.5mm}

    64

    65

    \vspace{\fill}

    66

    {\Large Florian Pose, \url{fp@igh-essen.com}\\[1ex]

    67

      Ingenieurgemeinschaft \IgH}

    68

    69

    \vspace{\fill}

    70

    {\Large Essen, \SVNDate\\[1ex]

    71

      Revision \SVNRevision}

    72

  \end{center}

    73

\end{titlepage}

    74

    75

%------------------------------------------------------------------------------

    76

    77

\tableofcontents

    78

\listoftables

    79

\listoffigures

    80

\lstlistoflistings

    81

    82

%------------------------------------------------------------------------------

    83

    84

\newpage

    85

\pagestyle{scrheadings}

    86

    87

\section*{Conventions}

    88

\addcontentsline{toc}{section}{Conventions}

    89

\markleft{Conventions}

    90

    91

The following typographic conventions are used:

    92

    93

\begin{itemize}

    94

\item \textit{Italic face} is used for newly introduced terms, file

    95

  names, parameter names and in-text source code elements.

    96

\item \texttt{Typewriter face} is used for code examples and

    97

  command line output.

    98

\item \texttt{\textbf{Bold typewriter face}} is used for user input in

    99

  command lines.

   100

\end{itemize}

   101

   102

Data values and addresses are specified as hexadecimal values with the

   103

prefix 0x. Example: 0x88A4. Unless otherwise noted, address values are

   104

specified as byte addresses.

   105

   106

Concerning bit operations, the phrase ``setting a bit'', stands for

   107

setting the bit to $1$, ``clearing a bit'' means setting it to $0$,

   108

respectively.

   109

   110

Function names are always printed with parentheses, but without

   111

parameters. So, if a function \textit{ecrt\_request\_master()} has

   112

empty parentheses, this does not mean, that it has no parameters.

   113

   114

If shell commands have to be entered, this is marked by a prompt:

   115

   116

\begin{lstlisting}[gobble=2]

   118

\end{lstlisting}

   119

   120

Further, if a shell command has to be entered as the superuser, the

   121

prompt ends with a mesh:

   122

   123

\begin{lstlisting}[gobble=2]

   125

\end{lstlisting}

   126

   127

%------------------------------------------------------------------------------

   128

   129

\chapter{The IgH EtherCAT Master}

   130

\label{chapter:master}

   131

\pagenumbering{arabic}

   132

   133

This section will first introduce the master's general features and

   134

the concepts used for master development and will then explain the

   135

master's general architecture and offer details of the different

   136

modules. In addition, it will cover state machine definitions, mailbox

   137

protocol implementation and the user space interface. The last section

   138

will deal with timing aspects.

   139

   140

%------------------------------------------------------------------------------

   141

   142

\section{Feature Summary}

   143

\label{sec:summary}

   144

\index{Master!Features}

   145

   146

The list below gives a short summary of the features of the

   147

implemented EtherCAT master.

   148

   149

\begin{itemize}

   150

\item The master runs as a kernel module for Linux 2.6.

   151

\item It comes with EtherCAT-capable network driver for RealTek

   152

  RTL8139 (and compatible) network interface cards.

   153

  \begin{itemize}

   154

  \item The Ethernet hardware is operated without interrupts.

   155

  \item Drivers for additional Ethernet hardware can easily be

   156

    implemented due to a common device interface provided by the

   157

    master.

   158

  \end{itemize}

   159

\item The master module supports multiple EtherCAT masters on one

   160

  machine.

   161

\item The master code supports any Linux realtime extension through

   162

  its independent architecture.

   163

  \begin{itemize}

   164

  \item RTAI\nomenclature{RTAI}{RealTime Application Interface},

   165

    ADEOS\nomenclature{ADEOS}{Adaptive Domain Environment for

   166

      Operating Systems}, etc.

   167

  \item It runs well even without realtime extensions.

   168

  \end{itemize}

   169

\item Common ``realtime interface'' for modules, that want to use

   170

  EtherCAT functionality.

   171

  \begin{itemize}

   172

  \item Synchronous and asynchronous sending and receiving of frames

   173

    is supported.

   174

  \item Avoidance of unnecessary copy operations for process data.

   175

  \end{itemize}

   176

\item \textit{Domains} are introduced, to allow grouping of process

   177

  data objects.

   178

  \begin{itemize}

   179

  \item Handling of multiple domains with different sampling rates.

   180

  \item Automatic calculation of process data mapping, FMMU and sync manager

   181

    configuration within each domain.

   182

  \end{itemize}

   183

\item Communication through serveral finite state machines.

   184

  \begin{itemize}

   185

  \item Bus monitoring possible during realtime operation.

   186

  \item Automatic reconfiguration of slaves on bus power failure

   187

    during realtime operation.

   188

  \item Controlling of single slaves during realtime operation.

   189

  \end{itemize}

   190

\item Master idle mode.

   191

  \begin{itemize}

   192

  \item Automatic scanning of slaves upon topology changes.

   193

  \item Bus visualisation and EoE processing without a realtime module

   194

    connected.

   195

  \end{itemize}

   196

\item Implementation of the CANopen-over-EtherCAT (CoE) protocol.

   197

  \begin{itemize}

   198

  \item Configuration of CoE-capable slaves via SDO interface.

   199

  \end{itemize}

   200

\item Implementation of the Ethernet-over-EtherCAT (EoE) protocol.

   201

  \begin{itemize}

   202

  \item Each master creates virtual network devices that are

   203

    automatically coupled to EoE-cap\-able slaves found.

   204

  \item This implementation natively supports either a switched or a

   205

    routed EoE network architecture.

   206

  \end{itemize}

   207

\item User space interface via the System Filesystem

   208

  (Sysfs)\nomenclature{Sysfs}{System Filesystem}.

   209

  \begin{itemize}

   210

  \item User space tool for bus visualisation.

   211

  \item Slave E$^2$PROM image reading and writing.

   212

  \end{itemize}

   213

\item Seamless system integration though LSB\nomenclature{LSB}{Linux

   214

    Standard Base} compliance.

   215

  \begin{itemize}

   216

  \item Master and network device configuration via Sysconfig files.

   217

  \item Linux Standard Base compatible init script for master control.

   218

  \end{itemize}

   219

\item Virtual read-only network interface for monitoring and debugging

   220

  purposes.

   221

\end{itemize}

   222

   223

%------------------------------------------------------------------------------

   224

   225

\section{License}

   226

\label{sec:license}

   227

   228

The master code is released under the terms and conditions of the GNU

   229

General Public License\index{GPL} \cite{gpl} (version 2). Other

   230

developers, that want to use EtherCAT with Linux systems, are invited

   231

to use the master code or even participate on development.

   232

   233

%------------------------------------------------------------------------------

   234

   235

\section{General Master Architecture}

   236

\label{sec:masterarch}

   237

\index{Master!Architecture}

   238

   239

The EtherCAT master is integrated into the Linux 2.6 kernel. This was

   240

an early design decision, which has been made for serveral reasons:

   241

   242

\begin{itemize}

   243

\item Kernel code has significantly better realtime characteristics,

   244

  i.~e. less jitter than user space code. It was foreseeable, that a

   245

  fieldbus master has a lot of cyclic work to do. Cyclic work is

   246

  usually triggered by timer interrupts inside the kernel. The

   247

  execution delay of a function that processes timer interrupts is

   248

  less, when it resides in kernel space, because there is no need of

   249

  time-consuming context switches to a user space process.

   250

\item It was also foreseeable, that the master code has to directly

   251

  communicate with the Ethernet hardware. This has to be done in the

   252

  kernel anyway (through network device drivers), which is one more

   253

  reason for the master code being in kernel space.

   254

\end{itemize}

   255

   256

A general overview of the master architecture can be seen in

   257

figure~\ref{fig:masterarch}.

   258

   259

\begin{figure}[htbp]

   260

  \centering

   261

  \includegraphics[width=.9\textwidth]{images/masterarch}

   262

  \caption{Master architecture}

   263

  \label{fig:masterarch}

   264

\end{figure}

   265

   266

\paragraph{Master Module}

   267

\index{Master module}

   268

   269

The EtherCAT master mainly consists of the master module, containing

   270

one or more EtherCAT masters (section~\ref{sec:mastermod}), the

   271

``Device Interface'' (section~\ref{sec:ecdev}) and the ``Realtime

   272

Interface'' (section~\ref{sec:ecrt}).

   273

   274

\paragraph{Device Modules}

   275

\index{Device modules}

   276

   277

Furthermore there are EtherCAT-capable network device driver

   278

modules\index{Device modules}, that connect to the EtherCAT master via

   279

the device interface. These modified network drivers can handle both

   280

network devices used for EtherCAT operation and ``normal'' Ethernet

   281

devices. The common case is, that the master module offers a single

   282

EtherCAT master: An EtherCAT-capable network device driver module

   283

connects one network device to this master, that is now able to send

   284

and receive EtherCAT frames, while all other network devices handled

   285

by the network driver get connected to the kernel's network stack as

   286

usual.

   287

   288

\paragraph{Realtime Modules}

   289

   290

A ``realtime module''\index{Realtime module} is a kernel module, that

   291

uses the EtherCAT master for cyclic exchange of process data with

   292

EtherCAT slaves. Realtime modules are not part of the EtherCAT master

   293

code\footnote{Although there are serveral examples provided in the

   294

  \textit{examples} directory, see chapter~\ref{chapter:usage} for

   295

  more information}, so anybody wanting to use the master has to write

   296

one. A realtime module can ``request'' a master through the realtime

   297

interface. If this succeeds, the module has the control over the

   298

master. It can now configure slaves and set up a process data image

   299

(see section~\ref{sec:processdata}) for cyclic exchange. This cyclic

   300

code has to be provided by the realtime module, so it is in hands of

   301

the developer, which mechanism to use for this. Moreover he has to

   302

decide, whether or not using a Linux realtime extension.

   303

   304

\paragraph{Why ``Realtime'' Module?}

   305

   306

The name shall not imply, that a linux realtime extension is

   307

mandatory: The master runs well even without realtime extensions, as

   308

section~\ref{sec:mini} shows. However, the code using the master is

   309

time-critical, because process data IO has to be done in cyclic work.

   310

Some EtherCAT slaves support watchdog units, that stop driving the

   311

outputs when process data was not exchanged for some time. So the

   312

names ``realtime interface'' and ``realtime module'' are quite

   313

appropriate.

   314

   315

%------------------------------------------------------------------------------

   316

   317

\subsection{Handling of Process Data}

   318

\label{sec:processdata}

   319

   320

\paragraph{Process Data Image}

   321

\index{Process data}

   322

   323

The slaves offer their inputs and outputs by presenting the master

   324

so-called ``Process Data Objects'' (PDOs\index{PDO}). The available

   325

PDOs can be determined by reading out the slave's TXPDO and RXPDO

   326

E$^2$PROM categories. The realtime module can register the PDOs for

   327

data exchange during cyclic operation. The sum of all registered PDOs

   328

defines the ``process data image'', which is exchanged via the

   329

``Logical ReadWrite'' datagrams introduced

   330

in~\cite[section~5.4.2.4]{dlspec}.

   331

   332

\paragraph{Process Data Domains}

   333

\index{Domain}

   334

   335

The process data image can be easily managed by creating co-called

   336

``domains'', which group PDOs and allocate the datagrams needed to

   337

exchange them. Domains are mandatory for process data exchange, so

   338

there has to be at least one. They were introduced for the following

   339

reasons:

   340

   341

\begin{itemize}

   342

\item The maximum size of a ``Logical ReadWrite'' datagram is limited

   343

  due to the limited size of an Ethernet frame: The maximum data size

   344

  is the Ethernet data field size minus the EtherCAT frame header,

   345

  EtherCAT datagram header and EtherCAT datagram footer: $1500 - 2 -

   346

  12 - 2 = 1484$ octets. If the size of the process data image exceeds

   347

  this limit, multiple frames have to be sent, and the image has to be

   348

  partitioned for the use of multiple datagrams. A domain manages this

   349

  automatically.

   350

\item Not every PDO has to be exchanged with the same frequency: The

   351

  values of PDOs can vary slowly over time (for example temperature

   352

  values), so exchanging them with a high frequency would just waste

   353

  bus bandwidth. For this reason, multiple domains can be created, to

   354

  group different PDOs and so allow separate exchange.

   355

\end{itemize}

   356

   357

There is no upper limit for the number of domains, but each domain

   358

occupies one FMMU in each slave involved, so the maximum number of

   359

domains is also limited by the slaves' capabilities.

   360

   361

\paragraph{FMMU Configuration}

   362

\index{FMMU!Configuration}

   363

   364

A realtime module can register PDOs for process data exchange. Every

   365

PDO is part of a memory area in the slave's physical memory, that is

   366

protected by a sync manager \cite[section~6.7]{dlspec} for

   367

synchronized access. In order to make a sync manager react on a

   368

datagram accessing its memory, it is necessary to access the last byte

   369

covered by the sync manager. Otherwise the sync manager will not react

   370

on the datagram and no data will be exchanged. That is why the whole

   371

synchronized memory area has to be included into the process data

   372

image: For example, if a certain PDO of a slave is registered for

   373

exchange with a certain domain, one FMMU will be configured to map the

   374

complete sync-manager-protected memory, the PDO resides in. If a

   375

second PDO of the same slave is registered for process data exchange

   376

within the same domain, and this PDO resides in the same

   377

sync-manager-protected memory as the first PDO, the FMMU configuration

   378

is not touched, because the appropriate memory is already part of the

   379

domain's process data image.  If the second PDO belongs to another

   380

sync-manager-protected area, this complete area is also included into

   381

the domains process data image. See figure~\ref{fig:fmmus} for an

   382

overview, how FMMU's are configured to map physical memory to logical

   383

process data images.

   384

   385

\begin{figure}[htbp]

   386

  \centering

   387

  \includegraphics[width=\textwidth]{images/fmmus}

   388

  \caption{FMMU configuration for serveral domains}

   389

  \label{fig:fmmus}

   390

\end{figure}

   391

   392

\paragraph{Process Data Pointers}

   393

   394

The figure also demonstrates the way, the realtime module can access the

   395

exchanged process data: At PDO registration, the realtime module has

   396

to provide the address of a process data pointer. Upon calculation of

   397

the domain image and allocation of process data memory, this pointer

   398

is redirected to the appropriate location inside the domain's process

   399

data memory and can later be easily dereferenced by the module code.

   400

   401

%------------------------------------------------------------------------------

   402

   403

\subsection{Operation Modes}

   404

\index{Master modes}

   405

   406

The EtherCAT master has serveral modes of operation:

   407

   408

\begin{description}

   409

\item[Orphaned Mode] This mode takes effect, when the master has no

   410

  EtherCAT-capable network device connected. No bus communication is

   411

  possible, so this mode is not of further interest.

   412

\item[Idle Mode]\index{Idle mode} takes effect when the master is

   413

  unused (i.~e. there is no realtime module, that reserved the

   414

  master). In this case, the master has the ability to scan the bus by

   415

  itsself and generously allow bus access from user space.  This mode

   416

  is meant for maintenance and visualisation.

   417

\item[Operation Mode]\index{Operation mode} The master is reserved for

   418

  exclusive access by a realtime module. In this mode, the master is

   419

  adjusted for availability and monitoring. Access from user space is

   420

  very restrictive and mostly limited to reading direction.

   421

   422

\end{description}

   423

   424

Figure~\ref{fig:modes} shows the three modes and the possible mode

   425

transitions.

   426

   427

\begin{figure}[htbp]

   428

  \centering

   429

  \includegraphics[width=.9\textwidth]{images/modes}

   430

  \caption{Master modes and transitions}

   431

  \label{fig:modes}

   432

\end{figure}

   433

   434

\subsubsection{Idle Mode}

   435

\index{Idle mode}

   436

   437

The master enters idle mode upon connection of a device module (see

   438

section~\ref{sec:device}) or releasing by a realtime module. The

   439

master owns a kernel workqueue and a suitable work structure, which is

   440

used to cyclically process the ``Idle state machine'' (see

   441

section~\ref{sec:fsm-idle}). This state machine automatically scans

   442

the bus for slaves (and re-scans upon topology changes), configures

   443

slaves for idle operation and executes pending operations from the

   444

user space interface (for example E$^2$PROM writing). On device

   445

disconnection or realtime request, the idle mode is stopped by

   446

cancelling the work and flushing the workqueue.

   447

   448

\subsubsection{Operation Mode}

   449

\index{Operation mode}

   450

   451

Operation mode is entered when a realtime module requests the master.

   452

The idle mode is stopped and the bus is scanned by getting the number

   453

of slaves and executing the ``Slave scan state machine'' (see

   454

section~\ref{sec:fsm-scan}) for each slave. The master is now ready to

   455

create domains and accept PDO registrations and slave configurations.

   456

After that, cyclic communication can be done by the realtime module.

   457

   458

\paragraph{Master Phases}

   459

   460

Every realtime module should use the master in three phases:

   461

   462

\begin{enumerate}

   463

\item \textit{Startup} - The master is requested and the bus is

   464

  validated. Domains are created and PDOs are registered. Slave

   465

  configurations are applied.

   466

\item \textit{Operation} - Cyclic code is run, process data is

   467

  exchanged and the master state machine is executed.

   468

\item \textit{Shutdown} - Cyclic code is stopped and the master

   469

  is released.

   470

\end{enumerate}

   471

   472

%------------------------------------------------------------------------------

   473

   474

\section{Device Modules}

   475

\label{sec:device}

   476

\index{Device modules}

   477

   478

Device modules are network device driver modules that handle Ethernet

   479

devices, which the master can use to connect to an EtherCAT bus.

   480

   481

Section~\ref{sec:networkdrivers} offers an overview of general Linux

   482

network driver modules, while section~\ref{sec:requirements} will show

   483

the requirements to an EtherCAT-enabled network driver. Finally,

   484

sections~\ref{sec:seldev} to~\ref{sec:patching} show how to fulfill

   485

these requirements and implement such a driver module.

   486

   487

%------------------------------------------------------------------------------

   488

   489

\subsection{Network Driver Basics}

   490

\label{sec:networkdrivers}

   491

\index{Network drivers}

   492

   493

EtherCAT relies on Ethernet hardware and the master needs a physical

   494

Ethernet device to communicate with the bus. Therefore it is necessary

   495

to understand how Linux handles network devices and their drivers,

   496

respectively.

   497

   498

\paragraph{Tasks of a Network Driver}

   499

   500

Network device drivers handle the lower two layers of the OSI model,

   501

that is the physical layer and the data-link layer. A network device

   502

itself natively handles the physical layer issues: It represents the

   503

hardware to connect to the medium and to send and receive data in the

   504

way, the physical layer protocol describes. The network device driver

   505

is responsible for getting data from the kernel's networking stack and

   506

forwarding it to the hardware, that does the physical transmission.

   507

If data is received by the hardware respectively, the driver is

   508

notified (usually by means of an interrupt) and has to read the data

   509

from the hardware memory and forward it to the network stack. There

   510

are a few more tasks, a network device driver has to handle, including

   511

queue control, statistics and device dependent features.

   512

   513

\paragraph{Driver Startup}

   514

   515

Usually, a driver searches for compatible devices on module loading.

   516

For PCI drivers, this is done by scanning the PCI bus and checking for

   517

known device IDs. If a device is found, data structures are allocated

   518

and the device is taken into operation.

   519

   520

\paragraph{Interrupt Operation}

   521

\index{Interrupt}

   522

   523

A network device usually provides a hardware interrupt that is used to

   524

notify the driver of received frames and success of transmittion, or

   525

errors, respectively. The driver has to register an interrupt service

   526

routine (ISR\index{ISR}\nomenclature{ISR}{Interrupt Service Routine}),

   527

that is executed each time, the hardware signals such an event. If the

   528

interrupt was thrown by the own device (multiple devices can share one

   529

hardware interrupt), the reason for the interrupt has to be determined