200 \item It runs well even without realtime extensions. |
198 \item It runs well even without realtime extensions. |
201 |
199 |
202 \end{itemize} |
200 \end{itemize} |
203 |
201 |
204 \item Common ``Application Interface'' for applications, that want to use |
202 \item Common ``Application Interface'' for applications, that want to use |
205 EtherCAT functionality (see chap.~\ref{sec:ecrt}). |
203 EtherCAT functionality (see chap.~\ref{chap:api}). |
206 |
204 |
207 \item \textit{Domains} are introduced, to allow grouping of process |
205 \item \textit{Domains} are introduced, to allow grouping of process |
208 data transfers with different slave groups and task periods. |
206 data transfers with different slave groups and task periods. |
209 |
207 |
210 \begin{itemize} |
208 \begin{itemize} |
331 |
328 |
332 \paragraph{Master Module} |
329 \paragraph{Master Module} |
333 \index{Master module} |
330 \index{Master module} |
334 |
331 |
335 Kernel module containing one or more EtherCAT master instances (see |
332 Kernel module containing one or more EtherCAT master instances (see |
336 section~\ref{sec:mastermod}), the ``Device Interface'' (see |
333 sec.~\ref{sec:mastermod}), the ``Device Interface'' (see sec.~\ref{sec:ecdev}) |
337 section~\ref{sec:ecdev}) and the ``Application Interface'' (see |
334 and the ``Application Interface'' (see chap.~\ref{chap:api}). |
338 chap.~\ref{sec:ecrt}). |
|
339 |
335 |
340 \paragraph{Device Modules} |
336 \paragraph{Device Modules} |
341 \index{Device modules} |
337 \index{Device modules} |
342 |
338 |
343 EtherCAT-capable Ethernet device driver modules\index{Device modules}, that |
339 EtherCAT-capable Ethernet device driver modules\index{Device modules}, that |
344 offer their devices to the EtherCAT master via the device interface (see |
340 offer their devices to the EtherCAT master via the device interface (see |
345 section~\ref{sec:ecdev}). These modified network drivers can handle network |
341 sec.~\ref{sec:ecdev}). These modified network drivers can handle network |
346 devices used for EtherCAT operation and ``normal'' Ethernet devices in |
342 devices used for EtherCAT operation and ``normal'' Ethernet devices in |
347 parallel. A master can accept a certain device and then is able to send and |
343 parallel. A master can accept a certain device and then is able to send and |
348 receive EtherCAT frames. Ethernet devices declined by the master module are |
344 receive EtherCAT frames. Ethernet devices declined by the master module are |
349 connected to the kernel's network stack as usual. |
345 connected to the kernel's network stack as usual. |
350 |
346 |
354 Kernel modules, that use the EtherCAT master (usually for cyclic exchange of |
350 Kernel modules, that use the EtherCAT master (usually for cyclic exchange of |
355 process data with EtherCAT slaves). These modules are not part of the EtherCAT |
351 process data with EtherCAT slaves). These modules are not part of the EtherCAT |
356 master code\footnote{Although there are some examples provided in the |
352 master code\footnote{Although there are some examples provided in the |
357 \textit{examples/} directory.}, but have to be generated or written by the |
353 \textit{examples/} directory.}, but have to be generated or written by the |
358 user. An application module can ``request'' a master through the application |
354 user. An application module can ``request'' a master through the application |
359 interface (see chap.~\ref{sec:ecrt}). If this succeeds, the module has the |
355 interface (see chap.~\ref{chap:api}). If this succeeds, the module has the |
360 control over the master: It can provide a bus configuration and exchange |
356 control over the master: It can provide a bus configuration and exchange |
361 process data. |
357 process data. |
362 |
358 |
363 %------------------------------------------------------------------------------ |
359 %------------------------------------------------------------------------------ |
364 |
360 |
379 master still waits for its Ethernet device to connect. No bus communication is |
375 master still waits for its Ethernet device to connect. No bus communication is |
380 possible until then. |
376 possible until then. |
381 |
377 |
382 \item[Idle phase]\index{Idle phase} takes effect when the master has accepted |
378 \item[Idle phase]\index{Idle phase} takes effect when the master has accepted |
383 an Ethernet device, but is not requested by any application yet. The master |
379 an Ethernet device, but is not requested by any application yet. The master |
384 runs its state machine (see section~\ref{sec:fsm-master}), that automatically |
380 runs its state machine (see sec.~\ref{sec:fsm-master}), that automatically |
385 scans the bus for slaves and executes pending operations from the userspace |
381 scans the bus for slaves and executes pending operations from the userspace |
386 interface (for example Sdo access). The command-line tool can be used to |
382 interface (for example Sdo access). The command-line tool can be used to |
387 access the bus, but there is no process data exchange because of the missing |
383 access the bus, but there is no process data exchange because of the missing |
388 bus configuration. |
384 bus configuration. |
389 |
385 |
429 \end{lstlisting} |
425 \end{lstlisting} |
430 |
426 |
431 The two masters can be addressed by their indices 0 and 1 respectively (see |
427 The two masters can be addressed by their indices 0 and 1 respectively (see |
432 figure~\ref{fig:masters}). The master index is needed for the |
428 figure~\ref{fig:masters}). The master index is needed for the |
433 \lstinline+ecrt_master_request()+ function of the application interface (see |
429 \lstinline+ecrt_master_request()+ function of the application interface (see |
434 chap.~\ref{sec:ecrt}) and the \lstinline+--master+ option of the |
430 chap.~\ref{chap:api}) and the \lstinline+--master+ option of the |
435 \textit{ethercat} command-line tool (see section~\ref{sec:ethercat}), which |
431 \textit{ethercat} command-line tool (see sec.~\ref{sec:tool}), which defaults |
436 defaults to $0$. |
432 to $0$. |
437 |
433 |
438 \begin{figure}[htbp] |
434 \begin{figure}[htbp] |
439 \centering |
435 \centering |
440 \includegraphics[width=.5\textwidth]{images/masters} |
436 \includegraphics[width=.5\textwidth]{images/masters} |
441 \caption{Multiple masters in one module} |
437 \caption{Multiple masters in one module} |
444 |
440 |
445 \paragraph{Init script} |
441 \paragraph{Init script} |
446 \index{Init script} |
442 \index{Init script} |
447 |
443 |
448 Most probably you won't want to load the master module and the Ethernet driver |
444 Most probably you won't want to load the master module and the Ethernet driver |
449 modules manually, but start the master as a service. See |
445 modules manually, but start the master as a service. See sec.~\ref{sec:system} |
450 section~\ref{sec:system} on how to do this. |
446 on how to do this. |
451 |
447 |
452 \paragraph{Syslog} |
448 \paragraph{Syslog} |
453 |
449 |
454 The master module outputs information about it's state and events to the |
450 The master module outputs information about it's state and events to the |
455 kernel ring buffer. These also end up in the system logs. The above module |
451 kernel ring buffer. These also end up in the system logs. The above module |
479 \paragraph{Process Data Image} |
475 \paragraph{Process Data Image} |
480 \index{Process data} |
476 \index{Process data} |
481 |
477 |
482 The slaves offer their inputs and outputs by presenting the master so-called |
478 The slaves offer their inputs and outputs by presenting the master so-called |
483 ``Process Data Objects'' (Pdos\index{Pdo}). The available Pdos can be |
479 ``Process Data Objects'' (Pdos\index{Pdo}). The available Pdos can be |
484 determined by reading out the slave's TXPDO and RXPDO E$^2$PROM categories. The |
480 determined by reading out the slave's TXPDO and RXPDO E$^2$PROM categories. |
485 application can register the Pdos for data exchange during cyclic operation. |
481 The application can register the Pdos for data exchange during cyclic |
486 The sum of all registered Pdos defines the ``process data image'', which is |
482 operation. The sum of all registered Pdos defines the ``process data image'', |
487 exchanged via the ``Logical ReadWrite'' datagrams introduced |
483 which is exchanged via the ``Logical ReadWrite'' datagrams introduced |
488 in~\cite[section~5.4.2.4]{dlspec}. |
484 in~\cite[sec.~5.4.2.4]{dlspec}. |
489 |
485 |
490 \paragraph{Process Data Domains} |
486 \paragraph{Process Data Domains} |
491 \index{Domain} |
487 \index{Domain} |
492 |
488 |
493 The process data image can be easily managed by creating so-called |
489 The process data image can be easily managed by creating so-called |
517 domains is also limited by the slaves' capabilities. |
513 domains is also limited by the slaves' capabilities. |
518 |
514 |
519 \paragraph{FMMU Configuration} |
515 \paragraph{FMMU Configuration} |
520 \index{FMMU!Configuration} |
516 \index{FMMU!Configuration} |
521 |
517 |
522 An application can register Pdos for process data exchange. Every |
518 An application can register Pdos for process data exchange. Every Pdo is part |
523 Pdo is part of a memory area in the slave's physical memory, that is |
519 of a memory area in the slave's physical memory, that is protected by a sync |
524 protected by a sync manager \cite[section~6.7]{dlspec} for |
520 manager \cite[sec.~6.7]{dlspec} for synchronized access. In order to make a |
525 synchronized access. In order to make a sync manager react on a |
521 sync manager react on a datagram accessing its memory, it is necessary to |
526 datagram accessing its memory, it is necessary to access the last byte |
522 access the last byte covered by the sync manager. Otherwise the sync manager |
527 covered by the sync manager. Otherwise the sync manager will not react |
523 will not react on the datagram and no data will be exchanged. That is why the |
528 on the datagram and no data will be exchanged. That is why the whole |
524 whole synchronized memory area has to be included into the process data image: |
529 synchronized memory area has to be included into the process data |
525 For example, if a certain Pdo of a slave is registered for exchange with a |
530 image: For example, if a certain Pdo of a slave is registered for |
526 certain domain, one FMMU will be configured to map the complete |
531 exchange with a certain domain, one FMMU will be configured to map the |
527 sync-manager-protected memory, the Pdo resides in. If a second Pdo of the same |
532 complete sync-manager-protected memory, the Pdo resides in. If a |
528 slave is registered for process data exchange within the same domain, and this |
533 second Pdo of the same slave is registered for process data exchange |
529 Pdo resides in the same sync-manager-protected memory as the first Pdo, the |
534 within the same domain, and this Pdo resides in the same |
530 FMMU configuration is not touched, because the appropriate memory is already |
535 sync-manager-protected memory as the first Pdo, the FMMU configuration |
531 part of the domain's process data image. If the second Pdo belongs to another |
536 is not touched, because the appropriate memory is already part of the |
532 sync-manager-protected area, this complete area is also included into the |
537 domain's process data image. If the second Pdo belongs to another |
533 domains process data image. See figure~\ref{fig:fmmus} for an overview, how |
538 sync-manager-protected area, this complete area is also included into |
534 FMMU's are configured to map physical memory to logical process data images. |
539 the domains process data image. See figure~\ref{fig:fmmus} for an |
|
540 overview, how FMMU's are configured to map physical memory to logical |
|
541 process data images. |
|
542 |
535 |
543 \begin{figure}[htbp] |
536 \begin{figure}[htbp] |
544 \centering |
537 \centering |
545 \includegraphics[width=\textwidth]{images/fmmus} |
538 \includegraphics[width=\textwidth]{images/fmmus} |
546 \caption{FMMU configuration for several domains} |
539 \caption{FMMU configuration for several domains} |
575 |
568 |
576 The application interface provides functions and data structures for |
569 The application interface provides functions and data structures for |
577 applications to access and use an EtherCAT master. The complete documentation |
570 applications to access and use an EtherCAT master. The complete documentation |
578 of the interface is included as Doxygen~\cite{doxygen} comments in the header |
571 of the interface is included as Doxygen~\cite{doxygen} comments in the header |
579 file \textit{include/ecrt.h}. You can either directly view the file comments |
572 file \textit{include/ecrt.h}. You can either directly view the file comments |
580 or generate an HTML documentation as described in section~\ref{sec:gendoc}. |
573 or generate an HTML documentation as described in sec.~\ref{sec:gendoc}. |
581 |
574 |
582 The following sections cover a general description of the application |
575 The following sections cover a general description of the application |
583 interface. |
576 interface. |
584 |
577 |
585 Every application should use the master in two steps: |
578 Every application should use the master in two steps: |
586 |
579 |
587 \begin{description} |
580 \begin{description} |
588 |
581 |
589 \item[Configuration] The master is requested and the configuration is applied. |
582 \item[Configuration] The master is requested and the configuration is applied. |
590 Domains are created Slaves are configured and Pdo entries are registered (see |
583 Domains are created Slaves are configured and Pdo entries are registered (see |
591 section~\ref{sec:masterconfig}). |
584 sec.~\ref{sec:masterconfig}). |
592 |
585 |
593 \item[Operation] Cyclic code is run, process data is exchanged (see |
586 \item[Operation] Cyclic code is run, process data is exchanged (see |
594 section~\ref{sec:cyclic}). |
587 sec.~\ref{sec:cyclic}). |
595 |
588 |
596 \end{description} |
589 \end{description} |
597 |
590 |
598 \paragraph{Example Applications} \index{Example Applications} There are a few |
591 \paragraph{Example Applications} \index{Example Applications} There are a few |
599 example applications in the \textit{examples/} subdirectory of the master |
592 example applications in the \textit{examples/} subdirectory of the master |
628 \section{Concurrent Master Access} % FIXME |
621 \section{Concurrent Master Access} % FIXME |
629 \label{sec:concurr} |
622 \label{sec:concurr} |
630 \index{Concurrency} |
623 \index{Concurrency} |
631 |
624 |
632 In some cases, one master is used by several instances, for example when an |
625 In some cases, one master is used by several instances, for example when an |
633 application does cyclic process data exchange, and there are EoE-capable slaves |
626 application does cyclic process data exchange, and there are EoE-capable |
634 that require to exchange Ethernet data with the kernel (see |
627 slaves that require to exchange Ethernet data with the kernel (see |
635 section~\ref{sec:eoe}). For this reason, the master is a shared resource, |
628 sec.~\ref{sec:eoe}). For this reason, the master is a shared resource, and |
636 and access to it has to be sequentialized. This is usually done by locking with |
629 access to it has to be sequentialized. This is usually done by locking with |
637 semaphores, or other methods to protect critical sections. |
630 semaphores, or other methods to protect critical sections. |
638 |
631 |
639 The master itself can not provide locking mechanisms, because it has no chance |
632 The master itself can not provide locking mechanisms, because it has no chance |
640 to know the appropriate kind of lock. For example if the application uses RTAI |
633 to know the appropriate kind of lock. For example if the application uses RTAI |
641 functionality, ordinary kernel semaphores would not be sufficient. For that, an |
634 functionality, ordinary kernel semaphores would not be sufficient. For that, an |
656 Figure~\ref{fig:locks} exemplary shows, how two processes share one master: |
649 Figure~\ref{fig:locks} exemplary shows, how two processes share one master: |
657 The application's cyclic task uses the master for process data exchange, while |
650 The application's cyclic task uses the master for process data exchange, while |
658 the master-internal EoE process uses it to communicate with EoE-capable |
651 the master-internal EoE process uses it to communicate with EoE-capable |
659 slaves. Both have to acquire the master lock before access: The application |
652 slaves. Both have to acquire the master lock before access: The application |
660 task can access the lock natively, while the EoE process has to use the |
653 task can access the lock natively, while the EoE process has to use the |
661 callbacks. See the application interface documentation (chap.~\ref{sec:ecrt} |
654 callbacks. See the application interface documentation (chap.~\ref{chap:api} |
662 of how to use the locking callbacks. |
655 of how to use the locking callbacks. |
663 |
656 |
664 %------------------------------------------------------------------------------ |
657 %------------------------------------------------------------------------------ |
665 |
658 |
666 \chapter{Ethernet Devices} |
659 \chapter{Ethernet Devices} |
806 network driver. |
799 network driver. |
807 |
800 |
808 %------------------------------------------------------------------------------ |
801 %------------------------------------------------------------------------------ |
809 |
802 |
810 \section{EtherCAT Device Drivers} |
803 \section{EtherCAT Device Drivers} |
811 \label{sec:ethercatdrivers} |
804 \label{sec:drivers} |
812 |
805 |
813 There are a few requirements for Ethernet network devices to function as |
806 There are a few requirements for Ethernet network devices to function as |
814 EtherCAT devices, when connected to an EtherCAT bus. |
807 EtherCAT devices, when connected to an EtherCAT bus. |
815 |
808 |
816 \paragraph{Dedicated Interfaces} |
809 \paragraph{Dedicated Interfaces} |
900 \section{Device Selection} |
893 \section{Device Selection} |
901 \label{sec:deviceselection} |
894 \label{sec:deviceselection} |
902 |
895 |
903 After loading the master module, at least one EtherCAT-capable network driver |
896 After loading the master module, at least one EtherCAT-capable network driver |
904 module has to be loaded, that offers its devices to the master (see |
897 module has to be loaded, that offers its devices to the master (see |
905 section~\ref{sec:ecdev}. The master module knows the devices to choose from the |
898 sec.~\ref{sec:ecdev}. The master module knows the devices to choose from the |
906 module parameters (see section~\ref{sec:mastermod}). If the init script is used |
899 module parameters (see sec.~\ref{sec:mastermod}). If the init script is used |
907 to start the master, the drivers and devices to use can be specified in the |
900 to start the master, the drivers and devices to use can be specified in the |
908 sysconfig file (see section~\ref{sec:sysconfig}). |
901 sysconfig file (see sec.~\ref{sec:sysconfig}). |
909 |
902 |
910 %------------------------------------------------------------------------------ |
903 %------------------------------------------------------------------------------ |
911 |
904 |
912 \section{EtherCAT Device Interface} |
905 \section{EtherCAT Device Interface} |
913 \label{sec:ecdev} |
906 \label{sec:ecdev} |
914 \index{Device interface} |
907 \index{Device interface} |
915 |
908 |
916 An anticipation to the section about the master module |
909 An anticipation to the section about the master module |
917 (section~\ref{sec:mastermod}) has to be made in order to understand |
910 (sec.~\ref{sec:mastermod}) has to be made in order to understand the way, a |
918 the way, a network device driver module can connect a device to a |
911 network device driver module can connect a device to a specific EtherCAT |
919 specific EtherCAT master. |
912 master. |
920 |
913 |
921 The master module provides a ``device interface'' for network device drivers. |
914 The master module provides a ``device interface'' for network device drivers. |
922 To use this interface, a network device driver module must include the header |
915 To use this interface, a network device driver module must include the header |
923 \textit{devices/ecdev.h}\nomenclature{ecdev}{EtherCAT Device}, coming with the |
916 \textit{devices/ecdev.h}\nomenclature{ecdev}{EtherCAT Device}, coming with the |
924 EtherCAT master code. This header offers a function interface for EtherCAT |
917 EtherCAT master code. This header offers a function interface for EtherCAT |
925 devices. All functions of the device interface are named with the prefix |
918 devices. All functions of the device interface are named with the prefix |
926 \lstinline+ecdev+. |
919 \lstinline+ecdev+. |
927 |
920 |
928 The documentation of the device interface can be found in the header file or in |
921 The documentation of the device interface can be found in the header file or |
929 the appropriate module of the interface documentation (see |
922 in the appropriate module of the interface documentation (see |
930 section~\ref{sec:gendoc} for generation instructions). |
923 sec.~\ref{sec:gendoc} for generation instructions). |
931 |
924 |
932 \ldots % FIXME general description of the device interface |
925 \ldots % FIXME general description of the device interface |
933 |
926 |
934 %------------------------------------------------------------------------------ |
927 %------------------------------------------------------------------------------ |
935 |
928 |
965 a \lstinline+net_device+ structure with a \lstinline+priv_data+ field to |
958 a \lstinline+net_device+ structure with a \lstinline+priv_data+ field to |
966 attach driver-dependent data to the structure. To distinguish between normal |
959 attach driver-dependent data to the structure. To distinguish between normal |
967 Ethernet devices and the ones used by EtherCAT masters, the private data |
960 Ethernet devices and the ones used by EtherCAT masters, the private data |
968 structure used by the driver could be extended by a pointer, that points to an |
961 structure used by the driver could be extended by a pointer, that points to an |
969 \lstinline+ec_device_t+ object returned by \lstinline+ecdev_offer()+ (see |
962 \lstinline+ec_device_t+ object returned by \lstinline+ecdev_offer()+ (see |
970 section~\ref{sec:ecdev}) if the device is used by a master and otherwise is |
963 sec.~\ref{sec:ecdev}) if the device is used by a master and otherwise is zero. |
971 zero. |
|
972 |
964 |
973 The RealTek RTL-8139 Fast Ethernet driver is a ``simple'' Ethernet driver and |
965 The RealTek RTL-8139 Fast Ethernet driver is a ``simple'' Ethernet driver and |
974 can be taken as an example to patch new drivers. The interesting sections can |
966 can be taken as an example to patch new drivers. The interesting sections can |
975 be found by searching the string ``ecdev" in the file |
967 be found by searching the string ``ecdev" in the file |
976 \textit{devices/8139too-2.6.24-ethercat.c}. |
968 \textit{devices/8139too-2.6.24-ethercat.c}. |
1002 the function is deprecated and stopped existing. Nevertheless it is adequate |
994 the function is deprecated and stopped existing. Nevertheless it is adequate |
1003 for showing it's own restrictions.}. Internally, it queues the specified |
995 for showing it's own restrictions.}. Internally, it queues the specified |
1004 datagram, invokes the \textit{ec\_master\_send\_datagrams()} function to send |
996 datagram, invokes the \textit{ec\_master\_send\_datagrams()} function to send |
1005 a frame with the queued datagram and then waits actively for its reception. |
997 a frame with the queued datagram and then waits actively for its reception. |
1006 |
998 |
1007 This sequential approach is very simple, reflecting in only three |
999 This sequential approach is very simple, reflecting in only three lines of |
1008 lines of code. The disadvantage is, that the master is blocked for the |
1000 code. The disadvantage is, that the master is blocked for the time it waits |
1009 time it waits for datagram reception. There is no difficulty when only |
1001 for datagram reception. There is no difficulty when only one instance is using |
1010 one instance is using the master, but if more instances want to |
1002 the master, but if more instances want to (synchronously\footnote{At this |
1011 (synchronously\footnote{At this time, synchronous master access will |
1003 time, synchronous master access will be adequate to show the advantages of an |
1012 be adequate to show the advantages of an FSM. The asynchronous |
1004 FSM. The asynchronous approach will be discussed in sec.~\ref{sec:eoe}}) use |
1013 approach will be discussed in section~\ref{sec:eoe}}) use the |
1005 the master, it is inevitable to think about an alternative to the sequential |
1014 master, it is inevitable to think about an alternative to the |
1006 model. |
1015 sequential model. |
|
1016 |
1007 |
1017 Master access has to be sequentialized for more than one instance |
1008 Master access has to be sequentialized for more than one instance |
1018 wanting to send and receive datagrams synchronously. With the present |
1009 wanting to send and receive datagrams synchronously. With the present |
1019 approach, this would result in having one phase of active waiting for |
1010 approach, this would result in having one phase of active waiting for |
1020 each instance, which would be non-acceptable especially in realtime |
1011 each instance, which would be non-acceptable especially in realtime |
1055 } |
1046 } |
1056 slave_states = EC_READ_U8(datagram->data); // process datagram |
1047 slave_states = EC_READ_U8(datagram->data); // process datagram |
1057 // state processing finished. |
1048 // state processing finished. |
1058 \end{lstlisting} |
1049 \end{lstlisting} |
1059 |
1050 |
1060 See section~\ref{sec:statemodel} for an introduction to the |
1051 See sec.~\ref{sec:statemodel} for an introduction to the state machine |
1061 state machine programming concept used in the master code. |
1052 programming concept used in the master code. |
1062 |
1053 |
1063 %------------------------------------------------------------------------------ |
1054 %------------------------------------------------------------------------------ |
1064 |
1055 |
1065 \section{State Machine Theory} |
1056 \section{State Machine Theory} |
1066 \label{sec:fsmtheory} |
1057 \label{sec:fsmtheory} |
1234 \paragraph{Mealy and Moore} |
1225 \paragraph{Mealy and Moore} |
1235 |
1226 |
1236 If a closer look is taken to the above listing, it can be seen that the |
1227 If a closer look is taken to the above listing, it can be seen that the |
1237 actions executed (the ``outputs'' of the state machine) only depend on the |
1228 actions executed (the ``outputs'' of the state machine) only depend on the |
1238 current state. This accords to the ``Moore'' model introduced in |
1229 current state. This accords to the ``Moore'' model introduced in |
1239 section~\ref{sec:fsmtheory}. As mentioned, the ``Mealy'' model offers a higher |
1230 sec.~\ref{sec:fsmtheory}. As mentioned, the ``Mealy'' model offers a higher |
1240 flexibility, which can be seen in the listing below: |
1231 flexibility, which can be seen in the listing below: |
1241 |
1232 |
1242 \begin{lstlisting}[gobble=2,language=C,numbers=left] |
1233 \begin{lstlisting}[gobble=2,language=C,numbers=left] |
1243 void state7(void *priv_data) { |
1234 void state7(void *priv_data) { |
1244 if (some_condition) { |
1235 if (some_condition) { |
1283 \paragraph{Using Sub State Machines} |
1274 \paragraph{Using Sub State Machines} |
1284 |
1275 |
1285 To avoid having too much states, certain functions of the EtherCAT master |
1276 To avoid having too much states, certain functions of the EtherCAT master |
1286 state machine have been sourced out into sub state machines. This helps to |
1277 state machine have been sourced out into sub state machines. This helps to |
1287 encapsulate the related workflows and moreover avoids the ``state explosion'' |
1278 encapsulate the related workflows and moreover avoids the ``state explosion'' |
1288 phenomenon described in section~\ref{sec:fsmtheory}. If the master would |
1279 phenomenon described in sec.~\ref{sec:fsmtheory}. If the master would instead |
1289 instead use one big state machine, the number of states would be a multiple of |
1280 use one big state machine, the number of states would be a multiple of the |
1290 the actual number. This would increase the level of complexity to a |
1281 actual number. This would increase the level of complexity to a non-manageable |
1291 non-manageable grade. |
1282 grade. |
1292 |
1283 |
1293 \paragraph{Executing Sub State Machines} |
1284 \paragraph{Executing Sub State Machines} |
1294 |
1285 |
1295 If a state machine starts to execute a sub state machine, it usually |
1286 If a state machine starts to execute a sub state machine, it usually |
1296 remains in one state until the sub state machine terminates. This is |
1287 remains in one state until the sub state machine terminates. This is |
1404 image memory. |
1395 image memory. |
1405 |
1396 |
1406 \item[SII Data] The SII contents are read into the master's image. |
1397 \item[SII Data] The SII contents are read into the master's image. |
1407 |
1398 |
1408 \item[PREOP] If the slave supports CoE, it is set to PREOP state using the |
1399 \item[PREOP] If the slave supports CoE, it is set to PREOP state using the |
1409 State change FSM (see section~\ref{sec:fsm-change}) to enable mailbox |
1400 State change FSM (see sec.~\ref{sec:fsm-change}) to enable mailbox |
1410 communication and read the Pdo configuration via CoE. |
1401 communication and read the Pdo configuration via CoE. |
1411 |
1402 |
1412 \item[Pdos] The Pdos are read via CoE (if supported) using the Pdo Reading FSM |
1403 \item[Pdos] The Pdos are read via CoE (if supported) using the Pdo Reading FSM |
1413 (see section~\ref{sec:fsm-pdo}). If this is successful, the Pdo information |
1404 (see sec.~\ref{sec:fsm-pdo}). If this is successful, the Pdo information from |
1414 from the SII (if any) is overwritten. |
1405 the SII (if any) is overwritten. |
1415 |
1406 |
1416 \end{description} |
1407 \end{description} |
1417 |
1408 |
1418 %------------------------------------------------------------------------------ |
1409 %------------------------------------------------------------------------------ |
1419 |
1410 |
1448 |
1439 |
1449 \item[PREOP] The state change FSM is used to bring the slave to PREOP state. |
1440 \item[PREOP] The state change FSM is used to bring the slave to PREOP state. |
1450 If this is the requested state, the state machine is finished. |
1441 If this is the requested state, the state machine is finished. |
1451 |
1442 |
1452 \item[Sdo Configuration] If there is a slave configuration attached (see |
1443 \item[Sdo Configuration] If there is a slave configuration attached (see |
1453 section~\ref{sec:masterconfig}), and there are any Sdo configurations are |
1444 sec.~\ref{sec:masterconfig}), and there are any Sdo configurations are |
1454 provided by the application, these are sent to the slave. |
1445 provided by the application, these are sent to the slave. |
1455 |
1446 |
1456 \item[Pdo Configuration] The Pdo configuration state machine is executed to |
1447 \item[Pdo Configuration] The Pdo configuration state machine is executed to |
1457 apply all necessary Pdo configurations. |
1448 apply all necessary Pdo configurations. |
1458 |
1449 |
1478 \index{FSM!State Change} |
1469 \index{FSM!State Change} |
1479 |
1470 |
1480 The state change state machine, which can be seen in |
1471 The state change state machine, which can be seen in |
1481 figure~\ref{fig:fsm-change}, leads through the process of changing a slave's |
1472 figure~\ref{fig:fsm-change}, leads through the process of changing a slave's |
1482 application-layer state. This implements the states and transitions described |
1473 application-layer state. This implements the states and transitions described |
1483 in \cite[section~6.4.1]{alspec}. |
1474 in \cite[sec.~6.4.1]{alspec}. |
1484 |
1475 |
1485 \begin{figure}[htbp] |
1476 \begin{figure}[htbp] |
1486 \centering |
1477 \centering |
1487 \includegraphics[width=.6\textwidth]{graphs/fsm_change} |
1478 \includegraphics[width=.6\textwidth]{graphs/fsm_change} |
1488 \caption{Transition Diagram of the State Change State Machine} |
1479 \caption{Transition Diagram of the State Change State Machine} |
1490 \end{figure} |
1481 \end{figure} |
1491 |
1482 |
1492 \begin{description} |
1483 \begin{description} |
1493 |
1484 |
1494 \item[Start] The new application-layer state is requested via the ``AL Control |
1485 \item[Start] The new application-layer state is requested via the ``AL Control |
1495 Request'' register (see ~\cite[section 5.3.1]{alspec}). |
1486 Request'' register (see ~\cite[sec. 5.3.1]{alspec}). |
1496 |
1487 |
1497 \item[Check for Response] Some slave need some time to respond to an AL state |
1488 \item[Check for Response] Some slave need some time to respond to an AL state |
1498 change command, and do not respond for some time. For this case, the command |
1489 change command, and do not respond for some time. For this case, the command |
1499 is issued again, until it is acknowledged. |
1490 is issued again, until it is acknowledged. |
1500 |
1491 |
1501 \item[Check AL Status] If the AL State change datagram was acknowledged, the |
1492 \item[Check AL Status] If the AL State change datagram was acknowledged, the |
1502 ``AL Control Response'' register (see~\cite[section 5.3.2]{alspec}) must be |
1493 ``AL Control Response'' register (see~\cite[sec. 5.3.2]{alspec}) must be read |
1503 read out until the slave changes the AL state. |
1494 out until the slave changes the AL state. |
1504 |
1495 |
1505 \item[AL Status Code] If the slave refused the state change command, the |
1496 \item[AL Status Code] If the slave refused the state change command, the |
1506 reason can be read from the ``AL Status Code'' field in the ``AL State |
1497 reason can be read from the ``AL Status Code'' field in the ``AL State |
1507 Changed'' registers (see~\cite[section 5.3.3]{alspec}). |
1498 Changed'' registers (see~\cite[sec. 5.3.3]{alspec}). |
1508 |
1499 |
1509 \item[Acknowledge State] If the state change was not successful, the master |
1500 \item[Acknowledge State] If the state change was not successful, the master |
1510 has to acknowledge the old state by writing to the ``AL Control request'' |
1501 has to acknowledge the old state by writing to the ``AL Control request'' |
1511 register again. |
1502 register again. |
1512 |
1503 |
1524 \section{The SII State Machine} |
1515 \section{The SII State Machine} |
1525 \label{sec:fsm-sii} |
1516 \label{sec:fsm-sii} |
1526 \index{FSM!SII} |
1517 \index{FSM!SII} |
1527 |
1518 |
1528 The SII\index{SII} state machine (shown in figure~\ref{fig:fsm-sii}) |
1519 The SII\index{SII} state machine (shown in figure~\ref{fig:fsm-sii}) |
1529 implements the process of reading or writing SII data via the |
1520 implements the process of reading or writing SII data via the Slave |
1530 Slave Information Interface described in \cite[section~6.4]{dlspec}. |
1521 Information Interface described in \cite[sec.~6.4]{dlspec}. |
1531 |
1522 |
1532 \begin{figure}[htbp] |
1523 \begin{figure}[htbp] |
1533 \centering |
1524 \centering |
1534 \includegraphics[width=.5\textwidth]{graphs/fsm_sii} |
1525 \includegraphics[width=.5\textwidth]{graphs/fsm_sii} |
1535 \caption{Transition Diagram of the SII State Machine} |
1526 \caption{Transition Diagram of the SII State Machine} |
1576 \label{sec:fsm-pdo} |
1567 \label{sec:fsm-pdo} |
1577 \index{FSM!Pdo} |
1568 \index{FSM!Pdo} |
1578 |
1569 |
1579 The Pdo state machines are a set of state machines that read or write the Pdo |
1570 The Pdo state machines are a set of state machines that read or write the Pdo |
1580 assignment and the Pdo mapping via the ``CoE Communication Area'' described in |
1571 assignment and the Pdo mapping via the ``CoE Communication Area'' described in |
1581 \cite[section 5.6.7.4]{alspec}. For the object access, the |
1572 \cite[sec. 5.6.7.4]{alspec}. For the object access, the CANopen-over-EtherCAT |
1582 CANopen-over-EtherCAT access primitives are used (see |
1573 access primitives are used (see sec.~\ref{sec:coe}), so the slave must support |
1583 section~\ref{sec:coe}), so the slave must support the CoE mailbox protocol. |
1574 the CoE mailbox protocol. |
1584 |
1575 |
1585 \paragraph{Pdo Reading FSM} This state machine (fig.~\ref{fig:fsm-pdo-read}) |
1576 \paragraph{Pdo Reading FSM} This state machine (fig.~\ref{fig:fsm-pdo-read}) |
1586 has the purpose to read the complete Pdo configuration of a slave. It reads |
1577 has the purpose to read the complete Pdo configuration of a slave. It reads |
1587 the Pdo assignment for each Sync Manager and uses the Pdo Entry Reading FSM |
1578 the Pdo assignment for each Sync Manager and uses the Pdo Entry Reading FSM |
1588 (fig.~\ref{fig:fsm-pdo-entry-read}) to read the mapping for each assigned Pdo. |
1579 (fig.~\ref{fig:fsm-pdo-entry-read}) to read the mapping for each assigned Pdo. |
1653 The master creates a virtual EoE network interface for every EoE-capable |
1644 The master creates a virtual EoE network interface for every EoE-capable |
1654 slave. These interfaces are called either |
1645 slave. These interfaces are called either |
1655 |
1646 |
1656 \begin{description} |
1647 \begin{description} |
1657 |
1648 |
1658 \item[eoeXsY] for a slave without an alias address (see |
1649 \item[eoeXsY] for a slave without an alias address (see sec.~\ref{sec:alias}), |
1659 section~\ref{sec:alias}), where X is the master index and Y is the slave's |
1650 where X is the master index and Y is the slave's ring position, or |
1660 ring position, or |
|
1661 |
1651 |
1662 \item[eoeXaY] for a slave with a non-zero alias address, where X is the master |
1652 \item[eoeXaY] for a slave with a non-zero alias address, where X is the master |
1663 index and Y is the decimal alias address. |
1653 index and Y is the decimal alias address. |
1664 |
1654 |
1665 \end{description} |
1655 \end{description} |
1701 up, the passing of new socket buffers is suspended with a call to |
1691 up, the passing of new socket buffers is suspended with a call to |
1702 \lstinline+netif_stop_queue()+. |
1692 \lstinline+netif_stop_queue()+. |
1703 |
1693 |
1704 \paragraph{Creation of EoE Handlers} |
1694 \paragraph{Creation of EoE Handlers} |
1705 |
1695 |
1706 During bus scanning (see section~\ref{sec:fsm-scan}), the master determines |
1696 During bus scanning (see sec.~\ref{sec:fsm-scan}), the master determines the |
1707 the supported mailbox protocols foe each slave. This is done by examining the |
1697 supported mailbox protocols foe each slave. This is done by examining the |
1708 ``Supported Mailbox Protocols'' mask field at word address 0x001C of the |
1698 ``Supported Mailbox Protocols'' mask field at word address 0x001C of the |
1709 SII\index{SII}. If bit 1 is set, the slave supports the EoE protocol. In this |
1699 SII\index{SII}. If bit 1 is set, the slave supports the EoE protocol. In this |
1710 case, an EoE handler is created for that slave. |
1700 case, an EoE handler is created for that slave. |
1711 |
1701 |
1712 \paragraph{EoE State Machine} |
1702 \paragraph{EoE State Machine} |
1774 \paragraph{EoE Processing} |
1764 \paragraph{EoE Processing} |
1775 |
1765 |
1776 To execute the EoE state machine of every active EoE handler, there must be a |
1766 To execute the EoE state machine of every active EoE handler, there must be a |
1777 cyclic process. The easiest solution would be to execute the EoE state |
1767 cyclic process. The easiest solution would be to execute the EoE state |
1778 machines synchronously with the master state machine (see |
1768 machines synchronously with the master state machine (see |
1779 section~\ref{sec:fsm-master}). This approach has the following disadvantage: |
1769 sec.~\ref{sec:fsm-master}). This approach has the following disadvantage: |
1780 |
1770 |
1781 Only one EoE fragment could be sent or received every few cycles. This |
1771 Only one EoE fragment could be sent or received every few cycles. This |
1782 causes the data rate to be very low, because the EoE state machines are not |
1772 causes the data rate to be very low, because the EoE state machines are not |
1783 executed in the time between the application cycles. Moreover, the data rate |
1773 executed in the time between the application cycles. Moreover, the data rate |
1784 would be dependent on the period of the application task. |
1774 would be dependent on the period of the application task. |
1785 |
1775 |
1786 To overcome this problem, an own cyclic process is needed to asynchronously |
1776 To overcome this problem, an own cyclic process is needed to asynchronously |
1787 execute the EoE state machines. For that, the master owns a kernel timer, that |
1777 execute the EoE state machines. For that, the master owns a kernel timer, that |
1788 is executed each timer interrupt. This guarantees a constant bandwidth, but |
1778 is executed each timer interrupt. This guarantees a constant bandwidth, but |
1789 poses the new problem of concurrent access to the master. The locking |
1779 poses the new problem of concurrent access to the master. The locking |
1790 mechanisms needed for this are introduced in section~\ref{sec:concurr}. |
1780 mechanisms needed for this are introduced in sec.~\ref{sec:concurr}. |
1791 |
1781 |
1792 \paragraph{Automatic Configuration} |
1782 \paragraph{Automatic Configuration} |
1793 |
1783 |
1794 By default, slaves are left in PREOP state, if no configuration is applied. If |
1784 By default, slaves are left in PREOP state, if no configuration is applied. If |
1795 an EoE interface link is set to ``up'', the requested slave's |
1785 an EoE interface link is set to ``up'', the requested slave's |
1814 |
1804 |
1815 \ldots |
1805 \ldots |
1816 |
1806 |
1817 \paragraph{Sdo Download State Machine} |
1807 \paragraph{Sdo Download State Machine} |
1818 |
1808 |
1819 The best time to apply Sdo configurations is during the slave's PREOP |
1809 The best time to apply Sdo configurations is during the slave's PREOP state, |
1820 state, because mailbox communication is already possible and slave's |
1810 because mailbox communication is already possible and slave's application will |
1821 application will start with updating input data in the succeeding |
1811 start with updating input data in the succeeding SAFEOP state. Therefore the |
1822 SAFEOP state. Therefore the Sdo configuration has to be part of the |
1812 Sdo configuration has to be part of the slave configuration state machine (see |
1823 slave configuration state machine (see section~\ref{sec:fsm-conf}): It |
1813 sec.~\ref{sec:fsm-conf}): It is implemented via an Sdo download state machine, |
1824 is implemented via an Sdo download state machine, that is executed |
1814 that is executed just before entering the slave's SAFEOP state. In this way, |
1825 just before entering the slave's SAFEOP state. In this way, it is |
1815 it is guaranteed that the Sdo configurations are applied each time, the slave |
1826 guaranteed that the Sdo configurations are applied each time, the |
1816 is reconfigured. |
1827 slave is reconfigured. |
|
1828 |
1817 |
1829 The transition diagram of the Sdo Download state machine can be seen |
1818 The transition diagram of the Sdo Download state machine can be seen |
1830 in figure~\ref{fig:fsm-coedown}. |
1819 in figure~\ref{fig:fsm-coedown}. |
1831 |
1820 |
1832 \begin{figure}[htbp] |
1821 \begin{figure}[htbp] |
1885 the master from userspace and allow a finer influence. It should be possible |
1874 the master from userspace and allow a finer influence. It should be possible |
1886 to view and to change special parameters at runtime. |
1875 to view and to change special parameters at runtime. |
1887 |
1876 |
1888 Bus visualization is another point: For development and debugging purposes it |
1877 Bus visualization is another point: For development and debugging purposes it |
1889 is necessary to show the connected slaves with a single command, for instance |
1878 is necessary to show the connected slaves with a single command, for instance |
1890 (see sec.~\ref{sec:ethercat}). |
1879 (see sec.~\ref{sec:tool}). |
1891 |
1880 |
1892 Another aspect is automatic startup and configuration. The master must be able |
1881 Another aspect is automatic startup and configuration. The master must be able |
1893 to automatically start up with a persistent configuration (see |
1882 to automatically start up with a persistent configuration (see |
1894 sec.~\ref{sec:system}). |
1883 sec.~\ref{sec:system}). |
1895 |
1884 |
1914 Each master instance will get a character device as a userspace interface. |
1903 Each master instance will get a character device as a userspace interface. |
1915 The devices are named \textit{/dev/EtherCATx}, where $x \in \{0 \ldots n\}$ is |
1904 The devices are named \textit{/dev/EtherCATx}, where $x \in \{0 \ldots n\}$ is |
1916 the index of the master. |
1905 the index of the master. |
1917 |
1906 |
1918 \paragraph{Device Node Creation} The character device nodes are automatically |
1907 \paragraph{Device Node Creation} The character device nodes are automatically |
1919 created, if the \lstinline+udev+ Package is installed. See section |
1908 created, if the \lstinline+udev+ Package is installed. See |
1920 \ref{sec:autonode} for how to install and configure it. |
1909 sec.~\ref{sec:autonode} for how to install and configure it. |
1921 |
1910 |
1922 %------------------------------------------------------------------------------ |
1911 %------------------------------------------------------------------------------ |
1923 |
1912 |
1924 \subsection{Setting Alias Addresses} |
1913 \subsection{Setting Alias Addresses} |
1925 \label{sec:alias} % FIXME |
1914 \label{sec:alias} % FIXME |
2074 |
2063 |
2075 The EtherCAT master init script conforms to the requirements of the ``Linux |
2064 The EtherCAT master init script conforms to the requirements of the ``Linux |
2076 Standard Base'' (LSB\index{LSB}, \cite{lsb}). The script is installed to |
2065 Standard Base'' (LSB\index{LSB}, \cite{lsb}). The script is installed to |
2077 \textit{etc/init.d/ethercat} below the installation prefix and has to be |
2066 \textit{etc/init.d/ethercat} below the installation prefix and has to be |
2078 copied (or better: linked) to the appropriate location (see |
2067 copied (or better: linked) to the appropriate location (see |
2079 section~\ref{sec:installation}), before the master can be inserted as a |
2068 sec.~\ref{sec:installation}), before the master can be inserted as a service. |
2080 service. Please note, that the init script depends on the sysconfig file |
2069 Please note, that the init script depends on the sysconfig file described |
2081 described below. |
2070 below. |
2082 |
2071 |
2083 To provide service dependencies (i.~e. which services have to be started before |
2072 To provide service dependencies (i.~e. which services have to be started before |
2084 others) inside the init script code, LSB defines a special comment block. |
2073 others) inside the init script code, LSB defines a special comment block. |
2085 System tools can extract this information to insert the EtherCAT init script at |
2074 System tools can extract this information to insert the EtherCAT init script at |
2086 the correct place in the startup sequence: |
2075 the correct place in the startup sequence: |
2405 This command will install the compiled kernel modules to |
2394 This command will install the compiled kernel modules to |
2406 \textit{/vol/nfs/root/lib/modules}, prepended by the kernel release. |
2395 \textit{/vol/nfs/root/lib/modules}, prepended by the kernel release. |
2407 |
2396 |
2408 If the EtherCAT master shall be run as a service\footnote{Even if the EtherCAT |
2397 If the EtherCAT master shall be run as a service\footnote{Even if the EtherCAT |
2409 master shall not be loaded on system startup, the use of the init script is |
2398 master shall not be loaded on system startup, the use of the init script is |
2410 recommended for manual (un-)loading.} (see section~\ref{sec:system}), the init |
2399 recommended for manual (un-)loading.} (see sec.~\ref{sec:system}), the init |
2411 script and the sysconfig file have to be copied (or linked) to the appropriate |
2400 script and the sysconfig file have to be copied (or linked) to the appropriate |
2412 locations. The below example is suitable for SUSE Linux. It may vary for other |
2401 locations. The below example is suitable for SUSE Linux. It may vary for other |
2413 distributions. |
2402 distributions. |
2414 |
2403 |
2415 % FIXME relative ln -s? |
2404 % FIXME relative ln -s? |
2419 # `\textbf{ln -s etc/init.d/ethercat /etc/init.d/}` |
2408 # `\textbf{ln -s etc/init.d/ethercat /etc/init.d/}` |
2420 # `\textbf{insserv ethercat}` |
2409 # `\textbf{insserv ethercat}` |
2421 \end{lstlisting} |
2410 \end{lstlisting} |
2422 |
2411 |
2423 Now the sysconfig file \texttt{/etc/sysconfig/ethercat} (see |
2412 Now the sysconfig file \texttt{/etc/sysconfig/ethercat} (see |
2424 section~\ref{sec:sysconfig}) has to be customized. The minimal customization |
2413 sec.~\ref{sec:sysconfig}) has to be customized. The minimal customization is |
2425 is to set the \lstinline+MASTER0_DEVICE+ variable to the MAC address of the |
2414 to set the \lstinline+MASTER0_DEVICE+ variable to the MAC address of the |
2426 Ethernet device to use (or \lstinline+ff:ff:ff:ff:ff:ff+ to use the first |
2415 Ethernet device to use (or \lstinline+ff:ff:ff:ff:ff:ff+ to use the first |
2427 device offered) and selecting the driver(s) to load via the |
2416 device offered) and selecting the driver(s) to load via the |
2428 \lstinline+DEVICE_MODULES+ variable. |
2417 \lstinline+DEVICE_MODULES+ variable. |
2429 |
2418 |
2430 After the basic configuration is done, the master can be started with |
2419 After the basic configuration is done, the master can be started with |
2475 \end{description} |
2464 \end{description} |
2476 |
2465 |
2477 \section{Automatic Device Node Creation} |
2466 \section{Automatic Device Node Creation} |
2478 \label{sec:autonode} |
2467 \label{sec:autonode} |
2479 |
2468 |
2480 The \lstinline+ethercat+ command-line tool (see section~\ref{sec:ethercat}) |
2469 The \lstinline+ethercat+ command-line tool (see sec.~\ref{sec:tool}) |
2481 communicates with the master via a character device. The corresponding device |
2470 communicates with the master via a character device. The corresponding device |
2482 nodes are created automatically, if the udev daemon is running. |
2471 nodes are created automatically, if the udev daemon is running. Note, that on |
2483 Note, that on some distributions, the \lstinline+udev+ package is not |
2472 some distributions, the \lstinline+udev+ package is not installed by default. |
2484 installed by default. |
|
2485 |
2473 |
2486 The device nodes will be created with mode \lstinline+0660+ and group |
2474 The device nodes will be created with mode \lstinline+0660+ and group |
2487 \lstinline+root+ by default. If you want to give normal users reading access, |
2475 \lstinline+root+ by default. If you want to give normal users reading access, |
2488 create a udev rule file (for example |
2476 create a udev rule file (for example |
2489 \textit{/etc/udev/rules.d/99-EtherCAT.rules} with the following content: |
2477 \textit{/etc/udev/rules.d/99-EtherCAT.rules} with the following content: |
2499 \begin{lstlisting} |
2487 \begin{lstlisting} |
2500 # `\textbf{ls -l /dev/EtherCAT0}` |
2488 # `\textbf{ls -l /dev/EtherCAT0}` |
2501 crw-rw-r-- 1 root root 252, 0 2008-09-03 16:19 /dev/EtherCAT0 |
2489 crw-rw-r-- 1 root root 252, 0 2008-09-03 16:19 /dev/EtherCAT0 |
2502 \end{lstlisting} |
2490 \end{lstlisting} |
2503 |
2491 |
2504 Now, the \lstinline+ethercat+ tool can be used (see |
2492 Now, the \lstinline+ethercat+ tool can be used (see sec.~\ref{sec:tool}) even |
2505 section~\ref{sec:ethercat}) even as a non-root user. |
2493 as a non-root user. |
2506 |
2494 |
2507 %------------------------------------------------------------------------------ |
2495 %------------------------------------------------------------------------------ |
2508 |
2496 |
2509 \begin{thebibliography}{99} |
2497 \begin{thebibliography}{99} |
2510 |
2498 |