GetPouVariables optimized with XSLTModelQuery

Added /plcopen/InstancesPathCollector.py, to move more model xslt querying code away from PLCController.py.

Fixed Exception dialog disapearing after a view second when exception occurs during app startup.
Problem was caused by splashscreen timeout, wx closing averything else if there is no more main frame.
Changes:
- no more timeout for splashscreen
- use wx.App OnInit method to give first operation to mainloop, object are then now created in mainloop
- main loop is then created _before_ showing splash screen
- no more wxyield or wx processevent tricks needed to display splash screen
- exception handler not blocking anymore on dialog (callafter)
- because of mainloop being there before everything, exit must be called explicitely if exception caught during startup -> exit parameter in handle_exception + try/except around startup calls

UNTESTED ON WINDOWS

Moved XSLTModelQuery class to own module in plcopen package.

Added a class to speedup all XSLT queries. WIP, now only for instances_path.xslt / GetInstanceList()

Stripped useless node creation in instances_path stylesheet.

This should really help GetInstanceList to perform better: resolver removed, no more deepcopy, no more useless xml serialization, no more node-set() calls.

YSLT auto indent trick is useless here, added yslt_noindent.yml2, regenerated XSLT files

Prevent compiling XSLT on each call of GetInstanceList. To be continued. More optimization needed here. 100x would be fine...

Changed meaning of -c commandline argument for Beremiz service : only gives DEFAULT wamp config.
If existing project already has a wamp config, it is taken instead.

Factorized duplicated code

Better variable naming, and small optimization.

Unclutter graphics package scope.
Moved some import statements from "graphics" package to "viewer" module.
It also avoid systematic import of everything in graphics when we only need one submodule.

Removed dead code

Fixed pyro timeout for runtime, was set too short and runtime was disconnecting when building PLC

Ask to stop PLC before transfer if it is running

Factorized some repeated refresh code into a method, to allow customisation.

Launcher : add a debug attribute, instead of BMZ_DBG, so that customization can use it.

Stripped old dead code.

More changes in Launcher. Modules to be loaded in the background are given in a list, and assigned as object attributes. Also splitted CreateUI into CloseSplashScreen and CreatueUI.

ClassImporter now accepts string, callable or class, for compatibility with some old extensions.

Launcher: ShowUI call moves from preStart to start. ShowUI() split into CreateUI() and ShowUI(). Start()'s arguments are passed to CreateUI()
This is meant to allow customizations to pass own pre-instanciated controller to UI at startup.

Launcher: allows customizations to specify what globals passed to extensions

Documentation

Another set of meaningless changes to satisfy PEP8 and PyLint.

PEP8 conformity for modbus. Most of it done by autopep8, plus some easy refactoring.

Add some class factory function to ease declaration of simple POU libraries in customizations (i.e. TC6 XML file only, no special C code generation)

Small pep8 conformity enforcement.

Better have globals passed to python extensions (-e) than locals.

launcher : move call to ShowSplashScreen and BackgroundInitialization to PreStart()
It doesn't change behaviour of launcher in beremiz, but may have some impact on some customizations.
Some customizations need to squeeze some bitmap folder init in between app creation and ShowSplashScreen

modbus extension code cleanup, from smarteh's patch

Merged Mario's modbus branch. Fixed line endings.

merge

Ignore bug report files.

Add support for Modbus (TCP and RTU) working as master & slave

merge

fixed erro introduced 2 commits ago (-e) : tuples have no reverse() method

Trying to get better logging/display of exceptions at runtime's startup

Changed way to deal with extensions arguments (-e) so that path and filename are stored independently and accessible for extensions themselves.

Textviewer : moved call to Colourise in a better place, so that ST highlighting gets also updated when typing in some case, not only IL

Main runtime template C code : Added HAVE_RETAIN preprocessor definition for customized build to signal it provides IEC-61131 Retain memory handling function. Removed targets/Xenomai/plc_Xenomai_noretain.c, now useless.

Clarify licensing, and packaging of runtime only files :
- moved and split targets/typemapping.py into runtime/typemapping.py and runtime/loglevels.py
- added runtime_files.list pointing to files necessary for runtime, and referring to runtime license

Enforced check_source.sh pep8 and pylint rules.

runtime/WAMP: Fixed secret loading into configuration. Fixed secret commandline switch handling.

runtime/WAMP : was 'Added time print on session left.' from dporopat. Purpose is to log (dis/re)connection, to help debugging connection problem

runtime/Wamp : adapt to unicode being enforced in latest crossbar API. Updated some documentation, to explain why there is unused definitions like SubscribedEvents or DoOnJoin.

Extend python<->PLC global variable description "GlobalDesc" to support Initial value setting.

Rebased from dporopat change "add initial value to runtime.py file", which means :
- GlobalDesc is used by some customized beremiz python runtime
- Those runtimes need to know what is initial value of these special variables

update tests/wamp README file, based on changes from dporopat.

Fixed WAMP test (crossbar confile upgrade and use same ID on both PLC and IDE side)

Fixed changes from dporopat and agregorcic about WAMP CRA and config.

WAMP runtime : changes from dporopat and agregorcic, adding CRA (-s) and changed way to load WAMP conf (-c)
- added Challenge Response Authentication,
- use -s commandline switch to give the shared secret.
- WAMP configuration now either given as -c commandline switch
or taken as default wampconf.json in project files.

Set WAMP registration option so that reconnecting while earlier session still didn't timeout is possible (in case of connection lost, for example).

More canonical overload of ReconnectingClientFactory (according twisted doc)

non significant print syntax change. ready for py3.

whitespace cosmetic for pep8

Added arbitrary timeout on Pyro, to force it to free memory after disconnect.

Moved code from to Beremiz_service.py Server class Start method to Loop method to allow customisation by hot patching. Also renamed Stop method as _stop and use Restart instead of Stop from everywhere else to make it clear that _stop actually does restart...

enforce PEP8 in previous commit

replaced normal panel with scrolled panel under project properties

fix missing root in empty search results panel

add possibility to specify subprocess stdout encoding

fix deadlock if method kill() is called after process is already finished

fix pylint py3k conversion warning: "(no-absolute-import) import missing `from __future__ import absolute_import`"

make CanFestival to be at the expected place in bitbucket pipelines container

fix pylint error '(unsubscriptable-object) Value 'X' is unsubscriptable'

fix pylint warning '(bad-continuation) Wrong hanging indentation before block'

force all python source code to have LF line ending

fix pylint warning '(expression-not-assigned) Expression "X" is assigned to nothing'

fix pylint warning '(redefine-in-handler) Redefining name 'X' from outer scope (line Y) in exception handler'

fix displaying result of update check during Beremiz startup

fix pylint warning '(no-init) Class has no __init__ method'

fix pylint error "(undefined-variable) Undefined variable 'X'"

fix problem happend when tests are running under python-coverage

$ python-coverage run ./tests/tools/test_application.py

Exception while connecting LOCAL://!
Traceback (most recent call last):
File "/home/developer/WorkData/PLC/beremiz/beremiz/ProjectController.py", line 1753, in _Connect
self._SetConnector(connectors.ConnectorFactory(uri, self)
File "/home/developer/WorkData/PLC/beremiz/beremiz/ProjectController.py", line 1702, in _SetConnector
if connector is not None:
PyAssertionError: C++ assertion "m_window" failed at ../src/gtk/dcclient.cpp(2043) in DoGetSize(): GetSize() doesn't work without window

fix pylint error '(too-many-function-args) Too many positional arguments for function call'

fix pylint error '(import-error) Unable to import X'

fix pylint error '(no-self-argument) Method should have "self" as first argument'

fix pylint error "(no-name-in-module) No name 'X' in module 'Y'"

fix pylint error '(bad-except-order) Bad except clauses order (X is an ancestor class of Y)'

fix pylint warning "(no-value-for-parameter) No value for argument 'X' in function call"

fix problems with unconnected input of InOut function variables and
enfoce InOut variables to be connected for functions and function blocks

fix error for functions with VAR_IN_OUT

The problem was found by pylint '(used-before-assignment) Using variable "X" before assignment'

fix pylint warning '(unidiomatic-typecheck) Using type() instead of isinstance() for a typecheck'

fix pylint warning '(misplaced-bare-raise) The raise statement is not inside an except clause'

fix format string argument error found by pylint '(too-many-format-args) Too many arguments for format string'

fix pylint warning '(wrong-import-position) Import "import X" should be placed at the top of the module'

fix pylint warning '(import-self) Module import itself'

remove old code useless code found by pylint "(unnecessary-pass) Unnecessary pass statement"

fix error found by pylint 'W0104(pointless-statement) Statement seems to have no effect'

fix pylint warning "(undefined-loop-variable) Using possibly undefined loop variable 'X'"

fix pylint warning 'C0112(empty-docstring) Empty function docstring'

fix pylint warning "(relative-import) Relative import 'Y', should be 'X.Y'"

fix pylint warning '(dangerous-default-value) Dangerous default value {} as argument'

fix pylint warning '(old-style-class) Old-style class defined.'

This warning is ignored for classes from PyJSLib, because library
overrides base object class.

fix pylint warning '(unused-import), Unused import connectors'

fix typos

comment out unused code SFC_Step.RefreshBoundingBox()

explicitly mark unused variables found by pylint with _ or dummy

pylint don't ignores unused variables with '_' in the name, even when
they are started with '_'. This bug is already fixed upstream, to fix
this in all versions of pylint custom dummy-variables-rgx is used
'_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy'

remove unused variables found by pylint

update .PHONE target in Makefile for mki18n.py

make mki18n return 0 on success

update list with files for internatialization, app.fil

move README content to Makefile

It's not necessary to copy commands from README anymore. Just use
'make source_list', 'make template' or 'make locales'.

fix pylint warning '(global-variable-not-assigned) Using global for 'X' but no assignment is done'

fix pylint error '(function-redefined) method already defined'

fix pylint warning '(unreachable) Unreachable code'

fix pylint warning (bad-indentation) Bad indentation. Found X spaces, expected Y

fix pylint warning '(pointless-string-statement) String statement has no effect'

Python doesn't support docstring for variables directly. Where
appropriate Sphinx style for docstrings for variables is used
'#: some-docstring'

fix error __init__ method from base class is not called

fix more than one statement on a single line

fix ungrouped imports from package X

fix unnecessary lambda and enable corresponding pylint check

fix wrong-import-order. first standard modules are imported, then others

fix (old-style-class) Old-style class defined error for most parts of
the code

only PyJS code is left, because it does some fancy tricks and can't be
easily fixed. So far PyJS doesn't support Python3 anyway.

remove in-tree version of Zeroconf and use upstream from
python-zeroconf package

remove reimported modules

fix unnecessary parens after keyword

enable pylint checker

so far only print statement is checked

fix usage of python2-only print statement

run python tests with non-english locale to catch bugs related to localization

make it possible to update XSD schema in project controller after new
target PLC is added or removed.

For example:


...
import targets
targets.targets["PLC"] = {"xsd": os.path.join(self.customBeremizPath, "PLCtarget", xsd),
"class": lambda: PLC_target,
"code": {"plc_PLC_main.c": os.path.join(self.customBeremizPath, "PLCtarget", "plc_PLC_main.c")}
}

from ProjectController import ProjectController, GetProjectControllerXSD
ProjectController.XSD = GetProjectControllerXSD()

fix commit "lazy initialization of highlight pens and brushes
for DebugVariableViewer" (5562f34f)

clean build example project after running test

fix problem if DataTypeEditor was imported before localization is
loaded

That doesn't affect systems with english locales.

add shell script to run all python tests and automatically handle Xvfb

add PLCOpenEditor tests

run application tests in full screen mode

clean sys.argv, because if test is running from pytest
sys.argv contains string "pytest --timeout 0 ./tests/tools"
Beremiz application parses it and throws exception.

rework Beremiz application "testCheckProject" test to allow easier reuse later

add application name for PLCOpenEditor

This is needed to work user configuration load/store, if PLCOpenEditor
is launched from test environment.

rewrite PLCOpenEditor. Add PLCOpenEditorApp class for easier customizations.

lazy locale initialization

wx.Locale() can't be called without wx.App. So don't call it at
import. It makes PLCOpenEditor simpler.

use CustomIntCtrl instead of wx.SpinCtrl because it doesn't support
long integer needed for UDINT, (U)LINT types.

This commit complements commit 74a3f4d "fix LINT and ULINT ranges".

add custom TextCtrl allowed to enter long integer with bounds checking

It's needed because wx.SpinCtrl supports only 'int' type. For UDINT
and (U)LINT long type is necessary.

use pytest for testing

run separate test on every project instead of one big long test all projects

move configuration for test environment into separate file conftest.py

lazy initialization of highlight pens and brushes for DebugVariableViewer

Constructors wx.Pen() and wx.Brush() require wx.App to exist already.
That causes crash during import of the control,
because import is done before calling main application code.

add '_' function to builtin dictionary at import time

This allows to import many Beremiz python modules before calling
util.misc.InstallLocalRessources().

remove builtin BMZ_DBG. It's not used.

update translation strings

updated all existing translations
added new languages: Portuguese (Portugal), Italian, Hungarian,
Spanish and Bengali. See version.py for information about translators.

fix LINT and ULINT ranges

found installed Inkscape on Windows

tested with Inkscape version 0.9 on Windows 7.

make possible to turn off at compile time online debugging, logging
and external sync functionality

Defines TARGET_DEBUG_DISABLE, TARGET_LOGGING_DISABLE, TARGET_EXT_SYNC_DISABLE
allow to use generated PLC code in low-cost microcontrollers,
integrate into external programs and so on.

main.c:
extern void __run(void);
int main(void)
{
for(;;) {
__run();
// sleep common_ticktime__ ns
// add common_ticktime__ ns to __CURRENT_TIME
}
return 0;
}

Compile for example for arm bare-metal:
../build$ arm-none-eabi-gcc \
-DTARGET_DEBUG_DISABLE -DTARGET_LOGGER_DISABLE
-DTARGET_EXT_SYNC_DISABLE \
-flto -ffunction-sections -fdata-sections -I../../../../matiec/lib/C \
*.c \
-nodefaultlibs --specs=nano.specs -Wl,--static -Wl,--gc-section -Wl,--start-group -lc -lm -lnosys -lgcc -Wl,--end-group

make default caption for PouNameDialog translatable

fix max value for UDINT and ULINT types

update Build icon

add couple Beremiz application tests

These tests trigger around 45% of python code.
Used in tests docker files are avaiable here:
https://bitbucket.org/skvorl/beremiz-dockerfiles

fix problem if _should_write is called by expired timer after
termination of wx application.

Initial Bitbucket Pipelines configuration

fix bug with unfinished debug thread
this bug occurs when user doesn't stop debugging process in current project, and opens another project

unify exception handling of Beremiz and PLCOpenEditor

fix problem running PLCOpenEditor

$ ./PLCOpenEditor.py
../src/common/stdpbase.cpp(62): assert "traits" failed in Get(): create wxApp before calling this

Regression was introduced by '3311eea' (clean-up: fix PEP8 E402 module level import not at top of file)

don't check any particular wx version

Beremiz works with any version of wx.

fix problem when Beremiz is running without matiec installed

set matiec settings only once on first use of ProjectController class and give
user-friendly message on compilation about missing matiec
installation.


Traceback (most recent call last):
File "Beremiz.py", line 197, in <module>
beremiz.Start()
File "Beremiz.py", line 192, in Start
self.CreateApplication()
File "Beremiz.py", line 130, in CreateApplication
self.BackgroundInitialization()
File "Beremiz.py", line 137, in BackgroundInitialization
self.ImportModules()
File "Beremiz.py", line 176, in ImportModules
import BeremizIDE
File
"BeremizIDE.py", line 76, in <module>
from ProjectController import ProjectController, GetAddMenuItems,
MATIEC_ERROR_MODEL, ITEM_CONFNODE
File
"ProjectController.py", line 177, in <module>
iec2c_cfg = Iec2CSettings()
File
"ProjectController.py", line 101, in __init__
self.ieclib_c_path = self.findLibCPath()
File
"ProjectController.py", line 135, in findLibCPath
os.path.join(self.ieclib_path, "C"),
File "/usr/lib/python2.7/posixpath.py", line 70, in join
elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'

Initial Bitbucket Pipelines configuration

add configuration for Bitbucket pipelines

add shell script to test Beremiz python code

currently only codestyle checking is done using pycodestyle (ex-pep8).
Later checks by flake8 and pylint maybe enabled.

clean-up: fix PEP8 W503 line break before binary operator

fix codestyle issues according to PEP8

clean-up: fix PEP8 E402 module level import not at top of file

clean-up: fix PEP8 E265 block comment should start with '# '

clean-up: fix PEP8 E266 too many leading '#' for block comment

clean-up: fix PEP8 E722 do not use bare except'

better it'd be to use particular exception instead of 'except Exception'.

clean-up: fix PEP8 E115 expected an indented block (comment)

clean-up: fix PEP8 E721 do not compare types, use 'isinstance()'

clean-up: fix PEP8 E131 continuation line unaligned for hanging indent

clean-up: fix PEP8 E122 continuation line missing indentation or outdented

clean-up: fix PEP8 E713 test for membership should be 'not in'

clean-up: fix PEP8 E272 multiple spaces before keyword

clean-up: fix PEP8 E124 closing bracket does not match visual indentation

clean-up: fix PEP8 W191 indentation contains tabs

clean-up: fix PEP8 E211 whitespace before '[' or '('

clean-up: fix PEP8 E114 indentation is not a multiple of four (comment)

clean-up: fix PEP8 E125 continuation line with same indent as next logical line

clean-up: fix PEP8 E128 continuation line under-indented for visual indent

clean-up: fix PEP8 E127 continuation line over-indented for visual indent

clean-up: fix PEP8 E129 visually indented line with same indent as next logical line

clean-up: fix PEP8 W602 deprecated form of raising exception

clean-up: fix PEP8 E502 the backslash is redundant between brackets

clean-up: fix PEP8 W601 .has_key() is deprecated, use 'in'

clean-up: fix PEP8 E731 do not assign a lambda expression, use a def

clean-up: fix PEP8 E203 whitespace before ','

clean-up: fix PEP8 E262 inline comment should start with '# '

clean-up: fix PEP8 E712 comparison to True should be 'if cond is True:' or 'if cond:'

clean-up: fix PEP8 E222 multiple spaces after operator

clean-up: fix PEP8 E111 indentation is not a multiple of four

clean-up: fix PEP8 E701 multiple statements on one line (colon)

clean-up: fix PEP8 E741 ambiguous variable name

clean-up: fix most PEP8 E221 multiple spaces before operator

clean-up: fix most PEP8 E266 too many leading '#' for block comment

clean-up: fix PEP8 E703 statement ends with a semicolon

clean-up: fix PEP8 E301 expected 1 blank line, found 0

clean-up: fix PEP8 E306 expected 1 blank line before a nested definition, found X

clean-up: fix PEP8 E305 expected 2 blank lines after class or function definition

clean-up: fix PEP8 W391 blank line at end of file

clean-up: fix PEP8 E201 whitespace after '{'

clean-up: fix PEP8 E202 whitespace before ')'

clean-up: fix PEP8 E227 missing whitespace around bitwise or shift operator

clean-up: fix PEP8 E251 unexpected spaces around keyword / parameter equals

clean-up: fix PEP8 E711 comparison to None should be 'if cond is not None:'

clean-up: fix PEP8 E225 missing whitespace around operator

clean-up: fix PEP8 E303 too many blank lines

clean-up: fix PEP8 E231 missing whitespace after ':' or ','

clean-up: fix PEP8 E203 whitespace before ':' and whitespace before ','

clean-up: fix PEP8 E271 multiple spaces after keyword

clean-up: fix PEP8 E261 at least two spaces before inline comment

clean-up: fix PEP8 E302 expected 2 blank lines, found 1

clean-up: fix PEP8 W293 blank line contains whitespace

clean-up: fix some PEP8 E228 missing whitespace around modulo operator

clean-up: fix some PEP8 E265 block comment should start with '# '

clean-up: fix PEP8 E401 multiple imports on one line

convert some files from CRLF to LF

It'll break for these files hg blame/annotate. Fortunately not many
files are affected.

clean-up for PEP8: fix W291 trailing whitespace

README.md edited online with Bitbucket

fix link to documentation in README.md

Add information about built official documentation to README.md

don't allow empty dimension for ARRAY types

ignore built documentation

manual: add untranslated po files for French and Korean

build missing pot files, when building HTML documentation

fix some typos

Changed title of new project dialog from:
Choose a project
to
Choose an empty directory for new project

Changed title of New Project dialog
from
Chose a project
to
Choose an empty directory for new project

skip hidden files and directories on permission check

To fix problems with git, because .git subdirectory contains
read-only repository objects.

change structure of filename for bug reports in Beremiz and
PLCOpenEditor so bug reports could be easily sorted by name

fix target Xenomai missing functions

update messages.pot and messages.po

mki18n.py: generate messages.po (contains English strings in msgstr)

This po file is necessary because Transifex translation platform
automatically can only import PO files. Usually messages.pot is
enough.

add to i18n/README forgotten commands to add some XML schemes as source
for strings and update app.fil

Backout commit 92537ed because of writing localized string in node *.xml file

Backout changeset 92537edeb2052d53ba5cad8b6da8463dbc660bb1

fixes double mouse capturing after forcing variable in debug mode
this problem appears on Linux

fix bug with TextViewer instance in debug mode, appears after transferring new program on PLC
TextViewer isn't able to display debug values, so calling for this class SubscribeAllDataConsumers method causes error

fix build problem when make command have trailing whitespace(s)

whitespaces at the end are replaced with "", that causes make problem

add dialog "add program", that appears after creating new project
also, sets one task and one instance of created program in main project resource

fix i18n string selection for combobox generated from XSD

update Russian and French (Fabien M) translations
added Slovenian (Janez Pregelj) and Portuguese (Thiago Alves) translations

l18n: update messages.pot and app.fil

add l18n support to variable table in CodeFileEditor (python/C extension)