OpenERP Automated Tests & Reports

The new automated test framework for OpenERP is finished in trunk. It consists of 1604 tests spread in 146 complete scenari. Every commit run a series of tests, according to the modules changed in the commit. These tests are validating the business logic, but also the reports of OpenERP.

The automated test server is built on buildbot. Here is an example of automated test that failed: http://test.openobject.com/builders/buildbot-openobject-addons/builds/38

If you want to see all the available reports in OpenERP, just launch the following queries on a trunk branch:
> createdb trunk --encoding=unicode
> openerp-server.py -i sale,purchase,mrp,project -d trunk --test-report-directory=/tmp --log-level=test

OpenERP will launch automatically all tests scenari and save the generated reports in the /tmp directory.

We are also working on an improved "bzr stats" to get information ala Google Analytics but for the commits on a bazaar branch. The very first prototype has been released here:
https://code.launchpad.net/~openerp-community/bzr-stats/bzr-stats-analytics

This will allow us to get statistics on contributions and developers, related to the automated tests quality report in order to better do continuous improvement in our development process.

Improved Automated Tests for OpenERP: reporting

Here are some news about our work in progress to improve automated tests an code review in OpenERP trunk. Let me first introduce how is our R&D organized.

At OpenERP SA, we use the SCRUM project management methodology to organize our developments on OpenERP. We use the OpenERP scrum module to organize the product backlogs. We have developers spread into 8 teams of about 6 developers per team:

• Framework: scrum master = Olivier
  
• Addons 1 (sale, stock, mrp, purchase): scrum master = Quentin
• Addons 2 (crm, dms, survey, misc): scrum master = Quentin
  
• Addons 3 (account, project, hr, events): scrum master = Quentin
  
• Usability: scrum master = Aline
  
• Support / Bugfixes: scrum master = Stéphane
  
• Web Client: scrum master = Xavier
  
• GTK Client: scrum master = Naresh
  

Each team work in sprints of 2 weeks. Each team works in his own launchpad branch during the sprint (example: lp:~openerp-dev/openobject-addons/trunk-dev-addons1). At the end of the sprint, the team makes a propose for mergal of his work in the OpenERP trunk official branch. In order to to control the quality of the code, every development is reviewed by the team'scrum master during the merge process. (code review)

In order to improve the quality of the tests, we setup a test server that launches a series of tests after each commit made on a team branch or the trunk official branch. This server is still under development, but you can already track the firsts results here: http://test.openobject.com/

The tests performed on each module by the test server are:

• Test scenarii to check all the features and workflows, example: http://test.openobject.com/buildersresource/addons-trunk/builds/147/steps/OpenERP-Test/logs/lunch.test/text
• Quality of the code and respect to OpenERP guidelines, example: http://test.openobject.com/buildersresource/addons-trunk/builds/151
• Modules installation and demo data: http://test.openobject.com/buildersresource/addons-trunk/builds/151
  

In the past three months, our teams developed 1495 YAML tests spread into 112 yaml files and modules. We can now ensure that most common enterprise flows are tested automatically by OpenERP at every commit and installation of the software.

Today we just added automated tests on OpenERP reports. Each time you install OpenERP, it will launch the generation of sample PDF reports. In the trunk version, you have the possibility to store the generated report into a directory using the new argument "--test-report-directory=/tmp/".

If you want to test this, on the latest OpenERP trunk, you can execute the following commands under Linux:

createdb mytestdb --encoding=unicode
./openerp-server.py --init=sale -d mytestdb --test-report-directory=/tmp
xpdf /tmp/sale-sale_order.pdf

You will get a sample PDF file of two sales orders generated by the OpenERP YAML test engine.

If you want to know how it has been implemented in the test scenario, check this file in your local installation: addons/sale/test/sale_order_print.yml

The automated test server is still in beta. We plan to release the final version and start using it massively by the end of september. Our current work in progress for the automated test engine are:

• get statistics on commits, test failures and contributions made by developers
• allows to monitor and test community or partners branches (everyone will be able to subscribe his own branches on the integration server)
  
• send a clear warning email to the developer that introduced a regression
• add more tests on the OpenERP reports (only the sale order has been made for the prototype)
• add sample of PDF reports in the documentation of the module on doc.openerp.com
  

We are also working on a stronger code review process to prepare ourself for the trunk Release Candidate 1. By the end of september, we will force every development to be done in the team branch and reviewed by the scrum master. (including the Bugfix team, which was commiting directly in the trunk currently.)