Changes between Version 15 and Version 16 of OpenVPN_QA

Timestamp:

06/04/14 12:19:24 (10 years ago)

Author:

Samuli Seppänen

Comment:

--

OpenVPN_QA

@@ -3 +3 @@
 = Introduction =
 
-This page outlines the efforts taken to maintain OpenVPN's quality without excessive compromises on development speed.
-
-The next major release (as of Sep 2011), OpenVPN 2.3, will contain significant low-level changes to OpenVPN 2.2.x and 2.1.x. Therefore lots of work is needed on the QA front to avoid regressions. The plan is to setup good QA environment for 2.3a -> 2.3GA development phase. This environment will contain several different parts:
+This page outlines the efforts taken to maintain OpenVPN's code quality without excessive compromises on development speed.
 
 = Static testing =
@@ -11 +9 @@
 == Peer review ==
 
-Static testing usually refers to [http://en.wikipedia.org/wiki/Static_code_analysis static code analysis], which is baked in into our [DeveloperDocumentation development process] in the form of mandatory ACK process every patch has to go through. The ACK process not only improves code quality, it also prevents highly specialized or rarely used features from polluting the codebase.
+Static testing usually refers to [http://en.wikipedia.org/wiki/Static_code_analysis static code analysis], which is baked in into our [DeveloperDocumentation development process] in the form of mandatory ACK process which every patch has to go through. The ACK process not only improves code quality, it also prevents highly specialized or rarely used features from polluting the codebase. Code reviews are especially important for patches coming from non-core contributors who may not be familiar with OpenVPN's coding practices.
 
 == Automated static testing ==
@@ -21 +19 @@
 == Dedicated black-box tests ==
 
-Dynamic black-box testing means trying out an application and verifying if it works as intended. In closed-source software development which is organized around a [http://en.wikipedia.org/wiki/Waterfall_model waterfall model] there are usually dedicated testers who do various scripted or intuitive tests to verify an application works as intended. This usually happens just before launch. In complex applications (such as OpenVPN) testing even a small fraction of functionality would be impractical and very costly. Fortunately, in [http://en.wikipedia.org/wiki/Lean_software_development Lean software development] methodologies such as [http://en.wikipedia.org/wiki/Scrum_%28development%29 Scrum] and especially in community-driven OSS development doing ''extensive'', dedicated testing is in general just a waste of time. It is replaced by
+Dynamic black-box testing means trying out an application and verifying if it works as intended. In closed-source software development which is organized around a [http://en.wikipedia.org/wiki/Waterfall_model waterfall model] there are usually dedicated testers who do various scripted or intuitive tests to verify that an application works as intended. This usually happens just before launch. In complex applications (such as OpenVPN) testing even a small fraction of functionality would be impractical and very costly. Fortunately, in [http://en.wikipedia.org/wiki/Lean_software_development Lean software development] methodologies such as [http://en.wikipedia.org/wiki/Scrum_%28development%29 Scrum] and especially in community-driven OSS development doing ''extensive'' dedicated testing is in general just a waste of time. It is replaced by
 
  * Constant quality assurance achieved with static whitebox technique (e.g. code reviews) 
  * Testing in real environments (by users)
 
-This said, a minimal amount of dedicated, dynamic testing (a.k.a. [http://en.wikipedia.org/wiki/Smoke_testing smoke testing]) goes into each release to catch the most obvious errors.
+This said, a small amount of dedicated, dynamic testing (a.k.a. [http://en.wikipedia.org/wiki/Smoke_testing smoke testing]) goes into each release to catch the most obvious errors:
+
+||'''Operating system'''||'''Architecture'''||'''List of tests'''||'''Tester'''||'''Notes'''||
+||Windows XP installer||i388||mattock||Uninstall/install, OpenVPN-GUI connect/disconnect, service connect/disconnect||||
+||Windows 7 installer||amd64||mattock||Same as for Windows XP, above||||
+||!Ubuntu/Debian||i386/amd64||mattock||Install, upgrade, service connect/disconnect||2-3 of the official packages are tested||
+
+All tests are performed against an OpenVPN 2.1.3 server that authenticates from LDAP and uses tls-auth.
+
+If you want to help with pre-release testing please don't hesitate to [wiki:GettingHelp contact the developers].
 
 == Performance tests ==
 
-To ensure good performance, performance tests are in place. For details, look at the [wiki:PerformanceTesting performance testing wiki page].
+In the past performance tests have been conducted to measure OpenVPN performance:
+
+ * [wiki:PerformanceTesting Performance testing wiki page]
+ * [wiki:OpenVPN_Speed Optimizing OpenVPN throughput]
+
+As no systematic performance tests have been conducted, minor performance regressions might go unnoticed.
 
 == Testing in real environments ==
 
-In OpenVPN (and most other open source projects), the stability of stable releases is ensured with ''real-life testing'' by it's users during all phases of software development, starting from development code in Git and leading into stable releases. There are at least two kinds of ''barriers'' to using pre-release code:
+In OpenVPN (and most other open source projects), the stability of stable releases is ensured with ''real-life testing'' by it's users during all phases of software development, starting from patches sent to the mailing list, followed by development code in Git and leading into stable releases. There are at least two kinds of ''barriers'' to using pre-release code:
 
  * '''Psychological barriers'''
@@ -49 +61 @@
 Use of snapshots help overcome some of the technical barriers. The only way to overcome psychological barriers is to speed up the release cycle. This results in new features get into wide circulation faster, which in turn results into issues being reported more quickly. This also gives more confidence in integrity of stable releases. On the flipside, more bugs will probably end up in the initial versions of the stable releases.
 
-In practice, the goal is to make live testing by users as easy as possible by providing snapshot packages regularly for common *NIX platforms and Windows.
+We do not currently provide development snapshot binaries, although doing that would definitely be beneficial.
 
 == Continuous integration ==
 
-The project also has a [http://trac.buildbot.net/ Buildbot] buildmaster, which drives several buildslaves. These together form a [http://en.wikipedia.org/wiki/Continuous_integration continuous integration] environment for OpenVPN. Each of these buildslaves is running a different operating system, and every commit to the OpenVPN Git repository triggers a build on each. After the build, each buildslave's newly compiled openvpn instance tries to connect to a test server using several different configurations. This ensures that
+The project also has a [http://trac.buildbot.net/ Buildbot] buildmaster, which drives several buildslaves. These together form a [http://en.wikipedia.org/wiki/Continuous_integration continuous integration] environment for OpenVPN. Each of these buildslaves is running a different operating system, and every commit to the OpenVPN Git repository triggers a build on each. After the build, each buildslave's newly compiled openvpn instance tries to connect to several test servers using several different configurations. This ensures that
 
  * OpenVPN builds properly on a variety of platforms
- * Basic functionality is unaffected by commits
+ * The very basic functionality is not horribly broken
 
 == Unit testing ==