Development/testing issues

Beaker test framework

Should we start configuring a dedicated testing framework for OpenVPN? This idea will be presented by mattock.

• Rationale

  • Would allow testing that OpenVPN works on a large number of predefined platforms and configurations.
  • Replaces part of the manual testing effort by testing the external behavior of the application automatically.
  • Would allow spotting bugs early.

• Implementation ideas

  • Beaker could perhaps be made to simulate poor connections for testing purposes. If not, there are plenty of tools and services available for this purpose (see full chatlog).

• Limitations

  • Only tests for external behavior of the application.
  • Does not replace human testing for those classes of problems which are difficult/impossible to test programmatically.

Continuous integration server

Should we build a continuous integration (CI) / automated release management server? This idea will be presented by mattock.

• Rationale

  • Would allow spotting build problems early on by building "allmerged" and reporting developers of build problems.
  • Would reduce developer frustration.
  • Would allow automated packaging of OpenVPN-testing for various platforms and publishing those on a web server.
  • The above would translate to increased use of OpenVPN-testing, leading to earlier bug reports and would thus make the release process (new code -> acceptance to testing -> acceptance to stable -> release) faster.
  • Could also allow centralized automated searching of code problems.

• Implementation ideas

  • Developers could be notified of problems using email.
  • The OpenVPN project has a license to Coverity service. As this service has an API, Coverity code analysis could be integrated into the CI server.

• Limitations

  • Does not replace human testing.
  • Only tests the build process (and possibly code problems).

Developer bounties

Should we have a bounty system for writing missing features? Similar systems are in use by several other projects, e.g. freeswitch, pfSense, and Funambol. This idea will be presented by ecrist.

• Rationale

  • Would allow users to prioritize feature additions through compensation.
  • Would help achieve greater interest of developers in users' requests and needs.

• Implementation ideas

  • Bounty funds would be set up and maintained by a non-developer (ecrist?) and held. This assures payment upon completion and assures the feature is completed before bounty is paid.
  • Multiple users could fund a bounty for intensive feature additions which may warrant higher payouts.
  • Developers set up accounts and 'claim' tasks.
  • After a specific timeout, a task becomes available to another user, unless progress can be shown.
  • Payment could be divided into multiple phases to motivate developers to maintain the code they've written. For example:
    • 50% payout for a commit to the main tree.
    • 25% upon release in production (provided bugs are fixed/etc).
    • 25% 6 months after release (provided bugs are fixed/etc).
  • The above would also guarantee the quality of the code.
  • Handling monetary transactions:
    • Could be handled through an external foundation/organization (e.g., SPI).
    • We could establish our own foundation for this purpose.
  • Licensing and copyright:
    • Developer would have to use the BSD license for the bounty features.
    • This would allow the project to relicense the code under GPLv2 (while mentioning the original author).
    • This would allow both developer (payee) and payer to use the code any way they wish.