Test Environment Requirements


This wiki page documents requirements for test environment. The need for such common tool was discussed during August2011Meeting?.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

There is a difference between test environment and test execution environment. These are closely related, but different things. Test environment (or a test tool) is something that allows execution of specific scenario, like sending DNS query and expecting response. Each test tool may be dedicated to specific feature, e.g. DHCPv4 configuration. For complex software validation, there may be many test tools used, often a mix of in-house developed and external. Test execution environment is something that works on top of test tools: deploys tested software version, deploys test tool, runs tests, gathers results, produces summary, statistics etc. Sometimes test tool and test execution environment are tightly integrated together.

Test Tools Requirements

  • MUST be implementation agnostic. For DNS, it must be able to test both BIND9 and BIND10. For DHCP the same is true for ISC DHCP4 and BIND10 DHCP (Kea). Although we don't need it now, it would be useful if we also could test other implementation. See Competitive comparison.
  • MUST produce deterministic and consistent results.
  • MUST produce consistent and readable log
  • MUST be versioned/labelled
  • MUST have test cases documented
  • SHALL have test documentation generator built-in

Test Execution Environment Requirements

  • MUST be able to build source code
  • MUST be able to deploy built binary to execution machines
  • MUST be able to deploy test tools
  • MUST be able to execute tests
  • MUST be able to retrieve result
  • MUST be able to recover from infinite loop. There should be a possibility to specify a timeout for each test.
  • SHOULD be able to recover from crash.
  • MUST be able to execute tests in parallel
  • MUST be scalable
  • MUST not influence test result
Last modified 6 years ago Last modified on Sep 6, 2011, 8:12:25 PM