wiki:AlphaBetaProgramPlan

BIND 10 Authoritative Server Alpha and Beta Program Plan

Overview : BIND 10 is in its fourth year of development, and has had little real world operational exposure. As we approach the first major production ready deliverable, (authoritative name server due out early 2012) the project seeks community input on features and implementation to date and going forward. To this end both an alpha and a beta test program are proposed.

Purpose of document: This document describes at a high level the objectives of these programs and further test documents which will be provided during the program duration.

Primary:

  • Real user testing of BIND 10 in real user environments
  • Extending knowledge about the BIND 10 project into the user community
  • Determining challenges in BIND 9 to 10 adoption, coexistence, and migration
  • Increasing adoption likelihood among existing customers

Secondary:

  • Introducing potential new customers to ISC support
  • Creating BIND 10 specific collateral and webinars etc
  • Training support and other staff on BIND 10
  • Finding quotable product advocates for release promotion, case studies, etc

Primary BIND 10 DNS Features to be tested

  • IXFR zone transfers in and out
  • AXFR zone transfers in and out
  • in-memory backend for zone data
  • SQLite3 backend for quick startup or huge amounts of zone data
  • DNSSEC: serving signed zones by authoritative server

o either in-memory or from DB backend

o including NSEC3

  • DDNS (Dynamic Updates) support
  • Access Control (for DDNS and outbound zone transfers)
  • TSIG support:

o Part of general ACLs (DDNS uses it) o authoritative server can verify and sign DNS messages with TSIG o Zone transfers can have signed transfers

Specific suggested use cases to be tested

  • Installation
  • Selection of modules
  • Selection of modules at runtime
  • Transfer zones
  • More as per usage patters of initial alpha participants

We may also get developers interested in the program, who might:

  • Build a new module
  • Add support for an additional database

User personas to target for Alpha:

  • Existing skilled BIND 9 users
  • BIND 10 sponsor organizations
  • BIND 9 vendors (OS and Appliance)

User personas to target for Beta:

  • Large ISP
  • Hosting company
  • Registrar/ccTLD
  • Large corporate enterprise network
  • Small business/home/hobbyist
  • Module builder
  • OEM
  • OS vendor
  • DNS developer
  • All interested participants!

Schedule and Milestones:

Planning o Milestone One: Draft Plan for Review: September 24th final plan Sept 27

o Milestone Two: isc.org beta program page and application form page plans and other collateral plans September 27

o Milestone Three: isc.org and BIND 10.isc.org alpha/beta program pages ready to accept applicants September 30

o Milestone Four : Launch Alpha

o Milestone Five: Select supported beta program participants by October 20th

o Milestone Six: Beta Preparation Webinar October 24th

o Milestone Seven: Beta Launch November 17 (beta software release)

  • Release schedule (all dates subject to change based on needs of the project):
  • Alpha Sept. 27
  • Beta 1 -- Nov. 17
  • Beta 2 -- Jan. 10 or 17?
  • Release Candidate -- ~Jan. 31 (or sooner!)
  • Goal month for official release – February 2013
  • (or sooner!)
  • Program Execution: Alpha:
  • Outreach to project sponsors and other “clueful users and DNS developers” primarily
  • Support through the bind10-users list
  • Registration for on isc.org for tracking (but not ongoing surveys)
  • Product Management will reach out to engaged participants for 1:1 meetings
  • Program Execution: Beta:

o Weekly emails to testers with option for weekly phone calls o In person meetings as possible (local areas, Grace Hopper, NANOG, LISA, RIPE, IETF) o Ongoing schedule of web seminars, blogs, etc to drive participation, inform testers of new features going into each beta, etc.

Phase Four: Postmortem, lessons learned, driving feedback into release – following each release, a postmortem within two weeks, and feedback goes into the next release cycle for consideration.

Methodology:

Three types of beta testers:

  • Current Support Customers – any customer who has a current support contract (open question, does this include forum members? All levels?)
  • Invited Supported Beta – all non-customer potential participants need to “apply” and some number will be selected (depending on how many support customers want in, up to about 50 total supported participants) to get full ISC support during the beta – there will be specific participation levels expected
  • General Community Beta – anyone may join , best effort support from email alias with engineers and team members on it

Data gathering mechanisms:

  • Registration, usage, and other surveys
  • Bug reports
  • Possibly “phone home“ option – opt in tool that sends isc data
  • isc-sysinfo
  • Support queues
  • bind10-users alias
  • Calls and/or face to face meetings with some participants

Incentives for Participants:

  • Participants who are invited to the “supported” program receive support benefits for duration of program
  • Participants who submit a registration form and two or more bugs or weekly/experience reports will receive a thank you gift
  • Another possible option is to use the “bounty” strategy where actual (small) cash prizes are awarded for participation – this might be good to use for substantial work like database ports of module development

Outreach Plan:

Finding test subjects:

  • ISC BIND, DHCP lists
  • Other DNS community lists:
  • NANOG and *NOG email campaign
  • Twitter, LinkedIn?, Facebook campaign

Engagement with test subjects:

  • Web seminars TBD
  • Weekly phone chat option (up to 5 participants can sign up to discuss, if more than 5 are interested in a week, more than one session is possible)
  • NANOG BIND 10 BoF and 1:1s
  • LISA BIND 10 BoF and 1:1s

Potential Blog topics (over the course of alpha and beta we will blog at least weekly about…):

  • BIND 10 Alpha Intro Blog
  • BIND 10 auth architectural overview Blog
  • Developer Documentation (APIs, etc) blog
  • Logging system blog
  • Configuration – philosophy and how it works blog
  • BIND 10 beta blog
  • Data sources in general and specific
  • Modularity – why it helps with performance, scaling, security, resiliency
  • So you want to build a module
  • Hooks: what we have now, what we want to do, how you can help
  • Quality Assurance in BIND 10
  • Using statistics in BIND 10
  • Protocol (RFC) issues and other DNS server bugs found by meticulous BIND 10 test-driven development.
  • Queries per second performance blog
  • Resource usage blog
  • Efficient in-memory datastore blog
  • Blog about development done by developers from four organizations and different timezones and how that affects our scrum sprints
  • lessons learned about nsec and wildcards?

Program Expenses:

  • staff time,
  • incentives for participants (prizes or bounty)
  • travel to NANOG and LISA (already planned)

Program Risks:

  • Not enough participants
  • Too many participants
  • Software found lacking by participants

Documents and Resources for testers: Will include but not be limited to:

  • Beta announcements
  • Beta test application
  • Participant guidelines
  • Program welcome letter
  • Beta release notes
  • Surveys
  • Appreciation email

Resources for internal support staff:

  • Documentation plan will outline tutorials and how-to guides for BIND 10.

Additional Planning documents to be provided

  • Documentation plan- to specify what documentation sets will be ready for beta and on what schedule
  • Pre-release code Test plan – to specify what standard and additional tests will be done on beta and production code to ensure it is ready
  • Beta Participant Test plan – to specify what scenarios in detail we want to ask beta participants to test. Lettuce system level test schemas and experience reports from alpha participants will both inform this plan.
Last modified 5 years ago Last modified on Sep 27, 2012, 7:53:32 PM