Skip to content

Quality Engineering

Overview

Quality engineering is about more than just catching bugs—it's about embedding reliability into our software from the start. At this early stage, our team needs a testing approach that maximizes impact without slowing down our development cycle. This guide outlines a testing strategy to help ensure that our product is stable, scalable, and easy to maintain.

Core Benefits

  • Faster Feedback Loops: Identify and resolve issues in minutes, not days.
  • Simplified Debugging: Clear test failures point directly to root causes
  • Confident Releases: Confidence in our deployments
  • Reduced Technical Debt: Prevent quality issues from compounding

The Test Pyramid in Practice

Unit Testing and Data Testing (Base Layer)

Target: 80% of all tests

Unit tests form the foundation of our testing approach, covering the small, isolated units that make up our code. Alongside, data tests ensure the accuracy and reliability of data transformations.

What to Unit Test

  • Individual functions and methods
  • Business logic and calculations
  • Data transformations
  • Utility functions
  • Service layer methods

Practical implementation

  • Unit tests should be small and test a single function or method.
  • Python unit tests need to be run as part of our CI/CD pipeline.
  • Tests should be written in pytest for Python.
  • Data tests should be orchestrated after transforms or ingestion jobs.
  • Data tests need to be written in dbt and surfaced via elementary.

Integration Testing (Middle Layer)

Target: 15% of all tests

Integration tests ensure that components function together seamlessly, verifying that interactions between modules behave as expected.

What to Integration Test

  • API endpoints (validating the integration with other services).
  • Database operations (ensuring accurate data retrieval, updates, and consistency).
  • Third-party service interactions (using mocks or stubs to isolate external dependencies).
  • Message queues (verifying successful processing).
  • Cache operations (confirming cached data consistency).

Practical Implementation

  • Run integration tests as part of the CI pipeline, ensuring they're executed before code is deployed.
  • Use mocking/stubbing for external service interactions to maintain fast test speeds without requiring live dependencies.

End-to-End Testing (Top Layer)

Target: 5% of all tests

End-to-end (E2E) tests verify that entire user flows function correctly, simulating real user interactions with the system. Given the complexity and runtime cost of E2E tests, they are applied sparingly and only for critical paths.

What to End-to-End Test

  • Critical user journeys, such as registration, protocol uploads, and account management.
  • Complex workflows that involve multiple steps or components.
  • Authentication and authorization flows, ensuring user roles function correctly.

Development Workflow

Development

  • Write tests alongside code
  • Run tests locally before pushing: Ensure all tests pass locally before pushing code.
  • Review test coverage: Verify that new code is covered by tests.

Code Review

  • Verify Coverage Standards: Ensure that the coverage level meets the team's goals for each new feature.
  • Check test quality: Evaluate reliability and clarity of tests to ensure they are maintainable and consistent.
  • Minimize flaky tests: Resolve or isolate any tests that produce inconsistent results to prevent undermining the CI system’s reliability.

Quality Gates

  • A PR should be rejected if it does not have tests for all new features.
  • A PR should be rejected if it does not pass all tests.
  • A PR should be rejected if it introduces flaky tests.