Skip to main content

Testing Stack

Gumroad uses a comprehensive testing stack for quality assurance:
  • RSpec 3.12 - Test framework
  • Capybara 3.38 - Integration testing
  • Selenium WebDriver - Browser automation
  • Factory Bot - Test data generation
  • Faker - Realistic fake data
  • VCR - HTTP interaction recording
  • WebMock - HTTP request stubbing

Running Tests

Setup Test Environment

Before running tests, ensure the test database is set up:
1

Start Docker services

make local
2

Setup test database

RAILS_ENV=test bin/rails db:setup
3

Export JavaScript constants

RAILS_ENV=test bin/rails js:export
The js:export command generates JavaScript constants for the test environment. Without this, tests may try to navigate to the development domain and encounter 502 errors.

Running Test Suite

bin/rspec

macOS Fork Error

If you encounter fork-related errors on macOS:
objc[11912]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
Solution: Disable Spring before running tests:
export DISABLE_SPRING=1
bin/rspec spec/requests/balance_pages_spec.rb

Test Organization

Tests are organized by type:
spec/
├── business/          # Business logic tests
├── controllers/       # Controller tests
├── factories/         # Factory Bot definitions
├── helpers/           # Helper method tests
├── jobs/             # Background job tests
├── models/           # Model tests
├── policies/         # Pundit policy tests
├── presenters/       # Presenter tests
├── requests/         # Integration/E2E tests
├── services/         # Service object tests
└── sidekiq/          # Sidekiq worker tests

Writing Tests

Test Naming Conventions

Don’t use “should” in test descriptions. Be descriptive about the behavior being tested.
it "displays the user's balance" do
  expect(page).to have_text("Balance $100")
end

it "redirects to login when not authenticated" do
  visit dashboard_path
  expect(current_path).to eq(login_path)
end

Using Factories

Factory Bot provides test data generation:
# Define factories in spec/factories/
FactoryBot.define do
  factory :user do
    email { "[email protected]" }
    username { "testuser" }
    password { "password" }
  end
  
  factory :named_seller, parent: :user do
    name { Faker::Name.name }
  end
end

# Use in tests
let(:seller) { create(:named_seller) }
let(:product) { create(:product, user: seller) }
Use @example.com for emails and example.com, example.org, example.net for custom domains in tests.

Integration Testing with Capybara

Capybara enables end-to-end testing through browser automation.

Writing Capybara Tests

RSpec.describe "Dashboard", js: true, type: :system do
  let(:seller) { create(:named_seller) }
  
  before do
    login_as seller
  end
  
  it "displays correct values and headings for stats" do
    visit dashboard_path
    
    within "main" do
      expect(page).to have_text("Balance $100", normalize_ws: true)
      expect(page).to have_text("Last 7 days $50", normalize_ws: true)
      expect(page).to have_text("Last 28 days $150", normalize_ws: true)
    end
  end
end

Element Selection Best Practices

Prefer semantic selectors over class names:
1

Text content (preferred)

click_on "Submit"
expect(page).to have_text("Success")
2

Input labels

fill_in "Email", with: "[email protected]"
3

Input placeholders

fill_in placeholder: "Search products", with: "eBook"
4

ARIA attributes

find("[aria-label='Close dialog']").click
5

Class names (last resort)

find(".product-card").click
# Only when above options don't work

Interaction Methods

# Preferred
click_on "Text"

# When text method doesn't work
find_and_click "selector", text: "Text"

Preventing Flaky Specs

# Good - Waits automatically
expect(page).to have_selector(".notification")

# Bad - No waiting
find(".notification")
# Bad
sleep 2

# Good - Wait for AJAX
wait_for_ajax

# Good - Implicit waiting
expect(page).to have_selector(".loaded")
for i in {1..10}; do
  echo "Run number $i\n-"
  bin/rspec spec/requests/product_creation_spec.rb:28
done
Capybara’s have_selector is smart and handles waiting automatically. Always prefer it over find when checking for element presence.

Browser Setup

Chrome and ChromeDriver

Integration specs use Google Chrome:
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
sudo apt-get update
sudo apt-get install google-chrome-stable

HTTP Mocking

VCR for Recording HTTP Interactions

VCR records HTTP requests and replays them in tests:
RSpec.describe PayPalService do
  it "processes payment" do
    VCR.use_cassette("paypal/successful_payment") do
      result = PayPalService.process_payment(amount: 100)
      expect(result).to be_success
    end
  end
end
Scope VCR cassettes to specific test files. Sharing cassettes across tests causes collisions where tests read incorrect cached responses.

WebMock for Stubbing

WebMock allows stubbing HTTP requests without recording:
stub_request(:get, "https://api.example.com/user")
  .to_return(status: 200, body: { name: "Test User" }.to_json)

Testing Background Jobs

Sidekiq Test Helpers

RSpec.describe ProcessOrderJob do
  it "enqueues the job" do
    expect {
      ProcessOrderJob.perform_async(order.id)
    }.to change { ProcessOrderJob.jobs.size }.by(1)
  end
  
  it "processes the order" do
    ProcessOrderJob.new.perform(order.id)
    expect(order.reload.status).to eq("processed")
  end
end
Avoid to_not have_enqueued_sidekiq_job or not_to have_enqueued_sidekiq_job because they’re prone to false positives. Make assertions on SidekiqWorkerName.jobs.size instead.

Testing Guidelines

General Principles

Descriptive Names

Write test names that explain the behavior being tested, not implementation details.

Group Related Tests

Use describe and context blocks to organize related test cases.

Keep Tests Independent

Each test should be able to run in isolation without depending on other tests.

Test the Fix

Tests must fail when the fix is reverted. If a test passes without the application code change, it’s invalid.

API Endpoint Testing

For API endpoints, test:
  1. Response status codes
  2. Response format (JSON structure)
  3. Response content accuracy
  4. Error cases and edge cases
RSpec.describe "Products API" do
  describe "GET /api/v2/products" do
    it "returns successful response" do
      get "/api/v2/products"
      expect(response).to have_http_status(:ok)
    end
    
    it "returns products array" do
      create(:product)
      get "/api/v2/products"
      expect(json_response["products"]).to be_an(Array)
    end
  end
end

Continuous Integration

Buildkite

Tests run automatically on Buildkite for every commit.

Reproducing CI Failures

To reproduce Buildkite test failures locally, ensure:
  1. Same Ruby and Node versions
  2. Fresh database setup
  3. Elasticsearch reindexed
  4. JavaScript exports updated

Troubleshooting

Problem: Tests get 502 errors when navigating to checkout.Solution: Run the JavaScript export command:
RAILS_ENV=test bin/rails js:export
The bin/dev script automatically generates JS constants for development, so tests need this command to use the test domain.
Problem: Integration spec times out with:
Selenium::WebDriver::Error::WebDriverError:
  unable to obtain stable chromedriver connection in 60 seconds
Solution: Ensure Chrome and ChromeDriver are properly installed and versions match.
Problem: Integration spec times out:
Rack::Timeout::RequestTimeoutError: Request ran for longer than 60000ms
Solution: Create .env.test.local with:
DISABLE_RACK_TIMEOUT="1"
Then restart Spring:
bin/spring stop

Visual Testing

Gumroad uses QA Wolf for visual regression testing.
  • Enabled by default on main branch
  • Not enabled by default on feature branches
  • Catches visual regressions before production

Test Data Best Practices

1

Use factories over fixtures

Factory Bot provides more flexibility than fixtures and makes tests clearer.
2

Use realistic data

Faker generates realistic test data that catches edge cases fixtures might miss.
3

Clean data between tests

RSpec’s database_cleaner ensures tests don’t interfere with each other.
4

Use let and let! strategically

let is lazy-loaded, let! is eager. Choose based on when data is needed.

Next Steps

Contributing

Learn testing requirements for pull requests

Architecture

Understand the system architecture

Authentication

Test authentication flows

Deployment

Deploy after tests pass

Build docs developers (and LLMs) love