Testing

Current State

There is no test suite yet. This document defines the standards to follow when tests are added.

---

Backend Testing (Python / pytest)

Framework

pip install pytest pytest-asyncio httpx

Test Location

backend/
├── app/
│   ├── api/
│   │   ├── exams.py
│   │   └── exams_test.py       # Tests next to source
│   ├── services/
│   │   ├── exam_verifier.py
│   │   └── exam_verifier_test.py
│   └── db/
│       └── models.py
└── tests/                      # Integration tests
    ├── conftest.py
    └── test_exam_flow.py

Unit Test Pattern

# exam_verifier_test.py
import pytest
from app.services.exam_verifier import ExamVerifier

@pytest.mark.parametrize("output,expected", [
    ("Directory exists\nPASS", True),
    ("Directory not found\nFAIL", False),
    ("PASS", True),
    ("FAIL", False),
    ("exit 0", False),           # No explicit PASS/FAIL
    ("", False),                  # Empty output
])
def test_parse_verification_output(output, expected):
    result = ExamVerifier._parse_output(output)
    assert result == expected

API Test Pattern

# tests/test_exam_flow.py
import pytest
from httpx import AsyncClient
from app.main import app

@pytest.fixture
async def client():
    async with AsyncClient(app=app, base_url="http://test") as client:
        yield client

@pytest.fixture
async def auth_headers(client):
    response = await client.post("/api/auth/login", data={
        "username": "testuser",
        "password": "testpass"
    })
    token = response.json()["access_token"]
    return {"Authorization": f"Bearer {token}"}

async def test_get_exams_requires_auth(client):
    response = await client.get("/api/exams")
    assert response.status_code == 401

async def test_get_exams_returns_list(client, auth_headers):
    response = await client.get("/api/exams", headers=auth_headers)
    assert response.status_code == 200
    assert isinstance(response.json(), list)

Edge Case Checklist

Before writing any feature, consider:

• Happy path
• Unauthenticated request (expects 401)
• Insufficient role (expects 403)
• Entity not found (expects 404)
• Invalid input (expects 422)
• Duplicate creation (expects 409 or 422)

---

Frontend Testing

No frontend test framework is set up yet. When added, use Vitest with Vue Test Utils.

npm install -D vitest @vue/test-utils

What to test

• API service methods (services/api.js) — mock axios, assert correct URLs and payloads
• Utility functions — pure functions with clear inputs/outputs
• Critical views — exam submission flow, auth redirect

What not to test

• Trivial template rendering
• Third-party component behavior (XTerm.js, Chart.js)
• CSS/styling

---

Verification Script Testing

Verification scripts (bash) are the core of exam grading. They should be tested manually before adding to the platform.

# Test a verification script locally on a container
ssh root@65.109.236.163 "lxc exec user-1-sandbox -- bash -c 'echo SCRIPT | base64 -d | bash'"

# Expected output ends with PASS or FAIL

Contract:

• Last line of stdout must be PASS or FAIL
• Exit code 0 is treated as PASS if stdout is ambiguous
• Scripts must complete in reasonable time (< 30 seconds for exam tasks)

---

Rules

1. Never merge without tests for business logic (verification output parsing, score calculation)
2. Test edge cases — not just the happy path
3. Deterministic — no random values, inject fixed timestamps where needed
4. No test pollution — each test manages its own setup/teardown data
5. API tests must cover auth — every protected endpoint gets an unauthenticated test case