14 minute read

So in this series I’ve been working towards presenting how I try to approach testing. In Part I, I mentioned that the tests in our test suite should have the following properties:

  1. Tests should stop bugs.
  2. Tests should be fast to run.
  3. Tests should be easy to read and understand.

In Part II1, I discussed that I feel the best way to achieve that is by focusing on “functional” or integration tests, as with a little bit of elbow grease, their ability to detect bugs is vastly superior to a naive file by file unit testing approach, and by focusing tests on a consistent layer of abstraction for testing you make it easier for humans to understand the system under test, and by extension have more confidence in changes that Generative AI makes to a code base.

In this part, I wanted to focus on a key technique that I’ve used to make the tests easy to read and understand, before the next part which focuses on how to implement it.

The Testing Wall

So as I mentioned in Part I, xUnit Test Patterns was super helpful in correcting a number of deficiencies in our tests. However, we hit a wall because fundamentally it felt like the techniques in the book would only let us test (with confidence) around 10% of our codebase. Specifically it worked well for testing leaf classes, or classes with minimal or simple dependencies, but didn’t really scale to the meat of our code base. The dependencies in our system, as you moved between layers, meant that effective tests were painful to write, both functional or unit tests, but for different reasons. Unit tests required mocking arbitrary classes and structures, whereas functional tests required the construction of lots of irrelevant objects and complex interactions with the framework we were using. In general, regardless of whether it was a unit or function test, it was difficult to write tests that follow Meszaros’ Law:

When something is important to understanding a test it is important that it be in the test.

When something is NOT important to understanding a test it is important that it NOT be in the test.

What finally broke the impasse in testing, and drove the system from 10% testability to >80% testability was when I read and started applying techniques from Domain Specific Languages by Fowler. In a nutshell the idea was to create a Domain Specific Language for specifying your fixtures, this would allow us to elide irrelevant details from the test, and keep the test simple to understand. As an aside, I would strongly recommend all senior professional developers – especially if they have no background in parsing and lexing – to read this book. It is certainly a book I have gone back to many times over the years.

Domain Specific Languages

Domain Specific Languages (DSL) are a technique/tool that can be used to concisely convey important meaning within a particular domain. Chess for instance has Algebraic Notation that can be used to efficiently, and unambiguously communicate game state, such as the following:

e4 e5
Qh5 Nc6
Bc4 Nf6??
Qxf7#

It’s important to note that while Algebraic Notation is useful for chess, it is not ideal for things outside its domain, for instance it can’t express a Shakespearean sonnet particularly well.

DSLs can be broken into two distinct groups:

  • External DSLs - Like the above, they are completely free-text, and to leverage them computationally you need to process them similar to how a compiler might operate on a source code file, via lexing and parsing. One of the most famous external DSLs is SQL, the Structured Query Language. Terraform, the infrastructure as code tool, uses a DSL called HCL, the HashiCorp Configuration Language.
  • Internal DSLs - These DSLs are written in your programming language (e.g., Java, Go, PHP) and leverage your language’s parser and lexer, and involve using (or abusing) your language’s syntax to concisely express something. A prototypical example of an internal DSL is a fluent builder, with method chaining, however on the other end of the spectrum are things like Gradle (one of the predominant build tools in Java). Each Gradle build file is in fact just a Groovy or Kotlin file.
plugins {
    id 'application'
}

java {
    // Compile against Java 21 language features.
    sourceCompatibility = JavaVersion.VERSION_21
}

dependencies {
    // A well-known utility library from Apache Commons.
    implementation 'org.apache.commons:commons-lang3:3.17.0'
}

application {
    // The class whose main() method starts the program.
    mainClass = 'com.example.Greeter'
}

The above Gradle file sets a language version, includes a library, as well as applies a plugin that makes the built jar runnable by pointing to the com.example.Greeter class. When interacting with a Gradle build file, your conceptual model is NOT supposed to be about the specific sequence of function calls, or anything to do with Groovy, but on the concepts pertaining to building applications, things like library dependencies, Java configuration, and the plugins applied.

DSLs for Fixtures

To talk about an internal DSL for testing, we are first going to need a domain to model, and as such we are going to use as an example a university software system. We need a non-trivial system to work with, as this series is about being able to cut through the noise and irrelevant details of a large system. It’s not particularly important that you understand the nuances of the domain, hopefully the tests can serve as an executable specification of the system under test 😁, but here is an overview of the domain and how various concepts relate to each other:

The examples I’m showing are available on GitHub, in the repo you will find a bunch of tests2, and a DSL implementation in both Java and Go, we will talk about the Java one mostly. The first requirement we want to validate is that you can’t register for a course, if you don’t satisfy the pre-requisites. Let’s just jump to the result and look at what a clean test might look like:

@Test
void rejectsMissingPrerequisite() {
  // Fixture Setup
  dsl()
    .aStudentNamed("Carol", "Miller", "carol.dsl@test.edu")
    .withDepartment()
      .aCourseWithCode("CS-100", "Intro to Programming")
      .withCourse("CS-200", "Data Structures")
        .aPrerequisite("CS-100")
        .withOfferingInANewRoom(o -> o.setSectionNumber("001"))
    .also()
    .install();

  // Execute SUT
  RegistrationResult result = new RegistrationService(emf)
      .enroll(studentIdByEmail("carol.dsl@test.edu"), offeringId("CS-200", "001"));

  // Verify
  assertThat(result.outcome()).isEqualTo(Outcome.REJECTED_MISSING_PREREQUISITES);
}

Now the important thing to focus on is not trying to understand how this code works (we will get to that in the next part), but instead understanding what the test fixture is trying to tell you about the intent of the test setup. Specifically:

There is a student named Carol Miller, a department with two courses, where CS-100 is a pre-req of CS-200, CS-200 has an offering.

After setting up the fixture, the test then tries to register the student, and as there is no reference to her having completed the course, the verification step expects the outcome to be a failure, with missing prerequisites as the error. I asked Claude to write a functional test for the same scenario and this is what came out:

@Test
void rejectsMissingPrerequisite_theHandBuiltWay() {
    // Fixture Setup
    Building sci = new Building();
    sci.setUuid("bld-sci");
    sci.setBuildingCode("SCI");
    sci.setName("Science Building");

    Room room = new Room();
    room.setUuid("room-101");
    room.setRoomNumber("101");
    room.setBuilding(sci);
    room.setSeatCapacity(30);

    Department cs = new Department();
    cs.setUuid("dept-cs");
    cs.setDepartmentCode("CS");
    cs.setName("Computer Science");
    cs.setOfficeBuilding(sci);

    Semester fall = new Semester();
    fall.setUuid("sem-fall-2024");
    fall.setYear(2024);
    fall.setTerm(Term.FALL);

    Course cs100 = new Course();
    cs100.setUuid("course-cs100");
    cs100.setCourseCode("CS-100");
    cs100.setTitle("Intro to Programming");
    cs100.setCredits(3);
    cs100.setDepartment(cs);

    Course cs200 = new Course();
    cs200.setUuid("course-cs200");
    cs200.setCourseCode("CS-200");
    cs200.setTitle("Data Structures");
    cs200.setCredits(3);
    cs200.setDepartment(cs);
    cs200.getPrerequisites().add(cs100);   // CS-200 requires CS-100 -- wire the @ManyToMany by hand

    CourseOffering offering = new CourseOffering();  // an offering of CS-200
    offering.setUuid("offering-cs200-001");
    offering.setSectionNumber("001");
    offering.setCourse(cs200);
    offering.setSemester(fall);
    offering.setRoom(room);
    offering.setSeatCapacity(30);
    // forget this and it's rejected as SECTION_CLOSED, not for the prereq
    offering.setOpen(true);

    Student carol = new Student();     // has NOT completed CS-100
    carol.setUuid("student-carol");
    carol.setStudentId("STU-1");
    carol.setFirstName("Carol");
    carol.setLastName("Miller");
    carol.setEmail("carol.hand@test.edu");


    persist(sci, room, cs, fall, cs100, cs200, offering, carol);

    // Execute SUT
    RegistrationResult result = new RegistrationService(emf)
            .enroll(carol.getId(), offering.getId());

    // Verification
    assertThat(result.outcome()).isEqualTo(Outcome.REJECTED_MISSING_PREREQUISITES);
}

It’s a lot more verbose, it has to ensure that transitively all classes and foreign key checks in the DB pass, even if they aren’t relevant to the test (e.g., it creates a building, rooms, semesters – things the DSL-based test didn’t specify) as well as unrelated invariants (e.g., credits on a course must be greater than 0). Now Claude generated the above code, and it did something that I see all too often, which is that all the UUIDs are set to things that aren’t actually UUIDs (e.g., student-carol instead of f789ff08-0ddd-4e80-b42c-d75935d434d4). If some code starts to rely on the fact that this should be a uuid, or that some other field shouldn’t ever really be null, and you have hundreds of tests that break that invariant, then it ends up often being the case that the tests, far from allowing you to make changes faster, become the very thing slowing you down.

I’ve seen examples of the above happen time and time again, and it exposes a fundamental tension, we want to keep the test streamlined and readable, but also benefit from ensuring that our tests are always valid scenarios on valid data (as opposed to being vacuous or invalid) which causes us to need to put in lots of noise that obscures the purpose of the test.

While I think the above is very much a fair representation of functional tests that I’ve seen, that’s not to say that you can’t do some simple things to improve them, leveraging a lot of the ideas from xUnit Test Patterns, if you are willing to take the time. For instance, using Creation Methods, you could simplify the above to:

@Test
void rejectsMissingPrerequisite_theCreationMethodWay() {
    // Fixture Setup
    Building sci = aBuilding();

    Room room = aRoom();
    room.setBuilding(sci);

    Department cs = aDepartment();
    cs.setOfficeBuilding(sci);

    Semester fall = aSemester();

    Course cs100 = aCourse();
    cs100.setDepartment(cs);

    Course cs200 = aCourse();
    cs200.setDepartment(cs);
    cs200.getPrerequisites().add(cs100);

    CourseOffering offering = anOffering();
    offering.setCourse(cs200);
    offering.setSemester(fall);
    offering.setRoom(room);

    Student carol = aStudent();

    persist(sci, room, cs, fall, cs100, cs200, offering, carol);

    // Execute SUT
    RegistrationResult result = new RegistrationService(emf)
            .enroll(carol.getId(), offering.getId());

    // Verification
    assertThat(result.outcome()).isEqualTo(Outcome.REJECTED_MISSING_PREREQUISITES);
}

The above, while certainly an improvement, does still leave the test to wire up certain things that are not relevant. In addition, while you could parameterize the methods, you need to manage the proliferation of utility methods, for instance when creating a course, it’s easy to pass a department in, but for prerequisites, there can be many, or courses might have equivalencies, so having dedicated parameterized utility methods can start becoming a challenge3. Still a desirable property is that as you evolve your system, the only tests that should need to be changed, are ones directly impacted by the test. If we added a new concept, say a Campus, and each Building is within a campus, then every test would need to be refactored (as the root dependency has changed).

    dsl()
      .aStudentNamed("Carol", "Miller", "carol.dsl@test.edu")
      .withDepartment()
        .aCourseWithCode("CS-100", "Intro to Programming")
        .withCourse("CS-200", "Data Structures")
          .aPrerequisite("CS-100")
          .withOfferingInANewRoom(o -> o.setSectionNumber("001"))
     .also()
     .install();

Fundamentally the difference between this DSL fixture (shown again for clarity), and what was shown previously is that even the streamlined functional test is still responsible for imperatively building the fixture, whereas this DSL is a declarative language just capturing the important details of what we want. In general, with a well-designed domain specific language, you can focus only on capturing intent. This is what the Gradle DSL does, for example, it assumes that your sources are in src/main/java unless you specifically say otherwise. Similarly, this testing DSL better captures only those details which are important to understand the test. Changes you make to the system under test shouldn’t require a huge amount of maintenance to otherwise unrelated tests. Considering the previous example, if we add a new Campus object at the root of the hierarchy, we can infer that by default all the other tests should take place in a single campus unless told otherwise, and so instead of changing every single test, just change a few lines of the fixture language, and add the new tests appropriate to this capability.

Further Examples

At this point I just want to show what a few other tests look like (the companion repo has around 21 tests implemented):

@Test
void flagsAProbationStudentExceedingTheCreditLimit() {
    // Fixture Setup
    dsl()
    .aStudentOnProbation(s -> s.setEmail("struggling@test.edu"))
    .withDepartment()
        .withCourse(c -> { c.setCourseCode("GEN-101"); c.setCredits(5); })
            .withOfferingInANewRoom()
                .enrollStudentByEmail("struggling@test.edu")
    .alsoWithDepartment()
        .withCourse(c -> { c.setCourseCode("GEN-102"); c.setCredits(5); })
            .withOfferingInANewRoom()
                .enrollStudentByEmail("struggling@test.edu")
    .alsoWithDepartment()
        .withCourse(c -> { c.setCourseCode("GEN-103"); c.setCredits(5); })
            .withOfferingInANewRoom()
                .enrollStudentByEmail("struggling@test.edu")
    .also()
    .install();

    // Execute SUT
    CreditLoadResult result = new AcademicStandingService(emf)
            .validateCreditLoad(studentIdByEmail("struggling@test.edu"));

    // Verification
    assertThat(result.outcome()).isEqualTo(CreditLoadResult.Outcome.EXCEEDS_PROBATION_LIMIT);
    assertThat(result.maxAllowed()).isEqualTo(12);
    assertThat(result.attempted()).isEqualTo(15);
}

    @Test
    void reportsAMultiRoleTeachingTeam() {
        // Fixture Setup
        dsl()
            .anInstructor(i -> { i.setFirstName("Dr"); i.setLastName("Lecturer"); })
            .anInstructor(i -> { i.setFirstName("Jane"); i.setLastName("TA"); })
            .anInstructor(i -> { i.setFirstName("Lab"); i.setLastName("Coordinator"); })
            .withDepartment()
                .withCourse("BIO-101", "Intro to Biology")
                    .withOfferingInANewRoom()
                        .assignInstructorByName("Dr Lecturer", TeachingRole.LECTURER)
                        .assignInstructorByName("Jane TA", TeachingRole.TEACHING_ASSISTANT)
                        .assignInstructorByName("Lab Coordinator", TeachingRole.LAB_COORDINATOR)
            .also()
            .install();

        // Execute SUT
        List<TeachingAssignment> team = new TeachingAssignmentService(emf)
                .getTeachingTeam(offeringId("BIO-101"));

        // Verification
        assertThat(team).hasSize(3);
        assertThat(team).extracting(TeachingAssignment::getRole)
                .containsExactlyInAnyOrder(TeachingRole.LECTURER,
                        TeachingRole.TEACHING_ASSISTANT, TeachingRole.LAB_COORDINATOR);
        assertThat(team).filteredOn(TeachingAssignment::isPrimary).hasSize(1);
    }

@Test
    void detectsTwoOfferingsClashingInTheSameRoom() {
    // Fixture Setup
    dsl()
        .withBuilding(b -> b.setBuildingCode("LIB"))
            .aRoom("100", r -> r.setSeatCapacity(50))
        .also()
        .withDepartment()
            .withCourse("HIST-101", "World History I")
                .withOfferingInRoom("LIB", "100", o -> {
                    o.setDayPattern(DayPattern.MWF);
                    o.setStartTime(LocalTime.of(10, 0));
                    o.setEndTime(LocalTime.of(10, 50));
            })
        .alsoWithDepartment()
            .withCourse("HIST-102", "World History II")
                .withOfferingInRoom("LIB", "100", o -> {
                    o.setDayPattern(DayPattern.MWF);
                    o.setStartTime(LocalTime.of(10, 0));
                    o.setEndTime(LocalTime.of(10, 50));
            })
    .also()
    .install();

    // Execute SUT
    List<ScheduleConflict> conflicts = new SchedulingService(emf).detectConflicts();

    // Verification
    assertThat(conflicts).hasSize(1);
    assertThat(conflicts.get(0).room()).isEqualTo("LIB-100");
    assertThat(conflicts.get(0).courseCodes())
            .containsExactlyInAnyOrder("HIST-101", "HIST-102");
}

The above tests span diverse areas such as instructor management, room management and course management, yet fundamentally all use the same language to express the state of the system.

To explain a bit more about the structure of the language, in general the language is structured around the following general ideas:

  1. Objects are created in a context (kind of like scope in a programming language), to enter a new context you use a function like withFoo(), which will create the specified object, and can fill in missing parent objects (e.g., withBuilding() upon seeing there is no Campus, could create one).
  2. If you want to create an object without entering a new context, you use aFoo().
  3. When you create an object (either using withFoo() or aFoo()) the default method accepts a closure that supplies you with a randomly generated valid object4 that you can mutate as necessary for the purposes of the test.
  4. If you want to go back to the root context you use also(), and to go back to some parent you use alsoWithFoo(). These methods do not take a closure.
  5. Beyond that you can create other utility methods to simplify common patterns, as closures can still be somewhat verbose, and aid readability of the tests.

In general, the capabilities and style of your DSL will very much be shaped by the language you are using. For example, in the examples above we use what Fowler calls Syntactic Indentation to convey different levels of context; however, up next we will see some examples in Go, which unfortunately can’t use this as gofmt won’t allow it.

func TestRejectsEnrollmentOnceTheSectionIsFull(t *testing.T) {
    db := openDB(t)
    defer db.Close()
    
    // Fixture Setup
    /* a 30-seat section already filled by 30 students; Carol wants the 31st seat */
    Dsl(db).
    ManyStudents(30).
    WithDepartment().
    WithCourseNamed("PSYCH-101", func(c *domain.Course) { c.Title = "Intro to Psychology" }).
    WithOfferingInANewRoom(func(o *domain.CourseOffering) { o.SeatCapacity = 30 }).
    EnrollAllStudents().
    Also().
    AStudentNamed("Carol", "Latecomer", "carol@test.edu").
    MustInstall()
    
    registrationSrv := registration.New(db)
    
    // Execute SUT
    decision, err := registrationSrv.Register(
    studentID(t, db, "carol@test.edu"), offeringID(t, db, "PSYCH-101", "001"))
    if err != nil {
        t.Fatalf("register failed: %v", err)
    }
    
    // Verification
    if decision.Allowed || decision.Reason != registration.ReasonSectionFull {
        t.Fatalf("expected %s, got allowed=%v reason=%s",  registration.ReasonSectionFull, decision.Allowed, decision.Reason)
    }
}

func TestRejectsAssigningAnInactiveInstructor(t *testing.T) {
    db := openDB(t)
    defer db.Close()

    // Fixture Setup
    Dsl(db).
    AnInstructor(Named("Retired", "Prof"), Inactive).
    WithDepartment().
    WithCourseNamed("PHIL-101", func(c *domain.Course) { c.Title = "Intro to Philosophy" }).
    WithOfferingInANewRoom().
    Also().
    MustInstall()

    teachingSrv := teaching.New(db)

    // Execute SUT
    result, err := teachingSrv.Assign(
    instructorID(t, db, "Retired Prof"), offeringID(t, db, "PHIL-101", "001"), domain.RoleLecturer)
    if err != nil {
        t.Fatalf("assign failed: %v", err)
    }

    // Verification
    if result.Assigned || result.Reason != teaching.ReasonInstructorInactive {
        t.Fatalf("expected %s, got assigned=%v reason=%s", teaching.ReasonInstructorInactive, result.Assigned, result.Reason)
    }
}

Now admittedly, I haven’t actually ever applied these ideas in Go deeply before and was worried about how verbose it was, but Claude suggested some nice tricks, such as the use of smaller utility lambdas that aid in readability (e.g., AnInstructor(Named("Retired", "Prof"), Inactive)). This pattern would be more challenging to implement in the Java DSL, as it has restrictions when mixing generics, varargs and interfaces. That said, the point of this series is to introduce you to the idea of DSLs, and show you one particular implementation/vision. I’m hardly the world’s foremost authority on them.

Conclusion

Anyway, this is the end of this part of the journey. In summary, DSL-based test fixtures allow you to concisely capture important details, and only those details necessary to express the intent of the test. One reason I was motivated to write this series is that I believe one of the limiting factors with scaling generative AI is our ability to express and convey intent to the LLM, as well as review and understand its intent. My belief is that a testing strategy like the above makes it much easier for the LLMs to learn and validate how the system works, and developers/reviewers to learn and validate what the LLM has built. In the next part, we will walk through the actual implementation of how this all works under the hood.

  1. I published Part III first, since I needed to know where the series was heading to do Part II. ↩

  2. All the example code was generated with Claude code, although I spent a bunch of time fine-tuning the specifics of the language, and it reverse engineered some patterns from other implementations I’ve done. ↩

  3. While the DSL will also have a large number of related methods one difference is that it is a centrally designed language, as opposed to random utility methods littered through the code base. ↩

  4. In some cases the objects might not be 100% valid depending on how the DSL is actually building. For instance, if the database is responsible for generating ids, the instances themselves may not have them at this point. ↩

Categories:

Updated:

You may also enjoy

Effective Testing Part I - Foundations

5 minute read

Software testing is an area that I have spent a large amount of time and energy in my career looking at. Often test infrastructure and frameworks are conside...