20 minute read

Welcome to Part IV (and hopefully final part) of my series on effective testing and here we are going to dig into the actual components of a DSL and how the one I presented in Part III was built. If you’d like to see code, it is available on GitHub and can be run with Docker.

Components

The above diagram is adapted from Figure 3.1 of Domain Specific Languages by Fowler, but extended to include our specific use cases of configuring test dependencies. An implementation consists of:

  1. DSL - The classes that together allow you to implement the language, or in other words are actually used directly in the tests to specify the fixtures and/or configuration of the system under test.
  2. Semantic Model - This is the in-memory representation of the system we are modeling. It basically is a giant repository of all the objects capturing the desired state of the system.
  3. Builder - This is the code that reads the semantic model and turns the specified intention into the actually realized version. It can:
    • Database - Populate the database with the entire desired state.
    • Mocks & Test Doubles - Abstract away the configuration of mocks and test doubles. In particular, I often find that having one flexible and configurable double, maintained alongside the implementation (by the team doing the implementation), is far preferable to having hundreds of tests mock an implementation in situ.
    • Application - Configure the rest of the system under test as well, for instance if you are using a framework initializing it properly with appropriate test doubles, but anything else necessary for bootstrapping your system in a test environment is done here.

We will examine each component in order from simplest to most complex, starting with the semantic model.

The Semantic Model

The semantic model (shown above with some details removed) is essentially just a giant in-memory repository and it is where all objects are accumulated as they are being defined by the DSL. It has a few responsibilities:

  1. Storing the actual objects.
  2. Providing efficient lookups for related objects (e.g., being able to find a Student by name or e-mail).
  3. Can provide some validation as necessary, although there is no right amount of validation. If for instance your service layer enforces some invariants, but you want your code to defensively handle it if there is old/bad data, you might want to tolerate the DSL creating it. You also don’t want to necessarily duplicate all the logic in your application validation in here as well.

Depending on the operations you want to support, the model will need to change over time, and store objects in more than one way to support efficient access. For instance, if you want your DSL to allow you to reference Students by student number, you would add additional data structures to support them in the semantic model.

The Builder

The builder itself is pretty straightforward, its responsibility is to simply walk through the model and populate the data. Some considerations when implementing the builder:

  • It will need to manage and know how to best persist data, for instance it might need to sort the records to ensure that foreign key constraints pass, or disable foreign key checks entirely.
  • Performance will be important here, so properly utilizing batch insertions and efficient queries will be important.
  • It might need to configure certain fixtures or interact with other parts of the system as well depending on the specifics of your application. For instance flushing caches, truncating existing data, replacing application components with test doubles in the dependency injection framework.

Conceptually, its job is to ensure that the state of the system specified declaratively above, is brought to life as much as possible for the purposes of testing. It is a mistake to think that it’s only responsible for persisting data, even though that is what I’ve focused on.

The first time I rolled this technique out, the builder was the key part that controlled what parts of our system could be tested effectively or not. The system at the time was a monolithic PHP application running a forked version of CodeIgniter in between migrations between Active Record and Doctrine, and between a flat MVC architecture and a pluggable component-based one. Comparatively small investments in the builder drastically increased the scope and size of the system that could be tested. The builder did need far-reaching knowledge and had to muck with the internals of a few legacy components, but the trade-off of going from being able to test ~10% to 80% of the code made it an easy decision for us.

The DSL Code

The final and most novel piece is the DSL code1 itself, the specific classes and methods that allow you to use (or abuse) the language to concisely represent the desired state. Now the purpose of this post is NOT to introduce you to internal DSLs in general. Exactly what you can do depends greatly on the specific language you are using. I’d recommend separately googling to find modern techniques in your language if you are interested. For its part the book Domain Specific Languages does give a big and broad overview of various techniques, but also it was published before Lambdas were a thing in Java, so might be dated (although it does cover them in other languages). It certainly won’t have the sleekest tricks available in the most recent version of your language. In what follows, I’m going to walk through a particular way of constructing a DSL and a few useful techniques, feel free to adapt and move responsibilities as it suits you.

Another word of warning is that the DSL will need to be maintained and evolved while remaining understandable. If your language is super flexible (e.g., Groovy) you can make your code too human-readable in my opinion, for instance in Groovy builder.with("foo").and("bar").execute() can also be written as builder with "foo" and "bar" execute, and that is a bridge too far for me in this context. Specifically I do like to be able to understand and debug DSL-related code, and relying on reflection or tricks that are too cute can hamper that. It can also be harder to understand the exact rules involved (e.g., why do foo and bar need to be quoted?). Finally, I fear that evolving the DSL over time can get tricky, for instance if you do that and need to support or, and operator precedence. If you needed to do that you might need to completely change the DSL, and refactor all your existing tests.

So with all that said, let’s look at a segment of the code, and the implementation.

Construction

dsl()
.aSemester(s -> {
    s.setTerm(Term.FALL);
    s.setYear(2024);
})
.aStudentNamed("Carol", "Miller", "carol.explicit@test.edu")
.withDepartment(d -> d.setDepartmentCode("CS"))
  .withCourse("CS-100", "Data Structures")
    .withOfferingInANewRoom()
    .alsoWithDepartment()
  .aCourse()
.also()
.withDepartment(d -> d.setDepartmentCode("MATH"))
.install();

The dsl() method is pretty straightforward, it just builds the DSL, and the implementation follows:

    protected NoContextDslContext dsl() {
        return DslBuilder.getNew()
                .builderFactory(model -> new HibernateUniversityModelBuilder(model, emf))
                .specify();
    }

So dsl() is essentially a factory method that wires up our various dependencies, and returns an instance of the NoContextDslContext interface, which we will talk about below in more detail, but essentially represents a blank canvas with an empty semantic model and nothing in scope.

The class/object dependencies are shown below, and are:

  1. Semantic Model - The object above that is responsible for storing all our desired objects, as talked about above.
  2. Primitive Value Generator - An object that is responsible for generating random data (like faker).
  3. Random Entity Generator - An object that is responsible for generating random entities, that importantly are valid and satisfy all their invariants.
  4. Model Builder - The object responsible for actually setting up the fixture as specified in the DSL, as talked about above.

An important thing to keep in mind, and not shown is that it is important to carefully control the seed and random values used. This can get fairly advanced and fairly niche, but I’ve often used a seed that changes based on the day, and print the seed at startup in the tests, so that if there is some subtle bug it is reproducible2.

The NoContext class serves as our entrypoint into the DSL, in the above Consumer<X> just means a lambda that takes X as an argument, and returns nothing. It has, as a dependency ContextDependencies which is just a POJO that contains a bunch of other dependencies all in one. Context here is a very important part of how this DSL is structured and affects what things are presently in scope much like a programming language scope. For instance when you create a Department with withDepartment, you switch from having no context, or nothing in scope, to having a Department in scope, every object created after is within the context of the Department, and as such the DSL offers new methods to callers so that they can specify things like Courses within the Department. We will talk about the specific structure of the interfaces in a second, but returning to our code for a second, we see the first few lines are:

Basic Interaction

Let’s look at the first few lines of the fixture we were specifying:

dsl()
   .aSemester(s -> {
     s.setTerm(Term.FALL);
     s.setYear(2024);
   }).aStudentNamed("Carol", "Miller", "carol.explicit@test.edu")
  //...

The aSemester() method is within the NoContext class and so let’s take a look at it:

public class NoContext extends AbstractContext implements NoContextDslContext {
    @Override
    public NoContextDslContext aSemester(final Consumer<Semester> semesterModifyingLambda) {
        Semester semester = getRandomEntityGenerator().getRandomSemester();
        semesterModifyingLambda.accept(semester);
        getSemanticModel().addSemester(semester);  // addSemester makes it the current semester
        return this;
    }
}

Nothing to see here, we randomly generate a Semester, then hand it to the code the user supplied (by calling accept()), where it can override what it wants, and then save it to the semantic model, in this case it sets the term and year. We return this, and the type is NoContextDslContext and so nothing has changed for callers(i.e., they can call the same methods as before). Using the scope analogy of before, nothing new is in scope.

The next thing we do is declare a Student, although note that this function doesn’t take a Lambda. Why? Well, there is in fact a function that takes a lambda, but for readability (perhaps because student names are logged and used in errors), it’s helpful to have a utility method so that you can write .aStudentNamed("Carol", "Miller", "carol.explicit@test.edu"), which is more readable than the Lambda would be. Looking at the implementation of this method, you’ll see it calls the aStudent method:

interface NoContextDerivedMethods extends NoContextOverloadedMethods, NoContextBaseDsl {
    default NoContextDslContext aStudentNamed(String firstName, String lastName, String email) {
        return this.aStudent(s -> {
            s.setFirstName(firstName);
            s.setLastName(lastName);
            s.setEmail(email);
        });
    }
}

You’ll note that this method is not implemented in the NoContext class. We are leveraging a feature of Java, default methods to keep the code organized. Essentially if you look closely at all the methods inherited by NoContextDslContext, you’ll notice ~20 or so methods. For specifying Students, depending on what your tests need there are lots of random utility methods to keep them readable, for example aStudent() which takes no arguments, manyStudents(int) which takes an int argument and will just generate that many Students, etc. To prevent the main class from becoming unwieldy, we split the definition into a few different cases:

  • The base class NoContext in general implements the most generic operation, the one that takes the lambda, interacts with the semantic model and random generators.
  • The base interface NoContextBaseDsl strictly defines all the base methods that should be visible to callers in NoContext.
  • The interface NoContextOverloadedMethods uses default methods and simply is where you can place simple method overrides for the base methods.
  • The interface NoContextDerivedMethods uses default methods and is where more complicated logic sits.

This kind of functionality is available in lots of languages, sometimes called a trait or a mixin. One of the advantages of doing the above, in Java anyway, is that the derived methods can’t circumvent the core logic. For instance, say you want to validate that all student names have length >=1 for example. The only way the derived methods in NoContextDerivedMethods can define a Student, is via the aStudent(Consumer<Student>) method. If you defined this method directly in the NoContext class, nothing would stop a developer or LLM from simply calling the semantic model directly. When you start having many derived methods the above is beneficial if you can do it. A more general takeaway is that constructing DSLs introduces its own trade-offs and software engineering challenges to tackle.

Contexts and Scope

Looking back at our DSL let’s see what’s next (denoted with // <--):

dsl()
.aSemester(s -> {
    s.setTerm(Term.FALL);
    s.setYear(2024);
})
.aStudentNamed("Carol", "Miller", "carol.explicit@test.edu")
.withDepartment(d -> d.setDepartmentCode("CS")) // <--
  .withCourse("CS-100", "Data Structures") // <--
    .withOfferingInANewRoom() // <--
//...

So here, we call the withDepartment(), withCourse() and withOfferingInANewRoom() methods. Looking at the first we see:

class NoContext {
    @Override
    public DepartmentDslContext withDepartment(final Consumer<Department> departmentModifyingLambda) {
        Department department = getRandomEntityGenerator().getRandomDepartment();
        department.setOfficeBuilding(getOrCreateCurrentBuilding()); 
        departmentModifyingLambda.accept(department);
        getSemanticModel().addDepartment(department);
        return new DepartmentContext(getContextDependencies(), this, department);
    }
}

This method largely mirrors the previous one, among other things it generates a valid Department, and then populates a reference to an office building so that the lambda can start with a valid object (we will talk more about the office building in a second), it then populates the semantic model with the object (after having been configured by the calling lambda with accept()). One new thing it does is return DepartmentContext instead of a NoContext instance, which changes the state of the DSL and gives callers the ability to call different methods, which construct things within this Department.

class DepartmentContext extends AbstractContext implements DepartmentDslContext {

    private final Department currentDepartment;

    DepartmentContext(
            final ContextDependencies contextDependencies,
            final AllContextDslMethods parentContext,
            final Department currentDepartment) {
        super(contextDependencies, parentContext);
        Objects.requireNonNull(currentDepartment);
        this.currentDepartment = currentDepartment;
    }

    @Override
    public DepartmentDslContext aCourse(final Consumer<Course> courseModifyingLambda) {
        //...
        return this;
    }
    // Rest of methods
}

The DepartmentContext passes the dependencies object and its parent context to the parent constructor, then stores the current Department in an instance variable.

We use the same convention for the interfaces here, as we did in the NoContext case, having a base interface specifying the “primitives” available, and then overloads and then derived utility methods such as aGraduateSeminar() to handle configuring more complex fixtures. The methods withCourse and withOfferingInANewRoom() largely proceed the same:

class DepartmentContext extends AbstractContext implements DepartmentDslContext {
    @Override
    public CourseDslContext withCourse(final Consumer<Course> courseModifyingLambda) {
        Course course = getRandomEntityGenerator().getRandomCourse(currentDepartment);
        courseModifyingLambda.accept(course);
        getSemanticModel().addCourse(course);
        return new CourseContext(getContextDependencies(), this, course);
    }

}

withCourse() returns a CourseContext, which again takes in all the dependencies, a parent context reference, and the current course, then sets an instance variable. It also implements withOfferingInRoom() which returns a CourseOfferingDslContext, which again lets callers call different sets of methods to specify things appropriate within a Course Offering.

class CourseContext extends AbstractContext implements CourseDslContext {

    private final Course currentCourse;

    CourseContext(
            final ContextDependencies contextDependencies,
            final AllContextDslMethods parentContext,
            final Course currentCourse) {
        super(contextDependencies, parentContext);
        Objects.requireNonNull(currentCourse);
        this.currentCourse = currentCourse;
    }

    @Override
    public CourseOfferingDslContext withOfferingInANewRoom(
            final Consumer<CourseOffering> offeringModifyingLambda) {
        return offeringIn(createRoomInCurrentBuilding(), offeringModifyingLambda);
    }

    /**
     * Creates an offering of the current course, in the current semester and the
     * given room, with seat capacity defaulting to the room's -- so the offering
     * is complete and valid BEFORE the lambda runs (the lambda has the full view).
     */
    private CourseOfferingDslContext offeringIn(
            final Room room,
            final Consumer<CourseOffering> offeringModifyingLambda) {
        CourseOffering offering = getRandomEntityGenerator().getRandomCourseOffering();
        offering.setCourse(currentCourse);
        offering.setSemester(getOrCreateCurrentSemester());
        offering.setRoom(room);
        offering.setSeatCapacity(room.getSeatCapacity());
        offeringModifyingLambda.accept(offering);
        getSemanticModel().addCourseOffering(offering);
        return new CourseOfferingContext(getContextDependencies(), this, offering);
    }
}

Implicit Dependencies

One thing above I glossed over was the line department.setOfficeBuilding(getOrCreateCurrentBuilding()) in withDepartment() and we see another instance here where we call offering.setSemester(getOrCreateCurrentSemester()). In Part III’s domain model, it showed Building as the root for basically every object, and it showed that a Course Offering needs both a Semester and a Course. What we are doing here is another perfectly valid way of managing this kind of context: have the Semantic Model keep track of the most recent instance of each relevant type for context instead. The implementation is straightforward:

abstract class AbstractContext implements AllContextDslMethods {
    /** The current building, creating a shared one on first use. */
    Building getOrCreateCurrentBuilding() {
        if (getSemanticModel().getCurrentBuilding() == null) {
            getSemanticModel().addBuilding(getRandomEntityGenerator().getRandomBuilding());
        }
        return getSemanticModel().getCurrentBuilding();
    }
}

public class UniversitySemanticModel {
    private Building currentBuilding;
    
    public void addBuilding(final Building building) {
        if (buildingsByCode.containsKey(building.getBuildingCode())) {
            throw new DslSpecificationException(
                    "Duplicate building code: " + building.getBuildingCode());
        }
        buildings.add(building);
        buildingsByCode.put(building.getBuildingCode(), building);
        this.currentBuilding = building;
    }

    public Building getCurrentBuilding() { return currentBuilding; }
    
}

This DSL design choice allows users to entirely skip specifying Buildings or Semesters most of the time, or manage their context. Contexts only have so much expressive power, and if you have something like a Course Offering which depends on both a Course and a Semester, then it can get complex to manage. You could make interfaces and types for each combination (CourseNoSemesterContext, SemesterNoCourseContext, CourseAndSemesterContext), or you can just do the above and rely on the Semantic Model.

You can build a DSL relying exclusively on the semantic model as above, and not use context juggling at all, but there are downsides, such as giving up compile-time safety, IDE autocompletion support, and perhaps understandability. Consider dsl().withTeachingAssignment(): it either throws an error at runtime, complaining about no Course Offering, Course, Department, or Instructor, or auto-creates them all, arguably leaving important details out of the test.

Context juggling has one additional benefit: by managing the context this way, you can in effect save state or handles to different contexts, for instance:

 DepartmentDslContext cs = dsl()
          .withDepartment(d -> d.setDepartmentCode("CS"))
              .aCourseWithCode("CS-101", "Intro to CS");

      cs.also()
          .withDepartment(d -> d.setDepartmentCode("MATH"))
              .withCourse("MATH-201", "Discrete Maths")
                  .aPrerequisite("CS-101");

      cs  // resume CS -- MATH-201 now exists, so CS-301 can depend on it
          .withCourse("CS-301", "Algorithms")
              .aPrerequisite("MATH-201")
          .also()
          .install();

In the above, we have dependencies between both Departments, Math and CS. Because each context is a distinct object, with an instance variable pointing to the object at the time it was created, you can just go back to it. Granted, the above is not ideal, probably a better way would be some kind of method on NoContext that allows you to link Courses outside of any context. My experience, both with my own and other DSLs, is that they can be great at handling 98% of cases, but occasionally you end up wanting to do something clunky. Perhaps an even cleaner implementation would be to introduce a forFoo(id) convention, like forDepartmentByCode("CS") that switches back to the DepartmentContext. These are the fun language design choices you get to make when implementing your DSL!

Returning To Earlier Contexts

Looking again at our test fixture, the only things we haven’t covered are the also____() methods (shown with // <--).

dsl()
.aSemester(s -> {
    s.setTerm(Term.FALL);
    s.setYear(2024);
})
.aStudentNamed("Carol", "Miller", "carol.explicit@test.edu")
.withDepartment(d -> d.setDepartmentCode("CS"))
  .withCourse("CS-100", "Data Structures")
    .withOfferingInANewRoom()
    .alsoWithDepartment() // <--
  .aCourse() //<--
.also() // <--
.withDepartment(d -> d.setDepartmentCode("MATH"))
.install();

These are pretty straightforward:

class CourseOfferingContext extends AbstractContext implements CourseOfferingDslContext {

    @Override
    public CourseDslContext alsoWithCourse() {
        if (getParentContext() instanceof CourseDslContext) {
            return (CourseDslContext) getParentContext();
        }
        throw new DslSpecificationException(
                "Cannot navigate to course -- this offering was not created within a course context");
    }

    
    @Override
    public DepartmentDslContext alsoWithDepartment() {
        AllContextDslMethods parent = getParentContext();
        if (parent instanceof DepartmentDslContext) {
            return (DepartmentDslContext) parent;
        }
        if (parent instanceof CourseDslContext) {
            // offering -> course -> department
            return ((CourseDslContext) parent).alsoWithDepartment();
        }
        throw new DslSpecificationException(
                "Cannot navigate to department -- this offering was not created within a department or course context");
    }
}


abstract class AbstractContext implements AllContextDslMethods {

    private final ContextDependencies contextDependencies;
    private final AllContextDslMethods parentContext;
    private final NoContextDslContext noContextDslContext;

    public NoContextDslContext also() {
        return noContextDslContext;
    }
}

The above don’t change any state in the semantic model, but change what methods are legal for the DSL to call, and primarily exist to support method chaining and readability. In the Course Offering context, you can see that it does allow for some flexibility for skipping a Course context, but beyond that there is nothing fancy about them.

Variants and Miscellany

So now that we have gone through the wringer with a basic implementation, there are a few thoughts I wish to leave you with for variations to explore.

  1. While the semantic model captures the objects directly today, there is actually no reason you can’t instead save the lambdas in the semantic model, and execute them later. This can be useful if you want to structure your DSL so that some aspects depend on things specified later.
  2. You can actually use more than one builder. You can have a builder that builds test doubles for repositories and does things without the database, and then have a different builder that is used for database testing. You can leave the DSL ambiguous with a generic install() method, or parameterize it with a builder, or even switch to a new DSL for configuring build (e.g., .install().withTestDoubles().noRedis().build()). You can also leave it unspecified and control with an environment variable whether the tests should run against the DB or in memory, and trigger a skip if not. A lot will depend on the language you are using, and the specifics of your testing goals, but with a DSL-based testing strategy you can use the same exact tests with different testing strategies simultaneously.
  3. You can use DSLs for not just the entity fixtures you need, but for anything you might want to control in your tests, including parts of the application, but also for executing the system under test.

Wrap Up

So that’s it all spelt out. We saw in this part the major pieces of building a DSL. The first is the Semantic Model, which captures fixture configuration and other intentions of the user. The second is the DSL Code itself, which uses and abuses the language to provide “code” that is easy to read and understand the domain. This happens, arguably at the expense of understanding exactly what the code is doing, but it is arguably a tradeoff worth making. Lastly, the Builder reads the state of the Semantic Model, and then does whatever is necessary to initialize the system under test, and provides the central place for engineering effort to ensure that the functional tests are fast3.

The first time I built this, it turned testing from one of the most hated parts of feature development to one of the most enjoyable parts. I have since employed these ideas several times over the years. Writing functional tests allows us to have tests that stop bugs. The DSL allows us to minimize irrelevant details, which keeps the tests understandable and satisfies Meszaros’ Law. Finally, the builder can leverage the fact that the DSL captures intention to provide high ROI on performance tweaks to functional tests. Again I’d recommend both xUnit Test Patterns and Domain Specific Languages as good reading, and DSLs are a broadly applicable tool to have in one’s toolbox. That said, in some ways, the above is maybe the coolest thing I ever built/discovered, and I hope you (or the LLM that you are feeding this to for your test strategy), enjoyed reading about it.

  1. Martin Fowler calls this part the parser, though I’ve avoided that because it’s in a different sense of the word than I’m used to. ↩

  2. Randomness in tests can be controversial. They can lead to erratic tests which are undesirable but they can also uncover subtle issues in your code base, or be used to test more code paths in the tests. I know in one instance, we had tests that failed if the UUID being generated started with ad due to an issue in the framework we were using. ↩

  3. When I first built this system, in 2016-17 we were able to get the overhead of functional tests down from about 10 seconds per test, to about 1 second of start-up overhead and 100 ms per test (some of the overhead was not due to functional testing). Most of this leveraged performance tricks outside of the scope of the article, but the centralized builder, and semantic model that captured the user’s intent, allowed us to better capture and understand everything that needed to be done, instead of, say, persisting entities one after another. ↩

Categories:

Updated:

You may also enjoy

Effective Testing Part III - Testing With DSLs

14 minute read

So in this series I’ve been working towards presenting how I try to approach testing. In Part I, I mentioned that the tests in our test suite should have the...

Effective Testing Part I - Foundations

5 minute read

Software testing is an area that I have spent a large amount of time and energy in my career looking at. Often test infrastructure and frameworks are conside...