Writing Tests in Bevy

I was lured to Rust by some of the impressive rendering projects in the ecosystem - wgpu, ash/volkano, and Kajiya. I enjoy working on games project and am big on automated tests. So here we are, a small project to bring Rust, Games, and Tests together using the Bevy Game Engine.

The game - snake

New to Rust, New to Bevy.

This article is a collection of thoughts that I've had while working on the project (focused on testing). There's a few examples that can hopefully help other people trying to test similar scenarios (e.g. how do I test keyboard input in my CI pipeline?).

This is not a review, this is more of a report, a summary, a post-motrem to my snake project.
Bevy is great - there's your review. I really look forward to seeing how it grows.

What I tested

I have selected a couple pieces of code that showcase how to test different scenarios in the game.

Fake input and test that Events are created

I have highlighted this test because it shows how to trigger keyboard inputs in tests using app.world.resource_mut::<Input>().press(KeyCode::Up) and how to retrieve events.

#[test]
fn up_key() {
    // Setup
    let mut app = App::default();
    app.add_event::<bevy::input::keyboard::KeyboardInput>();
    app.add_plugin(GameInputPlugin);
    app.world.insert_resource(Input::<KeyCode>::default());

    // Fake pressing the up key
    app.world.resource_mut::<Input<KeyCode>>().press(KeyCode::Up);

    // Force the app to process the run GameInputPlugin 
    // to convert KeyCode::Up to Direction::Up
    app.update();

    // Get all of the 'Direction' events
    let events = app.world
    .resource::<Events<Direction>>()
    .iter_current_update_events()
    .cloned()
    .collect()
    assert_eq!(events, vec![Direction::Up]);
}

Test that the game gets restarted after a timeout

This test shows how I handle dealing with time. I wasn't able to find a way to fake or mock Res<Time> (Time is a struct and not a trait) so instead I just use thread::sleep(Duration::from_millis(500)). Not ideal because it becomes a slow test, but as a once off it's ok.

#[test]
fn switch_from_dead_to_running_after_time() {
    let mut app = App::default();
    app.add_plugins(MinimalPlugins);
    app.add_plugin( GameStatePlugin { 
    	tick_time_sec: 1.0, 
        game_over_pause_sec: 1.0 
    });
    
    // Setup initial state
    app.update(); 
    assert_eq!(
    	app.world.resource::<CurrentState<GameState>>().0,
        GameState::RUNNING
    );
    
    // Set next iyes_loopless state
    app.world.insert_resource(NextState(GameState::DEAD));
    
    // process state change
    app.update(); 
    assert_eq!(
    	app.world.resource::<CurrentState<GameState>>().0, 
        GameState::DEAD
    );
    thread::sleep(Duration::from_millis(500));
    
    // tick timer
    app.update(); 
    assert_eq!(
    	app.world.resource::<CurrentState<GameState>>().0, 
        GameState::DEAD
    );
    thread::sleep(Duration::from_millis(500));
    app.update(); // tick + complete timer
    app.update(); // process state change
    assert_eq!(
    	app.world.resource::<CurrentState<GameState>>().0, 
        GameState::RUNNING
    );
}

Ensure that the snake movement controller sets the correct direction

This test injects an event to be processed by a system app.world.resource_mut::<Events>().send(Direction::Down). This test also shows how to run a query and retrieve the desired components app.world.query::<&MovementController>().iter(&app.world).next().unwrap(). I'm not sure why the &app.world is required and not infered from app.world.query like it is while running a system normally.

    #[test]
    fn handle_input_basic() {
        let mut app = App::default();
        app.add_event::<Direction>();
        app.world.insert_resource(board::Desc {
            grid_size: (5, 5),
            cell_size: 10
        });
        app.world
            .spawn()
            .insert(GridPosition::new(1, 0))
            .insert(MovementController{
                direction: Direction::Right,
                previous_position: GridPosition::new(-1, 0),
            })
            .insert(SnakeHead{});
        app.add_system(handle_input);

        app.world.resource_mut::<Events<Direction>>()
        	.send(Direction::Down);
        app.update();
        let movement_controller = app.world
            .query::<&MovementController>()
            .iter(&app.world)
            .next()
            .unwrap();
        assert_eq!(movement_controller.direction, Direction::Down);

        app.world.resource_mut::<Events<Direction>>()
        	.send(Direction::Right);
        app.update();
        let movement_controller = app.world
            .query::<&MovementController>()
            .iter(&app.world)
            .next()
            .unwrap();
        assert_eq!(movement_controller.direction, Direction::Right);
    }

Check that an entity is created with the expected components

I wasn't able to figure out how to use the Commands object without getting bevy to inject it. I've worked around this by creating an intermediary system to call my intended function.

fn test_system(
	head_params: Res<HeadParams>,
	mut commands: Commands) {
    spawn(
        &mut commands,
        head_params.start_position,
        head_params.cell_size
    );
}


#[test]
fn spawn_creates_head_entity() {
    let mut app = App::default();
    let head_params = HeadParams {
    	start_position: GridPosition{x:3, y:3},
    	cell_size: random::<f32>()
    };
    app.insert_resource(head_params);
    app.add_startup_system(test_system);
    app.update();
    let mut head_query = app.world.query::<(
        &SnakeHead,
        &GridPosition,
        &MovementController,
        With<Sprite>)>();
        
    // check we have exactly one of this entity
    assert_eq!(head_query.iter(&app.world).count(), 1);
    let (_, grid_pos, movement_controller, _) = head_query
    	.iter(&app.world)
        .next()
        .unwrap();
    assert_eq!(grid_pos, &head_params.start_position);
    assert_eq!(movement_controller.direction, Direction::Right);
    assert_eq!(
    	&movement_controller.previous_position, 
        &head_params.start_position
    );
}

A few things I couldn't figure out

Some of these are mentioned above.

How to fake time. It would be handy to be able to control,mock,fake what happens in the Res<Time> resource. I have a test that checks if the game restarts itself 2 seconds after a gameover. Ideally I would be able to set the Time resource to 2 seconds later and verify the behaviour immediately. Instead I just thread::sleep. This makes it costly to test any logic that depends on time.

Update: This is coming in Bevy 0.8 - Time::update_with_instant

How to get access to Commands. As shown above I end up creating a wrapper function to be able to test systems that rely on the Commands object. Commands are used here to spawn an entity and insert components.

Update: You can access Commands using SystemState

let mut state: SystemState<Commands> = SystemState::new(&mut app.world);
let mut commands = state.get_mut(&mut app.world);
let entity_id = spawn(
    &mut commands,
    head_params.start_position,
    head_params.cell_size
);
state.apply(&mut app.world);

How to call systems directly. This would remove some boilerplate from tests. Any test for a system needs to have an app setup and for app.update() being called to tick forwards. It might be handy to create an object, wrap it in a query and call the system directly.

How to create two parameter queries. In a system signature I can have a query of type Query<(&ComponentA, &ComponentB), With<ComponentC>> however when I use app.world.query I could only get a Query with one template parameter to work. The solution was to move With into the tuple like Query<(&ComponentA, &ComponentB, With<ComponentC>)>. Not a real issue, but would be good to better understand what happens here. It also means that I now have a mismatch between the two. If I want to use a type alias for the query parameter then I need to exclusively use the later (single tuple).

Areas for Improvement

Some of my tests are too big or complicated. This becomes a deterence to writing more tests and keeping them up to date. I write about reducing the scope of a single test here. This is done using Dependency Injection and Mocking to reduce and simplify individual tests. The technique leads to tests that are better focused and more resilient to change. Currently unsure on how  to inject Concrete+Mock dependencies into my systems (using static dispatch). One option is to use a Service Locator pattern and bypass bevy completely for them.

Creating instances with irrelevant values. I go into details here about only setting the values important to a test, everything else could be generated. I'm hoping to build a similar library soon to do the code generation. Excited to see what Rust Macros are capable of!

Tests are too far from logic. I have my regular code up top and if you scroll right down the bottom you can find the tests. What I end up needing to do is have the same file open twice in a vertical split to see both sections. This is because with a single mod tests there isn't the best locality between the two. This could be solved by having multiple test modules within the file (each close to the source), but they will need to have different module names.

Files are too big. I'm used to having regular code and tests in separate files. Either in a directory or next to the original in the file tree e.g. src/main.test.ts. Keeping the tests in the same file allows them to access the private elements of the file which is cool (and allows for better encapsulation). This is a trade off I'm still pondering.

Integration Testing

For this test I would run the game headlessly (no window or rendering), inject a stream of inputs and validate that the game state is as expected. In the case for Snake it wouldn't be too hard to create a script that automatically plays the game. I would then have it:

• Wait until the first piece of food is consumed (placed in front of the player)
• Check that the snake has grown by one tail segment
• Navigate the snake towards the next food
• Check that the snake has grown another tail segment
• Repeat for x number of rounds
• Force the snake to run into itself
• Check that the game state has switched to game over
• Check that the game restarts a few seconds later with the starting tail length

From this test we get a pretty good indication that all systems are working together and the state changes operate as expected.

Rendering Test

To perform visual regression testing we could use an approach similar to snapshot testing in React. Instead of taking a snapshot of a DOM tree we could snapshot PNGs. The process would go like this:

• Setup a scene / scenario e.g. snake bites own tail
• Run a single render frame of application
• Output it as a PNG
• Commit to the repository

We can setup different snapshots for different visual scenarios we want to cover. Whenever a change is pushed we generate new snapshots, if the images match then the test passes, if they differ (exact or using a threshold) then the test fails. This prompts the developer to manually check the new outputs and see if they're expected. If they are expected the the developer commits the new snapshot and CI will pass.

To get the rendering working in CI I may be able to use Mesas Lavapipe which I haven't tested yet but it sounds promising.