Tutorial¶

This tutorial explains how you can use this C# autograder to grade assignments programmed in the C# language. This grader works in the context of the Prairielearn framework. This framework is extensible by Docker containers and that is how this grader is supplied.

This tutorial is meant for instructors who want to add C# assignments. It supposes that you know the relevant parts of the Prairielearn instructor guide. It explains only the functionality that is particular for this grader.

We illustrate the C# autograder with two examples. One example uses the Xunit testing framework, augmented with the Shouldly library.

Xunit example¶

In this example we use Xunit for marking unit tests and Shouldly for assertions. Both these frameworks are available in the grader. The relevant files are in directory csharp_question_x:

+ questions
  + csharp_question_x
  - info.json
  - question.html
  + tests
    - test.py
    - test.cs

Every question in Prairielearn has an info.json file. The part that is particular for this grader is:

{
  ...
  "gradingMethod": "External",
  "externalGradingOptions": {
    "enabled": true,
    "image": "gjgiezeman/grader-csharp",
    "entrypoint": "python3 /grade/tests/test.py",
    "timeout": 30
  }
}

We have to set the gradingMethod to External and, in the options, the image to the value that is displayed. This points to the docker image. For the entry points, you must know that our directory tests will be available under /grade/tests in the image, and that a python interpreter can be used. The file test.py will be described next. You should set timeout (seconds) to a value that works in your environment.

File question.html contains:

<pl-question-panel>
  <p>Write a static method <em>twice</em> in class Answer that
    takes an int and returns an int twice as big.</p>
  <pl-file-editor file-name="answer.cs">

public class Answer
{
// Fill in your code here
}
  </pl-file-editor>
</pl-question-panel>

<pl-submission-panel>
  <pl-external-grader-results></pl-external-grader-results>
  <pl-file-preview></pl-file-preview>
</pl-submission-panel>

We expect the students to write a module Answer with a function twice. The code will be written to file answer.cs.

The C# autograder supplies a module csgrader which you should use in your own python script, here with the name test.py:

#! /usr/bin/python3

from csgrader import CsharpGrader

class QuestionGrader(CsharpGrader):
    def tests(self):
        with self.build_xunit("test1","answer.cs", ["test.cs"]) as prog:
            prog.run_with_status("test1", fail_msg="The function 'twice' returns wrong results")

g = QuestionGrader()
g.start()

The csgrader module supplies a class CsharpGrader. The idea is to derive a class from that (QuestionGrader, in the example), instantiate it and call start on it. This will, after setting things up and before reporting back to Prairielearn, call the method tests, which you should supply. In this method we can use methods in CsharpGrader for building and running programs.

The method build_xunit builds a program in which we can use Xunit and Shouldly. Its first parameter is a name that is passed to Prairielearn if the building fails. The second parameter is the name of the C# file that that the student supplies. The rhird parameter is a list of source files supplied by the instructor.

The return value of this method can be used in a with-statement. The variable prog is an instance of class Program.

The method run_with_status runs a program that was built before and checks its exit status. It awards a point if the exit status is correct. The first parameter is a name that is passed to Prairielearn to identify the test run. The second parameter is a message that will be presented to the student if the run fails.

Finally we show the contents of test.cs:

using Shouldly;
public class Tests
{
      [Theory]
      [InlineData(21, 42)]
      [InlineData(0, 0)]
      [InlineData(-403, -806)]
      public void TwoTimesTest(int value, int expected)
      {
              Answer.twice(value).ShouldBe(expected);
      }
}

We define a class with a function annotated with Theory, which is a Xunit test function with, in this case two, parameters. Next, we see three lines with data. Each line causes one call of the function with those values as parameters.

In the function body, we see a call to the function that the student should supply. The call Shouldbe comes from Shouldly and states the expected output.

An Example with File Input¶

A common use case is a function should take input from a stream and write output to a stream. To support this use case, the grader makes sure that

files under directory tests/input are available for test programs when running (read access)
there is a library in namespace GradingUtils that help to access those files

In this example we will see that how we can test code with different test programs and with different inputs for one program.

We expect the student to write a class Answer with static method AddAll:

public class Answer {
  public static void AddAll(Stream input_stream, TextWriter writer)
  { ... }
}

This input stream should contain

a line with an integer (n)
n lines with two integers

The output must be

a line with an integer (n)
n lines an integer, being the sum of the two integers on the input

For example, the input:

2
3 7
10 32

should result in output:

2
10
42

If the input is not correct, the program must throw an exception.

The relevant files for the question are in directory csharp_question_fi:

+ questions
  + csharp_question_fi
    - info.json
    - question.html
    + tests
      - test.py
      - check_correct.cs
      - check_invalid.cs
      + input
        - input1
        - input2
        - invalid1
        - invalid2
        - invalid3
      + expected
        - answer1
        - answer2

Files info.json and question.html are like in the previous example. In directory tests, things are different. The script test.py contains:

#! /usr/bin/python3

from csgrader import CsharpGrader, Equality

class QuestionGrader(CsharpGrader):
  def tests(self):
      with self.build_simple("test","answer.cs", ["check_correct.cs"]) as prog:
          prog.run_and_check("test1", "input1", "answer1")
          prog.run_and_check("test2", "input2", "answer2", cmp=Equality.LOOSE)
      with self.build_xunit("invalid_test","answer.cs", ["check_invalid.cs"]) as prog:
          prog.run_with_status("invalid_input_test", [])

g = QuestionGrader()
g.start()

There are several new things in the test script. There is another function (build_simple) to build a program that is not based on xunit, but where the files contain a main function. The files are a combination of student and provided files, just like before. This function also returns a manager.

The function run_and_check runs a program with one argument and checks the output that was written to standard output. The first argument is a name used in reporting. The second is the command line argument. The third is the filename for the answer. This file must reside in directory tests/expected. We also see a positional argument cmp, which influences how strict we compare for equality. By default, the output should be exactly equal to the expected output, but with LOOSE, empty lines or extra white space do not matter.

Next we compile a second program, with unit tests, that checks if an exception is thrown on invalid input. Building and running is like in the previous test. As second argument of run_with_status we explicitly pass an empty list of command line arguments. You can leave that out.

Currently, you can have at most three simple builds and at most three xunit builds in a script. If you try to do more, an exception is thrown. This, rather arbitrary, limitation comes from trying to keep build times low.

The contents of check_correct.cs is:

using GradingUtils;

public class Test
{
  static int Main(string[] args)
  {
      string input_file = args[0];
      using (FileStream input_stream = File.OpenRead(Paths.TestInput(input_file)))
      {
          // Execute student code
          Answer.AddAll(input_stream, Console.Out);
      }
      return 0;
  }
}

In the check_correct.cs file we see a main function which uses its command line argument. This is the argument that was supplied by the python script (input1 or input2, in the example). The function Paths.TestInput from GradingUtils, translates this argument to a path, where a file can be read. The argument must be a filename in tests/input. The rest of the file is standard C# code.

The contents of check_invalid.cs:

using GradingUtils;
using Shouldly;

public class Test1
{
  [Theory]
  [InlineData("invalid1")]
  [InlineData("invalid2")]
  [InlineData("invalid3")]
  public void InvalidInput(string filename)
  {
      using (MemoryStream output = new MemoryStream())
      {
          using (FileStream stream = File.OpenRead(Paths.TestInput(filename)))
          {
              Should.Throw<Exception>(() =>
                      {
                          Answer.AddAll(stream, new StreamWriter(output));
                      });
          }
      }
  }
}

This code checks if student code throws some exception if the input is invalid. We use the construct Should.Throw from the Shouldly library to do this. Because this test uses the xunit, which supplies its own Main method, we can’t use command line arguments. We can check with different input files. Here we use three. The whole test fails if one of the test cases fail.

Tutorial¶

Xunit example¶

An Example with File Input¶

C# Grader

Navigation

Related Topics