Advanced Evaluator Techniques

In CreateJavaParts you have seen a simple verison of an Evaluator. It has shown how to deal with a two case testing:
  • everything is right -> full score
  • at least one error occured -> zero score

This part will show you some techniques that have been developed for partial testing of the user's code. That means if not all tests are passed by the user's code he can still achieve a score greater than zero. This will be shown on a croncrete problem

problem task: compare two strings, and return which one is shorter

Test if user's code recognises following correctly
  • string1 is shorter than string2
  • string1 and string2 have the same length
  • string1 is shorter than string2

In the example code at the end of this article we used fixed testinputs for usercode to make it easier to read, but whereever possible you should use randomised testinputs.

We use the Variable "points" to determine the score/exitcode (which is used aquivilant here).

Each subtest that the usercode passes will raise the score.

Since history has shown that usercode can produce various exceptions, most often nullpointerExceptions even on very simple tasks, it is highly recommended to surround each subtest with its own try-catch-block. This way you are able to give some feedback on which part didn't work and also give some score for other subtests that succeeded.

---------------------------------------------------------------
Example Code for evaluator
---------------------------------------------------------------


import static net.mumie.srv.corrutil.CorrUtil.explanation;

public class CompareStringLengthEvaluator
{
    public static void main(String[] args)
    {

        CompareStringLengthWrapper wrapper = new CompareStringLengthWrapper();

        int points = 0;
        int expectedResult = -1;
        int userResult;
        String s1 = "xy";
        String s2 = "abc";
        String statement = "";

        try
        {
            try
            {
                userResult = wrapper.compareStringLength(s1, s2);
                if (expectedResult == userResult)
                {
                    statement += "This method works correctly when String s1 is shorter than s2. -> +1 point <br/>";
                    points += 2;
                }
                else
                {
                    statement += "This method does not work correctly when String s1 is shorter than s2. <br/>";}
                }
            catch (Exception e)
            {
                statement += "This method does not work correctly when String s1 is shorter than s2. <br/>";
            }

            try
            {
                s1 = "xyz";
                expectedResult = 0;
                userResult = wrapper.compareStringLength(s1, s2);
                if (expectedResult == userResult)
                {

                    statement += "This method works correctly when String s1 is as long as s2. -> +1 point <br/>";
                    points += 2;
                }
                else
                {
                    statement += "This method does not work correctly when String s1 is as long as s2. <br/>";
                }
            }
            catch (Exception e)
            {
                statement += "This method does not work correctly when String s1 is as long as s2. <br/>";
            }

            try
            {
                s2 = "ab";
                expectedResult = 1;
                userResult = wrapper.compareStringLength(s1, s2);
                if (expectedResult == userResult)
                {
                    statement += "This method works correctly when String s1 is longer than s2. -> +1 point <br/>";
                    points += 2;
                }
                else
                {
                    statement += "This method does not work correctly when String s1 is longer than s2. <br/>";
                }
            }
            catch (Exception e)
            {
                statement += "This method does not work correctly when String s1 is longer than s2. <br/>";
            }
        }

        catch (Throwable throwable)
        {
            points = 0;
            throwable.printStackTrace();
        }

        explanation(statement);
        System.exit(points);
    }
}

In this example we have used only one available option of the explanation command. A more extended command-usage can be found in this note

Add picture from clipboard (Maximum size: 500 MB)