Appendix E — Using AI in Student Assessment: the vpl-ai-feedback Project

This appendix describes vpl-ai-feedback, an open-source system developed to support the grading of code submissions sent to VPL (Moodle) through feedback generated by Artificial Intelligence (AI). Unlike a traditional automatic grader — which only compares outputs against test cases —, vpl-ai-feedback sends the student’s code, together with the question rubric, to a language model (LLM), which returns a qualitative analysis per criterion, in the style of Socratic feedback: the student receives comments pointing out what was done correctly, what can be improved and why, in both formative activities (practice exercises) and assessment activities (mock exams) in VPL.

This Socratic feedback mechanism for VPL activities was proposed by Zampirolli et al. (2026). The study describes the integration of AI into the assessment process for parameterized programming exercises in VPL/Moodle, using a set of seven LLMs adopted to analyze student code and generate automated feedback on every grading request, with preliminary results indicating a positive student perception regarding idea generation, clarity of explanations, and learning autonomy.

The rest of this appendix details the practical implementation of these ideas in the vpl-ai-feedback project — whose code is available at https://github.com/fzampirolli/vpl-ai-feedback — and specifically describes its use in the three PDI-VC classes between May and August 2026.

WarningWarning: AI does not replace instructor judgment

Using AI to grade students is not recommended as the sole source of a grade, since language models can hallucinate — that is, produce plausible but incorrect assessments, criteria, or justifications, even for correct or incorrect code. An LLM may give a high grade to code with a subtle bug, or a low grade to correct code written in an unusual way, simply because the generated explanation “sounds” coherent. AI should be used only as a complement to assessment, never as a substitute for the instructor. Appendix A and Appendix B describe the objective grading mechanisms (test cases) that remain the basis of the official grade; this appendix deals exclusively with the use of AI as an additional layer of qualitative feedback.

E.1 Context of use

vpl-ai-feedback was used in three sections of the PDI-VC course, taught between May and August 2026, totaling approximately one hundred students. AI feedback was used in three complementary ways:

1. Continuous feedback in formative activities — practice exercises submitted to VPL throughout the course, in which the student could resubmit code as many times as desired. On each submission, they received qualitative AI-generated feedback, in addition to the objective grading performed by VPL, described in Zampirolli et al. (2026).

2. Complementary feedback in mock exams (bonus assessment activities) — four mock exams held throughout the term, planned in the syllabus as bonus activities, each worth up to 5% of the final grade. Applied approximately 30 minutes before the end of the biweekly lab practical classes, the mock exams used vpl-ai-feedback to generate an individual assessment rubric, sent by e-mail together with a comparative summary between the grade assigned by Moodle/VPL and the assessment produced by the AI.

3. Complementary feedback on Exams 1 and 2 (official assessments) — the term’s two formal exams, each worth a larger share of the course’s final grade, also began using vpl-ai-feedback after they were applied. Unlike the mock exams (bonus), here the official grade is already determined by VPL’s objective grading and/or the instructor’s manual review before results are released; the AI-generated rubric is sent to the student only afterward, as supporting material for reviewing their own performance — reinforcing, also in the higher-weight assessments, the separation between official grade and complementary feedback detailed in Section D.7.

An opinion survey, answered by 102 students before the system was adopted, indicated high acceptance of the proposal. Only 2 students gave a rating of 1 (rejection) and 7 gave a rating of 2 regarding interest in receiving automatic AI code feedback. Another 19 declared themselves indifferent (rating 3), while the majority — 38 and 37 students — gave ratings of 4 and 5 (high approval), respectively, totaling the 102 respondents. This result motivated the adoption of the system as a complement to the formative process, not as a substitute for human grading.

ImportantThe official grade was never the AI’s grade

It is essential to stress that, as defined in the syllabus, the grade used both for calculating each mock exam’s bonus and for the official grade of Exams 1 and 2 was always the grade assigned by VPL (based on the test cases) and/or by the instructor’s manual review — and not the grade suggested by the AI. The rubric generated by vpl-ai-feedback was sent to the student only as supporting learning material — never as a criterion for assigning a grade, whether in bonus activities or official assessments. This separation between official grade (VPL/instructor) and complementary feedback (AI) is the central principle of this appendix and was revisited in Section D.8.

E.2 Project architecture

vpl-ai-feedback is an asynchronous Python script organized as follows:

.
├── config.yaml              ← configuration (credentials, provider, weights) — NEVER versioned
├── config.yaml.example      ← public configuration template
├── main.py                  ← main script (async), orchestrates the whole process
├── run.sh                   ← execution wrapper (bash run.sh config.yaml)
├── gerar_relatorio.py       ← converts the consolidated *_ALL.txt into CSV
├── enviar_email.py          ← sends the rubrics by e-mail to each student
├── providers/                ← LLM API clients
│   ├── __init__.py          ← client factory (Factory)
│   ├── base.py               ← retry, backoff, and fallback logic between models
│   ├── groq.py
│   ├── deepseek.py
│   └── gemini.py
├── core/                      ← business logic
│   ├── grader.py              ← identifies files by question, builds the prompt, calls the LLM
│   └── utils.py
└── <class_folder>/          ← submissions downloaded from Moodle VPL
    ├── Student name - login/
    │   ├── <timestamp>/                 ← student's latest submission
    │   │   ├── Q1.py                    ← pattern for exams with multiple questions
    │   │   ├── Q2.py
    │   │   └── rubrica.txt              ← generated by the LLM (only if the assessment is valid)
    │   └── <timestamp>.ceg/
    │       └── execution.txt            ← objective grade assigned by VPL
    └── ...

The system accepts three LLM providers — Groq, DeepSeek, and Gemini — configurable in config.yaml via the llm.provider key. Each provider can have multiple registered models; on each call, the list is shuffled, distributing the load among students processed in parallel and reducing the chance of rate-limit errors (HTTP 429). When a model fails, the system retries up to three times before moving to the next one in the list, respecting the wait time suggested by the provider. Authentication errors (401) or insufficient-balance errors (402) immediately abort the attempt with that provider.

An important aspect of the project concerns data privacy. The student’s name, e-mail, login, student ID, and national ID are never sent to the LLM API. Only the following are transmitted for processing: the name of the file containing the source code; the code itself (with comments removed); the rubric prompt — which contains no personal data —; and, when available in the execution.txt file generated by VPL, the official statement of the question drawn for that student and the raw result of the test cases actually executed (input, output produced by the code, and expected output). These last two items, although extracted from a file specific to each student, also contain no personal identification — only the question text and the input/output values of the automated tests —, and are used as objective evidence to reduce the risk of the AI incorrectly identifying the question type or assessing the code’s logic based solely on a static reading (Section D.3).

Even so, it is important to recognize that the three currently supported providers — Groq, DeepSeek, and Gemini — are commercial services operated by foreign companies, whose models run on infrastructure outside the country and are subject to usage, availability, and cost policies beyond the institution’s control. This dependence on external providers carries risks that go beyond one-off code grading: price changes, model discontinuation, temporary service unavailability, unilateral changes to terms of use, and, in more sensitive cases, geopolitical access restrictions could compromise the continuity of a grading system that becomes structurally dependent on these APIs. For this reason, it is strategic for Higher Education Institutions (HEIs) such as UFABC to invest in developing and hosting their own AI providers — for example, open-weight models run on local infrastructure or a sovereign cloud —, reducing dependence on external services, especially from other countries, and expanding institutional control over cost, availability, student data privacy, and long-term pedagogical continuity. The design of vpl-ai-feedback already favors this path: the providers/ architecture was built as an extensible factory, in which a new provider — including a model hosted locally by the institution itself, with a compatible interface — can be added with low integration effort.

E.3 The universal prompt and the two-stage rubric

Each exam type is described in a prompt file (for example, Simulado4.txt), referenced in grading.prompt_file in config.yaml. This file instructs the LLM to perform the assessment in two stages:

1. Question identification — the LLM determines the specific operation type drawn for that student (useful when there are parameterized questions drawn and shuffled per student, as described in Appendix A);
2. Applying the corresponding rubric — the LLM assesses the code against scored criteria, each with a maximum point range, ultimately returning a grade in the format FINAL GRADE: X/WEIGHT.

For parameterized exams, the prompt file usually contains several parallel rubrics, one for each possible question type, delimited by markers such as [START_RUBRICA_TIPO_A] / [END_RUBRICA_TIPO_A]. The LLM first identifies which type was drawn for that student and applies only the corresponding rubric, ignoring the others — which prevents criteria from a different type from contaminating the grade.

When the exam has more than one question per VPL activity, the code files must follow the pattern Q1.*, Q2.*, etc. — one file per question, with the extension free according to the language chosen by the student —, and each question has an individually configurable weight in grading.weights in config.yaml. When the exam has a single question and was not generated by MCTest, the system accepts any file name with a supported extension, as long as it is the only code file present in the submission folder.

In the case of Mock Exam 4 (topic Morphological Operators), for example, one of the possible types defined three criteria, with a total weight of 100 points for that question:

Criterion	Description	Maximum score
1 — Data input	Reading H, W, and the binary matrix (mm.readImg)	25 pts
2 — Morphological processing	Opening + Closing (mm.sebox()), mm.measure, sorting by bbox[0]/bbox[1], and reassigning IDs	50 pts
3 — Output and formatting	Displaying the cleaned image (mm.drawImg) and the formatted measurement table	25 pts

The prompt also requires a mandatory response format (line width, section order, the FINAL GRADE: marker), which makes automatic grade extraction and the consolidated report generation more reliable — although, as discussed in Section D.8, this does not eliminate the risk of semantic error in assessing the content.

E.4 Two additional layers of evidence

To reduce the risk of the LLM hallucinating the question type or the correctness of the code — the central risk discussed in Section D.7 —, vpl-ai-feedback started providing the AI with two sources of objective evidence, automatically extracted from VPL’s own execution.txt, in addition to the source code:

• Official question statement. Since questions are usually drawn and shuffled per student in MCTest (Appendix A), each student’s execution.txt already contains, in the “Question summary” field, the exact text of the question they were assigned. The system automatically extracts this excerpt and sends it to the LLM as primary evidence for identifying the question’s type/variation — more reliable than inferring solely from a static reading of the code, especially in incomplete submissions or those ambiguous between two close types (for example, “flat” erosion versus “weighted” erosion). When the submitted code diverges from what the statement asks, the system does not ignore the statement nor silently “correct” the reading: it keeps the identification based on the statement, lowers the assessment’s confidence to “Low,” and makes the divergence explicit in the feedback sent to the student.
• Actual VPL execution result. When available, the system also sends the LLM the test cases actually run on that code — input, output produced, and expected output — as objective evidence of correctness, to be used to confirm or refute the reading of the logic before scoring the processing and output criteria, instead of the AI relying solely on mentally simulating the code (which is especially error-prone in loops with min/max, index offsets, or Otsu threshold computation, where subtle errors go unnoticed in a static reading).

In addition, the LLM is instructed to explicitly state its confidence level (High, Medium, or Low) in identifying each question type, with justification when confidence is not High. This value is automatically extracted and feeds a Revisar_Manualmente (Review Manually) column in the consolidated CSV report (Section D.4), allowing the instructor to prioritize exactly the cases where the AI itself recognizes greater uncertainty — instead of treating all of a class’s assessments as equally reliable.

E.5 Execution flow

The typical run, done via bash run.sh config.yaml, follows these steps:

1. Loads config.yaml and locates the submissions folder (paths.student_base_dir);
2. For each student, identifies the latest submission (the folder with the most recent timestamp);
3. Extracts the objective Moodle grade from *.ceg/execution.txt;
4. For each question of the exam, locates the code file (Q1.*, Q2.*, or the single file, for an exam not generated by MCTest), builds the prompt (code + rubric), and sends it to a model drawn from the configured list;
5. Processes all students in parallel, with concurrency control;
6. Generates, per student, the rubrica.txt file — only if the AI returns at least one valid assessment for the questions that have code; otherwise, the file is not saved, forcing a retry on the next run;
7. Consolidates all rubrica.txt files into a single *_ALL.txt and generates a *_relatorio.csv report, comparing the Moodle grade, the AI grade, and the difference between them, per question and in total.

Situation	Is rubrica.txt saved?
All questions with code successfully assessed	✅ Yes
AI failed on at least one question with code	❌ No — reprocessed on the next run
Student submitted no code file	❌ No
Question type identified as UNKNOWN	❌ No

E.6 Illustrative example: rubric for a three-question exam

NoteAbout the origin of this example

The following excerpt does not reproduce any specific student’s code. It is an example reconstructed by the author, reproducing an error pattern repeatedly observed across PDI-VC classes — confusion between a 2D morphological function (mm.ero/mm.ero0) and a 1D operation on signals — without citing or identifying any actual submission. This choice avoids any risk of infringing a student’s copyright over their own source code, even anonymized, since authorship of the work remains with the original author even when name and login are removed.

The following excerpt illustrates the output generated by vpl-ai-feedback for a student on a three-question exam (Q1, Q2, Q3), using the deepseek-chat model. The comparative summary appears at the top of the file, followed by the official statement of each question, the submitted code, and the assessment per criterion.

┌─────────────────────────────────────────────────────────────────────┐
│ SUMMARY — AI (deepseek-chat)  x  MOODLE                             │
├─────────────────────────────────────────────────────────────────────┤
│ Total weight : 100 pts                                               │
│ ─────────────────────────────────────────────────────────────────── │
│ Q1 (AI) :  3.3 / 33 pts                                              │
│ Q2 (AI) :  6.7 / 33 pts                                              │
│ Q3 (AI) : 33.3 / 34 pts                                              │
│ ─────────────────────────────────────────────────────────────────── │
│ Moodle : (Q1=33 + Q2=0 + Q3=34) = 67 pts                             │
│ AI     : 43.3 / 100 pts                                              │
│ ─────────────────────────────────────────────────────────────────── │
│ Difference (AI - Moodle): -23.7 pts                                  │
│ ─────────────────────────────────────────────────────────────────── │
│ Q1: identified type = B  | confidence = Low (code does not          │
│   match the question statement) ⚠️ REVIEW                            │
│ Q2: identified type = A  | confidence = High (official statement    │
│   confirms the type and variation)                                   │
│ Q3: identified type = C1 | confidence = High (official statement    │
│   confirms 2D erosion with integrated OpenCV)                        │
└─────────────────────────────────────────────────────────────────────┘

Q1 — code does not match the statement (low confidence): the question’s official statement, extracted from execution.txt, asked for a 1D morphological erosion on a signal (reading N, M, the signal, and structuring element B, followed by the minimum of the active neighbors). The submitted code, however, only reads two integers and calls mm.ero0 — a 2D erosion function for images, unrelated to the requested operation.

Identified type: TYPE B (1D Erosion — flat)
Confidence: Low — code does not match the question statement (uses
mm.ero0, a 2D erosion function, instead of implementing flat 1D
erosion on a signal)

Criterion 1 - [3.33/10.00 pts]:
- The code reads two integers on a single line (which would correspond
  to N and M), but does not read the signal or the structuring element
  B as 1D vectors. Instead, it treats both as images via mm.readImg,
  which does not match the expected reading of a 1D signal and of B on
  separate lines.

Criterion 2 - [0.00/16.67 pts]:
- The operation applied is mm.ero0(img, element), a 2D erosion for
  images, not the 1D signal erosion requested in the statement. The
  logic of the minimum of active neighbors with border cropping is not
  implemented.

Criterion 3 - [0.00/6.67 pts]:
- The output is a 2D image via mm.drawImg, not the resulting 1D signal
  on a single line with integer values separated by spaces, as
  required.

FINAL GRADE: 3.33/33.33

Comments for the student: The submitted code does not match what the
statement asks. The drawn question requests a 1D morphological erosion
on a one-dimensional signal; the code uses a 2D erosion function for
images. Re-read the assigned statement carefully before choosing the
implementation.

Q2 — syntax error prevents execution (high confidence, low grade due to a real failure): here the AI correctly identified the type (Otsu thresholding), supported by the official statement, but the code contained an indentation error that prevented execution — evidenced both by the static reading and by the absence of valid output in the VPL tests.

Identified type: TYPE A (Thresholding — manual T + Otsu)
Confidence: High — official statement confirms the type and variation

Criterion 1 - [6.67/10.00 pts]:
- Correctly reads the three parameters L, C, and T on separate lines
  and the input image. However, the definition of a helper thresholding
  function is incorrectly indented within the global scope, causing an
  indentation error that prevents the program from running completely.

Criterion 2 - [0.00/16.67 pts]:
- The fixed-threshold T binarization is correctly structured, but the
  Otsu threshold computation depends on the malformed helper function,
  which references a variable out of scope. The indentation error
  prevents any processing.

Criterion 3 - [0.00/6.67 pts]:
- Due to the indentation error, no output is produced.

FINAL GRADE: 6.67/33.33

Comments for the student: The overall structure is correct (reading L,
C, T, and the image), but there is an indentation error that prevents
execution. Review the Python syntax before submitting and test the code
locally before sending it to VPL.

Q3 — function mismatch, but functional equivalence confirmed by real evidence (high confidence, fair grade): the statement explicitly asked for the use of mm.ero (OpenCV-integrated version), but the code uses mm.ero0 (manually implemented flat version). Since execution.txt showed all 10 VPL test cases passing, the AI did not penalize the function mismatch — it recognized the functional equivalence demonstrated by the real tests, but still noted the observation in the feedback.

Identified type: TYPE C1 (2D Erosion — flat, no weights)
Confidence: High — official statement confirms 2D erosion with
integrated OpenCV (mm.ero), and the code implements flat erosion via
mm.ero0

Criterion 1 - [10.00/10.00 pts]:
- Correctly reads height, width, the image matrix, and the dimensions
  and values of structuring element B.

Criterion 2 - [16.67/16.67 pts]:
- The statement asked for the use of mm.ero (integrated OpenCV), but
  the code uses mm.ero0 (flat, no weights). Despite the function
  mismatch, the operation was correctly implemented with the structuring
  element read: all 10 tests run on VPL passed completely (10/10),
  confirming functional equivalence for the tested cases.

Criterion 3 - [6.67/6.67 pts]:
- Prints the resulting image in the correct format; the tests confirmed
  the correct output.

FINAL GRADE: 33.34/33.33

Comments for the student: The code is correct and functional, with all
10 tests passing. Note: the statement explicitly asked for mm.ero
(OpenCV version), but you used mm.ero0 (flat version). Although the
results were equivalent in the tests, follow exactly the function
requested in the statement in future exams.

Note that, in this composite case, the AI assigned 23.7 fewer points than VPL’s objective grade, driven mainly by Q1. The aggregate difference alone would already be a signal for investigation (Section D.7), but the system itself hands the instructor the likely cause of each question: the Revisar_Manualmente column of the CSV is marked “YES” whenever at least one of the student’s questions receives low confidence, with the specific reason recorded in the corresponding confidence column — reducing the time the instructor needs to spend finding, among a hundred students, the few cases that actually deserve manual attention before the official grade is assigned. The case of Q3, in turn, illustrates the opposite care: the divergence between statement and code did not result in undue penalization, because the real execution evidence confirmed the functional equivalence — exactly the behavior that Section D.3 describes for these two layers of evidence.

E.7 Sending feedback by e-mail

After the rubrics are generated, the enviar_email.py script sends each student an e-mail with rubrica.txt attached. The e-mail body is defined in config.yaml, under templates.corpo, and is interpolated with {nome_pasta} and {login}:

email:
  smtp_server: smtp.ufabc.edu.br
  smtp_port: 587
  from_address: professor@ufabc.edu.br
  password: "YOUR_PASSWORD_HERE"     # never version real credentials
  use_tls: true

templates:
  assunto: "LLM-Generated Rubrics and Grading - Mock Exam - {login}@aluno.ufabc.edu.br"
  corpo: |
    Dear {nome_pasta},
    ...

ImportantCredential security

config.yaml contains real secrets (SMTP password, API keys). It must be listed in the project’s .gitignore and never be published, shared, or versioned — including in e-mail attachments, screenshots, or public repositories.

A real example of an e-mail sent to a student (data anonymized, with name and login replaced by a fictitious case) is reproduced below, to illustrate the didactic tone and the caveats made explicit to the student:

ImportantExample of an individual message (anonymized)

Dear [Student Name] - [login],

Your grade for Mock Exam 4 is available on Moodle.

Below I’m sending the competencies assessed and a detailed review of your code, automatically generated by Artificial Intelligence (AI).

The “rubrica.txt” attachment contains the AI’s full feedback (I recommend downloading it and opening it with a text editor or notepad).

Please note that this AI-generated review MAY contain inaccuracies or errors, but it serves as excellent support for your learning process in the course.

A pedagogical suggestion is to submit your code (contained in this file), together with the RUBRIC below, to other LLM models for comparison. This can help identify possible discrepancies, in addition to offering different perspectives on your code and on the assessment criteria.

If you have questions or notice any glaring inconsistency in the review shown on Moodle, I am available for clarification.

Best regards,

Prof. Francisco Zampirolli

PS.: Explaining the process: the latest version of your code saved on Moodle was attached to the prompt and sent to one of the LLMs chosen in an integrated way by our assessment system (models = (“deepseek-chat”)). The process is repeated until a response with a valid grade is obtained (between 0 and 100).

Three didactic elements deserve attention in this text: (i) the explicit warning that the review may contain errors; (ii) the invitation for the student to contrast the assessment with other LLMs, turning the AI into an active study tool rather than a passive verdict; and (iii) the transparent explanation of the automated process, including the model(s) used and the reprocessing policy until a valid grade is obtained.

E.8 Risks, ethical limits, and instructor responsibility

ImportantThe instructor cannot simply adopt the AI’s grade

This is the central point of this appendix: under no circumstances should the grade suggested by the AI be automatically assigned to the student as the official grade. Language models can hallucinate — award points for criteria not met, “invent” that a function was called correctly when it was not, or penalize correct code due to misinterpretation of the rubric. The final grade for any assessment activity must always result from the instructor’s manual review (or from objective grading via test cases, as in VPL/MCTest), with AI serving, at most, as a first pass or as support material for the student, never as the decision-making authority.

Some practical precautions adopted in the use of vpl-ai-feedback, recommended to any instructor wishing to adapt the system:

• Clear separation between the official grade and AI feedback. In the case reported, the mock-exam bonus grade always came from VPL, never from the AI (Section D.1). The AI rubric was communicated to the student as supporting material, with an explicit warning that it may contain errors.
• Transparency with the student. The e-mail sent (Section D.6) makes explicit that the assessment was AI-generated, states which model(s) were used, and invites the student to compare with other LLMs — reducing information asymmetry and encouraging critical thinking about the review they received.
• Reprocessing instead of an invalid cache. The system only writes rubrica.txt when it obtains a valid assessment (Section D.4); this prevents a malformed, incomplete, or visibly inconsistent AI response from being presented to the student as if it were final.
• Monitoring discrepancies. The consolidated CSV report (the Diferenca = Total_IA - Total_Moodle column) allows the instructor to quickly identify the cases where AI and objective grading disagree the most — as in the example in Section D.5 (+20 points) —, prioritizing those cases for manual review.
• Open channel for disputes. The final e-mail explicitly offers the student the option to report inconsistencies, keeping the instructor as the final authority over the grade.
• Personal data kept out of the prompt. As described in Section D.2, name, e-mail, login, student ID, and national ID are never sent to the LLM API, reducing privacy risks when using external AI providers.

E.9 Final remarks

vpl-ai-feedback shows how AI can enrich the feedback cycle in programming courses with a high volume of submissions, offering each student a qualitative, individualized reading of their own code — something unfeasible to do manually for a hundred students, across multiple questions and exams throughout the term. The opinion survey cited in Section D.1 suggests that this type of feedback is well received by students. Even so, responsible use of the system depends on a non-negotiable principle, reiterated throughout this appendix: AI complements, but does not replace, the instructor’s assessment, and the official grade for any assessment activity must continue to be defined by objective grading (VPL/MCTest, Appendices A and B) and/or human judgment — never by the raw grade returned by a language model.

ZAMPIROLLI, Francisco de Assis et al. Intelligent feedback for individualized introductory programming exercises. Computer Applications in Engineering Education, v. 34, n. 1, p. e70132, 2026.