Empowering Educators: Grading Automated Assignments with ChatGPT, an Azure OpenAI Service (2023)

Exceedaccording to the author

Learn how AI technology can simplify and improve the assessment process for teachers using Azure OpenAI ChatGPT. Cyrus Wong, Senior Lecturer at the Hong Kong Institute of Vocational Education, shares insights on using artificial intelligence to automate assessment of assignments and improve the quality of teaching. By using AI tools like Azure OpenAI Service ChatGPT, educators can save time grading assignments and focus on creating engaging learning experiences for students.

With the powerful combination of AI technology and assessment automation, educators can significantly reduce their workload, gain valuable insights into student responses, and provide meaningful feedback to enhance the learning experience. Explore the capabilities of Azure OpenAI ChatGPT to revolutionize the way assignments are graded in education.

Being a teacher, especially in Asia, often requires a lot of homework from students to meet parents' expectations and maintain the school's reputation. While I won't go into the implications of this cultural norm, I must admit that being a teacher in this environment can be challenging due to the enormous workload involved in grading countless assignments and providing feedback. This situation essentially multiplies the workload a hundredfold, similar to self-distributed denial of service (DDoS) attacks.

Teachers are often so busy grading assignments that they have little time to improve the quality of their teaching. Rather than focusing primarily on assessment, it is critical to take the time to design engaging and meaningful learning experiences for students.

Educators should consider using AI technologies such as Azure OpenAI Service ChatGPT to assist or automate assessment tasks. The AI ​​demonstrated consistent and reliable reasoning and processing and produced impressive results in automated job evaluation. Additionally, it can analyze student responses, providing valuable insights to improve their learning experience.

This is a use case where the Azure OpenAI service can be used to automatically assign reviews.

Evaluation of paper presentations in Microsoft Word and Adobe PDF format from Moodle LMS

Assume that each student's work is stored in a separate folder with their name and that folder contains at least one Word or PDF file.

For the Microsoft Word commands we usedocx2txt libraryConvert the content to text and store it in a Pandas DataFrame column labeled "Answers".

def filter_df_by_contains_docx(df): return df[(df["ContainsDocxFile"] == True)]words_df = filter_df_by_contains_docx(df)paths = words_df["Path"].valuesdef get_all_docx_files(path): 导入 glob return glob.glob(path ) + "/*.docx")Importeer docx2txtvan Functools uvoz reducestudents_words_files = popis (map (map (get_all_docx_files, staze)) # lijst 遇到 了 了 了 lIdsten van word datoteke (. y: x + y, karta(lambda f: docx2txt.process(f), word_files), "\n\n"))

For PDF files we use importPyPDF2 libraryConvert the content to text and store it in a Pandas DataFrame column labeled "Answers".

def filter_df_by_contains_pdf(df): return df[(df["ContainsPdfFile"] == True)]pdfs_df = filter_df_by_contains_pdf(df)paths = pdfs_df["Path"].valuesdef get_add_pdf_files(path): 导入 glob 返回 glob.glob(路径+ "/*.pdf") import PyPDF2from functools import reducef convert_pdf_all_pages_to_txt(path): pdfFileObj = open(path, 'rb') reader = PyPDF2.PdfReader(pdfFileObj) num_pages = len(reader.pages) count = 0 text = " " while count < num_pages: pageObj = reader.pages[count] count += 1 text += pageObj.extract_text() text += "\n\n" return textstudents_pdf_files = list(map(get_add_pdf_files, paths)) # 列表lijsten遇到了 woordbestandenfile_contents =[];voor pdf_bestanden in studenten_pdf_bestanden: file_contents.append(reduce(lambda x, y: x + y, map(convert_pdf_all_pages_to_txt, pdf_files), "\n\n"))pdfs_df.loc[:, " Bronnen "] = students_pdf_filespdfs_df.loc[:, "Odgovori"] = file_contents

We can then use Azure OpenAI Service ChatGPT 3.5 to evaluate the response.

import osimport jsonimport openaiopenai.api_type = "azure"openai.api_base = os.getenv("AZURE_OPENAI_ENDPOINT") openai.api_version = "2023-03-15-preview"openai.api_key = os.getenv("AZURE_OPENAI_KEY")def get_json_chatGpt(学生, 提示): response = openai.ChatCompletion.create( engine="gpt-35-turbo", # engine = "deployment_name". messages=[ {"role": "system", "content": "Je bent een onderwijsassistent."}, {"role": "user", "content": prompt}, ], temperature=0.9, max_tokens=1600, top_p=0​​.0, frequency_penalty=0, present_penalty=0, stop=Ništa ) # print ( antwoord) # 打印(antwoord['choices'][0]['message']['content']) write_text_to_file(f"tmp/{student}.json", json.dumps(antwoord)) tokeni = antwoord[ 'usage']['total_tokens'] 返回 json.loads(response['choices'][0]['message']['content']) , tokensdef grade_answer(student,student_answer, marking_schema): prompt= marking_scheme.replace ( “", student_answer) retry = 0; 而 True: try: content, tokens = get_json_chatGpt(student,prompt) break osim iznimke if e: if retry < 2: retry += 1 print(e) print("ponit: " + str (ponovi)) continue return 0, "Error", 0, 0, True, 0, True marks = content['marks'] comments = content['comments'] copyFromInternet = content['copyFromInternet'] generativeAI = content[ ' generativeAI'] manualReview = content['manualReview'] povratne ocjene, komentari, copyFromInternet, generativeAI, manualReview, tokeni, False def grade_answers(df_answers, rating_scheme): za indeks, red u df_answers.iterows(): student = row[" 学生"] print(student) answer = row["Answers"] ocjene, komentari, copyFromInternet, generativeAI, manualReview, tokens, error = grade_answer(student, answer, mark_schema) df_answers.loc[index, "Marks"] = ocjene df_answers. loc[index, "Komentari"] = komentari df_answers.loc[index, "CopyFromInternet"] = copyFromInternet df_answers.loc[index, "GenerativeAI"] = generativeAI df_answers.loc[index, "ChatGptTokens"] = tokeni df_answers.loc [ indeks, "manualReview"] = manualReview df_answers.loc [indeks, "pogreška"] = 错误 返回 返回 df_answersmarking_scheme = read_text_file ("marking_scheme.txt") df_marced = ocjena (df_answerma, maringswers, maring. , 指数=假)

Rapid design for markup schemes

Act as a teacher and provide grades and comments for written assignments. Task background: Note that you do not have to answer all of these questions. Job Responsibilities and Roles What are my job responsibilities? How do I feel about my place in the workplace? What (or who) influences the way I complete the task? Why do I feel good working at your workplace? Work experience What work experience have I gained? Any work experience that you can remember that is extremely memorable? Learning on the job What technical skills (hardware/software) did I learn? What soft skills did I learn? Have I gained skills relevant to your Advanced Diploma in Cloud and Data Center Management? Comments and suggestions for the workplace Do you have comments about your workplace? Any suggestions you can make to your supervisor/workplace mentor about your workplace's positive impact on WLA (work-based learning and assessment) (eg increasing efficiency, reducing human error, etc.) at school or work can help me. WLAE assessment How to feel about work performance? If I rate myself (out of 100), what is my score? What success have I achieved so far? How does my internship supervisor/mentor view my work? How do my colleagues perceive my work? Career Path/Future Goals What is the career path for my current position? What further goals do you hope to achieve in my current job? Attachments (optional) Certificates or proof of recognition that I received from the workplace? Is there evidence to support my claim? Comment on students' answers below in 150 words. Text separated by 20 equals signs is the student's answer. =============================================== Instructions: Rate the student responses above on a scale of 0-70. Rules: 1. Responsibilities at work and roles 10 points 2. Work experience 10 points 3. Learning at the workplace 10 points 4. Remarks and suggestions on the workplace 10 points 5. Evaluation of work (self/mentor/colleagues) 10 points 6. Career Path/Future Goals 7 out of 10 Format (minimum 800 words but less than 2000 words) 10 points 8. If the answer does not make sense, give 0 points and set manualReview to true. "copyFromInternet" is 0 - 1.0, which is the probability of copying from the Internet. "generativeAI" is 0 - 1.0, which is the probability that the answer was generated by AI. "Comment" includes. 1. Rank calculations are explained in comments 2. Mark for each line is shown line by line in "Comment" 3. Incentive style.4 in "Reviews". "Review" is less than 200 characters. 5. Escape all special characters in "Comment" according to the following 6 rules: Replace \b with \\b Replace \n with \\n Replace \r with \\r Replace \t with \\t Replace \" with \\" Replace \ \\ No interpretation is involved, just return an RFC8259 compliant JSON response that follows this format, not the format below. ++++++++++++++++++++++++ +++ +++ ++++++++++++++++++++ ++++ ++++++++++{ "markers": 30, "copyFromInternet": 0.1, "generativeAI": 0.5, "manualReview": false , "comments": "This is nice!" }++++ ++ +++++++++++++++++++++++++++++++++++++++++++++ +++++++ ++ + + + Don't return anything after the JSON object. JSON response:

1. to make "system informationInstructions: "Act like a teacher and give grades and comments for written assignments."
2. Supply "Provide grounding contextBy introducing "job background".
3. expand"Add clear syntaxBy specifying delimiters and inserting student answers.
4. run"start with clear instructionsBy giving scoring criteria.
5. implement one"Trigger a chain of thoughtReduce the possibility of incorrect grading by explaining the grade calculation and showing each line's labels line by line in the Comments section.
6. use "specify the output format" to produce valid JSON output. However, since it occasionally produces invalid JSON strings in the Comments section, we will adjust the "start with clear instructionsUse the commands "Escape all special characters in 'comment'" and "Return nothing after the JSON object."

We highly recommend that you take a look at Microsoft Learning's Tips Engineering Techniques as they can be effective in solving a variety of problems. Other interesting observations include ChatGPT's ability to detect content copied from the Internet and responses generated by artificial intelligence. However, we are currently using these findings as a reference. Sometimes ChatGPT can return invalid JSON because it ignores the "don't return anything after the JSON object" directive. To solve this, we implemented a retry mechanism andtemperatureto 0.9, so the return value will change slightly on retries.

Example output:

84 boda

"Excellent! Your response is well structured and clearly demonstrates your experience as a Business Analyst Trainee at Codefreesoft. Your reflection on your experience and the skills you have acquired is commendable. Your understanding of the work environment is insightful, your suggestions for improvement are practical and show a willingness to improvement. Your assessment of your own skills and performance on the job is fair and shows self-awareness. Overall, your answers are complete and satisfy the questions. All requirements. Keep it up!”

48 points

"This is a well-written reflective journal that captures an individual's learning experience as an IT system administrator. The responses highlight the importance of effective communication, collaboration, and continuous learning in the field of data analytics. The individual reflects on their work experience and demonstrated an understanding of the tools and software used in the industry .Responses were clear and within the required word count. However, there was improvement in providing more specific examples to support their points and expand on their recommendations for improving workplace effectiveness. Overall, individuals demonstrated a positive attitude toward their careers and future goals. It was clear that they are committed to continuous learning and development in the field of data analytics.”

Ato incorporateis a special data display format that can easily be used in machine learning models and algorithms. Embedding is an information-dense representation of the semantics of a part of the text. Each embedding is a vector of floating-point numbers, so the distance between two embeddings in the vector space is related to the semantic similarity between the two inputs in the source format. For example, if two texts are similar, then their vector representations must also be similar.

We followTutorial: Explore Azure OpenAI service embedding and find documentationMicrosoft Learn, get embeds for every student answer.

K-means uses clustering of student responses

从 sklearn.cluster 导入 KMeansmatrix = np.array(df_embeddings["ada_v2"].to_list())n_clusters = 7kmeans = KMeans(n_clusters=n_clusters, init="k-means++", random_state=42, n_init='auto') kmeans.fit(矩阵)labels = kmeans.labels_df_embeddings["Cluster"] = labels df_embeddings.head()

Clustering techniques group students with relatively similar answers into a cluster, which helps us identify problematic cases in smaller groups.

Visualize data to improve understanding of student performance

T-distributed embeddings of random neighbors.

t-SNE [1] is a tool for high-dimensional data visualization. It converts the similarity between data points into joint probabilities and attempts to minimize the Kullback-Leibler divergence between low-dimensional embeddings and joint probabilities for high-dimensional data. The cost function of t-SNE is not convex, that is, different initializations can give different results.

import seaborn kao snsfrom sklearn.manifold import TSNEimport matplotlib.pyplot kao pltplt.rcParams['figure.figsize'] = (15, tsne = TSNE(n_components=2, perplexity=5, random_state=42, init='random', learning_rate =200)vis_dims2 = tsne.fit_transform(array)x = [x za x,y u vis_dims2]y = [y za x,y u vis_dims2]paleta = sns.color_palette("inferno", 20).as_hex( ) za kategorije, boje u nabrajanju (paleta): xs = np.array(x)[df_embeddings["Cluster"]==category] ys = np.array(y)[df_embeddings["Cluster" ]==category] plt.scatter (xs, ys, color=boja, alpha=0.1) avg_x = xs.mean() av​​g_y = ys.mean() plt.scatter(avg_x, avg_y, marker= 'x', color=color, s= 100) plt .title("Ugradnje vizualizirane pomoću t-SNE")

Principal component analysis for embedding and visualization of clustering results

Principal Component Analysis (PCA) - Linear dimensionality reduction using singular value decomposition of data to project it into a lower dimensional space.

从 sklearn.decomposition import PCApca_df = df_embeddings.copy()matrix = pca_df["ada_v2"].to_list()pca = PCA(n_components=3)vis_dims = pca.fit_transform(matrix)pca_df["embed_vis"] = vis_dims.tolist ()pca_df

Calculates a ratio that captures the total variance of each principal component.

print(str(sum(pca.explained_variance_ratio_)*100)+"%")

Analysis of variation in the interpreted variance ratio

import numpy if npnums = np.arange(14) var_ratio = [] for num in numbers: pca = PCA(n_components=num) pca.fit(matrix) var_ratio.append(np.sum(pca.explained_variance_ratio_)) import matplotlib. pyplot is plt.figure(figsize=(4,2),dpi=150)plt.grid()plt.plot(nums,var_ratio,marker='o')plt.xlabel('n_components')plt.ylabel(' explained variance ratio') plt.title('n_components versus explained variance ratio')

3D diagram of the 3 main components

[ "Cluster"].to_list()# Plot elke steekproefcategorie afzonderlijk, zodat we label name.for i, clusterId kunnen instellen in opsommen(clusters): sub_matrix = np.array(pca_df[pca_df["Cluster"] = = clusterId] [ "embed_vis"].to_list()) x=sub_matrix[:, 0] y=sub_matrix[:, 1] z=sub_matrix[:, 2] kleuren = [cmap(i/len(clu​​sters) ))] * len( sub_matrix ) ax.scatter(x, y, zs=z, zdir='z', c=colors, label=clusterId) studenten = pca_df[pca_df["Cluster"] == clusterId].index .values.tolist() 对于i, txt u opsommen(studenten): ax.text(x[i], y[i], z[i], txt, size=8, zorder=1, color='k' )ax.set_xlabel(' x ')ax.set_ylabel('y')ax.set_zlabel('z')

source code:AzureOpenAIChatGTPAutoGrader

By using embedding, we can gain valuable insights into student responses, helping us better understand their thought processes, knowledge levels, and areas where additional support or guidance may be needed.

Project staff includesShine Seto,Liang Qichao,Du Jialiang,XU YUANexistHeng Ming (Leo) KwokIT114115Advanced Diploma in Cloud and Data Center Management.