COSC 329 Projects

Project Options Available

Build a mini-copilot to help you code (not as fancy as GitHub's co-pilot). Pick a programming language and find at least 10 projects of code in the same language for you to download as a dataset (document your sources!). (Note: this is a tiny dataset, so be careful of seeing new words that was never in the training model by using a smoothing technique.) Organize the code so that you have a training set (70%) and a testing set (30%). Train a letter n-gram model and vary n from 1 to 10, train a word n-gram and vary n from 1 to 10, get the abstract syntax tree (AST) of the programming language, then have these models make word prediction based on what the user is typing in a coding context. (Tricky: An AST parser will barf if the code has a syntax error. You will need to simulate the prediction experiment by feeding it the full file, truncating it to a word prediction context, use an error-tolerant parser to fill in the missing syntax tokens, then check for syntax compliance for the prediction.) Report on the accuracies of these models -- what metrics will you use? Even if a prediction is not perfect, is there a way to measure edit distance between what it is supposed to be versus what the suggestion is? Interpret the results and explain which is best for your purpose. Incorporate the best model as part of a text editor (any basic editor will do) to interactively demonstrate how the word predictions work while you are coding. What is the experience like? What else is missing?
- 1-person option: Define accuracy and edit distance as two metrics, train and test the models, show the evaluation results for each model (include individual results for each n in the letter and word n-gram models and results for the AST), identify the best model (explain your choice), incorporate the best model into a text editor and try out the coding experience, swap in another model of your choice and try that out, report on your experiment setup and results, then include your final text editor experience with the 2 models you tried out and document the experience (which model is better and why).
- 2-people option: Define accuracy and edit distance as two metrics, use 5-fold cross-validation to train and test your n-gram models, show the results of each n-gram model in the 5-fold setting (include average results for each metric with standard deviation), show the results for the AST model, consider 1 way to combine your best n-gram model with the AST model as context to make better predictions, explain how you combine the two approaches and include those results of this hybrid model, identify the best model (explain your choice), incorporate the best model into a text editor and try out the coding experience, swap in another model of your choice and try that out, make sure to design the text editor to present the top 5 suggestions while you type and code, report on your experiment setup and results, then include your final text editor experience with the 2 models you tried out and document the experience (which model is better and why).
Predict "at-risk" students after an early assessment using the Open University Learning Analytics Dataset (OULAD). Explore the dataset to identify courses, students, and their course-level data (engagement with course material, interaction in discussion forums, and assessment data). Massage the data into feature vectors that would be appropriate for classification models. Be careful how you engineer these features because that will impact your performance in the end, so document and discuss tradeoffs you make in this step. Classify your data using random forest, XGBoost, logistic regression, support vector machines, and naive Bayes. When you train the machine learning model, you will notice that most students pass and the model will therefore most likely predict a pass for all students. So use a synthetic minority over-sampling technique (SMOTE) as a precaution. The evaluation metrics include precision, recall, F1, and ROC-AUC score for all your classification algorithms.
- 1-person option: Transform the data into features (document choices and tradeoffs), simulate the midterm "cut-off" to predict final course grade, use 10-fold cross-validation and different splits to train and test your data for each classification algorithm (remember to use SMOTE for sample selection during training but not testing), use binary classification for students at risk of failing or dropping out, repeat this 10-fold cross validation across 25 independent trials with different random shuffles, report on average and standard deviations on 4 metrics above, include explanations and meaningful interpretation that reflects the type of data used and how the algorithms work.
- 2-person option: In addition to the steps in the 1-person option, use 5 evenly divided "cut-off" points in the course to measure prediction accuracy at each stage (e.g., a course that spans 100 days now will have a prediction after 20 days, 40 days, 60 days, 80 days, and 100 days). Use a graph with x-axis being time to show whether prediction accuracy improves with more information for a specific algorithm. Although at-risk student analyses typically focus on students failing the course, you will also want to predict academic trajectory drops by defining when students score high (e.g., 80+%) on an early assessment but drops 10+% on subsequent assessments. In this case, you will do multi-class classification to predict if the student experiences a major grade drop or fails or stays the same. Also, run the experiments with only engagement data, only interaction data, only assessment data, a combination of two of these, and all three data sources. Which features are more useful for the prediction? Include explanations and meaningful interpretation that reflects the type of data used and how the algorithms work.
Use a calendar API to build a personalized, Bayesian study planning system. The system should take as input upcoming exam dates and dynamically schedule study sessions based on the user's mastery of course concepts. These scheduling needs should be modeled using a Bayesian network that considers variables like concept difficulty, current stress level, and days remaining until the exam. The system's decision-making process must use a utility function that balances exam success against student stress. The system will choose between three or more specific actions (e.g., (i) schedule an intense, mandatory 5-hour study block today, (ii) schedule a light 1-hour review session, or (iii) clear the calendar for a rest day). To demonstrate your system works as intended, you'll need to simulate different types of student scenarios (e.g., a struggling student vs. an advanced student) and show that the system adapts its scheduling behavior accordingly.
- 1-person option: Create a simple Bayes net with variables that model difficulty of 3+ concepts and develops a study plan that maximizes exam success and minimizes last minute cramming stress (use additional variables in your model as needed, but keep your network to ~10 variables or less). Handcraft the model variables and CPTs clearly with an explanation of the parameters, identify the inference task and which variable(s) you query (be sure to document this), define 3+ specific actions for the system, define a utility function that expresses exam success and stress while depending on the query variables, use a built-in inference algorithm to make your query, compute the expected utility of each system action, take the best action, incorporate this into the calendar API and demonstrate this with different concept difficulties that will result in the system taking different actions.
- 2-people option: Basically do the 1-person option but with an expanded network that includes additional observable variables and associated data to learn the CPTs in the network. Once you expand the Bayesian network structure, identify 4+ variables that you will focus on for learning the CPT parameters empirically from data. The data should be gathered from online sources, or alternatively, built through a simulation of data generated by a reasonably meaningful mathematical function (document your data source and/or simulation approach clearly). If you simulate data, it should generate more than 10 data points per parameter you are learning (feel free to reach out with questions if you are stuck). The data should be processed into a CSV format. Once you have the CSV, use a simple learning approach to develop the CPT parameters for the variables of interest. Remember to use a smoothing technique such as Laplace smoothing.
Build and analyze a knowledge network from StackOverflow. Treat StackOverflow as a massive, open-source classroom and consider how users on the platform interact with each other by combining social network analysis concepts and text analysis to better understand the context.
- 1-person option: Extract data from StackOverflow using the official Stack Exchange API. Focus on a specific tag relevant to university students (e.g., #python, #java, or #datastructures) and extract the data into a CSV. Using the data you have, build a directed graph with users as nodes and directed edges from a user who answers another user who asked a question. The size of the node should reflect the amount of questioning and answering activity the user is involved in (more activity means bigger node) and the thickness of the edges should reflect the number of questions answered between a pair of users. If your graph has a lot of nodes, filter it by frequency so it only shows ~20 nodes so the visualization is clear. Calculate and report on these nodes' in-degree centrality, out-degree centrality, hub value, and authority value (explain what they say about your network). From the same dataset, analyze each question and categorize it at a Bloom's taxonomy level (use the verbs in the verbs table). The Bloom's level will then be incorporated into the edges of the graph, and you will want a separate edge for each level of the Bloom's taxonomy. For example, user A answers 10 questions from user B. In the original graph, there is 1 edge between A and B that is 10 "units" thick. Let's say 3 of these questions are Remembering, 2 are Analyzing, and 5 are Evaluating. Then in the Bloom's version of the graph, there are 3 edges between A and B corresponding to the 3 types of questions with appropriate thicknesses representing the frequency of each question type. If it gets too cluttered, you may choose to draw them as separate graphs so you have an original, full picture graph, then a separate graph for each of the Bloom's levels.
- 2-person option: Do the following in addition to the 1-person option: (i) verb set expansion and (ii) answer toxicity labeling. Part (i): Since the verbs table is limited in scale, you may come across words in your data set that you cannot find in the table. To fix this, use lemmatization (e.g., such as Morphy) and WordNet to come up with synonyms for verbs. If there are still verbs that do not exist after this approach, display that list of verbs and create a manual lookup mapping as a fallback (label these into appropriate Bloom's levels yourself). Part (ii): Analyze the answers provided in your data to detect rude and inappropriate responses (for example, use detoxify or s-nlp/roberta_toxicity_classifier). Normalize the toxicity level of each user's response from 0 (neutral) to 1 (toxic), and compute the average toxicity score of each edge in your graphs. Does toxicity correspond to the level of questions asked?
If there is another topic that you are passionate about, you can initiate a meeting with me and we can chat about how we might be able to relate it to learning analytics and scope it to be similar to the other options. If you are considering this option, you will need to meet with me before Week 6 so we can hash out the details and ensure the scope is feasiable. If we can't finalize the details by that time, then you will need to choose one of the other project options listed above.