COSC 419: Learning Analytics
A5: DBN Simulations [24 pts]
Due date: Mar 22, 2020, 11:59pm
What to submit:
Submit the following on Connect:
- A PDF report documenting:
- The DBN model and CPTs you created in mk_needhelp.m
- Other code files you used to generate the graphs, explain in a README
how to run it and with what settings
- all graphs required -- state clearly which graph is generated from which
part of your code (and which parameter setting was used)
- If your code doesn't fully work: document which part of the assignment
you were able to finish and which part(s) did not work. In your explanation,
be sure to reference precisely the code files that you got working.
- All the code you wrote (see grading criteria below): be sure it's well
documented so we know which file or which part of the file does what
Specific Instructions
Use Matlab and the BNT for this assignment. The slides referenced in these
instructions refer to the DBN Simulation lecture
(slides).
- Download the following and re-produce the examples in class (do it in
the order suggested):
- mk_hints.m
(defines a simple DBN reviewed in class)
- Run mk_hints to get a DBN from it
- Follow steps on Slide 21
- Additional helper files needed:
normalize.m
- sim_hints.m
(sets up a basic inference environment as reviewed in class)
- Run sim_hints with the DBN you get from mk_hints, try different
settings, make sure you understand the inference graphs that are
generated
- You should be able to replicate the graphs on Slides 46-49
- Additional helper files needed:
sampleHint_seq.m,
sampleRow.m
- sim_hints_decision.m
(sets up a decision making environment as reviewed in class)
- Run sim_hints_decision with the DBN you get from mk_hints, try
different settings, make sure you understand the expected utility graphs
and text otuput that are generated
- You should be able to replicate the graphs on Slides 62-64, and you
can try other settings as well
- Additional helper files needed:
util_hints.m,
get_meu_hints.m
- Create a new model (or adapt the one from the previous question) so
that you infer where the user needs help.
This DBN will be encoded in a file called mk_needhelp.m.
Specifically, your model should look like this:
where NeedHelp is a binary variable (false, true),
TaskTime is a tertiary variable (too short, on task, too long), and
Correct is a binary variable (false, true).
Define the necessary CPTs using meaningful numbers that reflect your
model intuitions.
Be sure to document your model intuitions above your CPT definitions.
Note: For simplicity, assume the observation functions are the same across
time, so you will only need to define 4 CPTs (2 observation functions, 1
transition function, 1 prior distribution) in total.
- Create a simulation environment called sim_needhelp_decision.m that is
modeled after sim_hints_decision.m so that it includes the following:
- Use T = 50 (so you can see the patterns over a longer time interval)
- For the part that generates a series of evidence in advance, mimic
the code in sim_hints_decision.m so that you also have 3 different ways of
generating evidence. You will need to try these out later.
- Print out the evidence that gets generated. (You don't have to
pretty-print this; just type the variable name without the semicolon so
the output is dumped onto the screen.)
- To complete the third setting for generating evidence: Write a
helper function in a separate file called sample_help_seq.m that takes as
input the DBN, a fixed parent value of neediness for help, and an interval
range T. The code creates an empty evidence cell structure for T time
steps, then incrementally samples a value for TaskTime and a value for
Correct at each time step by calling stoch_obs (this is a helper function
defined within the file that is not visible to any other file).
- To decide which action should be taken: Define a helper function in a
separate file called get_meu_help.m that takes Pr(NeedHelp = true) as
input. Only two actions will be considered: helping the user and doing
nothing. This function returns two things: the first is the calculated
best action to take (as a String), and the calculated expected utility of
help at that point in time. Both of these can then be displayed. In this
file, you will compute the expected utility of help and the expected
utility of doing nothing.
- Create a helper function in a separate file called util_help.m that
takes two inputs: an action (String) and a value of neediness (0 or 1).
This function defines the utility of help and the utility of doing nothing
in the cases when the user needs help and when the user doesn't need help.
All the utilities are defined in the range [-5,+5].
Be sure to document your model intuitions in the code.
- Generate plots for the setting 2 ("fixed evidence") where you can
have combinations of the user always doing exercises incorrectly (or
correctly) and always taking too short amount of time (or on task, or too
long). Considering all combinations of these observation variable values,
you have a total of 6 graphs to generate. Your graphs should look
something like this (I've combined them all onto one graph for
convenience):
Note that the purple and green lines overlap a lot so it seems like there
are only 5 cases when there really are 6.
Interpret your graphs so that they make sense to you. The graphs should
convince you that the CPTs in your model are defined in a sensible way.
- Similarly, generate plots for the "random evidence" setting and the
"controlled randomness" setting that uses your sampling code.
Grading Criteria
- [8 pts] for mk_needhelp.m (4 for model, 4 for CPTs)
- [2 pts] for sim_needhelp_decision.m
- [2 pts] for sample_help_seq.m
- [2 pts] for get_meu_help.m
- [2 pts] for util_help.m
- [8 pts] for 8 graphs generated from sim_needhelp_decision.m