University of Michigan · Natural-language-processing course project · Sept–Dec 2021

Checking XLNet's answers against its explanations

XLNet for Verifiable Commonsense Reasoning

An illustrative glass-breaking story is mapped to three evaluation levels: story choice, conflicting sentences, and physical states.
The three levels of TRIP reasoning, explained with a glass-breaking story created for this diagram. It isn't a dataset example or a saved model prediction.

I adapted the TRIP reasoning pipeline to XLNet to test whether it could identify why a short story was physically impossible. The evaluation checked its story choice, the conflicting sentences, and the physical states behind the conflict.

My role

I integrated XLNet-base, fine-tuned it, compared training losses, evaluated the results, and wrote the final report.

Levels of reasoning
3

Accuracy, consistency, and verifiability.

Supervision signals
4

Preconditions, effects, conflicts, and story choice.

Story accuracy
80%

Validation result with state-classification losses omitted.

Starting point

What the model had to explain

TRIP pairs a plausible story with an implausible one. To explain the conflict, the model has to identify the relevant sentences and the physical states before and after an action.

How it came together

  1. 01

    Adapt the model

    I added Hugging Face XLNet-base-cased to the published TRIP pipeline, adapting the tokenizer, model configuration, and representations.

  2. 02

    Compare training losses

    During fine-tuning, I removed selected losses to examine how training on story choice, sentence conflicts, and physical states affected each result.

  3. 03

    Evaluate the explanation

    I compared story accuracy with consistency and verifiability. Those two measures also check the conflicting sentence pair and the physical states that explain it.

Removing the state losses

Without the state-classification losses, story accuracy rose from 76.7% to 80.0%, while consistency fell from 12.7% to zero. Verifiability was zero in both configurations: neither met the full requirements for explaining the conflict in this validation table.

Story choices and supporting explanations

XLNet validation results from Table 1 of the December 2021 course report. Higher is better; the measures have different requirements.

All lossesNo state losses

Accuracy

76.7%
80%

Consistency

12.7%
0%

Verifiability

0%
0%

Bars share a zero baseline · scale 0–100%

Accuracy counts correct story choices. Consistency also requires the correct conflicting sentence pair; verifiability adds the relevant physical-state predictions.

These results come from the original course report. The archived notebook records my XLNet adaptation, and the experiments haven't been rerun for this portfolio. The two configurations don't establish statistical significance.

Reports, code, and credits

Course reportCode

Storks and colleagues created the TRIP dataset, reasoning framework, and original pipeline. My modified notebook is in the linked public fork.

The report's BERT, RoBERTa, and DeBERTa comparisons use baselines published by Storks et al.; I didn't rerun those models. The diagram above was created for this portfolio to explain the task.

Next projectEstimating 3D joints from clothed body scans