Testing if the Final Fantasy Tactics “Chance to Hit” Predictions Are Lying

Background

During battles, in the newly released video game “Final Fantasy Tactics: The Ivalice Chronicles”, when you select an ability, and then target an enemy, the game tells you the “Chance of Success” of whether the ability will hit or miss. That way, you can change your mind, before confirming.

But I don’t trust it.

For example, the “Orator” class, has a powerful ability called “Entice“, which, if successful, makes the target enemy become a traitor, and switch to your team, permanently. Pretty crazy! The drawback, is that it has a low chance to succeed–for my Orator, typically between 20% and 30%.

…Or at least, that’s what the in-battle tooltip would have me believe. Whenever I select “Entice” and then target an enemy, it usually says 26%.

But the first 4 times I ever used Entice, it succeeded! That’s crazy! What are the odds of that?? Well, about 0.46%, I think. (That my first 4 attempts all succeeded, assuming 26% predicted success of each one.)

But maybe it was a fluke? It’s still bound to happen to about 0.46% of players, maybe I’m just one of them.

Anyway, as I continued and finished the rest of the game, “Entice” did fail sometimes. But I still felt the success rate was much higher than the game’s predictions! It “felt” more like 70% success to me.

I did some research into various online guides, to see if anyone had figured out the true success rate. But every guide’s math formulas for the success rate, all matched up with the game’s in-battle predictions! The guide writers probably simply trusted the in-game predictions (why wouldn’t they?), and simply worked out the math behind them, and then put that math in the guides, without testing whether the predictions are accurate in the first place.

But I was still suspicious.

So one night, I decided to perform a quick statistical experiment, to put the game’s prediction math to the test.

The Experiment’s Conclusion

The experiment’s results imply that the game’s predictions are probably wrong.

I performed 30 “Entice” attempts, and 13 out of 30 succeeded. If the in-game predictions were correct, there’s only a 1.75% chance I’d have gotten that many successes, or more.

So according to the test, it’s much more likely that the predictions are wrong.

(But admittedly, 13 out of 30 is a lot less than the “70%” success rate that I “felt” I was experiencing.)

Experiment Methodology

Here’s how I did the test, so you can check for bias, or in case you would like to reproduce it.

I loaded my file, and went to random battles.

In each random battle, I left all my characters in one spot, with my Orator safely in the center, and cast Entice on the nearest enemy every turn, if an enemy was in range.

In a spreadsheet, I recorded each Entice attempt as 1 row. In each row I recorded the in-game predicted Success Chance of that attempt, and also, whether it actually succeeded, after I hit confirm. (And some other columns, that I didn’t end up using.)

If I got into a situation where the nearest enemy’s HP was less than 50%, or there were no more enemies, I reloaded my file and found a different random battle (just in case low HP has a secret effect on success chance).

I decided, in advance, to do this 30 times (30 Entices). (I chose 30 because I remember that 30 is a pretty decent sample size generally, as a crappy rule of thumb, also I didn’t care enough to do more). I also decided in advance, to share the results with others, regardless of if the results matched my hunch.

Here’s the resulting raw data. (Note: only the first 2 columns matter, the others I recorded just in case I wanted to analyze them later.) (The zodiac signs are here because I was curious if the game’s predictions take that compatibility into account–and they do!)

I succeeded in 13 out of 30 attempts, which is a 43.33% success rate. But according to the game’s predicted success rates, I should have been closer to a 24.4% success rate.

So I got more successes than predicted, but how lucky was my lucky result, assuming the game’s predictions are accurate?

Next, I wrote a simulation program, to run 1 million times. For each simulated run, it performs the same 30 Entice dice rolls, using the same Success Chances (20%, 20%, 20%, 26%, etc), using a random number generator, and records if each roll succeeds or fails. At the end of each run, it adds up the successes for that run (between 0 and 30). It increments a histogram bucket once (buckets 0 through 30), then starts over 1 million times

Here is the resulting histogram:

Here’s the actual histogram data, from the simulations:

Recall that in reality, I got 13 out of 30 successes.

According to the simulation program’s output, a result that good (or better) only occurred in 17,486 out of 1 million simulations, in other words, 1.75% of the time.

So either I got pretty lucky (luckier than 98.25% of the simulations!) OR, more likely, the game’s predictions are inaccurate.

But if you have some time, feel free to try to reproduce the experiment, and see if I really did simply get lucky!

Here is the Javascript I used, to run the simulations and generate the output histogram: https://jsfiddle.net/1fp3xscz/ (Just hit the “Run” button in the top right, to run it yourself. If you gather your own data, you might be able to modify the “input” variables, at the top of the code, in the “Javascript” area of the screen, and run it again.)

Additional Notes

1.75% is not quite low enough to definitively prove anything–this could still be a fluke.

On the other hand, the 1.75% coupled with the fact that my first 4 Entices ever, all succeeded, really really seems to imply something’s going on.

I could run the experiment again with a larger sample size than 30, but then I’d be falling into a trap of statistical bias, called “repeating the test until I get the results I want“

On the other other hand, if that “first 4 successes” didn’t occur with “Entice”, maybe it would have happened with another ability, and I would have noticed it, got curious, and wrote this same article, about that ability, instead of Entice… This was more likely to happen if I’m not looking at just Entice, and just the first 4 times I used any ability.

…BUT Entice is special–I didn’t really use other low-success-rate abilities, during my playthrough of the game. Entice just happened to so outrageously powerful, that I really wanted to try to use it a lot. It was very Enticing to use.