This is a misleading benchmark. IndexOf stops checking the array when index is found. By looking for 5000 in a 10,000 array your actually only checking an array of length 5000.
You need to compare them against the same search space, e.g make it try remove 10,0001 (item not in the array) and compare, then it’s the same size.
When doing perf comparisons you can’t make assumptions about things and need to take the worst case.
IndexOf still has some optimisations at a low level that make it faster in some cases, but not as dramatic as the benchmarks here