I may have derived this from the travellermap API document or come up with it myself, I don't remember now, but it works.
I have a mathematica method toMap[] which will take a vector of the x and y hex number and return a map-space value:
toMap[vec_] := Module[{ix, iy, x, y},
ix = vec[[1]] - 0.5;
iy = If[ EvenQ[ vec[[1]] ], vec[[2]] - 0.5, vec[[2]] ];
x = ix Cos[\[Pi] 6^-1];
y = -iy;
{x, y}
];
then just take the regular euclidean distance between them, e.g.,
In[972]:= EuclideanDistance[ toMap[{01,01}], toMap[{08,10}] ]
Out[972]= 10.4403
Asking mathematica for a general solution gives:
In[973]:= EuclideanDistance[ toMap[{ax,ay}], toMap[{bx,by}] ]
Out[973]= Sqrt[Abs[1/2 Sqrt[3] (-0.5+ax)-1/2 Sqrt[3] (-0.5+bx)]^2+Abs[-ay+by]^2]
where ax, ay are the hex coordinates of the first hex and bx, by are the coordinates of the second