Sorry I'm not able to help you on this. I'm just a mere user of AL as well and far from an expert. Maybe have a look at the Alphalens documentation on Github to try to figure it out. Or perhaps a true AL expert (such as @Luca, @Michael Matthews, @Thomas W.) could advise on this?
To me it looks like quite a decent factor (including autocorrelation and turnover), at least as a starting point (appears to be a shorter term mean-reversion type factor). Perhaps the returns discrepancies are due to extreme outliers? Maybe try to 'winsorize' the factor by x% on each side? Maybe also plot the factor distribution to see if there are 'fat tails' or extreme outliers?
See this post and help from Michael Matthews in a similar issue I was having. Unfortunately Delaney has moved on since then, so the thread is somewhat 'dead' unfortunately, but maybe you can find some solution hints to your issue there?