It's possible there is something floating around in there, or it could be just a sticky valve.
There is an additive that Honda transmissions use that works really well for sticky valves because they were so sensitive to that, and it could solve your problem.
It's called Lubegard RED, and we would add it if the trans hesitated to shift or got stuck in a gear after a rebuild. (It worked exceptionally well too)
The main reason I think it may be a sticky valve is because you said it free wheels when you let off the gas. (Called engine braking)
No engine braking usually means a sticking valve if I remember correctly.