We've been pounding on this problem for a while and I've been making reports in multiple threads. Since this looks like it's not an easy fix and to have a single place for me to post (and others to read) updates, I started a new thread. Please do not post general questions here so it can be clean with just updates.

I think we've had a breakthrough (well maybe that's too strong of a term), but certainly found something that will help. People should be getting more backlogged credits soon. We have to see whether this will fix all of the problems. I'm thinking it won't fix them all, but it is a step in the right direction.

Also, Joe is working on this today and may contact some of you for additional information to help us debug this.

People should be getting more backlogged credits soon.

Joe has made some good progress in tracking down the problem. He's found the bug that was recently introduced into the WS code that caused this problem and is now testing the fix to rollout to the NV GPU WS's.

He has also suggested a short term workaround which should allow many of the WUs that have been sitting in the queue to be sent back. We've instituted that fix this morning and are looking to see if that helps the situation.