So what would you be trying to achieve with multithreading Lua? As far as I can see you are trying to multi thread Lua by basically having multiple instances of Lua in their own threads; each Thread has it's own Lua state that is independant of others - this would mean that each instance of Lua in those threads would be separate from the other, would not be able to communicate with each other, etc. This would solve the concurrency problem, until..
Lua can have side-effects. This is basically when this happens:
Because we are using OOP as a base in Eluna, we pass around Entity objects that maintain an internal pointer to their C++ Entity. Whenever we call a function in Lua, we call the C function that is bound to the Lua object, which in turn can do some checking and then invoke the C function on the actual C entity later on down the road.
myCallback = function(event, entity)
So with the above sample you can see that we set the health of the entity passed into that function to 100. This invokes the SetHealth<T>(T) method for Lua, which will presumably check to make sure that the T parameter is not null (i.e, has not been destroyed) and then invoke T->SetHealth(100). This is all fine and dandy on a single-threaded solution, until you introduce multithreading. When you introduce multithreading (by having multiple Lua states in their own threads) and you have an event occur, first you would need to decide which Lua state to invoke the event on. Well, the easiest way to do this is just to pick a thread that isn't "busy". The problem here is what happens when you have an event for one entity invoked on one thread, and then another event for the same entity invoked on a different thread?
Potentially, here, we could end up with a happened-before relationship. That is, event A could call the above callback halfway through event B being processed, meaning that halfway through the event B, the HP of the entity changes. This could very well screw with event B's logic and cause an unexpected event, such as an expression not resolving true, causing an event to not happen.. which in turn would cause a bug.
The way to resolve this would be to ensure that each callback is an atomic action - i.e, use a mutual exclusion lock - a mutex. You would need to wrap this around every mutator on the entity, such that only one thread could ever invoke a mutator and such that a mutator could only be called if no threads were currently inside a "get" method... and then you end up with a load of mutex soup in your code. And mutexes are expensive to have. Well, I mean, creating a mutex is cheap, generally only a couple of bytes of memory. But the reason they are expensive is because most mutexes will cause a method to 'block' for X amount of time until it can gain access to the variable. That would mean that setters would have to block if you wanted to set a value, potentially for an indefinite amount of time causing the script to hang.
The other alternative would make it somehow impossible to alter state of an entity.. that is, to remove side effects.. but I don't know how you would achieve that given how the nature of the scripting engine is that it is one giant side effect.
As you can see, having multiple threads, each with their own Lua state, just moves the mutual exclusion problem to the Lua object methods, rather than just inside the the Lua scripts (like it is now). You're only moving the problem around so that the user doesn't see it, but in turn you are making the solution a lot more complex. :\ It's funny how the only reason we event need to worry about mutual exclusion is because the objects are mutable. If they were all immutable we wouldn't have to worry.
By keeping single-threaded Lua with coroutines and invoking every entity (or, alternatively, every script) from inside a coroutine you can maintain a separation of state by keeping state localized to each script and not allowing it to leak, which is a good first step. In a better world we wouldn't allow state to leak outside of functions (perhaps pass in a table into each callback that can be modified by users and would be continually passed in to each event for the same entity - a bit like a context object. That isn't totally removing state, but at the very least it isn't leaking outside of the callback into the rest of the script. IMO the more I talk about this the more I think that AI being inside a coroutine is a better and better idea. The only problem is when you need to maintain global state (which, while evil, is sometimes a necessity) :(