Note from 2022: This old article, dating from 2012, is about a video game and a C++ library for Lua I tried to make with friends.
For posterity, here's an archived zip file for the Pocsel source code, and here's an archived zip file for the C++ library source code.
If one day you use Lua to make your application scriptable, you will face a particular problem that we encountered with Pocsel: accessing your application's resources from the scripts.
In fact it's not really a problem. It's just that we think it's difficult to do correctly.
Typically, resources created by your application (such as images, entities…) will be allocated on the heap by a call to new and destroyed by a call to delete (in the case of C++). The simple approach of giving raw pointers to resources to your scripts, for example contained in LightUserData or UserData types, is really dangerous.
Let me demonstrate with a small Lua script:
entity = App.GetEntityById(1338) -- returns a LightUserData (the Lua raw pointer type)
... -- some time passes (control returns to C++, then comes back to Lua)
App.DoStuffWithEntity(entity) -- serious risk of crash (was the entity deleted?)
Each time your application receives a resource pointer from a Lua script, it has no way of knowing if it's valid or pointing to a deallocated space in memory.
A simple solution to this problem is to wrap the pointer in a custom object created by your application — exactly what the UserData type is for. This object will contain the raw resource pointer, and will register itself in the application resource manager. When a resource is destroyed, the resource manager will iterate over every registered UserData for this particular resource and invalidate it (probably by setting the pointer to 0). That way, when the application receives a UserData, it can check the validity of the resource with a simple test like ptr != 0.
Note that UserData objects are entirely managed by the Lua interpreter, that is, they are destroyed by the garbage collector. So it's also imperative to unregister UserData objects from the resource manager when they are collected (with the __gc metamethod) — otherwise the application would crash when destroying resources via the resource manager.
However, this technique has a disadvantage: the modder (let's call the script writer that) has no way of knowing if the resource is still alive.
Having a UserData as a resource pointer is nice because we can play with Lua's metamethods to add some functionality to each of our pointers.
For example we could fix the problem of not knowing if the resource is alive by overloading the call operator with the __call metamethod. Executing entity() would now call the application to test for the validity of the resource. We could have used # (__len, the length operator) or any other unary operator, like __unm (unary minus — resulting in -entity — but that would not have been very pretty).
We could also use the __index metamethod to return values stored in the resource or even functions interacting with it, thus creating a more real object and not a simple pointer.
Going further, we can use the __newindex metamethod to allow the modification of certain properties of the resources.
So now we have something like this:
entity = App.GetEntityById(1338)
... -- some time passes (control returns to C++, then comes back to Lua)
if entity() then
entity:DoStuff() -- '__index' called with 'DoStuff' as a parameter
-- the application returns a function taking a single UserData
-- which is immediatly called with entity, because of :
entity.health = 12 -- '__newindex' called with 'health' and 12
end
It's better, but still, there are problems. The modder has to remember to check for the validity of the resource every time he uses it. If entity:DoStuff() is executed with an entity that was previously destroyed, an error is thrown because ptr != 0 is false. The application catches it and does whatever it must do with the script that generated the error, so it's okay, but the modder is not happy — he forgot the obligatory if entity() then. Can't we make sure he never forgets it, or better, can't we make sure there is nothing to forget?
From the beginning, we do things wrong. We give the modder a direct (or almost direct) pointer to a resource that can be deleted at any moment. This is not good. The true way to handle this situation is to use weak pointers (also called weak references — Wikipedia link). Weak pointers can't be dereferenced implicitly. You can't interact with the pointed resource directly, you have to explicitly dereference the pointer before, usually by calling its Get() or Lock() method. The thing is, this method can fail. It either returns a good pointer to a resource, or 0 (nil in our case) if the resource doesn't exist anymore.
The application should only give the modder weak pointers, never direct resource pointers. This will force him to call Get() and check its return value. He won't be able to do anything else because he won't have any true pointer.
Another advantage of using weak pointers is that we get rid of the ptr != 0 condition executed every time a UserData wrapper is used. No condition is necessary because we know the resource exists after a successful call to Get(), so it's faster.
weakPtrToEntity = App.GetEntityById(1338)
... -- some time passes (control returns to C++, then comes back to Lua)
local entity = weakPtrToEntity:Get() -- returns nil or a UserData
if entity then
entity:DoStuff()
-- weakPtrToEntity:DoStuff() is not possible, it's only a weak pointer
entity.health = 12
end
Internally in Pocsel, we have a class named WeakResourceRefManager, which is a generic manager that generates weak pointers to resources (it's templated on the type of resource, and on the resource's manager type). Our weak pointers are UserData objects with a single method, almost exactly like those described above.
However, the modder can still screw up. You might have already guessed what can go wrong: nothing prevents him from keeping the return value of Get() for a very long time. He might check the return value once and assume the resource will stay alive forever. Notice in the last example how entity is declared local, meaning it will be lost at the end of the current scope block. Maybe the modder won't do that and keep entity for a long time, or worse, as a global… That's not at all what we want. Is there a way to discourage this behavior? Read on.
It's starting to get a little more complicated now. In Pocsel, when you dereference a weak pointer (Get()), you get what we call a "fake reference". It's another UserData wrapper, wrapping the resource's UserData wrapper. It acts like a proxy, forwarding anything the modder does to it to the resource's UserData wrapper. The modder doesn't need to know his method calls go through a fake reference/proxy. It doesn't affect anything for him — except one important thing!
Each fake reference registers itself in an internal C++ container when it's created (that is, when a weak pointer is dereferenced). That way, the application has the control over every fake reference ever given to the modder because it has their pointers stored in a C++ container (in our case, it's a simple std::list). At any moment, the application can invalidate a fake reference, making its use impossible by the modder, much like when we invalidated resource wrappers previously.
Every time control returns to the application, we iterate over the list of fake references and invalidate them all. If the modder kept a direct reference to a resource and not just a weak pointer, we can throw a nice error message the next time he uses it, like This reference was invalidated - you must not keep true references to resources, only weak references (actual message at the time of writing this article).
weakPtrToEntity = App.GetEntityById(1338)
entity = weakPtrToEntity:Get()
... -- some time passes (control returns to C++, then comes back to Lua)
entity:DoStuff() -- error!
With this method, the modder has almost immediate feedback on what he did wrong and he can correct his problems quickly.
Fake references do not always have to be used. We judged it to be a little too resource-intensive, so our fake references are only enabled in debug mode, that is, only when a modder is coding. In normal mode, everything is as fast as it can be (it's good for a video game!), and weak pointers return direct references, not fake ones.