This group describes class templates that provide shared object semantics for some kind of wrapped value that acts as "a form of reference" to some kind of computing resource.
At the time of writing, these constructs include:
These constructs act similarly to a simplified version of std::share_ptr (that also work on GPUs)
Common Operations
These constructs all provide a common set of operations:
Let's establish the basic semantics modeled by these constructs:
- we say that a construct exercises ownership of an underlying resource by wrapping a handle or pointer (depending on whether the construct is a SharedHandle or a SharedDevPtr). The wrapped handle or pointer can be accessed through the
get() member function
- a resource can be owned by one or more instance of a construct. When the number of owners of the resource drops to zero, the construct is deleted.
- an instance of a construct can also be "empty" (i.e. it doesn't own any construct)
Now, let's describe the way that this model is implemented
- the default constructor builds an empty instance
- the primary constructor builds an instance that wraps a previously created handle or pointer and a function-like callback for deleting the resource.
- The constructed instance takes ownership of the underlying resource referenced by the handle or pointer.
- This can only happen on the host
- the copy construction and copy assignment are mechanisms for sharing ownership of an underlying resource. Move construction and move assignment are mechanisms for transferring ownership of a resource.
- For any kind of assignment (
instance_a = instance_b; or instance_a = std::move(instance_b);), if instance_a and instance_b did not previously share ownership over the same resource, instance_a releases ownership of its previously owned resource in the course of the operation.
- ownership is released with the
reset method and in an instance's destructor
- the callback function-like deleter object is invoked for a wrapped value (e.g. handle or pointer) when ownership is released by the last construct that owned the value
IMPORTANTLY: these constructs can be used on the host and on GPUs
How it works:
At a high level, these constructs are implemented using (atomic) reference counting. Essentially, the constructs hold a pointer to a "control block" that holds a reference count. When the primary constructor is invoked, the reference count starts at one. Every drops to 0, the deleter callback is then invoked to delete the resource.
In slightly more detail, ownership is only tracked on the host. In case its not obvious why this is a viable strategy, let's make a simple assumption: let's assume for a moment that we are always extremely careful about releasing device resources until after all accesses to a resource are complete.
Under that assumption, let's consider the lifetime of a SharedHandle or a SharedDevPtr a GPU kernel:
- since the primary constructor can only be invoked on the host, the only way to get an instance of the construct on the device is if we pass it to a kernel function by value
- within a kernel function no matter how many copies we make of a construct the number of copies of a construct on the GPU that share ownership of a resource will drop to zero by time you exit the kernel
- thus, if you imagined counting the total number of instances that share ownership of a given resource across the CPU and GPU, the total number of instance that share ownership is equal to the number of instances that share ownership on the CPU both before you launch the kernel and after the kernel completes
- There are 2 relevant observations to add to this discussion:
- this discussion assumes that the primary constructor is always invoked to try to construct a SharedHandle or SharedDevPtr instance on the stack or the host's heap. Problems would arise if you tried to use placement new to initialize in device memory. For that matter, any attempt to track an instance of these constructs in one of the device's memory spaces that persists outside of a kernel would be problematic for the drawn conclusions (to my knowledge, isn't actually something we could really accomplish anyway without going out of our way to do something "bad" – there's no practical benefit to try this).
- It's also worth mentioning that even if we did want to track the total number of reference counts across the CPU and GPU, there isn't a straight-forward solution (if you pass a construct to a kernel by value, the copy constructor isn't technically invoked and the reference count won't get incremented). While there are workarounds, they aren't elegant.
What about our assumption? The degree of required care actually depends on the deleter callback. For example, deleters passed to SharedDevPtr that are based upon cudaFree and cudaFreeAsync will have distinct requirements.
Why Use These Constructs
These constructs are most useful as building blocks in larger components. For example, aspects of Cholla's feedback and cooling modules make use of resource allocations for the entirety of a simulation run. These constructs make it easier to build up constructs in a composable manner.
While alternatives are possible, they typically involve either (i) implementing data structures using move-semantics (like std::unique_ptr), or (ii) using global variables.
- The first option causes problems with wrapping the full command (e.g. modelling cooling) in a std::function. We could work around this issue by creating an analogue of C++23's std::move_only_function or creating a custom command base class and passing around a pointer to that class. Even then, are more issues:
- Unfortunately, move-semantics are a little intimidating for less-experienced C++ developers.
- Furthermore, an object with unique_ptr-like semantics should not be directly be passed to a kernel by value because it would be inconsistent with standard move-semantics. While I'm not fundamentally opposed to implementing "an
exception to the general rules of move-semantics," I think that would be a bad idea since I'm already concerned about some contributors learning about move-semantics. Thus, under this solution we should really be extracting all handles and pointers from a data structure before passing them to a kernel, which obviously hinders composability (which we desire for the cooling routines).
- The second option isn't composable in a manner desired for the cooling routines. Every time we would want to add support for a different kind of cooling table, we would need to define a brand new global variable
- if we want to track a new table tabulated cooling values, we would need to create a new global variable to track the new textures (thus every kind of cooling is a special case)
- plus, the current implementations (at the time of writing) that follow this kind of strategy, skip deallocation of the underlying memory. This is actually fine when running Cholla, but would be problematic if we just wanted to be able to write tests. The only robust way to properly deallocate this memory involves reference counting (if we're reference counting, we may as well just use SharedDevPtr or SharedHandle)
In the future, the internals of SharedDevPtr could be very useful. The View types adopted in libraries like Kokkos or Raja have the same shared object semantics as SharedDevPtr. We could reuse the machinery to accomplish similar goals:
- In the nearer term, we could rename SharedDevPtr to something like
Shared1DBuf use it to gradually replace every occurrence of cuda_utilities::DeviceVector (obviously, we would need to add on more methods to Shared1DBuf). This is beneficial since we could pass a Shared1DBuf directly to a kernel (making it possible to directly pass a cuda_utilities::DeviceVector would be inconsistent with standard copy-semantics of a vector-like thing).
- Longer term, if we wanted to support multi-dimensional views that internally convert a 3D index to a pointer-access (this could be very useful for reducing the amount of memory allocated when we use face-centered B-fields), we could start to make used of C++23's std::mdspan (backports of the library also exist). In that scenario, we would probably want to reuse our control-block logic to help implement a custom AccessorPolicy for std::mdspan that supports reference counting.
