...one of the most highly
regarded and expertly designed C++ library projects in the
world.
— Herb Sutter and Andrei
Alexandrescu, C++
Coding Standards
As we have seen, Boost.Interprocess offers some basic classes to create shared memory objects and file mappings and map those mappable classes to the process' address space.
However, managing those memory segments is not not easy for non-trivial tasks. A mapped region is a fixed-length memory buffer and creating and destroying objects of any type dynamically, requires a lot of work, since it would require programming a memory management algorithm to allocate portions of that segment. Many times, we also want to associate names to objects created in shared memory, so all the processes can find the object using the name.
Boost.Interprocess offers 4 managed memory segment classes:
operator
new
) memory buffer (basic_managed_heap_memory class).
The first two classes manage memory segments that can be shared between processes. The third is useful to create complex data-bases to be sent though other mechanisms like message queues to other processes. The fourth class can manage any fixed size memory buffer. The first two classes will be explained in the next two sections. basic_managed_heap_memory and basic_managed_external_buffer will be explained later.
The most important services of a managed memory segment are:
All Boost.Interprocess managed memory segment classes are templatized classes that can be customized by the user:
template < class CharType, class MemoryAlgorithm, template<class IndexConfig> class IndexType > class basic_managed_shared_memory / basic_managed_mapped_file / basic_managed_heap_memory / basic_external_buffer;
These classes can be customized with the following template parameters:
MemoryAlgorithm::mutex_family
)
to be used in all allocation operations. This allows the use
of user-defined mutexes or avoiding internal locking (maybe code
will be externally synchronized by the user).
MemoryAlgorithm::void_pointer
)
to be used by the memory allocation algorithm or additional helper
structures (like a map to maintain object/name associations).
All STL compatible allocators and containers to be used with
this managed memory segment will use this pointer type. The pointer
type will define if the managed memory segment can be mapped
between several processes. For example, if void_pointer
is offset_ptr<void>
we will be able to map the
managed segment in different base addresses in each process.
If void_pointer
is void*
only fixed address mapping could be used.
This way, we can use char
or wchar_t
strings to identify
created C++ objects in the memory segment, we can plug new shared memory
allocation algorithms, and use the index type that is best suited to our
needs.
As seen, basic_managed_shared_memory offers a great variety of customization. But for the average user, a common, default shared memory named object creation is needed. Because of this, Boost.Interprocess defines the most common managed shared memory specializations:
//!Defines a managed shared memory with c-strings as keys for named objects, //!the default memory algorithm (with process-shared mutexes, //!and offset_ptr as internal pointers) as memory allocation algorithm //!and the default index type as the index. //!This class allows the shared memory to be mapped in different base //!in different processes typedef basic_managed_shared_memory<char ,/*Default memory algorithm defining offset_ptr<void> as void_pointer*/ ,/*Default index type*/> managed_shared_memory; //!Defines a managed shared memory with wide strings as keys for named objects, //!the default memory algorithm (with process-shared mutexes, //!and offset_ptr as internal pointers) as memory allocation algorithm //!and the default index type as the index. //!This class allows the shared memory to be mapped in different base //!in different processes typedef basic_managed_shared_memory<wchar_t ,/*Default memory algorithm defining offset_ptr<void> as void_pointer*/ ,/*Default index type*/> wmanaged_shared_memory;
managed_shared_memory
allocates
objects in shared memory associated with a c-string and wmanaged_shared_memory
allocates objects in shared memory associated with a wchar_t null terminated
string. Both define the pointer type as offset_ptr<void>
so they can be used to map the shared
memory at different base addresses in different processes.
If the user wants to map the shared memory in the same address in all processes and want to use raw pointers internally instead of offset pointers, Boost.Interprocess defines the following types:
//!Defines a managed shared memory with c-strings as keys for named objects, //!the default memory algorithm (with process-shared mutexes, //!and offset_ptr as internal pointers) as memory allocation algorithm //!and the default index type as the index. //!This class allows the shared memory to be mapped in different base //!in different processes*/ typedef basic_managed_shared_memory <char ,/*Default memory algorithm defining void * as void_pointer*/ ,/*Default index type*/> fixed_managed_shared_memory; //!Defines a managed shared memory with wide strings as keys for named objects, //!the default memory algorithm (with process-shared mutexes, //!and offset_ptr as internal pointers) as memory allocation algorithm //!and the default index type as the index. //!This class allows the shared memory to be mapped in different base //!in different processes typedef basic_managed_shared_memory <wchar_t ,/*Default memory algorithm defining void * as void_pointer*/ ,/*Default index type*/> wfixed_managed_shared_memory;
Managed shared memory is an advanced class that combines a shared memory object and a mapped region that covers all the shared memory object. That means that when we create a new managed shared memory:
When we open a managed shared memory
To use a managed shared memory, you must include the following header:
#include <boost/interprocess/managed_shared_memory.hpp>
//1. Creates a new shared memory object // called "MyName". //2. Maps the whole object to this // process' address space. //3. Constructs some objects in shared memory // to implement managed features. //!! If anything fails, throws interprocess_exception // managed_shared_memory segment ( create_only , "MyName" //Shared memory object name , 65536); //Shared memory object size in bytes
//1. Opens a shared memory object // called "MyName". //2. Maps the whole object to this // process' address space. //3. Obtains pointers to constructed internal objects // to implement managed features. //!! If anything fails, throws interprocess_exception // managed_shared_memory segment (open_only, "MyName"); //Shared memory object name
//1. If the segment was previously created // equivalent to "open_only" (size is ignored). //2. Otherwise, equivalent to "create_only" //!! If anything fails, throws interprocess_exception // managed_shared_memory segment ( open_or_create , "MyName" //Shared memory object name , 65536); //Shared memory object size in bytes
When the managed_shared_memory
object is destroyed, the shared memory object is automatically unmapped,
and all the resources are freed. To remove the shared memory object from
the system you must use the shared_memory_object::remove
function. Shared memory object removing might fail if any process still
has the shared memory object mapped.
The user can also map the managed shared memory in a fixed address. This
option is essential when using using fixed_managed_shared_memory
.
To do this, just add the mapping address as an extra parameter:
fixed_managed_shared_memory segment (open_only ,"MyFixedAddressSharedMemory" //Shared memory object name ,(void*)0x30000000 //Mapping address
Windows users might also want to use native windows shared memory instead
of the portable shared_memory_object
managed memory. This is achieved through the basic_managed_windows_shared_memory
class. To use it just include:
#include <boost/interprocess/managed_windows_shared_memory.hpp>
This class has the same interface as basic_managed_shared_memory
but uses native windows shared memory. Note that this managed class has
the same lifetime issues as the windows shared memory: when the last process
attached to the windows shared memory is detached from the memory (or ends/crashes)
the memory is destroyed. So there is no persistence support for windows
shared memory.
To communicate between system services and user applications using managed_windows_shared_memory
, please
read the explanations given in chapter Native
windows shared memory.
Unix users might also want to use XSI (system V) instead of the portable
shared_memory_object
managed memory. This is achieved through the basic_managed_xsi_shared_memory
class. To use it just include:
#include <boost/interprocess/managed_xsi_shared_memory.hpp>
This class has nearly the same interface as basic_managed_shared_memory
but uses XSI shared memory as backend.
For more information about managed XSI shared memory capabilities, see basic_managed_xsi_shared_memory
class reference.
As seen, basic_managed_mapped_file offers a great variety of customization. But for the average user, a common, default shared memory named object creation is needed. Because of this, Boost.Interprocess defines the most common managed mapped file specializations:
//Named object creation managed memory segment //All objects are constructed in the memory-mapped file // Names are c-strings, // Default memory management algorithm(rbtree_best_fit with no mutexes) // Name-object mappings are stored in the default index type (flat_map) typedef basic_managed_mapped_file < char, rbtree_best_fit<mutex_family, offset_ptr<void> >, flat_map_index > managed_mapped_file; //Named object creation managed memory segment //All objects are constructed in the memory-mapped file // Names are wide-strings, // Default memory management algorithm(rbtree_best_fit with no mutexes) // Name-object mappings are stored in the default index type (flat_map) typedef basic_managed_mapped_file< wchar_t, rbtree_best_fit<mutex_family, offset_ptr<void> >, flat_map_index > wmanaged_mapped_file;
managed_mapped_file
allocates
objects in a memory mapped files associated with a c-string and wmanaged_mapped_file
allocates objects
in a memory mapped file associated with a wchar_t null terminated string.
Both define the pointer type as offset_ptr<void>
so they can be used to map the file
at different base addresses in different processes.
Managed mapped file is an advanced class that combines a file and a mapped region that covers all the file. That means that when we create a new managed mapped file:
When we open a managed mapped file
To use a managed mapped file, you must include the following header:
#include <boost/interprocess/managed_mapped_file.hpp>
//1. Creates a new file // called "MyMappedFile". //2. Maps the whole file to this // process' address space. //3. Constructs some objects in the memory mapped // file to implement managed features. //!! If anything fails, throws interprocess_exception // managed_mapped_file mfile ( create_only , "MyMappedFile" //Mapped file name , 65536); //Mapped file size
//1. Opens a file // called "MyMappedFile". //2. Maps the whole file to this // process' address space. //3. Obtains pointers to constructed internal objects // to implement managed features. //!! If anything fails, throws interprocess_exception // managed_mapped_file mfile ( open_only , "MyMappedFile"); //Mapped file name
//1. If the file was previously created // equivalent to "open_only". //2. Otherwise, equivalent to "open_only" (size is ignored) // //!! If anything fails, throws interprocess_exception // managed_mapped_file mfile (open_or_create , "MyMappedFile" //Mapped file name , 65536); //Mapped file size
When the managed_mapped_file
object is destroyed, the file is automatically unmapped, and all the resources
are freed. To remove the file from the filesystem you could use standard
C std::remove
or Boost.Filesystem's
remove()
functions, but file removing might fail if any process still has the file
mapped in memory or the file is open by any process.
To obtain a more portable behaviour, use file_mapping::remove(const char *)
operation, which will remove the file even if it's being mapped. However,
removal will fail in some OS systems if the file (eg. by C++ file streams)
and no delete share permission was granted to the file. But in most common
cases file_mapping::remove
is portable enough.
For more information about managed mapped file capabilities, see basic_managed_mapped_file
class reference.
The following features are common to all managed memory segment classes, but we will use managed shared memory in our examples. We can do the same with memory mapped files or other managed memory segment classes.
If a basic raw-byte allocation is needed from a managed memory segment,
(for example, a managed shared memory), to implement top-level interprocess
communications, this class offers allocate
and deallocate functions. The allocation
function comes with throwing and no throwing versions. Throwing version
throws boost::interprocess::bad_alloc (which derives from std::bad_alloc
) if there is no more memory
and the non-throwing version returns 0 pointer.
#include <boost/interprocess/managed_shared_memory.hpp> int main() { using namespace boost::interprocess; //Remove shared memory on construction and destruction struct shm_remove { shm_remove() { shared_memory_object::remove("MyName"); } ~shm_remove(){ shared_memory_object::remove("MyName"); } } remover; //Managed memory segment that allocates portions of a shared memory //segment with the default management algorithm managed_shared_memory managed_shm(create_only,"MyName", 65536); //Allocate 100 bytes of memory from segment, throwing version void *ptr = managed_shm.allocate(100); //Deallocate it managed_shm.deallocate(ptr); //Non throwing version ptr = managed_shm.allocate(100, std::nothrow); //Deallocate it managed_shm.deallocate(ptr); return 0; }
The class also offers conversions between absolute addresses that belong to a managed memory segment and a handle that can be passed using any interprocess mechanism. That handle can be transformed again to an absolute address using a managed memory segment that also contains that object. Handles can be used as keys between processes to identify allocated portions of a managed memory segment or objects constructed in the managed segment.
//Process A obtains the offset of the address managed_shared_memory::handle handle = segment.get_handle_from_address(processA_address); //Process A sends this address using any mechanism to process B //Process B obtains the handle and transforms it again to an address managed_shared_memory::handle handle = ... void * processB_address = segment.get_address_from_handle(handle);
When constructing objects in a managed memory segment (managed shared memory, managed mapped files...) associated with a name, the user has a varied object construction family to "construct" or to "construct if not found". Boost.Interprocess can construct a single object or an array of objects. The array can be constructed with the same parameters for all objects or we can define each parameter from a list of iterators:
//!Allocates and constructs an object of type MyType (throwing version) MyType *ptr = managed_memory_segment.construct<MyType>("Name") (par1, par2...); //!Allocates and constructs an array of objects of type MyType (throwing version) //!Each object receives the same parameters (par1, par2, ...) MyType *ptr = managed_memory_segment.construct<MyType>("Name")[count](par1, par2...); //!Tries to find a previously created object. If not present, allocates //!and constructs an object of type MyType (throwing version) MyType *ptr = managed_memory_segment.find_or_construct<MyType>("Name") (par1, par2...); //!Tries to find a previously created object. If not present, allocates and //!constructs an array of objects of type MyType (throwing version). Each object //!receives the same parameters (par1, par2, ...) MyType *ptr = managed_memory_segment.find_or_construct<MyType>("Name")[count](par1, par2...); //!Allocates and constructs an array of objects of type MyType (throwing version) //!Each object receives parameters returned with the expression (*it1++, *it2++,... ) MyType *ptr = managed_memory_segment.construct_it<MyType>("Name")[count](it1, it2...); //!Tries to find a previously created object. If not present, allocates and constructs //!an array of objects of type MyType (throwing version). Each object receives //!parameters returned with the expression (*it1++, *it2++,... ) MyType *ptr = managed_memory_segment.find_or_construct_it<MyType>("Name")[count](it1, it2...); //!Tries to find a previously created object. Returns a pointer to the object and the //!count (if it is not an array, returns 1). If not present, the returned pointer is 0 std::pair<MyType *,std::size_t> ret = managed_memory_segment.find<MyType>("Name"); //!Destroys the created object, returns false if not present bool destroyed = managed_memory_segment.destroy<MyType>("Name"); //!Destroys the created object via pointer managed_memory_segment.destroy_ptr(ptr);
All these functions have a non-throwing version, that is invoked with an additional parameter std::nothrow. For example, for simple object construction:
//!Allocates and constructs an object of type MyType (no throwing version) MyType *ptr = managed_memory_segment.construct<MyType>("Name", std::nothrow) (par1, par2...);
Sometimes, the user doesn't want to create class objects associated with a name. For this purpose, Boost.Interprocess can create anonymous objects in a managed memory segment. All named object construction functions are available to construct anonymous objects. To allocate an anonymous objects, the user must use "boost::interprocess::anonymous_instance" name instead of a normal name:
MyType *ptr = managed_memory_segment.construct<MyType>(anonymous_instance) (par1, par2...); //Other construct variants can also be used (including non-throwing ones) ... //We can only destroy the anonymous object via pointer managed_memory_segment.destroy_ptr(ptr);
Find functions have no sense here, since anonymous objects have no name. We can only destroy the anonymous object via pointer.
Sometimes, the user wants to emulate a singleton in a managed memory segment. Obviously, as the managed memory segment is constructed at run-time, the user must construct and destroy this object explicitly. But how can the user be sure that the object is the only object of its type in the managed memory segment? This can be emulated using a named object and checking if it is present before trying to create one, but all processes must agree in the object's name, that can also conflict with other existing names.
To solve this, Boost.Interprocess offers a "unique object" creation in a managed memory segment. Only one instance of a class can be created in a managed memory segment using this "unique object" service (you can create more named objects of this class, though) so it makes easier the emulation of singleton-like objects across processes, for example, to design pooled, shared memory allocators. The object can be searched using the type of the class as a key.
// Construct MyType *ptr = managed_memory_segment.construct<MyType>(unique_instance) (par1, par2...); // Find it std::pair<MyType *,std::size_t> ret = managed_memory_segment.find<MyType>(unique_instance); // Destroy it managed_memory_segment.destroy<MyType>(unique_instance); // Other construct and find variants can also be used (including non-throwing ones) //...
// We can also destroy the unique object via pointer MyType *ptr = managed_memory_segment.construct<MyType>(unique_instance) (par1, par2...); managed_shared_memory.destroy_ptr(ptr);
The find function obtains a pointer to the only object of type T that can be created using this "unique instance" mechanism.
One of the features of named/unique allocations/searches/destructions is
that they are atomic. Named allocations
use the recursive synchronization scheme defined by the internal mutex_family
typedef defined of the memory
allocation algorithm template parameter (MemoryAlgorithm
).
That is, the mutex type used to synchronize named/unique allocations is
defined by the MemoryAlgorithm::mutex_family::recursive_mutex_type
type. For shared memory, and memory mapped file based managed segments
this recursive mutex is defined as interprocess_recursive_mutex
.
If two processes can call:
MyType *ptr = managed_shared_memory.find_or_construct<MyType>("Name")[count](par1, par2...);
at the same time, but only one process will create the object and the other will obtain a pointer to the created object.
Raw allocation using allocate()
can be called also safely while executing
named/anonymous/unique allocations, just like when programming a multithreaded
application inserting an object in a mutex-protected map does not block
other threads from calling new[] while the map thread is searching the
place where it has to insert the new object. The synchronization does happen
once the map finds the correct place and it has to allocate raw memory
to construct the new value.
This means that if we are creating or searching for a lot of named objects, we only block creation/searches from other processes but we don't block another process if that process is inserting elements in a shared memory vector.
As seen, managed memory segments, when creating named objects, store the name/object association in an index. The index is a map with the name of the object as a key and a pointer to the object as the mapped type. The default specializations, managed_shared_memory and wmanaged_shared_memory, use flat_map_index as the index type.
Each index has its own characteristics, like search-time, insertion time, deletion time, memory use, and memory allocation patterns. Boost.Interprocess offers 3 index types right now:
As an example, if we want to define new managed shared memory class using boost::interprocess::map as the index type we just must specify [boost::interprocess::map_index map_index] as a template parameter:
//This managed memory segment can allocate objects with: // -> a wchar_t string as key // -> boost::interprocess::rbtree_best_fit with process-shared mutexes // as memory allocation algorithm. // -> boost::interprocess::map<...> as the index to store name/object mappings // typedef boost::interprocess::basic_managed_shared_memory < wchar_t , boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, offset_ptr<void> > , boost::interprocess::map_index > my_managed_shared_memory;
If these indexes are not enough for you, you can define your own index type. To know how to do this, go to Building custom indexes section.
All Boost.Interprocess managed memory segment classes construct in their respective memory segments (shared memory, memory mapped files, heap memory...) some structures to implement the memory management algorithm, named allocations, synchronization objects... All these objects are encapsulated in a single object called segment manager. A managed memory mapped file and a managed shared memory use the same segment manager to implement all managed memory segment features, due to the fact that a segment manager is a class that manages a fixed size memory buffer. Since both shared memory or memory mapped files are accessed though a mapped region, and a mapped region is a fixed size memory buffer, a single segment manager class can manage several managed memory segment types.
Some Boost.Interprocess classes require
a pointer to the segment manager in their constructors, and the segment
manager can be obtained from any managed memory segment using get_segment_manager
member:
managed_shared_memory::segment_manager *seg_manager = managed_shm.get_segment_manager();
Once an object is constructed using construct<>
function family, the programmer
can obtain information about the object using a pointer to the object.
The programmer can obtain the following information:
Here is an example showing this functionality:
#include <boost/interprocess/managed_shared_memory.hpp> #include <cassert> #include <cstring> class my_class { //... }; int main() { using namespace boost::interprocess; //Remove shared memory on construction and destruction struct shm_remove { shm_remove() { shared_memory_object::remove("MyName"); } ~shm_remove(){ shared_memory_object::remove("MyName"); } } remover; managed_shared_memory managed_shm(create_only, "MyName", 10000*sizeof(std::size_t)); //Construct objects my_class *named_object = managed_shm.construct<my_class>("Object name")[1](); my_class *unique_object = managed_shm.construct<my_class>(unique_instance)[2](); my_class *anon_object = managed_shm.construct<my_class>(anonymous_instance)[3](); //Now test "get_instance_name" function. assert(0 == std::strcmp(managed_shared_memory::get_instance_name(named_object), "Object name")); assert(0 == std::strcmp(managed_shared_memory::get_instance_name(unique_object), typeid(my_class).name())); assert(0 == managed_shared_memory::get_instance_name(anon_object)); //Now test "get_instance_type" function. assert(named_type == managed_shared_memory::get_instance_type(named_object)); assert(unique_type == managed_shared_memory::get_instance_type(unique_object)); assert(anonymous_type == managed_shared_memory::get_instance_type(anon_object)); //Now test "get_instance_length" function. assert(1 == managed_shared_memory::get_instance_length(named_object)); assert(2 == managed_shared_memory::get_instance_length(unique_object)); assert(3 == managed_shared_memory::get_instance_length(anon_object)); managed_shm.destroy_ptr(named_object); managed_shm.destroy_ptr(unique_object); managed_shm.destroy_ptr(anon_object); return 0; }
Sometimes the programmer must execute some code, and needs to execute it with the guarantee that no other process or thread will create or destroy any named, unique or anonymous object while executing the functor. A user might want to create several named objects and initialize them, but those objects should be available for the rest of processes at once.
To achieve this, the programmer can use the atomic_func()
function offered by managed classes:
//This object function will create several named objects create_several_objects_func func(/**/); //While executing the function, no other process will be //able to create or destroy objects managed_memory.atomic_func(func);
Note that atomic_func
does
not prevent other processes from allocating raw memory or executing member
functions for already constructed objects (e.g.: another process might
be pushing elements into a vector placed in the segment). The atomic function
only blocks named, unique and anonymous creation, search and destruction
(concurrent calls to construct<>
, find<>
, find_or_construct<>
, destroy<>
...) from other processes.
These functions are available to obtain information about the managed memory segments:
Obtain the size of the memory segment:
managed_shm.get_size();
Obtain the number of free bytes of the segment:
managed_shm.get_free_memory();
Clear to zero the free memory:
managed_shm.zero_free_memory();
Know if all memory has been deallocated, false otherwise:
managed_shm.all_memory_deallocated();
Test internal structures of the managed segment. Returns true if no errors are detected:
managed_shm.check_sanity();
Obtain the number of named and unique objects allocated in the segment:
managed_shm.get_num_named_objects(); managed_shm.get_num_unique_objects();
Once a managed segment is created the managed segment can't be grown. The limitation is not easily solvable: every process attached to the managed segment would need to be stopped, notified of the new size, they would need to remap the managed segment and continue working. Nearly impossible to achieve with a user-level library without the help of the operating system kernel.
On the other hand, Boost.Interprocess offers off-line segment growing. What does this mean? That the segment can be grown if no process has mapped the managed segment. If the application can find a moment where no process is attached it can grow or shrink to fit the managed segment.
Here we have an example showing how to grow and shrink to fit managed_shared_memory
:
#include <boost/interprocess/managed_shared_memory.hpp> #include <boost/interprocess/managed_mapped_file.hpp> #include <cassert> class MyClass { //... }; int main() { using namespace boost::interprocess; //Remove shared memory on construction and destruction struct shm_remove { shm_remove() { shared_memory_object::remove("MyName"); } ~shm_remove(){ shared_memory_object::remove("MyName"); } } remover; { //Create a managed shared memory managed_shared_memory shm(create_only, "MyName", 1000); //Check size assert(shm.get_size() == 1000); //Construct a named object MyClass *myclass = shm.construct<MyClass>("MyClass")(); //The managed segment is unmapped here } { //Now that the segment is not mapped grow it adding extra 500 bytes managed_shared_memory::grow("MyName", 500); //Map it again managed_shared_memory shm(open_only, "MyName"); //Check size assert(shm.get_size() == 1500); //Check "MyClass" is still there MyClass *myclass = shm.find<MyClass>("MyClass").first; assert(myclass != 0); //The managed segment is unmapped here } { //Now minimize the size of the segment managed_shared_memory::shrink_to_fit("MyName"); //Map it again managed_shared_memory shm(open_only, "MyName"); //Check size assert(shm.get_size() < 1000); //Check "MyClass" is still there MyClass *myclass = shm.find<MyClass>("MyClass").first; assert(myclass != 0); //The managed segment is unmapped here } return 0; }
managed_mapped_file
also offers a similar function to grow or shrink_to_fit the managed file.
Please, remember that no process should be modifying
the file/shared memory while the growing/shrinking process is performed.
Otherwise, the managed segment will be corrupted.
As mentioned, the managed segment stores the information about named and unique objects in two indexes. Depending on the type of those indexes, the index must reallocate some auxiliary structures when new named or unique allocations are made. For some indexes, if the user knows how many named or unique objects are going to be created it's possible to preallocate some structures to obtain much better performance. (If the index is an ordered vector it can preallocate memory to avoid reallocations. If the index is a hash structure it can preallocate the bucket array).
The following functions reserve memory to make the subsequent allocation
of named or unique objects more efficient. These functions are only useful
for pseudo-intrusive or non-node indexes (like flat_map_index
,
iunordered_set_index
).
These functions have no effect with the default index (iset_index
)
or other indexes (map_index
):
managed_shm.reserve_named_objects(1000); managed_shm.reserve_unique_objects(1000);
managed_shm.reserve_named_objects(1000); managed_shm.reserve_unique_objects(1000);
Managed memory segments also offer the possibility to iterate through constructed named and unique objects for debugging purposes. Caution: this iteration is not thread-safe so the user should make sure that no other thread is manipulating named or unique indexes (creating, erasing, reserving...) in the segment. Other operations not involving indexes can be concurrently executed (raw memory allocation/deallocations, for example).
The following functions return constant iterators to the range of named and unique objects stored in the managed segment. Depending on the index type, iterators might be invalidated after a named or unique creation/erasure/reserve operation:
typedef managed_shared_memory::const_named_iterator const_named_it; const_named_it named_beg = managed_shm.named_begin(); const_named_it named_end = managed_shm.named_end(); typedef managed_shared_memory::const_unique_iterator const_unique_it; const_unique_it unique_beg = managed_shm.unique_begin(); const_unique_it unique_end = managed_shm.unique_end(); for(; named_beg != named_end; ++named_beg){ //A pointer to the name of the named object const managed_shared_memory::char_type *name = named_beg->name(); //The length of the name std::size_t name_len = named_beg->name_length(); //A constant void pointer to the named object const void *value = named_beg->value(); } for(; unique_beg != unique_end; ++unique_beg){ //The typeid(T).name() of the unique object const char *typeid_name = unique_beg->name(); //The length of the name std::size_t name_len = unique_beg->name_length(); //A constant void pointer to the unique object const void *value = unique_beg->value(); }
Sometimes it's interesting to be able to allocate aligned fragments of memory because of some hardware or software restrictions. Sometimes, having aligned memory is a feature that can be used to improve several memory algorithms.
This allocation is similar to the previously shown raw memory allocation but it takes an additional parameter specifying the alignment. There is a restriction for the alignment: the alignment must be power of two.
If a user wants to allocate many aligned blocks (for example aligned to 128 bytes), the size that minimizes the memory waste is a value that's is nearly a multiple of that alignment (for example 2*128 - some bytes). The reason for this is that every memory allocation usually needs some additional metadata in the first bytes of the allocated buffer. If the user can know the value of "some bytes" and if the first bytes of a free block of memory are used to fulfill the aligned allocation, the rest of the block can be left also aligned and ready for the next aligned allocation. Note that requesting a size multiple of the alignment is not optimal because lefts the next block of memory unaligned due to the needed metadata.
Once the programmer knows the size of the payload of every memory allocation, he can request a size that will be optimal to allocate aligned chunks of memory maximizing both the size of the request and the possibilities of future aligned allocations. This information is stored in the PayloadPerAllocation constant of managed memory segments.
Here is a small example showing how aligned allocation is used:
#include <boost/interprocess/managed_shared_memory.hpp> #include <cassert> int main() { using namespace boost::interprocess; //Remove shared memory on construction and destruction struct shm_remove { shm_remove() { shared_memory_object::remove("MyName"); } ~shm_remove(){ shared_memory_object::remove("MyName"); } } remover; //Managed memory segment that allocates portions of a shared memory //segment with the default management algorithm managed_shared_memory managed_shm(create_only, "MyName", 65536); const std::size_t Alignment = 128; //Allocate 100 bytes aligned to Alignment from segment, throwing version void *ptr = managed_shm.allocate_aligned(100, Alignment); //Check alignment assert(std::size_t(static_cast<char*>(ptr)-static_cast<char*>(0)) % Alignment == 0); //Deallocate it managed_shm.deallocate(ptr); //Non throwing version ptr = managed_shm.allocate_aligned(100, Alignment, std::nothrow); //Check alignment assert(std::size_t(static_cast<char*>(ptr)-static_cast<char*>(0)) % Alignment == 0); //Deallocate it managed_shm.deallocate(ptr); //If we want to efficiently allocate aligned blocks of memory //use managed_shared_memory::PayloadPerAllocation value assert(Alignment > managed_shared_memory::PayloadPerAllocation); //This allocation will maximize the size of the aligned memory //and will increase the possibility of finding more aligned memory ptr = managed_shm.allocate_aligned (3u*Alignment - managed_shared_memory::PayloadPerAllocation, Alignment); //Check alignment assert(std::size_t(static_cast<char*>(ptr)-static_cast<char*>(0)) % Alignment == 0); //Deallocate it managed_shm.deallocate(ptr); return 0; }
Caution | |
---|---|
This feature is experimental, interface and ABI are unstable |
If an application needs to allocate a lot of memory buffers but it needs
to deallocate them independently, the application is normally forced to
loop calling allocate()
.
Managed memory segments offer an alternative function to pack several allocations
in a single call obtaining memory buffers that:
This allocation method is much faster than calling allocate()
in a loop. The downside is that the segment
must provide a contiguous memory segment big enough to hold all the allocations.
Managed memory segments offer this functionality through allocate_many()
functions. There are 2 types of allocate_many
functions:
//!Allocates n_elements of elem_bytes bytes. //!Throws bad_alloc on failure. chain.size() is not increased on failure. void allocate_many(size_type elem_bytes, size_type n_elements, multiallocation_chain &chain); //!Allocates n_elements, each one of element_lengths[i]*sizeof_element bytes. //!Throws bad_alloc on failure. chain.size() is not increased on failure. void allocate_many(const size_type *element_lengths, size_type n_elements, size_type sizeof_element, multiallocation_chain &chain); //!Allocates n_elements of elem_bytes bytes. //!Non-throwing version. chain.size() is not increased on failure. void allocate_many(std::nothrow_t, size_type elem_bytes, size_type n_elements, multiallocation_chain &chain); //!Allocates n_elements, each one of //!element_lengths[i]*sizeof_element bytes. //!Non-throwing version. chain.size() is not increased on failure. void allocate_many(std::nothrow_t, const size_type *elem_sizes, size_type n_elements, size_type sizeof_element, multiallocation_chain &chain); //!Deallocates all elements contained in chain. //!Never throws. void deallocate_many(multiallocation_chain &chain);
Here is a small example showing all this functionality:
#include <boost/interprocess/managed_shared_memory.hpp> #include <boost/move/utility_core.hpp> //boost::move #include <cassert>//assert #include <cstring>//std::memset #include <new> //std::nothrow #include <vector> //std::vector int main() { using namespace boost::interprocess; typedef managed_shared_memory::multiallocation_chain multiallocation_chain; //Remove shared memory on construction and destruction struct shm_remove { shm_remove() { shared_memory_object::remove("MyName"); } ~shm_remove(){ shared_memory_object::remove("MyName"); } } remover; managed_shared_memory managed_shm(create_only,"MyName", 65536); //Allocate 16 elements of 100 bytes in a single call. Non-throwing version. multiallocation_chain chain; managed_shm.allocate_many(std::nothrow, 100, 16, chain); //Check if the memory allocation was successful if(chain.empty()) return 1; //Allocated buffers std::vector<void*> allocated_buffers; //Initialize our data while(!chain.empty()){ void *buf = chain.pop_front(); allocated_buffers.push_back(buf); //The iterator must be incremented before overwriting memory //because otherwise, the iterator is invalidated. std::memset(buf, 0, 100); } //Now deallocate while(!allocated_buffers.empty()){ managed_shm.deallocate(allocated_buffers.back()); allocated_buffers.pop_back(); } //Allocate 10 buffers of different sizes in a single call. Throwing version managed_shared_memory::size_type sizes[10]; for(std::size_t i = 0; i < 10; ++i) sizes[i] = i*3; managed_shm.allocate_many(sizes, 10, 1, chain); managed_shm.deallocate_many(chain); return 0; }
Allocating N buffers of the same size improves the performance of pools and node containers (for example STL-like lists): when inserting a range of forward iterators in a STL-like list, the insertion function can detect the number of needed elements and allocate in a single call. The nodes still can be deallocated.
Allocating N buffers of different sizes can be used to speed up allocation
in cases where several objects must always be allocated at the same time
but deallocated at different times. For example, a class might perform
several initial allocations (some header data for a network packet, for
example) in its constructor but also allocations of buffers that might
be reallocated in the future (the data to be sent through the network).
Instead of allocating all the data independently, the constructor might
use allocate_many()
to speed up the initialization, but it still can deallocate and expand
the memory of the variable size element.
In general, allocate_many
is useful with large values of N. Overuse of allocate_many
can increase the effective memory usage, because it can't reuse existing
non-contiguous memory fragments that might be available for some of the
elements.
When programming some data structures such as vectors, memory reallocation becomes an important tool to improve performance. Managed memory segments offer an advanced reallocation function that offers:
The expansion can be combined with the allocation of a new buffer if the expansion fails obtaining a function with "expand, if fails allocate a new buffer" semantics.
Apart from this features, the function always returns the real size of
the allocated buffer, because many times, due to alignment issues the allocated
buffer a bit bigger than the requested size. Thus, the programmer can maximize
the memory use using allocation_command
.
Here is the declaration of the function:
enum boost::interprocess::allocation_type { //Bitwise OR (|) combinable values boost::interprocess::allocate_new = ..., boost::interprocess::expand_fwd = ..., boost::interprocess::expand_bwd = ..., boost::interprocess::shrink_in_place = ..., boost::interprocess::nothrow_allocation = ... }; template<class T> std::pair<T *, bool> allocation_command( boost::interprocess::allocation_type command , std::size_t limit_size , size_type &prefer_in_recvd_out_size , T *&reuse_ptr);
Preconditions for the function:
boost::interprocess::shrink_in_place
it can't contain any of these values: boost::interprocess::expand_fwd
,
boost::interprocess::expand_bwd
.
boost::interprocess::expand_fwd
or boost::interprocess::expand_bwd
, the parameter reuse_ptr
must be non-null and returned
by a previous allocation function.
boost::interprocess::shrink_in_place
,
the parameter limit_size
must be equal or greater than the parameter preferred_size
.
command
contains any of these values: boost::interprocess::expand_fwd
or boost::interprocess::expand_bwd
, the parameter limit_size
must be equal or less
than the parameter preferred_size
.
Which are the effects of this function:
boost::interprocess::shrink_in_place
,
the function will try to reduce the size of the memory block referenced
by pointer reuse_ptr
to the value preferred_size
moving only the end of the block. If it's not possible, it will try
to reduce the size of the memory block as much as possible as long
as this results in size(p) <= limit_size
. Success is reported only
if this results in preferred_size
<= size(p)
and size(p) <= limit_size
.
command
only contains the value boost::interprocess::expand_fwd
(with optional additional boost::interprocess::nothrow_allocation
),
the allocator will try to increase the size of the memory block referenced
by pointer reuse moving only the end of the block to the value preferred_size
. If it's not possible,
it will try to increase the size of the memory block as much as possible
as long as this results in size(p) >= limit_size
. Success is reported only
if this results in limit_size
<= size(p)
.
command
only contains the value boost::interprocess::expand_bwd
(with optional additional boost::interprocess::nothrow_allocation
),
the allocator will try to increase the size of the memory block referenced
by pointer reuse_ptr
only moving the start of the block to a returned new position new_ptr
. If it's not possible, it
will try to move the start of the block as much as possible as long
as this results in size(new_ptr) >= limit_size
. Success is reported only
if this results in limit_size
<= size(new_ptr)
.
command
only contains the value boost::interprocess::allocate_new
(with optional additional boost::interprocess::nothrow_allocation
),
the allocator will try to allocate memory for preferred_size
objects. If it's not possible it will try to allocate memory for at
least limit_size
objects.
command
only contains a combination of boost::interprocess::expand_fwd
and boost::interprocess::allocate_new
, (with optional additional
boost::interprocess::nothrow_allocation
) the allocator
will try first the forward expansion. If this fails, it would try a
new allocation.
command
only contains a combination of boost::interprocess::expand_bwd
and boost::interprocess::allocate_new
(with optional additional
boost::interprocess::nothrow_allocation
), the allocator
will try first to obtain preferred_size
objects using both methods if necessary. If this fails, it will try
to obtain limit_size
objects using both methods if necessary.
command
only contains a combination of boost::interprocess::expand_fwd
and boost::interprocess::expand_bwd
(with optional additional
boost::interprocess::nothrow_allocation
), the allocator
will try first forward expansion. If this fails it will try to obtain
preferred_size objects using backwards expansion or a combination of
forward and backwards expansion. If this fails, it will try to obtain
limit_size
objects
using both methods if necessary.
command
only contains a combination of allocation_new, boost::interprocess::expand_fwd
and boost::interprocess::expand_bwd
, (with optional additional
boost::interprocess::nothrow_allocation
) the allocator
will try first forward expansion. If this fails it will try to obtain
preferred_size objects using new allocation, backwards expansion or
a combination of forward and backwards expansion. If this fails, it
will try to obtain limit_size
objects using the same methods.
received_size
.
On failure the allocator writes in received_size
a possibly successful limit_size
parameter for a new call.
Throws an exception if two conditions are met:
boost::interprocess::nothrow_allocation
.
This function returns:
boost::interprocess::nothrow_allocation
the first member
will be 0 if the allocation/expansion fails or there is an error in
preconditions.
Notes:
char
as template argument the returned buffer will be suitably aligned to
hold any type.
char
as template argument and a backwards expansion is performed, although
properly aligned, the returned buffer might not be suitable because
the distance between the new beginning and the old beginning might
not multiple of the type the user wants to construct, since due to
internal restrictions the expansion can be slightly bigger than the
requested bytes. When performing backwards expansion,
if you have already constructed objects in the old buffer, make sure
to specify correctly the type.
Here is a small example that shows the use of allocation_command
:
#include <boost/interprocess/managed_shared_memory.hpp> #include <cassert> int main() { using namespace boost::interprocess; //Remove shared memory on construction and destruction struct shm_remove { shm_remove() { shared_memory_object::remove("MyName"); } ~shm_remove(){ shared_memory_object::remove("MyName"); } } remover; //Managed memory segment that allocates portions of a shared memory //segment with the default management algorithm managed_shared_memory managed_shm(create_only, "MyName", 10000*sizeof(std::size_t)); //Allocate at least 100 bytes, 1000 bytes if possible managed_shared_memory::size_type min_size = 100; managed_shared_memory::size_type first_received_size = 1000; std::size_t *hint = 0; std::size_t *ptr = managed_shm.allocation_command<std::size_t> (boost::interprocess::allocate_new, min_size, first_received_size, hint); //Received size must be bigger than min_size assert(first_received_size >= min_size); //Get free memory managed_shared_memory::size_type free_memory_after_allocation = managed_shm.get_free_memory(); //Now write the data for(std::size_t i = 0; i < first_received_size; ++i) ptr[i] = i; //Now try to triplicate the buffer. We won't admit an expansion //lower to the double of the original buffer. //This "should" be successful since no other class is allocating //memory from the segment min_size = first_received_size*2; managed_shared_memory::size_type expanded_size = first_received_size*3; std::size_t * ret = managed_shm.allocation_command (boost::interprocess::expand_fwd, min_size, expanded_size, ptr); //Check invariants assert(ptr != 0); assert(ret == ptr); assert(expanded_size >= first_received_size*2); //Get free memory and compare managed_shared_memory::size_type free_memory_after_expansion = managed_shm.get_free_memory(); assert(free_memory_after_expansion < free_memory_after_allocation); //Write new values for(std::size_t i = first_received_size; i < expanded_size; ++i) ptr[i] = i; //Try to shrink approximately to min_size, but the new size //should be smaller than min_size*2. //This "should" be successful since no other class is allocating //memory from the segment managed_shared_memory::size_type shrunk_size = min_size; ret = managed_shm.allocation_command (boost::interprocess::shrink_in_place, min_size*2, shrunk_size, ptr); //Check invariants assert(ptr != 0); assert(ret == ptr); assert(shrunk_size <= min_size*2); assert(shrunk_size >= min_size); //Get free memory and compare managed_shared_memory::size_type free_memory_after_shrinking = managed_shm.get_free_memory(); assert(free_memory_after_shrinking > free_memory_after_expansion); //Deallocate the buffer managed_shm.deallocate(ptr); return 0; }
allocation_command
is a
very powerful function that can lead to important performance gains. It's
specially useful when programming vector-like data structures where the
programmer can minimize both the number of allocation requests and the
memory waste.
When mapping a memory segment based on shared memory or files, there is
an option to open them using open_copy_on_write
option. This option is similar to open_only
but every change the programmer does with this managed segment is kept
private to this process and is not translated to the underlying device
(shared memory or file).
The underlying shared memory or file is opened as read-only so several processes can share an initial managed segment and make private changes to it. If many processes open a managed segment in copy on write mode and not modified pages from the managed segment will be shared between all those processes, with considerable memory savings.
Opening managed shared memory and mapped files with open_read_only maps the underlying device in memory with read-only attributes. This means that any attempt to write that memory, either creating objects or locking any mutex might result in an page-fault error (and thus, program termination) from the OS. Read-only mode opens the underlying device (shared memory, file...) in read-only mode and can result in considerable memory savings if several processes just want to process a managed memory segment without modifying it. Read-only mode operations are limited:
find<>
member function avoids using
internal locks and can be used to look for named and unique objects.
Here is an example that shows the use of these two open modes:
#include <boost/interprocess/managed_mapped_file.hpp> #include <fstream> //std::fstream #include <iterator>//std::distance int main() { using namespace boost::interprocess; //Define file names const char *ManagedFile = "MyManagedFile"; const char *ManagedFile2 = "MyManagedFile2"; //Try to erase any previous managed segment with the same name file_mapping::remove(ManagedFile); file_mapping::remove(ManagedFile2); remove_file_on_destroy destroyer1(ManagedFile); remove_file_on_destroy destroyer2(ManagedFile2); { //Create an named integer in a managed mapped file managed_mapped_file managed_file(create_only, ManagedFile, 65536); managed_file.construct<int>("MyInt")(0); //Now create a copy on write version managed_mapped_file managed_file_cow(open_copy_on_write, ManagedFile); //Erase the int and create a new one if(!managed_file_cow.destroy<int>("MyInt")) throw int(0); managed_file_cow.construct<int>("MyInt2"); //Check changes if(managed_file_cow.find<int>("MyInt").first && !managed_file_cow.find<int>("MyInt2").first) throw int(0); //Check the original is intact if(!managed_file.find<int>("MyInt").first && managed_file.find<int>("MyInt2").first) throw int(0); { //Dump the modified copy on write segment to a file std::fstream file(ManagedFile2, std::ios_base::out | std::ios_base::binary); if(!file) throw int(0); file.write(static_cast<const char *>(managed_file_cow.get_address()), (std::streamsize)managed_file_cow.get_size()); } //Now open the modified file and test changes managed_mapped_file managed_file_cow2(open_only, ManagedFile2); if(managed_file_cow2.find<int>("MyInt").first && !managed_file_cow2.find<int>("MyInt2").first) throw int(0); } { //Now create a read-only version managed_mapped_file managed_file_ro(open_read_only, ManagedFile); //Check the original is intact if(!managed_file_ro.find<int>("MyInt").first && managed_file_ro.find<int>("MyInt2").first) throw int(0); //Check the number of named objects using the iterators if(std::distance(managed_file_ro.named_begin(), managed_file_ro.named_end()) != 1 && std::distance(managed_file_ro.unique_begin(), managed_file_ro.unique_end()) != 0 ) throw int(0); } return 0; }
Boost.Interprocess offers managed shared
memory between processes using managed_shared_memory
or managed_mapped_file
. Two
processes just map the same the memory mappable resource and read from and
write to that object.
Many times, we don't want to use that shared memory approach and we prefer
to send serialized data through network, local socket or message queues.
Serialization can be done through Boost.Serialization
or similar library. However, if two processes share the same ABI (application
binary interface), we could use the same object and container construction
capabilities of managed_shared_memory
or managed_heap_memory
to
build all the information in a single buffer that will be sent, for example,
though message queues. The receiver would just copy the data to a local buffer,
and it could read or modify it directly without deserializing the data .
This approach can be much more efficient that a complex serialization mechanism.
Applications for Boost.Interprocess services using non-shared memory buffers:
To help with this management, Boost.Interprocess
provides two useful classes, basic_managed_heap_memory
and basic_managed_external_buffer
:
Sometimes, the user wants to create simple objects, STL compatible containers, STL compatible strings and more, all in a single buffer. This buffer could be a big static buffer, a memory-mapped auxiliary device or any other user buffer.
This would allow an easy serialization and we-ll just need to copy the buffer to duplicate all the objects created in the original buffer, including complex objects like maps, lists.... Boost.Interprocess offers managed memory segment classes to handle user provided buffers that allow the same functionality as shared memory classes:
//Named object creation managed memory segment //All objects are constructed in a user provided buffer template < class CharType, class MemoryAlgorithm, template<class IndexConfig> class IndexType > class basic_managed_external_buffer; //Named object creation managed memory segment //All objects are constructed in a user provided buffer // Names are c-strings, // Default memory management algorithm // (rbtree_best_fit with no mutexes and relative pointers) // Name-object mappings are stored in the default index type (flat_map) typedef basic_managed_external_buffer < char, rbtree_best_fit<null_mutex_family, offset_ptr<void> >, flat_map_index > managed_external_buffer; //Named object creation managed memory segment //All objects are constructed in a user provided buffer // Names are wide-strings, // Default memory management algorithm // (rbtree_best_fit with no mutexes and relative pointers) // Name-object mappings are stored in the default index type (flat_map) typedef basic_managed_external_buffer< wchar_t, rbtree_best_fit<null_mutex_family, offset_ptr<void> >, flat_map_index > wmanaged_external_buffer;
To use a managed external buffer, you must include the following header:
#include <boost/interprocess/managed_external_buffer.hpp>
Let's see an example of the use of managed_external_buffer:
#include <boost/interprocess/managed_external_buffer.hpp> #include <boost/interprocess/allocators/allocator.hpp> #include <boost/container/list.hpp> #include <cstring> #include <boost/aligned_storage.hpp> int main() { using namespace boost::interprocess; //Create the static memory who will store all objects const int memsize = 65536; static boost::aligned_storage<memsize>::type static_buffer; //This managed memory will construct objects associated with //a wide string in the static buffer wmanaged_external_buffer objects_in_static_memory (create_only, &static_buffer, memsize); //We optimize resources to create 100 named objects in the static buffer objects_in_static_memory.reserve_named_objects(100); //Alias an integer node allocator type //This allocator will allocate memory inside the static buffer typedef allocator<int, wmanaged_external_buffer::segment_manager> allocator_t; //Alias a STL compatible list to be constructed in the static buffer typedef boost::container::list<int, allocator_t> MyBufferList; //The list must be initialized with the allocator //All objects created with objects_in_static_memory will //be stored in the static_buffer! MyBufferList *list = objects_in_static_memory.construct<MyBufferList>(L"MyList") (objects_in_static_memory.get_segment_manager()); //Since the allocation algorithm from wmanaged_external_buffer uses relative //pointers and all the pointers constructed int the static memory point //to objects in the same segment, we can create another static buffer //from the first one and duplicate all the data. static boost::aligned_storage<memsize>::type static_buffer2; std::memcpy(&static_buffer2, &static_buffer, memsize); //Now open the duplicated managed memory passing the memory as argument wmanaged_external_buffer objects_in_static_memory2 (open_only, &static_buffer2, memsize); //Check that "MyList" has been duplicated in the second buffer if(!objects_in_static_memory2.find<MyBufferList>(L"MyList").first) return 1; //Destroy the lists from the static buffers objects_in_static_memory.destroy<MyBufferList>(L"MyList"); objects_in_static_memory2.destroy<MyBufferList>(L"MyList"); return 0; }
Boost.Interprocess STL compatible allocators can also be used to place STL compatible containers in the user segment.
basic_managed_external_buffer
can be also useful to build small databases for embedded systems limiting
the size of the used memory to a predefined memory chunk, instead of letting
the database fragment the heap memory.
Note: The external memory supplied by
the user shall be aligned to the maximum value between alignof(max_align_t)
and the alignment of the strictest over-aligned
type to be built inside that memory.
The use of heap memory (new/delete) to obtain a buffer where the user wants to store all his data is very common, so Boost.Interprocess provides some specialized classes that work exclusively with heap memory.
These are the classes:
//Named object creation managed memory segment //All objects are constructed in a single buffer allocated via new[] template < class CharType, class MemoryAlgorithm, template<class IndexConfig> class IndexType > class basic_managed_heap_memory; //Named object creation managed memory segment //All objects are constructed in a single buffer allocated via new[] // Names are c-strings, // Default memory management algorithm // (rbtree_best_fit with no mutexes and relative pointers) // Name-object mappings are stored in the default index type (flat_map) typedef basic_managed_heap_memory < char, rbtree_best_fit<null_mutex_family>, flat_map_index > managed_heap_memory; //Named object creation managed memory segment //All objects are constructed in a single buffer allocated via new[] // Names are wide-strings, // Default memory management algorithm // (rbtree_best_fit with no mutexes and relative pointers) // Name-object mappings are stored in the default index type (flat_map) typedef basic_managed_heap_memory< wchar_t, rbtree_best_fit<null_mutex_family>, flat_map_index > wmanaged_heap_memory;
To use a managed heap memory, you must include the following header:
#include <boost/interprocess/managed_heap_memory.hpp>
The use is exactly the same as basic_managed_external_buffer
,
except that memory is created by the managed memory segment itself using
dynamic (new/delete) memory.
basic_managed_heap_memory also offers
a grow(std::size_t extra_bytes)
function that tries to resize internal
heap memory so that we have room for more objects. But be
careful, if memory is reallocated, the old buffer will be copied
into the new one so all the objects will be binary-copied to the new buffer.
To be able to use this function, all pointers constructed in the heap buffer
that point to objects in the heap buffer must be relative pointers (for
example offset_ptr
). Otherwise,
the result is undefined. Here is an example:
#include <boost/container/list.hpp> #include <boost/interprocess/managed_heap_memory.hpp> #include <boost/interprocess/allocators/allocator.hpp> #include <cstddef> using namespace boost::interprocess; typedef boost::container::list<int, allocator<int, managed_heap_memory::segment_manager> > MyList; int main () { //We will create a buffer of 1000 bytes to store a list managed_heap_memory heap_memory(1000); MyList * mylist = heap_memory.construct<MyList>("MyList") (heap_memory.get_segment_manager()); //Obtain handle, that identifies the list in the buffer managed_heap_memory::handle_t list_handle = heap_memory.get_handle_from_address(mylist); //Fill list until there is no more memory in the buffer try { while(1) { mylist->insert(mylist->begin(), 0); } } catch(const bad_alloc &){ //memory is full } //Let's obtain the size of the list MyList::size_type old_size = mylist->size(); //To make the list bigger, let's increase the heap buffer //in 1000 bytes more. heap_memory.grow(1000); //If memory has been reallocated, the old pointer is invalid, so //use previously obtained handle to find the new pointer. mylist = static_cast<MyList *> (heap_memory.get_address_from_handle(list_handle)); //Fill list until there is no more memory in the buffer try { while(1) { mylist->insert(mylist->begin(), 0); } } catch(const bad_alloc &){ //memory is full } //Let's obtain the new size of the list MyList::size_type new_size = mylist->size(); assert(new_size > old_size); //Destroy list heap_memory.destroy_ptr(mylist); return 0; }
Note: Heap memory is not guaranteed to be compatible with over-aligned types so it is not supported to use memory allocation algorithms or construct objects that need over-aligned memory:
// 64 byte aligned memory might not be supported typedef basic_managed_heap_memory< char, rbtree_best_fit<mutex_family, offset_ptr<void, long, unsigned long, 64>, 64>, iset_index > managed_heap_with_overalignment_t;
All managed memory segments have similar capabilities (memory allocation inside the memory segment, named object construction...), but there are some remarkable differences between managed_shared_memory, managed_mapped_file and managed_heap_memory, managed_external_file.
To see the utility of managed heap memory and managed external buffer classes, the following example shows how a message queue can be used to serialize a whole database constructed in a memory buffer using Boost.Interprocess, send the database through a message queue and duplicated in another buffer:
//This test creates a in memory data-base using Interprocess machinery and //serializes it through a message queue. Then rebuilds the data-base in //another buffer and checks it against the original data-base bool test_serialize_db() { //Typedef data to create a Interprocess map typedef std::pair<const std::size_t, std::size_t> MyPair; typedef std::less<std::size_t> MyLess; typedef node_allocator<MyPair, managed_external_buffer::segment_manager> node_allocator_t; typedef boost::container::map<std::size_t, std::size_t, std::less<std::size_t>, node_allocator_t> MyMap; //Some constants const std::size_t BufferSize = 65536; const std::size_t MaxMsgSize = 100; //Allocate a memory buffer to hold the destiny database using vector<char> std::vector<char> buffer_destiny(BufferSize, 0); message_queue::remove("MyName"); { //Create the message-queues message_queue mq1(create_only, "MyName", 1, MaxMsgSize); //Open previously created message-queue simulating other process message_queue mq2(open_only, "MyName"); //A managed heap memory to create the origin database managed_heap_memory db_origin(buffer_destiny.size()); //Construct the map in the first buffer MyMap *map1 = db_origin.construct<MyMap>("MyMap") (MyLess(), db_origin.get_segment_manager()); if(!map1) return false; //Fill map1 until is full try { std::size_t i = 0; while(1){ (*map1)[i] = i; ++i; } } catch(boost::interprocess::bad_alloc &){} //Data control data sending through the message queue std::size_t sent = 0; message_queue::size_type recvd = 0; message_queue::size_type total_recvd = 0; unsigned int priority; //Send whole first buffer through the mq1, read it //through mq2 to the second buffer while(1){ //Send a fragment of buffer1 through mq1 std::size_t bytes_to_send = MaxMsgSize < (db_origin.get_size() - sent) ? MaxMsgSize : (db_origin.get_size() - sent); mq1.send( &static_cast<char*>(db_origin.get_address())[sent] , bytes_to_send , 0); sent += bytes_to_send; //Receive the fragment through mq2 to buffer_destiny mq2.receive( &buffer_destiny[total_recvd] , BufferSize - recvd , recvd , priority); total_recvd += recvd; //Check if we have received all the buffer if(total_recvd == BufferSize){ break; } } //The buffer will contain a copy of the original database //so let's interpret the buffer with managed_external_buffer managed_external_buffer db_destiny(open_only, &buffer_destiny[0], BufferSize); //Let's find the map std::pair<MyMap *, managed_external_buffer::size_type> ret = db_destiny.find<MyMap>("MyMap"); MyMap *map2 = ret.first; //Check if we have found it if(!map2){ return false; } //Check if it is a single variable (not an array) if(ret.second != 1){ return false; } //Now let's compare size if(map1->size() != map2->size()){ return false; } //Now let's compare all db values MyMap::size_type num_elements = map1->size(); for(std::size_t i = 0; i < num_elements; ++i){ if((*map1)[i] != (*map2)[i]){ return false; } } //Destroy maps from db-s db_origin.destroy_ptr(map1); db_destiny.destroy_ptr(map2); } message_queue::remove("MyName"); return true; }