Implementing shm_get_var and shm_put_var in Java using JNI
Posted: Wed Feb 19, 2003 1:22 am
This is my goal.
I want to leverage the source code for these methods in PHP so that they can be reused from Java via JNI. After Java, it would seem reasonable to go after Python and PERL. The end game is to achieve a persistent dictionary object that transcends things like web servers, application servers and standalone applications--basically, fast access durable objects that exist until reboot or removed using ipcrm. Incidentally, I also use N-number of memory mapped files (but could be simple files) based on a hash code to route specific chunks of serialized data to discrete units of storage, and sort by these chunks so that resource utilization is kept to a minimum when a subset of key/value pairs are accessed, as each cache file is consumed and freed before the next one, and key/value pairs accessed are merged into a local array. This is done by queuing up keys requested and then requesting them all at once. The benefit of all this is that we now end up with a scalable dictionary object that minimizes page swaps through discrete chunks of memory, can be tuned for memory availability, and (using semaphores) handles concurrency. The developer is free to code in a language of preference with the ability to efficiently pass data back and forth to other processes--the IPC dream made simple. For Java, I would rewrite this PHP class as C++ code.
Recently, I developed a name/value pair dictionary class that uses these methods and some more traditional IPC methods found in C++. I simply found that the PHP methods were a very convenient way to store and fetch data with one small drawback. These methods store values in a manner that somewhat resembles serialized data with some additional padding and metadata that defines various lengths of objects. One other slight hurdle was developing a hash code generator that yields 32bit integer values that can be exactly reproduced in Java (and other languages), and an algorithm that ensures uniqueness. All this I achieved other than the wrapper I seek for shm_get_var and shm_put_var. I am assuming that these functions are components maintained in some CVS repository.
I have done plenty of custom PHP builds, so, I am sure the answer is right under my nose, but I cannot admit to any profound understanding of specific PHP functions and how they are implemented.
If anyone can confirm this and provide the references to this specific code, and/or assist me in obtaining the code snippets that uniquely define these functions, I would greatly appreciate it. I am not interested in reinventing the methods used to interpret the shared memory that shm_put_var manages and how shm_get_var is optimized to do what it does, unless that is the last resort--it seems doable after looking at a hex dump, just not very fun knowing that these methods could change in another PHP minor release.
I hope someone sees the value in this. I spent days looking at a number of solutions published on the web. None of them provided the level of tuning, scalability, concurrency and efficiency that I needed. I will be publishing the source code and a white paper on this and a few other template engine/caching models on my website at http://www.e-freelance-network.com after I finalize a story board for my site and add a few channels. Once I conquer this, I will add a durable queuing object with overflow to database, and a frequently accessed cached resultset object to the list of goodies published. I already have these in Java, and, of course, would like to have them in PHP.
BTW, the caching algorithms I have implemented are producing dynamic pages anywhere from 40% to 300% faster than using the normal implode and replace methods typically used to process html templates. I will provide the source used to obtain these benchmarks.
My intention is to give away these solutions, and maintain a CVS repository for them.
Thanks in advance for your help,
Jerry
I want to leverage the source code for these methods in PHP so that they can be reused from Java via JNI. After Java, it would seem reasonable to go after Python and PERL. The end game is to achieve a persistent dictionary object that transcends things like web servers, application servers and standalone applications--basically, fast access durable objects that exist until reboot or removed using ipcrm. Incidentally, I also use N-number of memory mapped files (but could be simple files) based on a hash code to route specific chunks of serialized data to discrete units of storage, and sort by these chunks so that resource utilization is kept to a minimum when a subset of key/value pairs are accessed, as each cache file is consumed and freed before the next one, and key/value pairs accessed are merged into a local array. This is done by queuing up keys requested and then requesting them all at once. The benefit of all this is that we now end up with a scalable dictionary object that minimizes page swaps through discrete chunks of memory, can be tuned for memory availability, and (using semaphores) handles concurrency. The developer is free to code in a language of preference with the ability to efficiently pass data back and forth to other processes--the IPC dream made simple. For Java, I would rewrite this PHP class as C++ code.
Recently, I developed a name/value pair dictionary class that uses these methods and some more traditional IPC methods found in C++. I simply found that the PHP methods were a very convenient way to store and fetch data with one small drawback. These methods store values in a manner that somewhat resembles serialized data with some additional padding and metadata that defines various lengths of objects. One other slight hurdle was developing a hash code generator that yields 32bit integer values that can be exactly reproduced in Java (and other languages), and an algorithm that ensures uniqueness. All this I achieved other than the wrapper I seek for shm_get_var and shm_put_var. I am assuming that these functions are components maintained in some CVS repository.
I have done plenty of custom PHP builds, so, I am sure the answer is right under my nose, but I cannot admit to any profound understanding of specific PHP functions and how they are implemented.
If anyone can confirm this and provide the references to this specific code, and/or assist me in obtaining the code snippets that uniquely define these functions, I would greatly appreciate it. I am not interested in reinventing the methods used to interpret the shared memory that shm_put_var manages and how shm_get_var is optimized to do what it does, unless that is the last resort--it seems doable after looking at a hex dump, just not very fun knowing that these methods could change in another PHP minor release.
I hope someone sees the value in this. I spent days looking at a number of solutions published on the web. None of them provided the level of tuning, scalability, concurrency and efficiency that I needed. I will be publishing the source code and a white paper on this and a few other template engine/caching models on my website at http://www.e-freelance-network.com after I finalize a story board for my site and add a few channels. Once I conquer this, I will add a durable queuing object with overflow to database, and a frequently accessed cached resultset object to the list of goodies published. I already have these in Java, and, of course, would like to have them in PHP.
BTW, the caching algorithms I have implemented are producing dynamic pages anywhere from 40% to 300% faster than using the normal implode and replace methods typically used to process html templates. I will provide the source used to obtain these benchmarks.
My intention is to give away these solutions, and maintain a CVS repository for them.
Thanks in advance for your help,
Jerry