FFI: :malloc-free allocation (was: FFI guru/wizard award to be re warded)

Hi,

FFI has had :malloc-free allocation since ever. It would be unfortunate if it 
would not work with Berkely DB. Let's see.

Aurelio Bignoli wrote:
>I'm trying to interface Berkeley DB C API
>(http://www.sleepycat.com). It uses "Data Base Thangs" to read and
>write data to/from DB files:

I went to consult their doc and found:
http://www.sleepycat.com/docs/index.html
http://www.sleepycat.com/docs/api_java/c_index.html

(def-c-struct DBT
  (data c-string) ;; truly a polymorphic pointer type
  (size uint32)
  (ulen uint32) (dlen uint32) (doff uint32) (flags uint32))

One problem I see is that CLISP's FFI has :malloc-free allocation on a per 
parameter basis. Here it looks like one would wish for it on a per-pointer 
basis,e.g.
 (c-ptr :alloca
   DBT  :malloc-free)

So how can we mix that?
I'll talk about the 100% solution later. Here comes a solution that should work 
in current CLISP.

We're going to use DEF-C-VAR with :alloc :malloc-free. The variable itself will 
not be freed, yet some of its pointers may.

(def-c-var DBT-pool (:type (c-array DBT 10))
  (:name "DBT_pool") (:alloc :malloc-free))

Using (setf (slot (element DBT-pool 0) 'data) "foobar")
invokes malloc(). We can free() that memory using
(setf (slot (element DBT-pool pool-index) 'data) nil)

This can be used together with both Bekerley DB's free and realloc flags.

This yields to the following schema:
(def-call-out DB-get
  (:arguments (key c-pointer))
  (:return-type ...)
  (:language :stdc))

(defun get-as-string (pool-index)
  (prog1
      (values
       (DB-get (c-var-address (element DBT-pool pool-index)))
       ;;TODO error checking
       (slot (element DBT-pool pool-index) 'data))
    ;; setting to NIL frees memory
    (setf (slot (element DBT-pool pool-index) 'data) nil)))

We will use the fact that free() can be called on anything:
(get-using-c-type (pool-index c-type)
  (prog1
      (values
       (DB-get (c-var-address (element DBT-pool pool-index)))
       ;;TODO error checking
       ;;uhoh, run-time parse-c-type...
       (cast (slot (element DBT-pool pool-index) 'data) c-type))
    (setf (slot (element DBT-pool pool-index) 'data) nil)))
;(get-using-c-type 0 `(c-ptr (c-array uint ,size)))


On symbol-macros vs. foreign-variable objects

Here (slot (element- ...)) is recomputed each and multiple times. If we could 
use the underlying foreign-variable objects instead of the symbol macros, we 
could use memoization and other speed up techniques.
We'd keep a pool of 10 DBT represented as foreign-variable objects. We could 
keep a memoized set of foreign-variables, each corresponding to a specific data 
type, for each DBT.

That's also a value of making foreign-variable objects go public. Places are 
nice, however Lisp (and Smalltalk) programmers know how to manipulate objects.


On creating and using foreign-objects dynamically instead of the def-c-var trick
More on this another time. Time is running out on me.

Regards,
        Jorg Hohle.


-------------------------------------------------------
This SF.net email is sponsored by: ValueWeb: 
Dedicated Hosting for just $79/mo with 500 GB of bandwidth! 
No other company gives more support or power for your dedicated server
http://click.atdmt.com/AFF/go/sdnxxaff00300020aff/direct/01/