done processing and token.c

Today I started fixing problem using compute results.

Mainly in token.c there are 5 function to process tokens
 tds_process_cancel
 tds_process_result_tokens
 tds_process_row_tokens
 tds_process_row_tokens_ct
 tds_process_trailing_tokens
(I removed tds_do_until_done just some minutes ago... it was not used
anymore)

Main differences are in how DONE tokens are handled. For example
tds_process_row_tokens handle DONE while tds_process_row_tokens_ct
don't. I don't remember who but someone reported that in ODBC it seems
that MS driver accept query after last SUCCESS SQLFetch (that is if you
have a simple select query which return 3 rows after reading 3 rows you
can issue another query). Our ODBC behave differently. From dblib source
it seems that even dblib should read trailing token

    if (dbproc->tds_socket->state == TDS_PENDING) {

        if (tds_process_trailing_tokens(dbproc->tds_socket) !=
TDS_SUCCEED) {
            _dblib_client_msg(dbproc, 20019, 7, "Attempt to initiate a
new SQL Server operation with results pending.");
            dbproc->command_state = DBCMDSENT;
            return FAIL;
        }
    }

(code from dbwritetext and dbsqlsend, even dbnumrets use similar code
and I added tds_process_trailing_tokens call to ODBC too to handle
compute correctly)

As usually I never liked the use of tds_peek/tds_get_byte and similar in
higher libraries. During the coding of compute (not in final code
however) I added the use of tds_peek too. It seems that in some case
it's useful to read only some type of tokens (for example only rows in
ct_fetch/SQLFetch or skip DONE* in dbsqlok). 

I think that it would fine to define a unique function that:
- handle some fixed token trasparently (TDS_RETURNSTATUS_TOKEN, param
tokens)
- handle or just return (selecting behavior with a flag) the type of
other tokens. For example I would define some class:
 - metadata (ROWFMT, COMPUTEFMT)
 - param
 - normal row
 - compute row
 - status ??

This would also fix another problem raised during debug. Calling
tds_process_row_tokens* on a metadata token it fails with an "Unknown
marker" cause tds_process_default_tokens do not handle all tokens (I
think that this can also happen calling tds_process_cancel.. not that
fine).

freddy