Odd data returned in respose to readfilerequest

Hey, all!

Long-time listener, first-time caller.  (Well, "sorta" long-time.  I tried
to do my research before posting.)

I do apologize in advance if this post is too long.  I can be a verbose
and detailed person sometimes, but I am this way because I believe that
only by communicating all of the details that one knows as clearly as
possible can true communication and exchange take place.

I have a phone that is not officially supported by BitPim.  It is a
Kyocera Koi/KX2.

It came in the box with a straight USB cable (no serial converter in-line
with the cable itself), but the phone is not a "USB phone" as far as I can
tell...I think that the USB<->Serial interface is built into the phone
itself (I say this because the drivers included with the phone expose two
"serial" interfaces...a "USB Data Port" which Windows sees as a modem, and
a "USB Control Port" which Windows sees as a straight serial port; that
and the phone has a menu option that allows you to change the rate at
which the "serial" interface communicates).

I am using BitPim 0.7.29 on WinXP SP2.  I selected "Other CDMA Phone" as
the driver to use, explicitly specified the COM# to BitPim that the "USB
Control Port" is set to (BitPim couldn't automatically determine where the
phone was), and then tried to use the filesystem browser.

BitPim has no problem successfully querying the phone for information on
the filsystem itself; a complete tree of directories and files shows up
without BitPim hitting any roadblocks.

The problem comes when I try to actually read one of the files.  I'm not
fully acquainted with the Brew protocol by any means, and so I'm not
*completely* sure how to interpret what I'm looking at, but when I compare
the results I get from my phone with results I saw posted to the "devel"
list at various places in the archives, it seems pretty obvious that
something is Broken(tm).  I realize that test12 introduced some
"weirdness-agnostic" brew response parsing changes in response to a
Kyocera phone that was also doing some strange things, but it appears that
those changes are not able to deal with this situation.

Here is what happens:

Whenever I try to read a file, the following exception is thrown (I
picked /CLK_DB for the following examples, but I have tried multiple
different files in multiple different locations, believe me):

- - - - - - - - - - - - - -

An unexpected exception has occurred.
Please see the help for details on what to do.

Traceback (most recent call last):
  File "gui.pyo", line 158, in run
  File "gui.pyo", line 98, in __call__
  File "gui.pyo", line 1454, in getfile
  File "com_brew.pyo", line 241, in getfilecontents
  File "com_brew.pyo", line 426, in sendbrewcommand
CommsDataCorruption: Other CDMA Phone: Brew packet failed CRC check

Variables by last 8 frames, innermost last

Frame __bootstrap in threading.pyo at line 442
           self =  <WorkerThread(BitPim helper, started daemon)>

Frame run in gui.pyo at line 164
              e =  <common.CommsDataCorruption instance at 0x01F760D0>
            res =  None
           self =  <WorkerThread(BitPim helper, started daemon)>
           item =  (<gui.Request instance at 0x01F76058>, <gui.Callback
instance at 0x01F761C0>)
           call =  <gui.Request instance at 0x01F76058>
             ex =  <common.CommsDataCorruption instance at 0x01F760D0>
       resultcb =  <gui.Callback instance at 0x01F761C0>
          first =  0

Frame __call__ in gui.pyo at line 98
           self =  <gui.Request instance at 0x01F76058>
           args =  ()
              d =  Keys []
                   {}
         kwargs =  Keys []
                   {}

Frame getfile in gui.pyo at line 1454
           path =  u'CLK_DB'
           self =  <WorkerThread(BitPim helper, started daemon)>

Frame getfilecontents in com_brew.pyo at line 241
           self =  <com_othercdma.Phone instance at 0x01F76328>
            req =  <p_brew.readfilerequest object at 0x00909050>
          start =  1112249895.5929999
           file =  u'CLK_DB'
           data =  <cStringIO.StringO object at 0x01F5C0C0>
           desc =  u'Reading CLK_DB'

Frame sendbrewcommand in com_brew.pyo at line 426
       origdata =  'BY\x04\x00\x07CLK_DB\x00A\xe1~'
              d =  1
  responseclass =  <class 'p_brew.readfileresponse'>
         buffer =  <prototypes.buffer instance at 0x01F76300>
        request =  <p_brew.readfilerequest object at 0x00909050>
    callsetmode =  True
       firsttwo =  'Y\x04'
            crc =  'A\xe1'
           data =  'Y\x04\x00\x07CLK_DB\x00'
           self =  <com_othercdma.Phone instance at 0x01F76328>

- - - - - - - - - - - - - -

This series of steps generates the following log:

- - - - - - - - - - - - - -

22:18:07.180 COM7: Opening port COM7, 115200 baud, timeout 3.000000,
hardwareflow 0, softwareflow 0
22:18:07.180 COM7: Open of comm port suceeded
22:18:07.180 Other CDMA Phone: Listing dir ''
22:18:07.200 Other CDMA Phone: Now in brew mode
22:18:15.592 Other CDMA Phone: Getting file contents 'CLK_DB'
22:18:15.602 Other CDMA Phone: Junk at begining of packet, data at 1
22:18:15.612 Exception: An unexpected exception has occurred.
Please see the help for details on what to do.

[redundant traceback information snipped]

- - - - - - - - - - - - - -

Notice the next-to-the-last entry in the above log referring to data that
it identifies as "junk."  As we can see in these captures of the actual
communication that takes place between BitPim and the phone, there is one
extra byte that the phone is sending back (hex value 42) in front of what
I'm guessing is the normal Brew response header that I see on all "good"
responses (which starts with hex value 59):

(grrr...Pine might try to reformat this...hopefully this looks alright)

- - - - - - - - - - - - - -

22:18:15.592 Other CDMA Phone: Getting file contents 'CLK_DB'
22:18:15.592 Other CDMA Phone: brew request Data - 11 bytes
<#! p_brew.readfilerequest !#>
00000000 59 04 00 07 43 4c 4b 5f 44 42 00                    Y...CLK_DB.

22:18:15.602 Other CDMA Phone: Junk at begining of packet, data at 1
22:18:15.602 Other CDMA Phone: Original data Data - 15 bytes
00000000 42 59 04 00 07 43 4c 4b 5f 44 42 00 41 e1 7e        BY...CLK_DB.A.~

22:18:15.602 Other CDMA Phone: Working on data Data - 15 bytes
00000000 42 59 04 00 07 43 4c 4b 5f 44 42 00 41 e1 7e        BY...CLK_DB.A.~

22:18:15.602 Other CDMA Phone: Original data Data - 15 bytes
00000000 42 59 04 00 07 43 4c 4b 5f 44 42 00 41 e1 7e        BY...CLK_DB.A.~

22:18:15.602 Other CDMA Phone: Working on data Data - 11 bytes
00000000 59 04 00 07 43 4c 4b 5f 44 42 00                    Y...CLK_DB.

22:18:15.612 Exception: An unexpected exception has occurred.
Please see the help for details on what to do.

- - - - - - - - - - - - - -

You will also notice that in response to the readfilerequest command, the
phone does not respond with file data, but rather with what looks like a
mere re-echoing of the request that was just sent to it moments before
with the following differences:

1) The response is prepended with hex value 42.
2) The response is always appended with 2 seemingly-random byte values
   (perhaps some checksum?) and then terminated with a third byte that is
   always hex value 7e.

Not sure how much of this is "normal" and how much of it is not, but I'm
pretty sure that first byte isn't since BitPim is complaining about it.

But there is one futher oddity which you should be aware of, and I'm not
sure if there is any way to overcome this or not.  Let's try to grab a
different file...perhaps one with a longer full path name:

- - - - - - - - - - - - - -

22:40:52.344 Other CDMA Phone: Getting file contents 
'ui/background_images/textmode_entry_bk.png'
22:40:52.344 Other CDMA Phone: brew request Data - 47 bytes
<#! p_brew.readfilerequest !#>
00000000 59 04 00 2b 75 69 2f 62 61 63 6b 67 72 6f 75 6e     Y..+ui/backgroun
00000010 64 5f 69 6d 61 67 65 73 2f 74 65 78 74 6d 6f 64     d_images/textmod
00000020 65 5f 65 6e 74 72 79 5f 62 6b 2e 70 6e 67 00        e_entry_bk.png.

22:40:52.344 Other CDMA Phone: Junk at begining of packet, data at 1
22:40:52.344 Other CDMA Phone: Original data Data - 20 bytes
00000000 42 59 04 00 2b 75 69 2f 62 61 63 6b 67 72 6f 75     BY..+ui/backgrou
00000010 6e 1b 75 7e                                         n.u~

22:40:52.344 Other CDMA Phone: Working on data Data - 20 bytes
00000000 42 59 04 00 2b 75 69 2f 62 61 63 6b 67 72 6f 75     BY..+ui/backgrou
00000010 6e 1b 75 7e                                         n.u~

22:40:52.344 Other CDMA Phone: Original data Data - 20 bytes
00000000 42 59 04 00 2b 75 69 2f 62 61 63 6b 67 72 6f 75     BY..+ui/backgrou
00000010 6e 1b 75 7e                                         n.u~

22:40:52.354 Other CDMA Phone: Working on data Data - 16 bytes
00000000 59 04 00 2b 75 69 2f 62 61 63 6b 67 72 6f 75 6e     Y..+ui/backgroun
00000010 

22:40:52.354 Exception: An unexpected exception has occurred.
Please see the help for details on what to do.

- - - - - - - - - - - - - -

Well, now isn't that strange?  In the phone's response to my
readfilerequest, the request string that it tried to echo back has been
truncated.  This happens every single time without fail: the response to a
readfilerequest returns a maximum of 20 bytes.  Any more than that, and
it's chopped off (actually, it cuts down the original text that it manages
to echo back to only 16 bytes...as you can see, it still manages to 
prepend an ASCII 'B' and append 2 "random" bytes plus an ASCII '~', which
only leaves 16 out of the total 20 returned bytes).

BitPim, in the last line of output there before the exception is thrown,
shows us that it is doing an admirable job in *trying* (vainly) to
interpret the nonsense being thrown back at it by successfully managing to
strip off the 1 byte from the front of the response and the three at the
end (I'm still guessing that the three at the end make up the checksum
somehow since BitPim never complains about those and yet seems to know
that they don't comprise any of the actual data from the response).

I'm pretty sure that this is not a cable or driver issue since responses
to a listdirectoriesrequest are NOT being truncated in this fashion, nor
do they have weird bytes before or after the response (and one that was
returned from the phone -- in response to asking about the contents of
/ui/background_images -- was 263 bytes long).  Though if you'd like, I'm
sure I can try to rig up BitPim under Linux (assuming that whatever
USB-to-Serial chipset that the phone uses internally is compatible with
the Linux 'acm' module) in order to confirm that there are no undue
influences being exercised against BitPim and/or the phone's USB 
"interface" under my Windows environment.

I would be happy to test out any modifications to BitPim code on my phone 
that anyone happens to generate in response to this issue; I'll even set
up the BitPim Python development environment somewhere if need be.  Just
let me know what I can do.

Thank you for taking the time to read this, and thank you for any and all
responses!

Regards,

-- 
-- Nathan Anderson           <mailto:nathan@xxxxxxxxxxxxxxxx>
   <xri:=nathan.anderson>      <http://www.anderson-net.com/>

   "You can't appreciate Shakespeare until you've read him in
   the original Klingon."
     -- General Chang, Star Trek VI: The Undiscovered Country



-------------------------------------------------------
This SF.net email is sponsored by Demarc:
A global provider of Threat Management Solutions.
Download our HomeAdmin security software for free today!
http://www.demarc.com/Info/Sentarus/hamr30
_______________________________________________
BitPim-user mailing list BitPim-user@xxxxxxxxxxxxxxxxxxxxx
Unsubscribe or change options at
https://lists.sourceforge.net/lists/listinfo/bitpim-user

There are several hundred people on this list.  Please be
courteous, on topic and follow the instructions before posting
http://bitpim.sf.net/testhelp/support.htm